Forecasting

Scheduling is about the future, and you need some knowledge / expectations about the future to do it.

In FlexMeasures, this knowledge often comes in the form of forecasts — data-driven estimates of what’s likely to happen next.

https://github.com/FlexMeasures/screenshots/raw/main/tut/PV-forecasting-example.png

Example of a 24-hour horizon forecast for solar power production.

Of course, the nicest forecasts are the ones you don’t have to make yourself (it’s not an easy field), so do use price or usage forecasts from third parties if available, and load them into FlexMeasures. There are even existing plugins for importing weather forecasts or market data.

If you need to make your own predictions, forecasting algorithms can be used within FlexMeasures, for instance, to arrive at an expected profile of future solar power production at the site.

FlexMeasures provides a CLI command and an API endpoint to generate forecasts (see below).

FlexMeasures provides a fixed viewpoint forecasting infrastructure. This means that from one point in time (the fixed viewpoint), we forecast a range of events into the future (e.g. 24 hourly events for a span of one day). While the first forecast (one hour ahead) has a small horizon (1H), the last one has a large horizon (24H) and the accuracy between the two will usually differ (it is easier to forecast small horizons).

At the same time, the design we implemented in FlexMeasures is inspired by rolling forecasts, as training and prediction can be repeated in cycles until a user-specified end date is reached. If you ask FlexMeasures for a fixed viewpoint forecast (one cycle), the model is trained once on the most recent applicable historical period and then produces predictions for the requested future period in one go. This is controlled by the forecast-frequency parameter, which specifies how often predictions are generated during the forecast period.

How a forecasting cycle works

A single forecasting cycle consists of the following steps:

  1. Training: Fit the model on a historical window defined by train-start and train-end.

  2. Prediction: Produce forecasts for a time window defined by predict-start and predict-end.

  3. Repeat: Extend the training window and move the prediction window forward, both by forecast-frequency.

Cycles repeat until predict-end reaches the global to-date. This way, forecasts can cover long ranges while still being based on updated training data in each cycle.

CLI Command

You can create forecasts from the command line using:

flexmeasures add forecasts --from-date 2024-02-02 --to-date 2024-02-02 --max-forecast-horizon 6 --sensor 12 --as-job

This command asks FlexMeasures to generate forecasts for one day (2 February 2024) with a forecast horizon of 6 hours for the sensor with ID 12. If you include --as-job, the forecasting task is added to the job queue to be processed by a worker.

The main CLI parameters that control this process are:

  • from-date: Defines the first predict-start.

  • to-date: The global cutoff point. Training and prediction cycles continue until the predict-end reaches this date.

  • max-forecast-horizon: The maximum length of a forecast into the future.

  • forecast-frequency: Determines the number of prediction cycles within the forecast period (e.g. daily, hourly).

  • train-period: Define a window of historical data to use for training.

Note that:

forecast-frequency together with max-forecast-horizon determine how the forecasting cycles advance through time. train-period, from-date and to-date allow precise control over the training and prediction windows in each cycle.

Forecast post-processing

Forecaster configuration can clip and snap forecast values before they are stored. Use lower and upper to clip values to bounds, and snap to replace values inside a configured interval with a target value. Units are optional; unitless values are interpreted in the output sensor unit.

For example, this configuration clips forecasts to the 0-20 kW range and snaps values in [0 kW, 4 kW) to 0 kW:

{
  "lower": "0 kW",
  "snap": {
    "0 kW": ["0 kW", "4 kW"]
  },
  "upper": "20 kW"
}

The snap target must lie within its interval (on a bound or inside it), so values never snap to a value outside the interval. A target inside the interval snaps the whole band to that value, for example {"2 kW": ["0 kW", "4 kW"]} snaps everything in [0 kW, 4 kW) to 2 kW. The first bound is inclusive and the second is exclusive, so an interval reads as [first, second). Listing the bounds in reverse order flips the closed side: ["10 kW", "4 kW"] means (4 kW, 10 kW]. This keeps adjacent intervals unambiguous — a value on a shared boundary belongs to the interval that opens at it. For instance, given {"0 kW": ["0 kW", "4 kW"], "10 kW": ["4 kW", "10 kW"]}, a forecast of exactly 4 kW snaps to 10 kW.

Snapping runs first and clipping runs afterwards, so lower/upper always take precedence: a snap target outside the bounds is clipped back into range.

Pass the same object as the API config payload, or place it in a JSON or YAML file and pass it to flexmeasures add forecasts with --config.

Forecasting via the UI

The quickest way to create a one-off forecast is the Create forecast button on the sensor page (see Creating a forecast). The button is available to users with permission to record data on sensors, provided at least two days of historical data exist. The forecast duration defaults to 48 hours (configured via FLEXMEASURES_PLANNING_HORIZON) but can be adjusted up to 7 days in the panel. No further configuration is needed — one click queues the job and the page shows progress messages until the forecast is ready.

For more control over what and how to forecast, use the API.

Forecasting via the API

In addition to the CLI command, FlexMeasures provides API endpoints for triggering forecasts and retrieving their results.

These endpoints live under the Sensor API (/api/v3_0/sensors).

A typical workflow is:

  1. Trigger a forecasting job for a sensor.

  2. Poll the job until it finishes.

  3. Retrieve the forecast results.

For the exact API endpoints, parameters, and response formats, refer to the API v3 documentation:

or try the endpoints interactively with the Swagger UI at /api/v3_0/docs.

Technical specs

In a nutshell, FlexMeasures uses a LightGBM regression model (darts.models.LightGBMModel) as its base model to forecast future values.

Note that the most important factor is often the features provided to the model ― lagged values (e.g., the value at the same time yesterday) and regressors (e.g., wind speed prediction to forecast wind power production). Most assets have yearly seasonality (e.g. wind, solar) and therefore forecasts benefit from at least two years of historical data.

Here are more details:

  • The main model is a LightGBM model, which can be wrapped to produce probabilistic forecasts if required.

  • Lagged outcome variables are selected based on the periodicity of the asset (e.g. hourly, daily and/or weekly).

  • Missing data is filled using linear interpolation (via the Darts MissingValuesFiller, which wraps pandas.DataFrame.interpolate).

  • The model is trained once per cycle for each asset and can forecast up to the maximum forecast horizon in a single run.

  • Forecasts are fixed viewpoint forecasts — the model is trained on a given history and produces predictions for a future window in one go. Training and prediction can then be repeated in cycles until a user-specified end date is reached. This cycle-based design is inspired by rolling forecasts while keeping a fixed viewpoint. The forecast-frequency parameter controls how often predictions are generated during the forecast period.

A use case: automating solar production prediction

We’ll consider an example that FlexMeasures supports ― forecasting an asset that represents solar panels. Here is how you can ask for forecasts to be made in the CLI:

flexmeasures add forecasts --from-date 2024-02-02 --to-date 2024-02-02 --max-forecast-horizon 6 --sensor 12  --as-job  # add train-start

Sensor 12 would represent the power readings of your solar power, and here you ask for forecasts for one day (2 February, 2024), with a forecast of 6 hours.

The --as-job parameter is optional. If given, the computation becomes a job which a worker needs to pick up. There is some more information at How forecasting jobs are queued.

Fixed viewpoint vs rolling viewpoint

Unlike previous rolling forecasts, where each prediction covers the same relative forecast horizon (but the origin keeps moving forward), the new infrastructure generates fixed viewpoint forecasts:

  • One reference timestamp.

  • Predictions are made for multiple future horizons from that point.

  • Periodic retraining ensures forecasts remain accurate.

Regressors

If you want to take regressors into account, in addition to merely past measurements (e.g. weather forecasts, see above).

  • past regressors : sensors that only have realizations (historical data).

  • future regressors : sensors that only have forecasts (e.g. weather forecasts).

  • regressors : sensors that have both historical data and forecasts (e.g. weather forecasts).

Including regressors can significantly improve forecasting accuracy, especially when they are highly correlated with the target variable. For example, using irradiation forecasts as regressors can substantially improve solar production predictions. In this weather forecast plugin, we enable you to collect regressor data for ["temperature", "wind speed", "cloud cover", "irradiance"], at a location you select.

Annotation regressors

In addition to sensor-based regressors, you can use annotation regressors to let the forecasting model learn from binary signals derived from annotation data. Holiday flags, factory shutdowns, or any other event stored as an annotation can be passed as future covariates.

Annotation regressors are configured in the annotation-regressors key of the forecasting config. Each entry is a dict with:

  • account, asset, or sensor (required): the database ID of the account, asset, or sensor whose annotations to use.

  • annotation-type (optional, default "holiday"): filter to annotations of this type ("holiday", "label", "alert", etc.).

  • name (optional): a human-readable column name for the regressor. Defaults to annotation_regressor_<index>.

The annotation data is converted to a binary 0/1 time series at the target sensor’s resolution: 1 for every time step that falls within an annotation period, 0 otherwise. Since holidays and scheduled events are typically known in advance, annotation regressors are treated as future covariates.

Example config (passed via --config file):

{
  "annotation-regressors": [
    {"account": 1, "annotation-type": "holiday", "name": "public_holidays"},
    {"asset": 5, "annotation-type": "label", "name": "factory_shutdown"}
  ]
}

Usage:

flexmeasures add forecasts \
  --from-date 2024-01-01T00:00:00+00:00 \
  --to-date 2024-12-31T00:00:00+00:00 \
  --max-forecast-horizon PT24H \
  --sensor 42 \
  --annotation-regressors \
    '{"account": 1, "annotation-type": "holiday", "name": "public_holidays"}'

Note

Create the annotations you want to use as regressors before running the forecast. For holidays, use flexmeasures add holidays, which supports both workalendar and holidays. See Annotations for details.

Automating forecasts

Instead of asking for forecasts one at a time, you can set up an automation: a recurring task defined on an asset. On each run, the automation queues forecasting jobs (so make sure a worker is processing the forecasting queue, see Redis Queues). When the automation was created, its forecast parameters (see above) were stored, and validated with the same schema that the CLI and API use. Timing parameters are resolved on each run — for instance, the forecast start defaults to the time the automation runs, so each run produces fresh forecasts. The sensor on which forecasts are saved (sensor-to-save, falling back to sensor) must belong to the automation asset or one of its descendants. This relationship is checked both when the automation is created and immediately before each run.

Here is how you create an automation in the CLI, asking for daily (at 6 AM) forecasts of sensor 12:

flexmeasures add automation --asset 3 --name "Daily PV forecasts" --cron "0 6 * * *" --timezone Europe/Amsterdam --sensor 12

The recurrence is defined by a standard five-field cron string (minute, hour, day of month, month, and day of week). It is interpreted in the automation’s IANA timezone. If --timezone is omitted, the current FLEXMEASURES_TIMEZONE value is copied to the automation. Changing that configuration later does not change existing automations. Cron aliases and optional seconds or year fields are not supported. Automations are active by default (use --inactive to create them in deactivated state). Use flexmeasures edit automation to rename, re-schedule (--cron), change the timezone, activate or deactivate an automation, and flexmeasures delete automation to remove one. These changes are recorded in the asset’s audit log. The stored data generator is required while the automation exists, so its data source cannot be deleted until the automation is removed.

Each active automation stores a scheduling cursor in the database, so restarting the runner does not lose due occurrences. The cursor is a UTC scheduling watermark: occurrences at or before it are ineligible for another automatic queueing attempt. It does not indicate that queueing or forecast computation succeeded. A new automation starts scheduling from its creation minute and does not replay earlier occurrences. Changing its cron expression or timezone, or reactivating it, starts scheduling from the time of that change. Deactivated automations do not accumulate catch-up work. After upgrading an existing installation, historical occurrences before the migration watermark are not replayed.

If the runner misses one occurrence, it queues that occurrence once when it resumes. If it misses several forecast occurrences, it queues only the latest one and marks the older occurrences handled instead of replaying stale forecasts. Timing parameters that default to the run time are resolved when this catch-up run is actually queued, producing a current forecast.

Daylight-saving-time transitions follow wall-clock semantics. If the clock skips a scheduled local time in spring, that occurrence is handled once at the transition boundary. If a scheduled local time occurs twice in autumn, the first instance is the canonical occurrence and the repeated instance is not queued again.

For automations to actually run, let a cron job execute the following command once per minute:

* * * * * flexmeasures jobs run-automations

Each due automation then queues its forecasting jobs. Each scheduled occurrence receives at most one automatic queueing attempt. The scheduling cursor is committed before queueing starts. If the process crashes, or queueing fails after creating some jobs, that occurrence is not retried automatically because a retry could duplicate partial work. Durable run records and safe retries are outside this feature. The jobs record how they were created, which is shown on the asset’s status page (UI), where recent jobs are listed.

Automations defined on an asset can be viewed on the asset’s Automations page in the UI, and listed with the API endpoint [GET] /assets/(id)/automations.