PVForecast User’s Guide

A high level introduction to the project is given here. This README file provides a full description of installation and configuration

Introduction

An extensive set of forecasts relevant to PV rooftop installations is supported:

Upgrade Notice: incompatible changes - see Version History for details

Table of Content

Installation

Setup

Installation mainly ensures that all necessary python packages are available. We assume a Raspberry host here - although the instructions are probably quite generic. The scrip requires Python 3.8 or newer.

Required packages are described in requirements.txt. These packages can be installed manually (with python -m pip install package) or with python -m pip install -r requirements.txt. Referring to this file, which contains comments:

Influx must be installed separately. SQLite is supported natively by Python. However, an SQLite browser maybe useful.

Additional help for installation is in the project wiki.

Running the Script

After downloading the script from Github, into a directory of your choosing (eg. \home\pi\PV), you should have these files (and some more):

./PVForecasts.py
./config.ini
  |- ./PVForecast/*py
  |- ./docs
  |- ./emissionFactors
  +- ./data
./LICENSE

A typical crontab entry can look like so (assuming you have downloaded into \home\pi\PV):

*/15 * * * * cd /home/pi/PV && /usr/bin/python3 PVForecasts.py >> /home/pi/PV/data/err.txt 2>&1

which would run the script every 15min

A great explanation of cron is from crontab guru. Crontab entries are made with crontab -e and checked with crontab -l.

Note: The script doesn’t do much in terms of housekeeping (eg., limit size of SQLite database or err.txt file used above to redirect error messages).

Configuration

.\config.ini is a configuration file parsed with python’s configparser. It consists of Sections and key = value pairs. Most importantly:

Sections

Section Description
[Default] If a key-value pair is not found in a specific section, the corresponding value in the default section is used.
[Forecasts] Forecasts to be run. If this section is missing, all forecasts for which a specific section exists is run
Forecast configs Each forecast source has its own section: Solcast, VisualCrossing, DWD, OpenWeatherMap, Entso-E, CO2signal, FileInput
[PVSystem] describes the PV system (for forecast sources which require modelling: VisualCrossing, DVD, OpenWeatherMap. For split-array configurations, additional sections can be created
[DBRepo] configuration of SQLite storage
Influx] configuration of Influx storage

Default Section

The following parameters are used by many forecast sources and hence typically placed into the [Default] section.

[DEFAULT]
    # ----------------------------------------------------- Storage locations
    storePath         = ./data/   # storage location for files (.csv, .kml, ..._ and SQLite database)
    # following parameters could be overwritten for individual forecast providers
    storeDB           = 0         # store to SQLite database (see [DBRepo] for name)
    storeCSV          = 0         # store .csv files (mainly for debugging)
    storeInflux       = 1         # store DC power output estimates in Influx (see [Influx] for name)
    # dropWeather     = 1         # drop weather parameters irrelevant for PV forecasting for 'storeDB', 'storeCSV'
    # force           = 0         # force downloading of new data

    # ----------------------------------------------------- Location of PV system
    Latitude          = <latitude_of_your_system>
    Longitude         = <longitude_of_your_system>
    # Altitude        = 0            # altitude of system (meters above sea level)

Parameters storeXX all default to 0 (False), but at least one must be set to 1. For dropWeather, see SQLite Storage force overwrites time-based blocking of downloading new data, if, for a data source, last data was downloaded not too long ago. Blocking time intervals are different per data source.

Configuring Data Sources

Forecast Sources

Source Description Look-ahead
Solcast Solar forecast by Solcast Default 7 days
VisualCrossing Weather and solar forecast from VisualCrossing 15 days
DWD provided by Deutscher Wetterdienst (primarily for Germany). Two flavours exist: 10 days
MOSMIX_L single station forecast  
MOSMIX_S all weather stations (will be downfiltered to single station after data download)  
OpenWeatherMap Weather forecast from OpenWeatherMap.org with approx. 10 parameters. Cloud coverage is used for PV output power forecast 2 days
Entso-E CO2 intensity for grid power, Electricity auction prices (EU only, based on data from Entsoe Transparency Platform) ~1 day
CO2signal actual CO2 intensity for grid power, provided by Electricity Maps na

VisualCrossing, DWD and OpenWeatherMap need modeling as described below

Depending on the data source, various forecast algorithms are available. The configuration happens in the respective sections described below.

Solcast Configuration

[SolCast]
    resource_id       = <resource_id_from_solcast.com>
    # resource_id_2   = <second_resource_id_from_solcast.com>
    api_key           = <api_id_from_solcast.com>
    # interval        =   0       # interval at which Solcast is read (during daylight only)
    # hours           = 168       # forecast period defaults to 7 days, up to 14 days (336h)
    # apiCalls        =  10       # number of API calls supported by Solcast (new default in v2.11.00)

Solcast allows for the free registration of a residential rooftop PV installation of up to 1MWp and allows for up to 10 API calls/day (legacy users may benefit from more credits and can set their entitlement with apiCalls). Note that split array configurations require one credit per array - hence essentially reducing the calls to 5/day.

Solcast is considering to provide more calls as a paid service to hobbyists - this questionaire helps them establish the market acceptance - hence, if you are interested, please fill in the form.

The registration process provides a 12-digit resource_id (xxxx-xxxx-xxxx-xxxx) and a 32 character API key. Solcast also supports dual array systems (eg., east/west) through a second resource_id_2.

Solcast directly provides a PV forecasts (in kW) for 30min intervals, with 10% and 90% confidence level. Hence, no further modelling is needed. Forecasts are updated every 15min (for Eurasia), but it is recommended to call Solcast no more than every 30min.

To stay within the limits of apiCalls per day, the script calls the API only between sunrise and sunset, except in the 24h configuration below. It can further manage the calling interval to the API automatically or explicitly through the value assigned to interval:

value meaning
0 Default: call API every 15min (single array) or 30min (dual-array). To not exceed maximum API calls, extend interval to 30min (60min) after sunrise and before sunset on long days. Hence, this provides most accurate (short-term) forecasts during mid-day.
early same as 0, but all interval extensions are done before sunset only. Hence, this provides most accurate forecasts in the morning (this is not useful for apiCalls < 25)
late same as 0, but all interval extensions are done after sunrise only. Hence, this provides most accurate forecasts in the afternoon (this is not useful for apiCalls < 25)
24h downloads over the full day, but intervals are longer than in the previous configuration
number a positive number (eg. 15, 30, 60, …) ensures that the API is not called more frequently than the stated number of minutes. It is the users responsability to stay within the limits of apiCalls supported by Solcast

There is obviously an interaction between the interval settings and the crontab entry used to run the script (see above). It is suggested to configure crontab to run the script every 30min (for apiCalls >=25, every 15min is possible). The interested user can use the script ./debug/solcast_timeInterval.py to learn how download intervals are calculated - self-study is required).

Parameters Latitude and Longitude are only used to calculate daylight time. Defaults are for Frankfurt, Germany. (The Solcast service has it’s own location information, associated with the api_key.)

hours defines the forecast period and defaults to 168h, but can be extended up to 14 days (336h)

VisualCrossing Configuration

[VisualCrossing]
    api_key           = <api_id_from_visualcrossing.com>
    # Irradiance      = disc        # default irradiation model

VisualCrossing offers free access to their API to regularly download weather forecasts. The registration process provides a 25 character API key.

The Weather Timeline API provides a 15-day forecast of approx. 18 parameters, including solarradiation (or GHI). This can be converted to a PV output power forecast (see Forecast Models). The modelling strategy is controlled with the Irradiance parameter as described below.

dropWeather, Latitude and Longitude are typically provided in [Default] section.

DWD configuration

Deutscher Wetterdienst (DWD) supports two (file based) interfaces (without the requirement of authentication):

Although both interfaces are supported, it is strongly suggested to use MOSMIX_L, since MOSMIX_S causes a download volume of ~1GByte/day without improving the forecast quality (despite the shorter update interval). MOSMIX_S can only be called from the Forecast section and downloaded data is immediatly downfiltered to the selected DWDStation.

[DWD]
    DWDStation        = <station_number>    # Station number
    # DWD_URL_L       = https://opendata.dwd.de/weather/local_forecasts/mos/MOSMIX_L/single_stations/
    # DWD_URL_S       = https://opendata.dwd.de/weather/local_forecasts/mos/MOSMIX_S/all_stations/kml/
    # Irradiance      = disc    # default irradiation model
    # storeKMZ        = 0       # store downloaded .kmz files (.kml compressed as .zip)
    # keepKMZ_S       = 0       # keep MOSMIX_S original file after downloading - note that these are many big files!

Valid DWDStation values are defined on the MOSMIX website

storeKMZ: The files downloaded are named *.kmz which is inadequate in two ways: First, the files are simple .zip files (so, why are they not called that way?) and second, a .zip file is meant to contain multiple files, which clearly the .kmz files never do. Hence, with storeKMZ = 1, downloaded data is stored in the more adequate .gz format. For MOSMIX_L, the downloaded files for the selected station are stored. For MOSMIX_S, an extract for the selected station is stored in a self-contained compressed .xml file. That format is very similar to MOSMIX_L files.

keepKMZ_S: in case of downloading the (huge) MOSMIX_S file, they can be stored by enabling this option.

The modelling strategy used to convert weather data to irradiance is controlled with the Irradiance parameter as described in the next section. Not all MOSMIX stations support irradiance data (inconveniently labeled Rad1h). If the chosen station does not have it, irradiance based models won’t work, but cloud-based models still do.

dropWeather is typically provided in [Default] section.

OpenWeatherMap Configuration

[OpenWeatherMap]
    api_key           = <api_id_from_openweathermap.org>
    # Irradiance      = clearsky_scaling    # default irradiation model

OpenWeatherMap offers free access to their API to regularly download weather forecasts. The registration process provides a 32 character API key.

The weather forecast consists of approx. 10 parameters, including cloud coverage, which can be modelled to a PV forecast (see Forecast Models). The modelling strategy is controlled with the Irradiance parameter as described below.

dropWeather, Latitude and Longitude are typically provided in [Default] section.

Entso-E Configuration

new - this works only for EU area. A detained introduction to the ideas behind this is given on a separate CO2 Intensity page

Entso-E, the European Network of Transmission System Operators for Electricity, operates the EU Transparency Platform with the goal to promote transparency goals to stakeholders.

[Entso-E]
    api_key             = <api_from_Entso-E>
    zones               = DE, DE_AMPRION                       # comma separated list of zones to be analyzed
    # resolution        = 60T                                  # some countries offer bidding prices for different time intervals, typically 15T and 60T
    # verbose           = 0                                    # verbosity level: 0=default, 1=basic, 2=max; forced =2 if start/end date are given
    # keepRaw           = 0                                    # default 0, together with start/end can be used to dump Entso-E data to .csv
    # start             = 2023-01-01T23:00Z                    # - see User's Guid
    # end               = 2023-02-18T23:00Z                    # -
    # loop              = 0                                    # -

An api_key can be requested as described in the User’s Guide, chapter 2.

Data can then be downloaded for a comma separated list of zones. Depending on selected zone(s), different data is available and calculated. A list of zones - and available data per zone - is here. For more details, refer to the CO2 Intensity page, where also the other parameters are explained.

To get accurate data, a rolling linear correlation fit between forecasts and actuals is used. Due to this, the system needs to run for a couple of days before accurate forecasts are achieved.

CO2signal Configuration

new - A detained introduction to the ideas behind this is given on a separate CO2 Intensity page

[CO2signal]
    api_key             = <api_from_www.co2signal.com>
    zones               = DE      # comma separated list of zones to be downloaded

ElectricityMaps generate CO2 intensity data for many regions of the world. A free API is available at CO2signal, which provides hourly data of the CO2 footprint of grid electricity (there is no forecast available), where a free api_key can be registered.

zones is a comma-separated list of zones to be downloaded, from a list of supported zones.

FileInput Configuration

[FileInput]
    ...

This forecast source is for mainly for debugging purposes and allows to read .kmz or .csv files with weather data. Refer to comments in sample config.ini file and source code ForecastManager.processFileInput for further guidance.

Configuring PV Output Power Forecast Modelling

Data from all PV output power related data sources (except Solcast) do not directly contain PV output power. This needs be modelled using functionality provided by pvlib.

Essentially, the modelling consists of a two-step approach:

  1. convert weather data to irradiation data (GHI, DNI, DHI). Multiple conversion strategies are available and controlled with the Irradiance parameter in the config section for [VisualCrossing], [DWD] and [OpenWeatherMap] respectively.

  2. convert such irradiation data into PV output power. This is controlled in the config section [PVSystem]

Convert Weather Data to Irradiation Data

Model Input parameter Applicable to Comment
disc GHI MOSMIX (*), VisualCrossing default if GHI available
dirint GHI MOSMIX (*), VisualCrossing  
dirindex GHI MOSMIX (*), VisualCrossing some numerical instabilities at very low values of GHI
erbs GHI MOSMIX (*), VisualCrossing  
campbell_norman clouds OWM, MOSMIX  
clearsky_scaling clouds OWM, MOSMIX default if GHI not available,
clearsky NA all (except Solcast), output agnostic to weather forecast clear sky estimation of PV output power; uses simplified_solis
all NA NA calculate all applicable models for provided weather data

(*) not all MOSMIX stations provide GHI data

Where needed, DHI is calculated from GHI and DNI using the fundamental equation DNI = (GHI - DHI)/cos(Z) where Z is the solar zenith angle (see eg. Best Practices Handbook). Weather parameters considered in above models include:

Parameter VisualCrossing MOSMIX OpenWeatherMap unit
ghi solarradiation Rad1h - W/m2
temp_air temp TTT temp K
temp_dew dew Td dew_point K
wind_speed windspeed FF wind_speed m/s
pressure pressure PPPP pressure Pa
clouds cloudcover Neff clouds 0 .. 100

Where needed, unit conversion and parameter renaming is performed. Parameter correspond to same-named pvlib parameters and are stored in the SQLite Storage, if enabled.

MOSMIX Rad1h is (according to DWD customer service) the integrated radiation over the last hour prior to the forecast time stamp. For VisualCrossing, the documentation states that solarradiation is the power at the instantaneous moment of the forecast. Hence, it probably best reflects the average radiation for a period beginning 30min before and ending 30min after the forecast timestamp. To account for this, the forecast time stamp period_end is corrected by +30min (which is then slightly misleading for the secondary weather parameters reported) once it gets written out

Convert Irradiation Data to PV Output Power

In this section, we first describe how to model a single array PV System. The software also supports the configuration of split array systems. The necessary extensions are described in the next section.

pvlib supports two modelling strategies for a PV system:

Both approaches are supported and selected based on Model, but PVWatts is simpler to configure:

PVWatts Modelling

[PVSystem]
    # Model            = PVWatts  # modeling strategy for PV: 'CEC' or 'PVWatts' 
    # TemperatureModel = open_rack_glass_glass
    # clearsky_model   = simplified_solis
    
    # --------------------------- PVWatts definition
    InverterPower     = 10000     # name-plate inverter max. power
    NominalEfficiency = 0.965     # nominal European inverter efficiency
    SystemPower       =  9750     # system power [Wp]
    TemperatureCoeff  = -0.0036   # temperature coefficient (efficiency loss per 1C)

The PVWatts model considers considerably less inefficiencies (~2.5%) than PVWatts defaults (~14%):

CEC Modelling

To use actual PV system component data, the CEC model must be used instead:

[PVSystem]
    Model            = CEC        # modeling strategy for PV: 'CEC' or 'PVWatts' 
    # TemperatureModel = open_rack_glass_glass
    # clearsky_model   = simplified_solis
    
    # --------------------------- physical definition of PV System, using CEC database
    # based on .csv files at .../pvlib/data, special characters to be replaced by '_'
    ModuleName        = LG_Electronics_Inc__LG325N1W_V5
    InverterName      = SMA_America__SB10000TL_US__240V_
    NumStrings        =   2       # number of strings 
    NumPanels         =  15       # number of panels per string

The location of the pvlib library can be determined with python -m pip show pvlib. The two .csv files sam-library-cec-inverters-2019-03-05.csv and sam-library-cec-modules-2019-03-05.csv list inverters and modules respectively. The first column contain the names of supported inverters and modules. Special characters and blanks need replaced with _ in the config file. Hence, eg. SMA America: SB10000TL-US [240V] becomes SMA_America__SB10000TL_US__240V_

The selected model should at a minimum match the nameplate power of the installed panels (eg. 325Wp). The selected inverter is uncritical as long as the nameplate power is same or higher as installed inverter (eg. 10kW) - the modeling of inverters is relatively poor in pvlib, considering only a NominalEfficency.

pvlib models panel temperature (and related efficiency loss) based on TemperatureModel and weather parameter temp_air. clearsky_model is used for irradiation model clearsky. ineichen and simplified_solis are supported, haurwitz is not.

Both models also need basic parameters of the system location and orientation:

    # Latitude, Longitude, Altitude required, but typically taken from [Default] section
    Tilt              =  30
    Azimuth           = 127       # 270=West, 180=South, 90=East

Split Array System Configuration

The above allows the definition of a single array PV system. Split array systems (eg. with a west and east looking set of panels) can be configured as follows:

[PVSystem]
    # define one array as explained in previous section
    # additionally, following two parameters are supported:
    suffix     = West             # value = name of this array; default '1'
    storage    = both             # legal values: individual, both, sum (default)

[PVSystem_East]
    # define settings applicable to this array

[PVSystem_South]
    # define settings applicable to this array

There is no limit to the number of splits that can be defined.

Names of the sub-arrays are arbitrary - anything after the _ serves as a suffix (here eg. East, South). Since the first section does not contain such a name (the section is strictly named [PVSystem]) a suffix can be provided separately (eg. West)

The secondary arrays ([PVSystem_East], [PVSystem_South], …) inherit all settings from [PVSystem] except those which are explicitly overwritten. Typically, one wants to overwrite at least Azimuth and Tilt, likely also NumStrings, NumPanels and possibly panel types.

PV output is calculated for each sub-array and creates parameters dc_<irradiation_model>_<suffix> and ac_<irradiation_model>_<suffix>. Parameters are added as columns to the same table a single-array PV system would have created. The parameter storage controls what is handed to the data storage module. Valid values are:

Value Function
sum default: only sum of all sub-arrays is stored (as dc_/ac_<irradiation_model>)
individual only the individual sub-array results are stored, but sum is not calculated
both individual results and sum are stored

Configuring Data Storage

Forecasting PV output power would be pointless, if the resulting data wouldn’t be stored anywhere. The application supports three storage models:

  1. SQLite (file based relational database)
  2. Influx
  3. csv files

The following configuration parameters control what is stored where and can be configured separately for each forecast provider or, more commonly, in section [Default] (0 = disable, 1 = enable)

Parameter Function
storeDB enable SQLite storage
storeInflux enable Influx storage
storeCSV enable CSV file storage
storePath storage location of SQlite database and other files stored

Databases and tables are created dynamically.

Influx stores a reduced set of data, aimed at displaying forecast data in a dashboard or similar. SQLite is tailored to build a data repository useful for deeper analysis and learning.

All times stored or reported are in UTC. All period time stamps are aligned to show the periodEnd. Hence, at times, data appears to be mis-aligned with directly downloaded data from the data source. This is because some sources report period timestamps as periodStart.

SQLite Storage

[DBRepo]
    dbName  = pvforecasts.db      # SQLite database name (at 'storePath')

An SQLite database is dynamically created with above defined name at storePath with name dbName. It is sufficient to remove the database to cause a re-creation of a fresh database. If the configuration is changed on an existing data, new tables are added dynamically. Fields which no longer exist are left empty. But new fields are not added dynamically.

The following features are only available in SQLite storage:

Influx Storage

Influx contains a reduced set of data, compared to SQLite:

Influx has undergone a major, largely not backward compatible upgrade between version 1.x and 2.x. However, both version are supported (though not in parallel). Influx 1.x is out of maintenance since 2021. Hence, for new installations, it is suggested to move to Influx 2.6 or newer. Influx 3.x is not supported however.

Influx v2.x Storage

Instead of Influx v1.x storage, Influx v2.x can be used. For this to work, the config file section must adhere to the following:

[Influx]
    influx_V2       = 1              # enable Influx 2.x support (default is to use 1.x)
    token           = <your token>
    org             = <your org>
    bucket          = <your bucket>  # fall-back: use key `database`

Tables are called buckets but largely serve the same purpose. Note that in Influx 2.x token based authentication is mandatory. Tokens can be generated in the Influx GUI.

Influx v1.x Storage

[Influx]
    host              = <your_hostname>         # default: localhost
    # port            = 8086
    database          = <your_influx_db_name>
    # username        = root
    # password        = root
    # retention       = None                    # retention policy

Tables are called measurements but serve the same purpose.

If authentication is required, optional username and password can be provided in the [Influx] config section. Default is root / root (as is the default for Influx 1.x). Authentication is not SSL encrypted though.

If the database is configured to support multiple retention policies, one for the PVForecast data can be selected with retention.

.csv File Storage

storeCSV = 1 store output in .csv files at storePath. This is mainly for debugging.

Solcast can only store to csv files if at least one other storage model (SQlite, Influx) is enabled.

Version History

v2.11.04 2024-01-21

v2.11.03 2023-12-28 Bug fixes

v2.11.02 2023-08-06 Bug fixes

v2.11.00 2023-04-21 Bug fixes

Compatibility notes on v2.11.00

Solcast has changed number of allowed apiCalls per day to 10, but it appears that legacy users still can use their previous entitlements. The default value for apiCalls has changed, so that legacy users now need explicitly set their entitlement value.

v2.10.00 2023-02-28

If you plan to continue using only SolcastLight, there is no reason to update - but you miss out on the new capabilities on CO2 intensity forecast

Compatibility notes on v2.10.00

v2.01.00 2022-12-03

v2.00.00 2022-07-24

Compatibility notes on v2.00.00

There are no changes if you are using SolcastLight and hence, there is no reason to update. However, if the full version is used, the following changes apply:

As a consequence, if the SQLite Storage model is used, the pre-existing database (referenced by DBRepo.dbName) has to be deleted, so that a new version, with new tables and fields, will automatically be re-created on first execution of the script.

v1.03.00

v1.02.00

v1.01.00 2021-03-28

v1.00.00 2021-02-06 initial public release

Deprecations

Acknowlegements

Thanks to all who raised issues or helped in testing!

License and Disclaimer

Distributed under the terms of the GNU General Public License v3

The software pulls data from various weather sources. It is the users responsibility to adhere to the use conditions of these sources.

The author cannot provide any warranty concerning the availability, accessability or correctness of such weather data and/or the correct computation of derived data for any specific use case or purpose. Further warranty limitations are implied by the license