Upstream Data Sources¶

All grid data served by this project comes from three public EirGrid "Smart Grid Dashboard" backends. This page documents every HTTP call the pipeline makes, the parameters we send, the shape of the response rows we store, and how those raw rows flow through Parquet into the JSON summaries served by the API.

How this dashboard depends on EirGrid¶

Every figure on this dashboard comes straight from EirGrid, the official operator of Ireland's electricity grid. We faithfully collect, store, and present their published data, so what you see here mirrors the authoritative source rather than any independent estimate of our own.

Official source. All values are sourced directly from EirGrid's public feeds, the same data EirGrid uses to operate the grid.
Up to the latest 15-minute interval. We fetch through the most recent interval EirGrid has published, including today. EirGrid only finalises each day's figures once the day is complete, so the most recent hours are preliminary and settle as the official numbers are confirmed.
Always in step with EirGrid. Whenever EirGrid revises or backfills its data, those updates flow through to this dashboard on the next pipeline run, keeping it aligned with the official record.

The sections below document each backend, what we request, and how the raw rows become the JSON summaries the dashboard reads.

API backends at a glance¶

Backend	Base URL	Areas covered	Date format
vargen	`https://www.vargen.smartgriddashboard.com/api/export`	`WIND`, `SOLAR`	`YYYYMMDDHHII` (path segment)
interconn	`https://www.interconn.smartgriddashboard.com/api/Interconnector`	interconnector flows (EWIC, Greenlink, Moyle, Net)	`YYYYMMDDHHII` (path segment)
dashboard	`https://www.smartgriddashboard.com/DashboardService.svc/data`	`demandActual`, `fuelMix`, `co2Emission`, `frequency`, `SnspAll`	`DD-Mon-YYYY` (query string)

The dashboard backend is known to be flaky — 5xx responses are common. The pipeline retries only on transient errors (HTTP 5xx or network transport errors); 4xx responses are treated as terminal. See _is_retryable for the exact rule.

Supported (area, region) combinations on the dashboard backend are declared in DASHBOARD_AREA_REGIONS:

Area	Allowed regions	Notes
`demandActual`	`ROI`, `NI`	`ALL` is derived in export from `ROI + NI`
`fuelMix`	`ROI`	Snapshot-only; ignores date range
`co2Emission`	`ROI`	`ALL`/`NI` return 403 or null
`frequency`	`ROI`	Fetched in weekly chunks — monthly requests time out
`SnspAll`	`ALL`	Only meaningful for the whole island

Fetches for the vargen backend cover all three regions (ROI, NI, ALL).

1. `vargen` — wind and solar generation¶

Endpoint¶

GET https://www.vargen.smartgriddashboard.com/api/export/{start}/{end}/{region}/{area}?FORMAT=JSON

Path / query parameters¶

Parameter	Source	Example	Notes
`start`	path	`202603010000`	`YYYYMMDDHHII` — always `0000` for the start of day
`end`	path	`202603311615`	`YYYYMMDDHHII` - for a window ending today, the current time (captures the latest 15-min interval); historical windows use `23:59`
`region`	path	`ROI`, `NI`, `ALL`	Fetched for all three
`area`	path	`WIND`, `SOLAR`	One call per area + region pair
`FORMAT`	query	`JSON`	Always `JSON`

The pipeline issues 6 calls per calendar month (2 areas × 3 regions) and iterates across months from date.today() - DAYS_BACK to date.today().

Response row shape¶

{
  "Rows": [
    {
      "EffectiveTime": "01-Mar-2026 00:00",
      "FieldName":     "WIND_FCAST",
      "Value":         615.2,
      "Region":        "ROI",
      "Effective_Date": "01-Mar-2026 00:00"
    }
  ]
}

FieldName distinguishes actual vs forecast measurements (e.g. WIND_ACTUAL / WIND_FCAST). The summarise.py export keys output by FieldName.
Value is megawatts at 15-minute resolution.

Mapping¶

Stage	Path
Parquet key	`grid-data/eirgrid/area=wind/region={ROI\\|NI\\|ALL}/year={YYYY}/month={MM}/data.parquet` (same pattern for `area=solar`)
JSON summary	`grid-data/summary/wind/latest.json`, `grid-data/summary/solar/latest.json`
API endpoint	`GET /api/wind`, `GET /api/solar`

2. `interconn` — cross-border interconnector flows¶

Endpoint¶

GET https://www.interconn.smartgriddashboard.com/api/Interconnector/{start}/{end}/{region}?format=json

Path / query parameters¶

Parameter	Source	Example	Notes
`start`	path	`202603010000`	Same `YYYYMMDDHHII` format as `vargen`
`end`	path	`202603311615`	For a window ending today, the current time; historical windows use `23:59`
`region`	path	`ALL`	Hardcoded to `ALL` — the backend embeds each connector's own region in the response
`format`	query	`json`	Always `json`

One call per calendar month covers every interconnector.

Response row shape¶

{
  "Rows": [
    {
      "Effective_Date": "01-Mar-2026 00:00",
      "Field_Name":     "INTER_EWIC",
      "Value":          250.0,
      "Region":         "ROI"
    }
  ]
}

Field_Name is one of INTER_EWIC, INTER_GRNLK, INTER_MOYLE, INTER_NET.
The Region column varies per connector (EWIC/GRNLK → ROI, MOYLE → NI, NET → ALL) — the export groups by Field_Name, never by Region.
summarise.py strips the INTER_ prefix: the JSON summary exposes EWIC, GRNLK, MOYLE, NET.

Mapping¶

Stage	Path
Parquet key	`grid-data/eirgrid/area=interconnection/region=ALL/year={YYYY}/month={MM}/data.parquet`
JSON summary	`grid-data/summary/interconnection/latest.json`
API endpoint	`GET /api/interconnection`

3. `dashboard` — demand, generation mix, CO₂, frequency, SNSP¶

Endpoint¶

GET https://www.smartgriddashboard.com/DashboardService.svc/data?area={area}&region={region}&datefrom={DD-Mon-YYYY}&dateto={DD-Mon-YYYY}

All requests share the same four query parameters — only area and region differ.

Parameter	Source	Example	Notes
`area`	query	`demandActual`	See table below
`region`	query	`ROI`, `NI`, `ALL`	Constrained by `DASHBOARD_AREA_REGIONS`
`datefrom`	query	`01-Mar-2026`	`DD-Mon-YYYY` (locale-independent month abbreviation)
`dateto`	query	`01-Apr-2026`	The backend treats `dateto` as an exclusive midnight boundary, so a window ending today requests `today + 1 day` to include today's intra-day rows. If that request fails or returns empty, the pipeline retries once with `dateto = yesterday` (graceful degradation) - see `_fetch_dashboard`

Areas¶

Area	Region(s)	Cadence	Fetch strategy	Notes
`demandActual`	`ROI`, `NI`	15-min	Per month	`ALL` is computed in export (`ROI + NI`)
`co2Emission`	`ROI`	15-min	Per month	`ALL`/`NI` variants return 403 or `Value: null`
`SnspAll`	`ALL`	30-min	Per month	System Non-Synchronous Penetration (%)
`frequency`	`ROI`	5-second	Weekly chunks	~51k rows/month — monthly requests time out; summary downsamples to hourly mean
`fuelMix`	`ROI`	Snapshot	Per month (date range ignored)	Always returns 5 rows reflecting the current fuel split

Response row shape¶

{
  "Rows": [
    {
      "EffectiveTime": "01-Mar-2026 00:00",
      "FieldName":     "FUEL_GAS",
      "Value":         24000.0,
      "Region":        "ROI"
    }
  ]
}

Column names vary slightly by area: the pipeline accepts any of EffectiveTime, Effective_Time, or Effective_Date as the timestamp column when parsing the DataFrame.

`fuelMix` — special handling¶

The datefrom/dateto parameters are ignored by the upstream API; it always returns the current 5 fuel rows (FUEL_GAS, FUEL_RENEW, FUEL_COAL, FUEL_OTHER_FOSSIL, FUEL_NET_IMPORT).
Values are cumulative MWh, not MW. summarise.py normalises them to percentages.
The upstream EffectiveTime is stale, so the pipeline overwrites it with the run timestamp (datetime.utcnow()) before persisting, letting daily-partitioned snapshots form a time series.

Mapping¶

Area	Parquet key	JSON summary	API endpoint
`demandActual`	`area=demandactual/region={ROI\\|NI}/year=.../month=...` (monthly)	`summary/demand/latest.json`	`GET /api/demand`
`co2Emission`	`area=co2emission/region=ROI/year=.../month=...` (monthly)	`summary/co2/latest.json`	`GET /api/co2`
`SnspAll`	`area=snspall/region=ALL/year=.../month=...` (monthly)	`summary/snsp/latest.json`	`GET /api/snsp`
`frequency`	`area=frequency/region=ROI/year=.../month=.../day={DD}/data.parquet` (daily — one per weekly chunk, keyed by week-start)	`summary/frequency/latest.json` (hourly avg)	`GET /api/frequency`
`fuelMix`	`area=fuelmix/region=ROI/year=.../month=.../day={DD}/data.parquet` (daily — one per pipeline run)	`summary/generation/latest.json`	`GET /api/generation`

Call volume per pipeline run¶

With the default DAYS_BACK=31, a single pipeline run issues roughly:

Backend	Calls per month	Monthly windows	Total
vargen	6 (`2 areas × 3 regions`)	2 (spans month boundary)	12
interconn	1	2	2
dashboard (non-frequency)	5 (`demandActual×2 + fuelMix + co2Emission + SnspAll`)	2	10
dashboard (frequency)	1 per week	~5	5

Typical total: ~29 HTTP calls per run, executed concurrently up to MAX_CONCURRENCY (default 10). See configuration.md for environment variables that tune concurrency, retry count, and request timeout.

Transforming upstream rows into JSON summaries¶

src/export/summarise.py reads the Parquet partitions with DuckDB and produces seven JSON files under grid-data/summary/. The mapping per area:

Summary JSON	Source area(s)	Key transformation
`wind/latest.json`	vargen `WIND` × (ROI, NI, ALL)	Split by `FieldName` into `actual` / `forecast` per region
`solar/latest.json`	vargen `SOLAR` × (ROI, NI, ALL)	Same as wind
`demand/latest.json`	dashboard `demandActual` × (ROI, NI)	`ALL` = merge-on-timestamp of `ROI + NI` values
`interconnection/latest.json`	interconn (ALL)	Grouped by `Field_Name`; `INTER_` prefix stripped
`co2/latest.json`	dashboard `co2Emission/ROI`	Region-keyed (`ROI` only — no upstream ALL/NI series)
`frequency/latest.json`	dashboard `frequency/ROI`	`date_trunc('hour', EffectiveTime)` + `avg(Value)` (DuckDB query)
`snsp/latest.json`	dashboard `SnspAll/ALL`	Passed through, filtered to `Region = 'ALL'`
`generation/latest.json`	dashboard `fuelMix/ROI` (daily snapshots)	Grouped by `FieldName`; values normalised to % of total MWh

All summaries follow the shape:

{
  "generated_at": "2026-04-21T15:22:34Z",
  "window_days": 30,
  "series": {
    "ROI": [ { "t": "2026-04-20T14:30:00", "value": 3866 }, ... ]
  }
}

Wind and solar swap "value" for "actual"/"forecast". Generation keys each series by fuel type rather than region.

Upstream Data Sources¶

How this dashboard depends on EirGrid¶

API backends at a glance¶

1. vargen — wind and solar generation¶

Endpoint¶

Path / query parameters¶

Response row shape¶

Mapping¶

2. interconn — cross-border interconnector flows¶

Endpoint¶

Path / query parameters¶

Response row shape¶

Mapping¶

3. dashboard — demand, generation mix, CO₂, frequency, SNSP¶

Endpoint¶

Areas¶

Response row shape¶

fuelMix — special handling¶

Mapping¶

Call volume per pipeline run¶

Transforming upstream rows into JSON summaries¶

See also¶

1. `vargen` — wind and solar generation¶

2. `interconn` — cross-border interconnector flows¶

3. `dashboard` — demand, generation mix, CO₂, frequency, SNSP¶

`fuelMix` — special handling¶