Data Model

S3 key structure

All Parquet files use Hive-style partitioning for compatibility with AWS Athena, DuckDB, and any Parquet-aware query tool.

graph TD
    ROOT["{s3_prefix}/{source_id}/"] --> AREA["area={area}/"]
    AREA --> REGION["region={region}/"]
    REGION --> YEAR["year={YYYY}/"]
    YEAR --> MONTH["month={MM}/"]
    MONTH --> MONTHLY["data.parquet<br/>(most areas)"]
    MONTH --> DAY["day={DD}/"]
    DAY --> DAILY["data.parquet<br/>(fuelMix, frequency)"]

Monthly partitions

Used for all areas except fuelMix and frequency:

grid-data/eirgrid/area=wind/region=ROI/year=2026/month=03/data.parquet
grid-data/eirgrid/area=wind/region=NI/year=2026/month=03/data.parquet
grid-data/eirgrid/area=demandactual/region=ROI/year=2026/month=03/data.parquet
grid-data/eirgrid/area=co2emission/region=ROI/year=2026/month=03/data.parquet
grid-data/eirgrid/area=snspall/region=ALL/year=2026/month=03/data.parquet
grid-data/eirgrid/area=interconnection/region=ALL/year=2026/month=03/data.parquet

Daily partitions

Used for snapshot-only APIs that must accumulate over time:

grid-data/eirgrid/area=fuelmix/region=ROI/year=2026/month=03/day=10/data.parquet
grid-data/eirgrid/area=frequency/region=ROI/year=2026/month=03/day=07/data.parquet

fuelMix is partitioned daily because the EirGrid API always returns the current snapshot regardless of the requested date range. Stamping each run with the pipeline's wall-clock time and writing to a daily key turns a snapshot API into a time series.

frequency is partitioned daily (weekly chunks) because it is fetched in 7-day chunks to avoid timeouts; the chunk-start date is used as the partition key.

Parquet schema per area

wind / solardemandActualinterconnectionco2EmissionfuelMixfrequencySnspAll

Column	Type	Notes
Effective_Date	timestamp	15-min interval start
MW_ACTUAL	float	Measured output (MW)
MW_FORECAST	float	Forecast output (MW)
Region	string	ROI / NI / ALL

Column	Type	Notes
EffectiveTime	timestamp	15-min interval start
Value	float	System demand (MW)
Region	string	ROI / NI

Column	Type	Notes
Effective_Date	timestamp	15-min interval start
Value	float	Flow (MW) — positive = import
Field_Name	string	INTER_EWIC / INTER_GRNLK / INTER_MOYLE / INTER_NET
Region	string	ROI / NI / ALL (per connector)

Column	Type	Notes
EffectiveTime	timestamp	15-min interval start
Value	float	Emissions (tonnes/hour)
Region	string	ROI

Column	Type	Notes
EffectiveTime	timestamp	Pipeline run time (not API value)
Value	float	Cumulative MWh
FieldName	string	FUEL_GAS / FUEL_RENEW / FUEL_COAL / FUEL_OTHER_FOSSIL / FUEL_NET_IMPORT
Region	string	ROI

Column	Type	Notes
EffectiveTime	timestamp	5-second measurement time
Value	float	System frequency (Hz)
Region	string	ROI

Column	Type	Notes
EffectiveTime	timestamp	15-min interval start
Value	float	SNSP (%)
Region	string	ALL

JSON summary format

After each pipeline run, run_export() reads all Parquet partitions via DuckDB and writes seven pre-computed JSON summaries to S3. The API loads these into memory; they are not computed at request time.

grid-data/summary/wind/latest.json
grid-data/summary/solar/latest.json
grid-data/summary/demand/latest.json
grid-data/summary/interconnection/latest.json
grid-data/summary/co2/latest.json
grid-data/summary/frequency/latest.json
grid-data/summary/snsp/latest.json
grid-data/summary/generation/latest.json

Common envelope

{
  "generated_at": "2026-03-10T14:00:00Z",
  "window_days": 33,
  "series": {
    "<key>": [
      {"t": "2026-02-05T10:00:00", "value": 3866.0},
      {"t": "2026-02-05T10:15:00", "value": 3901.0}
    ]
  }
}

null values indicate readings not yet available.

Series key conventions

Summary	Series keys	Point fields
wind, solar	ROI, NI, ALL	actual, forecast
demand	ROI, NI, ALL	value
interconnection	EWIC, GRNLK, MOYLE, NET	value
co2	ROI	value
frequency	ROI	value (hourly average)
snsp	ALL	value
generation	FUEL_GAS, FUEL_RENEW, FUEL_COAL, FUEL_OTHER_FOSSIL, FUEL_NET_IMPORT	value

Notes

• demand/ALL is derived by the export layer (ROI + NI aligned by timestamp). The EirGrid API returns 403 for demandActual/ALL.
• frequency is downsampled from 5-second to hourly average entirely in DuckDB; raw rows never load into Python.
• generation values are cumulative MWh from the fuelMix API snapshot. The export layer normalises them to percentages for the dashboard.
• interconnection series are grouped by connector name (Field_Name), not by Region. Each connector uses its own region in the raw data (EWIC/GRNLK → ROI, MOYLE → NI, NET → ALL).

Querying Parquet directly

Because the files use Hive partitioning you can query them directly with DuckDB or AWS Athena without going through the API:

import duckdb

con = duckdb.connect()
con.execute("INSTALL httpfs; LOAD httpfs;")
# configure S3 credentials ...

df = con.execute("""
    SELECT Effective_Date, MW_ACTUAL, MW_FORECAST, Region
    FROM read_parquet(
        's3://eirgrid-data/grid-data/eirgrid/area=wind/**/*.parquet',
        hive_partitioning=true
    )
    WHERE Region = 'ALL'
      AND Effective_Date >= current_date - INTERVAL 7 DAYS
    ORDER BY Effective_Date
""").df()