API Tutorial 01 - Download data from the Open Energy Platform using the API¶

Learnings¶

This tutorials will enable you to:

• Download data
• Download filtered data
• Download metadata

Requirements¶

• Python
• Python packages:
  • requests (See the packages documentation for more help)
  • jupyter notebook, e.g. notebook or jupyterlab
  • shapely (only for geometry data conversion)
• Skills:
  • use python with jupyter notebooks
  • some understanding of JSON encoded data
• Internet: obviously, but also make sure your network settings allow https traffic from python

Feedback¶

You can provide feedback on this tutorial in this GitHub issue

Setup¶

In [1]:

# install required packages with: pip install requests
# import required packages
import json

import requests as req

# TODO: explain schema/topic/sandbox
# TODO: explain randint

topic = "model_draft"
table = "tutorial_example_table"
table_api_url = f"https://openenergyplatform.org/api/v0/schema/{topic}/tables/{table}/"
table_web_url = f"https://openenergyplatform.org/dataedit/view/{topic}/{table}"

print(f"you can see the table in your browser here: {table_web_url}")

# install required packages with: pip install requests # import required packages import json import requests as req # TODO: explain schema/topic/sandbox # TODO: explain randint topic = "model_draft" table = "tutorial_example_table" table_api_url = f"https://openenergyplatform.org/api/v0/schema/{topic}/tables/{table}/" table_web_url = f"https://openenergyplatform.org/dataedit/view/{topic}/{table}" print(f"you can see the table in your browser here: {table_web_url}")

you can see the table in your browser here: https://openenergyplatform.org/dataedit/view/model_draft/tutorial_example_table

Download data¶

requires execution of setup

In [2]:

res = req.get(table_api_url + "rows/")
data = res.json()

# show results in notebook
print(json.dumps(data, indent=4))

res = req.get(table_api_url + "rows/") data = res.json() # show results in notebook print(json.dumps(data, indent=4))

[
    {
        "id": 1,
        "name": "unit1",
        "is_active": true,
        "capacity_mw": 1.2,
        "installation_datetime_utc": "2010-02-03T00:00:00",
        "location": "01010000008FC2F5285C8F4A40713D0AD7A3B02040"
    },
    {
        "id": 2,
        "name": "unit2",
        "is_active": false,
        "capacity_mw": 2.1,
        "installation_datetime_utc": "2010-01-08T00:00:00",
        "location": null
    },
    {
        "id": 3,
        "name": "unit3",
        "is_active": true,
        "capacity_mw": 100.0,
        "installation_datetime_utc": "2010-01-02T10:30:00",
        "location": "0101000000EC51B81E85AB4B405C8FC2F5285C1D40"
    }
]

In [ ]:

Notes on data types¶

The api returns json serializable data, which is only bool, numerical, or string. In the example, the value of installation_datetime_utc is a string representation of a datetime object, the value of location a WKB string representation of a geometry object.

Depending on the use case, the data needs to be converted after the download

In [3]:

from shapely import wkb
from datetime import datetime

data_converted = []
for row in data:
    row_converted = row.copy()
    # convert datetime string to datetime object
    row_converted["installation_datetime_utc"] = datetime.strptime(
        row_converted["installation_datetime_utc"], "%Y-%m-%dT%H:%M:%S"
    )
    # convert wkb string to shapely geoemtry object
    row_converted["location"] = wkb.loads(row_converted["location"])
    data_converted.append(row_converted)

from shapely import wkb from datetime import datetime data_converted = [] for row in data: row_converted = row.copy() # convert datetime string to datetime object row_converted["installation_datetime_utc"] = datetime.strptime( row_converted["installation_datetime_utc"], "%Y-%m-%dT%H:%M:%S" ) # convert wkb string to shapely geoemtry object row_converted["location"] = wkb.loads(row_converted["location"]) data_converted.append(row_converted)

Download filtered data¶

• requires execution of setup
• read more about query strings

In [3]:

# Add (multiple) where filters to the url
filter = "where=is_active=true&where=capacity_mw>10"

res = req.get(table_api_url + f"rows/?{filter}")
data = res.json()

# show results in notebook
print(json.dumps(data, indent=4))

# Add (multiple) where filters to the url filter = "where=is_active=true&where=capacity_mw>10" res = req.get(table_api_url + f"rows/?{filter}") data = res.json() # show results in notebook print(json.dumps(data, indent=4))

[
    {
        "id": 3,
        "name": "unit3",
        "is_active": true,
        "capacity_mw": 100.0,
        "installation_datetime_utc": "2010-01-02T10:30:00",
        "location": "0101000000EC51B81E85AB4B405C8FC2F5285C1D40"
    }
]

Download metadata¶

requires execution of setup

In [4]:

res = req.get(table_api_url + "meta/")
metadata = res.json()

# show (partial) results in notebook
print(json.dumps(metadata, indent=4))

res = req.get(table_api_url + "meta/") metadata = res.json() # show (partial) results in notebook print(json.dumps(metadata, indent=4))

{
    "id": "test_table",
    "_comment": {
        "null": "If not applicable use: null",
        "todo": "If a value is not yet available, use: todo",
        "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)",
        "units": "Use a space between numbers and units (100 m)",
        "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/blob/master/data-review/manual/review_manual.md)",
        "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)",
        "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata)",
        "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)"
    },
    "keywords": [
        "energy",
        "installations",
        "geo"
    ],
    "resources": [
        {
            "name": "test_table",
            "schema": {
                "fields": [
                    {
                        "name": "id",
                        "type": "integer",
                        "description": "numerical id of this data point"
                    },
                    {
                        "name": "name",
                        "type": "string",
                        "description": "name of installation"
                    },
                    {
                        "name": "is_active",
                        "type": "boolean",
                        "description": "true/false if installation is active"
                    },
                    {
                        "name": "capacity_mw",
                        "type": "number",
                        "description": "installed capacity in MW"
                    },
                    {
                        "name": "installation_datetime_utc",
                        "type": "string",
                        "description": "date (and time) when installation was build"
                    },
                    {
                        "name": "location",
                        "type": "geojson",
                        "description": "point location of installation"
                    }
                ],
                "foreignKeys": []
            }
        }
    ],
    "metaMetadata": {
        "metadataLicense": {
            "name": "CC0-1.0",
            "path": "https://creativecommons.org/publicdomain/zero/1.0/",
            "title": "Creative Commons Zero v1.0 Universal"
        },
        "metadataVersion": "OEP-1.5.2"
    }
}

About this tutorial¶

• Author: Christian Winger
• Copyright: Öko-Institut (2024)
• License: CC BY 4.0
• Attribution: Öko-Institut (2024): API Tutorial 01- Download data from the Open Energy Platform using the API.
• Last update 2024-11-18