OEP Client Tutorial 02 - Upload data using the OEP Client in a python script¶

Learnings¶

This tutorials will enable you to:

• Create a table on the OEP
• Upload data to that table
• Upload metadata accompanying the data
• Delete the table and the corresponding metadata

Requirements¶

To execute the following code you will need to need to be a registered user on the OEP and

• have your OEP API token at hand. You find this in settings tab in your profile page

You will also need to have the following installed on your computer:

• Python

• Python packages:

  • oep-client>=0.14 (installs also the command line tool)
  • jupyter notebook, e.g. notebook or jupyterlab

• Ideally you exhibit the following skills:

  • use python with jupyter notebooks

Setup¶

This part of the code sets up all you need to execute the further code below. It imports the required packages into python, sets up the test table and asks you for the OEP API token so that you can identify with the OEP, which is mandatory if you want to upload to the OEP. The setup is a necessary step.

In [1]:

# install required packages with: pip install "oep-client>=0.14"
# import required packages
import json
from random import randint
from getpass import getpass
from os import environ

import requests as req
from oep_client import OepClient

topic = 'sandbox'
table = f'tutorial_example_table_{randint(0, 100000)}'
token = environ.get("OEP_API_TOKEN") or getpass('Enter your OEP API token:')

cli = OepClient(token=token, default_schema=topic)

# install required packages with: pip install "oep-client>=0.14" # import required packages import json from random import randint from getpass import getpass from os import environ import requests as req from oep_client import OepClient topic = 'sandbox' table = f'tutorial_example_table_{randint(0, 100000)}' token = environ.get("OEP_API_TOKEN") or getpass('Enter your OEP API token:') cli = OepClient(token=token, default_schema=topic)

Create table¶

This part of the code creates the table you will later upload to the OEP.

To be able to execute this part you will need to have executed the setup

In [2]:

# TODO: explain / link to data types
table_schema = {
    "columns": [
        # NOTE: first column should be numerical column named `id` .
        # Use `bigserial` if you want the database to create the re
        {"name": "id", "data_type": "bigserial", "primary_key": True},
        {"name": "name", "data_type": "varchar(18)", "is_nullable": False},
        {"name": "is_active", "data_type": "boolean"},
        {"name": "capacity_mw", "data_type": "float"},
        {"name": "installation_datetime_utc", "data_type": "datetime"},        
        {"name": "location", "data_type": "geometry"},
    ]
}

# TODO: explain / link to data types table_schema = { "columns": [ # NOTE: first column should be numerical column named `id` . # Use `bigserial` if you want the database to create the re {"name": "id", "data_type": "bigserial", "primary_key": True}, {"name": "name", "data_type": "varchar(18)", "is_nullable": False}, {"name": "is_active", "data_type": "boolean"}, {"name": "capacity_mw", "data_type": "float"}, {"name": "installation_datetime_utc", "data_type": "datetime"}, {"name": "location", "data_type": "geometry"}, ] }

In [3]:

cli.create_table(table, table_schema)

cli.create_table(table, table_schema)

Out[3]:

{}

Upload data¶

This part of the code uploads the table that you have created above to the OEP.

To be able to execute this part you will need to have executed the setup and you will need to have created a table.

In [4]:

# TODO: explain required data structure

# get example data
data = req.get("https://raw.githubusercontent.com/OpenEnergyPlatform/academy/production/docs/data/tutorial_example_table.data.json").json()

# show results in notebook
print(json.dumps(data, indent=4))

# TODO: explain required data structure # get example data data = req.get("https://raw.githubusercontent.com/OpenEnergyPlatform/academy/production/docs/data/tutorial_example_table.data.json").json() # show results in notebook print(json.dumps(data, indent=4))

[
    {
        "name": "unit1",
        "is_active": true,
        "capacity_mw": 1.2,
        "installation_datetime_utc": "2010-02-03 00:00:00",
        "location": "POINT(53.12 8.345)"
    },
    {
        "name": "unit2",
        "is_active": false,
        "capacity_mw": 2.1,
        "installation_datetime_utc": "2010-01-08",
        "location": null
    },
    {
        "name": "unit3",
        "is_active": true,
        "capacity_mw": 100.0,
        "installation_datetime_utc": "2010-01-02 10:30:00",
        "location": "Point(55.34 7.34)"
    }
]

In [5]:

cli.insert_into_table(table, data)

cli.insert_into_table(table, data)

Upload metadata¶

This part of the code will upload metadata that described the data in your table to the OEP.

To be able to execute this part you will need to have executed the setup and you will need to have created a table.

In [6]:

# get metadata (from example file)
metadata = req.get("https://raw.githubusercontent.com/OpenEnergyPlatform/academy/production/docs/data/tutorial_example_table.metadata.json").json()

metadata = cli.set_metadata(table, metadata)
print(json.dumps(metadata, indent=4))

# get metadata (from example file) metadata = req.get("https://raw.githubusercontent.com/OpenEnergyPlatform/academy/production/docs/data/tutorial_example_table.metadata.json").json() metadata = cli.set_metadata(table, metadata) print(json.dumps(metadata, indent=4))

{
    "id": "test_table",
    "keywords": [
        "energy",
        "installations",
        "geo"
    ],
    "resources": [
        {
            "name": "test_table",
            "schema": {
                "foreignKeys": [],
                "fields": [
                    {
                        "name": "id",
                        "description": "numerical id of this data point",
                        "type": "integer"
                    },
                    {
                        "name": "name",
                        "description": "name of installation",
                        "type": "string"
                    },
                    {
                        "name": "is_active",
                        "description": "true/false if installation is active",
                        "type": "boolean"
                    },
                    {
                        "name": "capacity_mw",
                        "description": "installed capacity in MW",
                        "type": "number"
                    },
                    {
                        "name": "installation_datetime_utc",
                        "description": "date (and time) when installation was build",
                        "type": "string"
                    },
                    {
                        "name": "location",
                        "description": "point location of installation",
                        "type": "geojson"
                    }
                ]
            }
        }
    ],
    "metaMetadata": {
        "metadataVersion": "OEP-1.5.2",
        "metadataLicense": {
            "name": "CC0-1.0",
            "title": "Creative Commons Zero v1.0 Universal",
            "path": "https://creativecommons.org/publicdomain/zero/1.0/"
        }
    },
    "_comment": {
        "metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/oemetadata)",
        "dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss\u00b1hh)",
        "units": "Use a space between numbers and units (100 m)",
        "languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)",
        "licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)",
        "review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/blob/master/data-review/manual/review_manual.md)",
        "null": "If not applicable use: null",
        "todo": "If a value is not yet available, use: todo"
    }
}

Delete table¶

This part of the code deletes the table (including the metadata) that you have uploaded to the OEP.

To be able to execute this part you will need to have executed the setup and you will need to have created a table.

In [7]:

cli.drop_table(table)

cli.drop_table(table)

Out[7]:

{}