Dataset Architect¶

The Dataset Architect is mainly used to plan and structure datasets before they are transformed or published.

It detects the available CSV resources, describes their structure, classifies their role, groups tables with identical column structures and creates an target structure.

Learnings¶

This tutorial will enable you to:

• inspect the resources of a CSV dataset,
• understand rows, columns and resource types,
• classify resources as Data, Metadata, Additional Data or Ignore,
• group tables with identical column structures,
• create and review a current and planned dataset structure,
• review the merge overview

Requirements¶

To follow this tutorial, you need:

• Python 3
• Git
• the open-energy-database-compliance-manager repository
• a dataset containing CSV files

Setup¶

Clone the repository:

In [ ]:

git clone https://github.com/OpenEnergyPlatform/open-energy-database-compliance-manager.git

git clone https://github.com/OpenEnergyPlatform/open-energy-database-compliance-manager.git

Move into the repository:

In [ ]:

cd open-energy-database-compliance-manager

cd open-energy-database-compliance-manager

Create a virtual environment:

In [ ]:

python3 -m venv .venv

python3 -m venv .venv

Install the required packages:

In [ ]:

.venv/bin/python -m pip install -r requirements.txt

.venv/bin/python -m pip install -r requirements.txt

When working directly in a terminal, activate the environment with source .venv/bin/activate.

Example dataset¶

This tutorial uses three CSV files:

test/test_data/
├── Bad-File Name!.csv
├── energy_consumption_2023.csv
└── energy_consumption_2024.csv

The two energy files use the same column structure so that the planner can group them. The third file is used later to demonstrate naming issues.

Example: energy_consumption_2023.csv¶

timestamp,energy_kwh,temperature_c
2023-01-01,150.5,11.2
2023-01-02,145.2,11.8
2023-01-03,152.1,13.5
2023-01-04,149.8,12.7
2023-01-05,151.3,12.9
2023-01-06,148.7,12.4
2023-01-07,150.1,
2023-01-08,149.5,12.6
2023-01-08,149.5,12.6
2023-01-10,151.0,13.0
2023-01-11,149.9,12.8
2023-01-12,320.0,13.1

Example: energy_consumption_2024.csv¶

timestamp,energy_kwh,temperature_c
2024-01-01,151.2,11.5
2024-01-02,148.7,12.0
2024-01-03,153.1,13.2
2024-01-04,150.4,12.8
2024-01-05,149.9,12.6

The number of rows may differ. For structural grouping, the important point is that both tables use the same column structure.

Example: invalid naming¶

Bad-File Name!.csv contains:

Date,Energy (kWh),Temperature
2023-01-01,150.5,12.3

The filename and column names intentionally violate the expected naming convention.

Plan the dataset structure¶

The structure-planning workflow is started with data_plan/1_plan_structure_hsrm.py.

For this tutorial, the configured dataset path must point to the example data:

dataset_path = Path("test/test_data")

The workflow performs the following steps:

1. load the CSV resources,
2. create a resource catalog,
3. review the resource types,
4. compare Data resources,
5. group identical column structures,
6. create the current and planned structure,
7. save grouped structure files,
8. generate a merge overview,

Run the planning workflow¶

Run the script from a terminal.

python data_plan/1_plan_structure_hsrm.py

Step 1: Load the dataset¶

The first step scans the configured directory and detects the available CSV files.

Example output:

📦 Step 1: Loading dataset...
Dataset Path: test/test_data
Files Found: 3
Found 3 resources

📊 ANALYSIS COMPLETE

The tool now knows which resources are available and records basic properties such as rows and columns.

Continue with y when the workflow asks whether the planning phase should start.

Step 2: Create and review the catalog¶

The planner creates two catalog files:

• *_catalog_draft.csv — automatically generated reference,
• *_catalog.csv — editable working copy used by the planner.

The catalog contains information such as filename, type, rows, columns, size_mb, format, encoding, target_group and notes.

For this example, classify the two energy files as Data and exclude the naming-test file from grouping:

Bad-File Name!.csv          → Ignore
energy_consumption_2023.csv → Data
energy_consumption_2024.csv → Data

After editing the catalog, confirm the resource types with y.

Step 3: Group identical table structures¶

The planner compares resources classified as Data or Additional Data.

Both energy files contain the same columns:

timestamp, energy_kwh, temperature_c

The planner therefore assigns them to the same target group:

Group 1:
- energy_consumption_2023.csv
- energy_consumption_2024.csv

The output confirms that one group with an identical column structure was found. Review the target_group values in the catalog and confirm the assignments.

Step 4: Create the structure plan¶

The planner creates two dataset-level YAML files:

• *_structure_dataset_current.yaml — current structure,
• *_structure_dataset_plan.yaml — planned target structure.

For the example, the plan contains two Data resources that belong to one structural group.

Step 5: Save grouped structures and create the merge overview¶

For each detected group, the workflow creates:

• *_structure_group1_draft.yaml — automatically generated group structure,
• *_structure_group1_target.yaml — editable target structure.

It also creates:

data/1_planning/test_data/plots/
└── test_data_v0.9.0_merge_overview.png

The graphic shows which existing resources belong together and how they are planned to be used.

Result¶

The Dataset Architect workflow can be summarized as:

CSV resources → catalog → classification → structural groups → current/target structure → merge overview → metadata drafts

This tutorial focuses on dataset structure and naming. Value-level data-quality checks are handled separately.