Dataset Validation¶

This tutorial focuses on validating CSV datasets after the structure has been planned.

It combines two checks that are demonstrated in examples/example_usage.py:

1. Naming validation — checks filenames and column names.
2. Data validation — checks the actual values inside each CSV resource.

The structure-planning workflow itself is covered separately in the Dataset Architect tutorial.

Learnings¶

This tutorial will enable you to:

• detect invalid filenames,
• detect invalid column names,
• understand the dataset validation report,
• validate individual CSV resources,
• identify missing values,
• identify duplicate rows,
• detect duplicate and invalid timestamps,
• detect potential outliers using the IQR method,
• calculate descriptive statistics,
• generate histograms and boxplots.

Requirements¶

To follow this tutorial, you need:

• Python 3
• Git
• the open-energy-database-compliance-manager repository
• a dataset containing CSV files

Setup¶

Clone the repository:

In [ ]:

git clone https://github.com/OpenEnergyPlatform/open-energy-database-compliance-manager.git

git clone https://github.com/OpenEnergyPlatform/open-energy-database-compliance-manager.git

Move into the repository:

In [ ]:

cd open-energy-database-compliance-manager

cd open-energy-database-compliance-manager

Create a virtual environment:

In [ ]:

python3 -m venv .venv

python3 -m venv .venv

Install the required packages:

In [ ]:

.venv/bin/python -m pip install -r requirements.txt

.venv/bin/python -m pip install -r requirements.txt

Activate the virtual environment:

In [ ]:

source .venv/bin/activate

source .venv/bin/activate

Example dataset¶

The example uses two CSV files:

test/test_data/
├── Bad-File Name!.csv
├── energy_consumption_2023.csv

The files are prepared so that different validation checks can be demonstrated.

Example: invalid naming¶

Bad-File Name!.csv contains:

Date,Energy (kWh),Temperature
2023-01-01,150.5,12.3

The filename and its column names intentionally violate the expected naming convention.

Example: data-quality issues¶

energy_consumption_2023.csv contains:

timestamp,energy_kwh,temperature_c
2023-01-01,150.5,11.2
2023-01-02,145.2,11.8
2023-01-03,152.1,13.5
2023-01-04,149.8,12.7
2023-01-05,151.3,12.9
2023-01-06,148.7,12.4
2023-01-07,150.1,
2023-01-08,149.5,12.6
2023-01-08,149.5,12.6
2023-01-10,151.0,13.0
2023-01-11,149.9,12.8
2023-01-12,320.0,13.1

This file intentionally contains:

• one missing value in temperature_c,
• one duplicated entry for 2023-01-08,
• an unusually high energy_kwh value of 320.0.

These values are used later to demonstrate the data-quality checks.

Run the validation example¶

Run the complete validation workflow:

python examples/example_usage.py

The script first validates the dataset naming and then validates every CSV resource individually.

Part 1: Naming validation¶

The first part of the example checks the dataset as a whole.

It detects:

• available CSV files,
• invalid filenames,
• invalid column names.

Example output:

Total files: 2

❌ FILENAME ISSUES (1):
  • Bad-File Name!.csv
    → Contains uppercase, spaces, or special characters

⚠️ COLUMN NAME ISSUES (1):
  • Bad-File Name!.csv
    → Invalid: Date, Energy (kWh), Temperature

❌ Validation failed - fix issues above.

Naming rules¶

File and column names should:

• use lowercase letters,
• use underscores instead of spaces,
• avoid special characters and brackets,
• use clear and consistent names.

Examples:

Bad-File Name!.csv → bad_file_name.csv
Energy (kWh) → energy_kwh
Date → date
Temperature → temperature

Part 2: Validate individual resources¶

After the naming check, the script processes every CSV resource individually.

For each resource, the script asks:

Do you want to generate plots for this resource? (y/n):

Selecting y generates histograms and boxplots for suitable numeric columns.

First resource: Bad-File Name!.csv¶

This file contains only one data row.

The validation is therefore limited:

• no meaningful outlier analysis is possible,
• no timestamp column is available,
• no missing values are detected,
• no duplicate rows are detected,
• basic descriptive statistics are calculated for numeric columns.

Because there is only one value per numeric column, minimum, maximum, mean and median are identical.

Second resource: energy_consumption_2023.csv¶

This resource contains the intentionally added data-quality issues and demonstrates the validator more clearly.

The validator checks:

• outliers,
• timestamps,
• missing values,
• duplicate rows,
• descriptive statistics,
• optional plots.

Outlier detection¶

Potential outliers are detected with the IQR method.

The validator calculates:

• Q1,
• Q3,
• the interquartile range (IQR),
• a lower bound,
• an upper bound.

Values outside these bounds are reported as potential outliers.

For energy_kwh, the example reports:

145.20
320.00

as potential outliers according to the calculated IQR bounds.

Timestamp validation¶

The timestamp column is checked for invalid and duplicated values.

In the example:

2023-01-08

occurs twice and is therefore reported as a duplicate timestamp.

Missing values¶

The validator counts missing entries for every column and calculates their percentage.

In energy_consumption_2023.csv, one value is missing in:

temperature_c

The file contains 12 rows, so the missing-value percentage is:

1 / 12 = 8.33 %

Duplicate rows¶

The validator also checks whether complete rows occur more than once.

The duplicated 2023-01-08 entry causes:

Duplicate Rows: 1

Descriptive statistics¶

For suitable numeric columns, the validator calculates:

• count,
• minimum,
• maximum,
• mean,
• median,
• standard deviation.

These statistics provide a compact overview of the numerical distribution of each column.

Histograms and boxplots¶

If plot generation is enabled, the validator creates visualizations for suitable numeric columns.

Histograms show the distribution of values.

Boxplots show:

• the median,
• Q1 and Q3,
• the interquartile range,
• whiskers,
• potential outliers.

The generated files are stored in a plots directory associated with the test dataset.

Result¶

The example workflow combines two levels of validation:

1. Naming validation
   Checks filenames and column names for consistency.

2. Data validation
   Checks individual CSV resources for missing values, duplicates, timestamp problems, descriptive statistics and potential outliers.

The Dataset Architect tutorial covers the planning and structural organization of the dataset. This tutorial covers the validation of naming and data quality.