Skip to content

Maintenance & troubleshooting#

You are here

Final page of the Production deployment (Podman) guide, after Update. Day-to-day operations plus the failure modes seen during bring-up.

Everyday operations#

# Status of one service / all services
systemctl --user status oep-oeplatform
systemctl --user status 'oep-*'

# Follow logs
journalctl --user -u oep-oeplatform -f

# Restart one service (dependencies keep running)
systemctl --user restart oep-oeplatform

# Stop / start everything
systemctl --user stop  oep-postgres oep-fuseki oep-oeplatform oep-ontop oep-lookup
systemctl --user start oep-postgres oep-fuseki oep-oeplatform oep-ontop oep-lookup

Open a shell in the app container:

podman exec -it oeplatform bash

Ontop operations live on their own page

For applying/updating the ontop mapping, rebuilding with a different JDBC version, or the lazy-init behaviour, see Ontop — that procedure is not repeated here.

Access log#

The app container's Apache writes one line per request to stdout, so it lands in the journal next to the error log. The line is Common Log Format with two fields appended at the end:

10.0.2.100 - - [07/Oct/2026:15:37:20 +0000] "GET /database/ HTTP/1.1" 200 5120 "Mozilla/5.0 ..." 882
Field (awk) Meaning
$1 client address (behind the reverse proxy: the proxy's)
$4 $5 time
$7 path, with query string
$9 HTTP status
$10 response size in bytes, - for an empty body (e.g. 304)
quoted user agent
$NF time to serve the request, in microseconds

Referer, cookies and the Authorization header are deliberately not logged.

Total server time per path today, the most useful ranking when the platform is slow (a path called often can cost more than a slow one called rarely):

journalctl --user -u oep-oeplatform --since today -o cat \
  | awk '$9 ~ /^[0-9][0-9][0-9]$/ && $NF ~ /^[0-9]+$/ {p=$7; sub(/\?.*/, "", p); t[p]+=$NF; n[p]++}
         END {for (p in t) printf "%9.1fs %6d  %s\n", t[p]/1e6, n[p], p}' \
  | sort -rn | head -15

Columns: total seconds, number of requests, path. The filter on $9 and $NF skips error-log lines, which share the journal.

Backups#

Persistent state is in named volumes (physically under the service user's home). At minimum, back up the Postgres data, the Fuseki volume (the OEKG) and the media volume regularly, and always before a release that runs migrations. List volumes and their locations:

podman volume ls
podman volume inspect systemd-pgdata

Quadlet prefixes every volume name with systemd-.

Use pg_dump from inside/against the postgres container for logical database backups, and snapshot the media and Fuseki volumes. Stop oep-oeplatform, the only writer, while copying the Fuseki volume, so the copy is consistent. (Physical volume placement per server is a separate storage-configuration concern, out of scope for this generic guide.)

Reset the database (destructive)#

This deletes all database data. Only for a throwaway/dev-like environment.

systemctl --user stop oep-oeplatform oep-postgres
podman volume rm systemd-pgdata   # confirm the exact name with: podman volume ls
systemctl --user start oep-postgres oep-oeplatform

The postgres container recreates the schema on a fresh volume; the app re-runs migrations on start.

Troubleshooting — common failure modes#

These are the issues most likely to bite during a first deployment.

CSRF "origin not trusted" / config error at startup#

Symptom: Django rejects form posts, or refuses to start complaining about CSRF_TRUSTED_ORIGINS.

Cause: OEP_CSRF_TRUSTED_ORIGINS values must include a scheme.

Fix: use https://your-domain.org, not your-domain.org. Multiple origins are comma-separated. Restart the app after editing oep.env.

DisallowedHost / blank page after DNS resolves#

Symptom: Invalid HTTP_HOST header in the logs, or the site 400s.

Cause: the request's host isn't in OEP_ALLOWED_HOSTS.

Fix: set OEP_ALLOWED_HOSTS to the exact public domain (comma-separated if several), then systemctl --user restart oep-oeplatform.

App can't reach the database / could not translate host name "postgres"#

Symptom: the app or ontop logs a DNS failure for postgres.

Cause: the postgres container isn't running, or the service isn't on the oep network.

Fix: systemctl --user status oep-postgres — start it if it's down. All services must be on the oep network (Quadlets attach them automatically); confirm with podman network inspect oep.

Postgres won't start / POSTGRES_PASSWORD not specified#

Symptom: the postgres container exits immediately on first start.

Cause: empty or mistyped DB credentials in oep.env.

Fix: fill in POSTGRES_USER / POSTGRES_PASSWORD (and the LOCAL_DB_* values) — watch for typos. Restart the service.

Service fails with exit code 125#

Symptom: systemctl --user status shows the unit failed with status 125.

Cause: Podman couldn't find the image under the name the unit expects (image-name mismatch) — common when you built an image under one tag but the unit references another.

Fix: confirm the image exists under the exact name the unit uses (podman images), and re-tag if needed, e.g. podman tag localhost/oeplatform:latest <expected-name>.

Lookup service fails to start / can't find its config#

Symptom: the lookup unit fails, complaining about a missing config path.

Cause: the config bind-mount path didn't resolve to your checkout.

Fix: the install script substitutes the checkout path into the unit for you — re-run bash podman/quadlets/install.sh from the checkout you want to use, then systemctl --user daemon-reload and restart the service. (Do not add symlinks — the path is resolved automatically.)

Upstream proxy shows "Error during SSL Handshake with remote server"#

Symptom: the browser gets a proxy error; the upstream proxy log mentions a backend SSL handshake failure.

Cause: the upstream proxy re-encrypts to this host over HTTPS, but nginx here isn't terminating TLS (e.g. it's listening plain HTTP, or has no certificate).

Fix: ensure nginx listens 443 ssl with a certificate present under /etc/nginx/ssl/oeplatform/. Re-run sudo bash podman/nginx/install-nginx.sh to regenerate the self-signed cert and site config, then reload nginx.

Services vanish after logout / don't come back after reboot#

Symptom: everything stops when you log out, or nothing runs after a reboot.

Cause: linger isn't enabled for the service user.

Fix: sudo loginctl enable-linger <service-user>, and make sure the units are enabled (systemctl --user enable oep-*).

Where to look#

Symptom area First command
App behaviour journalctl --user -u oep-oeplatform -f
Database journalctl --user -u oep-postgres -f
SPARQL-over-SQL journalctl --user -u oep-ontop -f + Ontop
TLS / proxy nginx error log + sudo nginx -t
Networking / DNS podman network inspect oep

That's the full production deployment path: Overview → Install → Ontop → Update → Maintenance. Optional next: Monitoring. For the OEP team's concrete server values, see the internal runbook (kept outside this public documentation).