Infrastructure & Monitoring

GeoLens exposes Prometheus-compatible metrics on the API container at /metrics and on the worker container at port 8001 (the API’s port is published on the host loopback — 127.0.0.1:8001 by default — and is not reachable from the public edge; the worker’s port is internal to the Docker network only; see Metrics endpoint), plus a connectivity health endpoint at /api/health (through the bundled Nginx; /health on the API container directly). The admin web UI surfaces both at /admin/overview along with catalog statistics. This page covers the operator-facing monitoring surface; service-level diagnostics (Docker logs, database size queries) are at the bottom.

Replace https://geolens.example.com with your GeoLens instance’s URL in every example below.

Metrics endpoint

The /metrics endpoint serves Prometheus-format metrics, gzipped, response-buffered, with scrape paths excluded from histogram contamination. It is served on the API container port (:8000) inside the Docker network and is deliberately not exposed at the public edge: the bundled Nginx returns 404 for /api/metrics, so a scrape aimed at the public hostname reaches nothing. Point Prometheus at the internal service address (http://api:8000/metrics) instead. The endpoint is unauthenticated; if you deliberately republish that port outside the Docker network, put it behind a reverse-proxy IP allowlist or basic auth.

The worker serves its own /metrics on port 8001, and the job-queue series live there rather than on the API’s endpoint: the loop that refreshes those gauges runs inside the worker process. That port is internal-only and is not published by the bundled compose files. Scrape both, or the job rows below are simply absent.

Metric	Served on	Type	Labels	Description
http_requests_total	api	counter	method, status, handler	Total HTTP requests served
http_request_duration_seconds	api	histogram	method, status, handler	Request latency distribution
http_requests_inprogress	api	gauge	method, handler	In-flight requests
geolens_jobs_queue_depth	worker	gauge	queue	Pending jobs (Procrastinate status=todo)
geolens_jobs_active	worker	gauge	queue	Running jobs (Procrastinate status=doing)
geolens_jobs_completed_total	worker	counter	queue	Completed jobs (since process start)
geolens_jobs_failed_total	worker	counter	queue	Failed jobs (since process start)
geolens_db_pool_checkedout	api	gauge	(none)	Connections currently checked out
geolens_db_pool_checkedin	api	gauge	(none)	Connections currently available in pool
geolens_db_pool_overflow	api	gauge	(none)	Overflow connections currently open
geolens_db_pool_size	api	gauge	(none)	Configured pool size

Sample Prometheus scrape configuration, matching the reference config shipped at infra/monitoring/prometheus.yml:

scrape_configs:
  - job_name: geolens-api
    metrics_path: /metrics
    static_configs:
      # Internal Docker-network address of the API container, not the public host.
      - targets: ['api:8000']
        labels:
          service: api

  - job_name: geolens-worker
    metrics_path: /metrics
    static_configs:
      - targets: ['worker:8001']
        labels:
          service: worker

For Grafana dashboards, the geolens_jobs_queue_depth and geolens_db_pool_overflow series are the most actionable. The alert rules shipped at infra/monitoring/alerts.yml fire on queue depth above 100 sustained for 15 minutes (worker undersizing) and on any non-zero pool overflow sustained for 10 minutes (DB-connection contention) — start from those thresholds rather than inventing your own.

Health endpoint

GET /api/health returns 200 (healthy) or 503 (degraded), with a JSON body covering each provider:

Caution

Through the bundled Nginx, the probe lives at /api/health. A bare /health on the proxied URL is answered by the frontend SPA catch-all with HTML and status 200 — an uptime monitor pointed there will report healthy no matter what. (/health without the prefix is correct only on the API container directly; see the note at the bottom.)

{
  "status": "healthy",
  "providers": {
    "database": { "status": "ok", "latency_ms": 12.3 },
    "storage": { "status": "ok", "latency_ms": 45.2 },
    "cache": { "status": "ok", "latency_ms": 1.1 }
  },
  "version": "<running version>",
  "build": "<image commit SHA, or null for source builds>"
}

version reports the running application version and build the release image’s commit SHA (null for local/source builds), so a deployment can be verified over HTTP — production instances hide /api/docs, which was previously the only surface exposing the version.

The probes:

• database: exercises a live SELECT to_regclass('catalog.datasets') (catches hung DB, broken search_path).
• storage: calls the configured storage provider’s health_check() (S3 HeadBucket or local writability test).
• cache: calls Valkey/Redis PING.

Use this endpoint as the upstream health check for load balancers and for a Kubernetes readiness probe: 503 is intentional, and it signals “do not route traffic here” without 5xx-class application errors that would page on-call.

# Basic check
curl -fsS https://geolens.example.com/api/health || echo "unhealthy"

# Detailed JSON with latency breakdown
curl -s https://geolens.example.com/api/health | jq

For internal/private endpoints, the FastAPI process exposes the same health check directly at /health on the API container port (:8000 inside the Docker network, e.g. http://api:8000/health; published on the host at :8001 by default). Use this when nginx/the frontend container is itself the failure point.

Liveness: point your own probe at /health/live, not /health

/health checks the database, object store, and cache — a readiness signal. Since 1.18.0 there’s a separate GET /health/live (also proxied at /api/health/live) that answers only “is the process up”, with no dependency calls: {"status": "ok"}, always 200 if the process can respond at all. It’s unauthenticated and exempt from rate limiting — a kubelet probing every second should never see a 429 that gets the pod killed. The shipped Docker healthchecks and the compose file’s own depends_on: service_healthy gates target /health/live. If you run a custom orchestrator with its own liveness probe pointed at /health, move it to /health/live: a database, MinIO, or Valkey outage should not restart-loop an otherwise-healthy API process, which is what a dependency-probing endpoint behind a liveness check causes. Keep readiness probes on /health.

OIDC connectivity validation

OIDC connectivity validation runs separately from the standard /health probe: IdP discovery URLs are checked on demand rather than on every health poll, since cold-cache IdP fetches add 200 to 500 ms latency.

Trigger validation via the admin UI:

1. Navigate to Admin -> Config Ops.
2. Click Validate Connectivity.
3. The panel reports per-service and per-provider latency, status, and any error details (e.g., DNS failure, expired discovery cache, certificate mismatch).

Or via the API:

curl -X POST https://geolens.example.com/api/config-ops/validate/ \
  -H "Authorization: Bearer $TOKEN"

The response probes storage and cache, plus one entry per enabled OIDC provider keyed by slug:

{
  "storage": { "name": "storage", "status": "ok", "latency_ms": 45.2 },
  "cache": { "name": "cache", "status": "ok", "latency_ms": 1.1 },
  "oidc_providers": {
    "google": { "name": "google", "status": "ok", "latency_ms": 142.7 },
    "keycloak": { "name": "keycloak", "status": "error", "latency_ms": 0.0, "error": "Connection refused" }
  }
}

Run validation after any of: (1) adding a new OAuth provider, (2) rotating client secrets, (3) network changes affecting outbound HTTPS to IdP endpoints, (4) certificate renewals on self-hosted IdPs.

Admin overview UI

/admin/overview shows real-time health badges for database, storage, cache, and each enabled OIDC provider, alongside catalog statistics:

• Total datasets and total storage bytes
• Recent additions (last 30 days)
• By-geometry-type breakdown (Point, LineString, Polygon, Raster, etc.)
• By-visibility breakdown (private, internal, restricted, public)
• Users by status (active, deactivated, pending) and total users

The health badges poll /api/admin/infrastructure/ every 30 seconds — the authenticated admin probe, which is also what supplies the per-provider OIDC rows. The catalog statistics are computed live on each request, so they are current rather than cached. For real-time queue/worker metrics, use Prometheus + Grafana (the badges are intentionally coarse-grained).

Catalog statistics endpoint

For programmatic access to the same statistics:

curl https://geolens.example.com/api/admin/stats \
  -H "Authorization: Bearer $TOKEN"

Returns total datasets, recent additions (30 days), total storage bytes, datasets by geometry type, and datasets by visibility.

For database-level size queries:

docker compose exec db psql -U geolens -d geolens -c "
  SELECT pg_size_pretty(pg_database_size('geolens')) AS db_size;
"

Per-table sizes (largest datasets first):

docker compose exec db psql -U geolens -d geolens -c "
  SELECT table_name,
         pg_size_pretty(pg_total_relation_size(reg)) AS size
  FROM (SELECT table_name,
               to_regclass(format('%I.%I', 'data', table_name)) AS reg
        FROM catalog.datasets) t
  WHERE reg IS NOT NULL
  ORDER BY pg_total_relation_size(reg) DESC;
"

to_regclass returns NULL rather than erroring for a catalog row whose physical table is missing, which is why the app’s own storage-usage query uses it — a bare pg_total_relation_size('data.' || table_name) aborts the whole statement as soon as one dataset’s table is absent.

Audit log

Every admin action is recorded in the audit log table. Inspect via the UI at Admin -> Audit Log (filterable by action, user, resource, and date range) or via the API:

# All audit logs
curl https://geolens.example.com/api/admin/audit-logs \
  -H "Authorization: Bearer $TOKEN"

# Filter by action
curl "https://geolens.example.com/api/admin/audit-logs?action=dataset.export" \
  -H "Authorization: Bearer $TOKEN"

# Filter by user and date range
curl "https://geolens.example.com/api/admin/audit-logs?user_id={user_id}&date_from=2024-01-01" \
  -H "Authorization: Bearer $TOKEN"

Audit actions span authentication and sessions (user.login.success, user.login.failure, user.logout, oauth.login.init, oauth.login.success, oauth.login.failure), user lifecycle (user.register, user.create, user.update, user.approve, user.reject, user.deactivate, user.delete, user.change_password, user.verify_email), API keys (api_key.create, api_key.revoke), datasets (dataset.create, dataset.view, dataset.export, dataset.download_cog, dataset.delete, metadata.edit), collections (collection.create, collection.update, collection.delete), maps (map.create, map.share, map.revoke_share), features and layers (feature.insert, feature.update, feature.delete, layer.add, layer.remove, layer.add_column, layer.rename_column), AI queries (query.execute, query.reject), embed tokens (embed_token.create, embed_token.revoke), OAuth providers (oauth_provider.create, oauth_provider.update, oauth_provider.delete), exports (audit.export, user.export), and config operations (config_export, config_import, update, reset, probe_service). This is a representative list, not the complete vocabulary — filter by prefix rather than assuming an exact set.

Audit events can be downloaded in bulk — up to 100,000 rows per request — as CSV or JSON, either from the Admin -> Audit Log page or via the API (same filters as the list endpoint):

curl -o audit-export.csv \
  "https://geolens.example.com/api/admin/audit-logs/export/csv?date_from=2026-01-01" \
  -H "Authorization: Bearer {admin_token}"

There is no built-in retention/archival policy (rows are never deleted automatically); for long-term retention, schedule a periodic export or archive the audit table to S3.

Docker logs & debugging

For service-level debugging beyond the metrics endpoint, use Docker Compose log streaming:

# Follow all logs
docker compose logs -f

# Follow specific service logs
docker compose logs -f api
docker compose logs -f db
docker compose logs -f worker

# Last 100 lines
docker compose logs --tail=100 api

For service health (Docker-level, not application-level):

# View all service statuses
docker compose ps

# Check specific service
docker compose ps db
docker compose ps api

Service health here reflects container restart status and entrypoint health checks; it does not exercise the application’s own provider probes. Use /health for application-level connectivity checks; use docker compose ps for “is the container running.”

Tip

For production deployments without SSH access to the host, route Docker logs to a centralized log aggregator (CloudWatch, Loki, Datadog) via the Docker logging driver. The default json-file driver works for local development but rotates aggressively under sustained load.

See also

• OAuth/OIDC setup: the Validate Connectivity probe checks IdP reachability
• Backups & restore: operational health includes verifying backup status
• API authentication: for /api/admin/* endpoint authentication details