Infrastructure & Monitoring

GeoLens exposes Prometheus-compatible metrics on the API container at /metrics and on the worker container at port 8001 (the API’s port is published on the host loopback — 127.0.0.1:8001 by default — and is not reachable from the public edge; the worker’s port is internal to the Docker network only; see Metrics endpoint), plus a connectivity health endpoint at /api/health (through the bundled Nginx; /health on the API container directly). The admin web UI surfaces both at /admin/overview along with catalog statistics. This page covers the operator-facing monitoring surface; service-level diagnostics (Docker logs, database size queries) are at the bottom.

Replace https://geolens.example.com with your GeoLens instance’s URL in every example below.

Metrics endpoint

The /metrics endpoint serves Prometheus-format metrics, gzipped, response-buffered, with scrape paths excluded from histogram contamination. It is served on the API container port (:8000) inside the Docker network and is deliberately not exposed at the public edge: the bundled Nginx returns 404 for /api/metrics, so a scrape aimed at the public hostname reaches nothing. Point Prometheus at the internal service address (http://api:8000/metrics) instead. The endpoint is unauthenticated; if you deliberately republish that port outside the Docker network, put it behind a reverse-proxy IP allowlist or basic auth.

The worker serves job-queue metrics from its own /metrics endpoint on port 8001. The port is internal and is not published by the bundled compose files. Scrape both the API and worker endpoints; otherwise the job series below will be absent.

Metric	Served on	Type	Labels	Description
http_requests_total	api	counter	method, status, handler	Total HTTP requests served
http_request_duration_seconds	api	histogram	method, status, handler	Request latency distribution
http_requests_inprogress	api	gauge	method, handler	In-flight requests
geolens_jobs_queue_depth	worker	gauge	queue	Pending jobs (Procrastinate status=todo)
geolens_jobs_active	worker	gauge	queue	Running jobs (Procrastinate status=doing)
geolens_jobs_completed_total	worker	counter	queue	Completed jobs (since process start)
geolens_jobs_failed_total	worker	counter	queue	Failed jobs (since process start)
geolens_db_pool_checkedout	api	gauge	(none)	Connections currently checked out
geolens_db_pool_checkedin	api	gauge	(none)	Connections currently available in pool
geolens_db_pool_overflow	api	gauge	(none)	Overflow connections currently open
geolens_db_pool_size	api	gauge	(none)	Configured pool size

Sample Prometheus scrape configuration, matching the reference config shipped at infra/monitoring/prometheus.yml:

scrape_configs:
  - job_name: geolens-api
    metrics_path: /metrics
    static_configs:
      # Internal Docker-network address of the API container, not the public host.
      - targets: ['api:8000']
        labels:
          service: api

  - job_name: geolens-worker
    metrics_path: /metrics
    static_configs:
      - targets: ['worker:8001']
        labels:
          service: worker

For Grafana dashboards, the geolens_jobs_queue_depth and geolens_db_pool_overflow series are the most actionable. The alert rules shipped at infra/monitoring/alerts.yml fire on queue depth above 100 sustained for 15 minutes (worker undersizing) and on any non-zero pool overflow sustained for 10 minutes (DB-connection contention) — start from those thresholds rather than inventing your own.

Health endpoint

GET /api/health returns 200 (healthy) or 503 (degraded), with a JSON body covering each provider:

Attention

Through the bundled Nginx, the probe lives at /api/health. A bare /health on the proxied URL is answered by the frontend SPA catch-all with HTML and status 200 — an uptime monitor pointed there will report healthy no matter what. (/health without the prefix is correct only on the API container directly; see the note at the bottom.)

{
  "status": "healthy",
  "providers": {
    "database": { "status": "ok", "latency_ms": 12.3 },
    "storage": { "status": "ok", "latency_ms": 45.2 },
    "cache": { "status": "ok", "latency_ms": 1.1 }
  },
  "version": "<running version>",
  "build": "<image commit SHA, or null for source builds>"
}

version reports the running application version and build the release image’s commit SHA (null for local/source builds), so a deployment can be verified over HTTP — production instances hide /api/docs, which was previously the only surface exposing the version.

The probes:

• database: exercises a live SELECT to_regclass('catalog.datasets') (catches hung DB, broken search_path).
• storage: calls the configured storage provider’s health_check() (S3 HeadBucket or local writability test).
• cache: calls Valkey/Redis PING.

Use this endpoint as the upstream health check for load balancers and for a Kubernetes readiness probe: 503 is intentional, and it signals “do not route traffic here” without 5xx-class application errors that would page on-call.

# Basic check
curl -fsS https://geolens.example.com/api/health || echo "unhealthy"

# Detailed JSON with latency breakdown
curl -s https://geolens.example.com/api/health | jq

For internal/private endpoints, the FastAPI process exposes the same health check directly at /health on the API container port (:8000 inside the Docker network, e.g. http://api:8000/health; published on the host at :8001 by default). Use this when nginx/the frontend container is itself the failure point.

Liveness: point your own probe at /health/live, not /health

/health checks the database, object store, and cache — a readiness signal. Since 1.18.0 there’s a separate GET /health/live (also proxied at /api/health/live) that answers only “is the process up”, with no dependency calls: {"status": "ok"}, always 200 if the process can respond at all. It’s unauthenticated and exempt from rate limiting — a kubelet probing every second should never see a 429 that gets the pod killed. The shipped Docker healthchecks and the compose file’s own depends_on: service_healthy gates target /health/live. If you run a custom orchestrator with its own liveness probe pointed at /health, move it to /health/live: a database, MinIO, or Valkey outage should not restart-loop an otherwise-healthy API process, which is what a dependency-probing endpoint behind a liveness check causes. Keep readiness probes on /health.

OIDC connectivity validation

OIDC connectivity validation runs separately from the standard /health probe: IdP discovery URLs are checked on demand rather than on every health poll, since cold-cache IdP fetches add 200 to 500 ms latency.

Trigger validation via the admin UI:

1. Navigate to Admin -> Config Ops.
2. Click Validate Connectivity.
3. The panel reports per-service and per-provider latency, status, and any error details (e.g., DNS failure, expired discovery cache, certificate mismatch).

Or via the API:

curl -X POST https://geolens.example.com/api/config-ops/validate/ \
  -H "Authorization: Bearer $TOKEN"

The next release adds the shared credential store to the existing storage, cache, and OIDC-provider probes:

{
  "storage": { "name": "storage", "status": "ok", "latency_ms": 45.2 },
  "cache": { "name": "cache", "status": "ok", "latency_ms": 1.1 },
  "credential_store": { "name": "credential_store", "status": "ok", "latency_ms": 1.2 },
  "oidc_providers": {
    "google": { "name": "google", "status": "ok", "latency_ms": 142.7 },
    "keycloak": { "name": "keycloak", "status": "error", "latency_ms": 0.0, "error": "Connection refused" }
  }
}

The credential_store entry is not available in v1.19.1. It writes and claims a short-lived test secret through the same path used by protected refreshes. An unset REDIS_URL reports the missing configuration; connection or authentication failures report the store as unreachable. Check this field even when cache is healthy, because the cache can fall back to process memory while credential handoff cannot.

Run validation after any of: (1) adding a new OAuth provider, (2) rotating client secrets, (3) network changes affecting outbound HTTPS to IdP endpoints, (4) certificate renewals on self-hosted IdPs.

Admin overview UI

/admin/overview shows real-time health badges for database, storage, cache, and each enabled OIDC provider, alongside catalog statistics:

• Total datasets and total storage bytes
• Recent additions (last 30 days)
• By-geometry-type breakdown (Point, LineString, Polygon, Raster, etc.)
• By-visibility breakdown (private, internal, restricted, public)
• Users by status (active, deactivated, pending) and total users

The health badges poll /api/admin/infrastructure/ every 30 seconds — the authenticated admin probe, which is also what supplies the per-provider OIDC rows. The catalog statistics are computed live on each request, so they are current rather than cached. For real-time queue/worker metrics, use Prometheus + Grafana (the badges are intentionally coarse-grained).

Catalog statistics endpoint

For programmatic access to the same statistics:

curl https://geolens.example.com/api/admin/stats \
  -H "Authorization: Bearer $TOKEN"

Returns total datasets, recent additions (30 days), total storage bytes, datasets by geometry type, and datasets by visibility.

For database-level size queries:

docker compose exec db psql -U geolens -d geolens -c "
  SELECT pg_size_pretty(pg_database_size('geolens')) AS db_size;
"

Per-table sizes (largest datasets first):

docker compose exec db psql -U geolens -d geolens -c "
  SELECT table_name,
         pg_size_pretty(pg_total_relation_size(reg)) AS size
  FROM (SELECT table_name,
               to_regclass(format('%I.%I', 'data', table_name)) AS reg
        FROM catalog.datasets) t
  WHERE reg IS NOT NULL
  ORDER BY pg_total_relation_size(reg) DESC;
"

to_regclass returns NULL rather than erroring for a catalog row whose physical table is missing, which is why the app’s own storage-usage query uses it — a bare pg_total_relation_size('data.' || table_name) aborts the whole statement as soon as one dataset’s table is absent.

Audit log

Every admin action is recorded in the audit log table. Inspect via the UI at Admin -> Audit Log (filterable by action, user, resource, and date range) or via the API:

# All audit logs
curl https://geolens.example.com/api/admin/audit-logs \
  -H "Authorization: Bearer $TOKEN"

# Filter by action
curl "https://geolens.example.com/api/admin/audit-logs?action=dataset.export" \
  -H "Authorization: Bearer $TOKEN"

# Filter by user and date range
curl "https://geolens.example.com/api/admin/audit-logs?user_id={user_id}&date_from=2024-01-01" \
  -H "Authorization: Bearer $TOKEN"

Audit actions span authentication and sessions (user.login.success, user.login.failure, user.logout, oauth.login.init, oauth.login.success, oauth.login.failure), user lifecycle (user.register, user.create, user.update, user.approve, user.reject, user.deactivate, user.delete, user.change_password, user.verify_email), API keys (api_key.create, api_key.revoke), datasets (dataset.create, dataset.view, dataset.export, dataset.download_cog, dataset.delete, metadata.edit), collections (collection.create, collection.update, collection.delete), maps (map.create, map.share, map.revoke_share), features and layers (feature.insert, feature.update, feature.delete, layer.add, layer.remove, layer.add_column, layer.rename_column), AI queries (query.execute, query.reject), embed tokens (embed_token.create, embed_token.revoke), OAuth providers (oauth_provider.create, oauth_provider.update, oauth_provider.delete), exports (audit.export, user.export), and config operations (config_export, config_import, update, reset, probe_service). This is a representative list, not the complete vocabulary — filter by prefix rather than assuming an exact set.

Audit events can be downloaded in bulk — up to 100,000 rows per request — as CSV or JSON, either from the Admin -> Audit Log page or via the API (same filters as the list endpoint):

curl -o audit-export.csv \
  "https://geolens.example.com/api/admin/audit-logs/export/csv?date_from=2026-01-01" \
  -H "Authorization: Bearer {admin_token}"

There is no built-in retention/archival policy (rows are never deleted automatically); for long-term retention, schedule a periodic export or archive the audit table to S3.

Docker logs & debugging

For service-level debugging beyond the metrics endpoint, use Docker Compose log streaming:

# Follow all logs
docker compose logs -f

# Follow specific service logs
docker compose logs -f api
docker compose logs -f db
docker compose logs -f worker

# Last 100 lines
docker compose logs --tail=100 api

For service health (Docker-level, not application-level):

# View all service statuses
docker compose ps

# Check specific service
docker compose ps db
docker compose ps api

Service health here reflects container restart status and entrypoint health checks; it does not exercise the application’s own provider probes. Use /health for application-level connectivity checks; use docker compose ps for “is the container running.”

Astuce

For production deployments without SSH access to the host, route Docker logs to a centralized log aggregator (CloudWatch, Loki, Datadog) via the Docker logging driver. The default json-file driver works for local development but rotates aggressively under sustained load.

See also

• OAuth/OIDC setup: the Validate Connectivity probe checks IdP reachability
• Backups & restore: operational health includes verifying backup status
• API authentication: for /api/admin/* endpoint authentication details