Infrastructure & Monitoring

GeoLens exposes Prometheus-compatible metrics on the API container at /metrics (internal to the Docker network; see Metrics endpoint) and a connectivity health endpoint at /api/health (through the bundled Nginx; /health on the API container directly). The admin web UI surfaces both at /admin/overview along with catalog statistics. This page covers the operator-facing monitoring surface; service-level diagnostics (Docker logs, database size queries) are at the bottom.

Replace https://geolens.example.com with your GeoLens instance’s URL in every example below.

Metrics endpoint

The /metrics endpoint serves Prometheus-format metrics, gzipped, response-buffered, with scrape paths excluded from histogram contamination. It is served on the API container port (:8000) inside the Docker network and is deliberately not exposed at the public edge: the bundled Nginx returns 404 for /api/metrics, so a scrape aimed at the public hostname reaches nothing. Point Prometheus at the internal service address (http://api:8000/metrics) instead. The endpoint is unauthenticated; if you deliberately republish that port outside the Docker network, put it behind a reverse-proxy IP allowlist or basic auth.

Metric	Type	Labels	Description
http_requests_total	counter	method, status, handler	Total HTTP requests served
http_request_duration_seconds	histogram	method, status, handler	Request latency distribution
http_requests_inprogress	gauge	method, handler	In-flight requests
geolens_jobs_queue_depth	gauge	queue	Pending jobs (Procrastinate status=todo)
geolens_jobs_active	gauge	queue	Running jobs (Procrastinate status=doing)
geolens_jobs_completed_total	counter	queue	Completed jobs (since process start)
geolens_jobs_failed_total	counter	queue	Failed jobs (since process start)
geolens_db_pool_checkedout	gauge	(none)	Connections currently checked out
geolens_db_pool_checkedin	gauge	(none)	Connections currently available in pool
geolens_db_pool_overflow	gauge	(none)	Overflow connections currently open
geolens_db_pool_size	gauge	(none)	Configured pool size

Sample Prometheus scrape configuration:

scrape_configs:
  - job_name: geolens
    metrics_path: /metrics
    static_configs:
      # Internal Docker-network address of the API container, not the public host.
      - targets: ['api:8000']

For Grafana dashboards, the geolens_jobs_queue_depth and geolens_db_pool_checkedout series are the most actionable: sustained queue depth above 50 indicates worker undersizing; sustained pool checkout near pool_size indicates DB-connection contention.

Health endpoint

GET /api/health returns 200 (healthy) or 503 (degraded), with a JSON body covering each provider:

Caution

Through the bundled Nginx, the probe lives at /api/health. A bare /health on the proxied URL is answered by the frontend SPA catch-all with HTML and status 200 — an uptime monitor pointed there will report healthy no matter what. (/health without the prefix is correct only on the API container directly; see the note at the bottom.)

{
  "status": "healthy",
  "providers": {
    "database": { "status": "ok", "latency_ms": 12.3 },
    "storage": { "status": "ok", "latency_ms": 45.2 },
    "cache": { "status": "ok", "latency_ms": 1.1 }
  },
  "version": "1.4.11",
  "build": "e90d350"
}

version reports the running application version and build the release image’s commit SHA (null for local/source builds), so a deployment can be verified over HTTP — production instances hide /api/docs, which was previously the only surface exposing the version.

The probes:

• database: exercises a live SELECT to_regclass('catalog.datasets') (catches hung DB, broken search_path).
• storage: calls the configured storage provider’s health_check() (S3 HeadBucket or local writability test).
• cache: calls Valkey/Redis PING.

Use this endpoint as the upstream health check for load balancers and Kubernetes liveness/readiness probes. The 503 response is intentional: it signals “do not route traffic here” without 5xx-class application errors that would page on-call.

# Basic check
curl -fsS https://geolens.example.com/api/health || echo "unhealthy"

# Detailed JSON with latency breakdown
curl -s https://geolens.example.com/api/health | jq

For internal/private endpoints, the FastAPI process exposes the same health check directly at /health on the API container port (:8000 inside the Docker network, e.g. http://api:8000/health; published on the host at :8001 by default). Use this when nginx/the frontend container is itself the failure point.

OIDC connectivity validation

OIDC connectivity validation runs separately from the standard /health probe: IdP discovery URLs are checked on demand rather than on every health poll, since cold-cache IdP fetches add 200 to 500 ms latency.

Trigger validation via the admin UI:

1. Navigate to Admin -> Config Ops.
2. Click Validate Connectivity.
3. The panel reports per-service and per-provider latency, status, and any error details (e.g., DNS failure, expired discovery cache, certificate mismatch).

Or via the API:

curl -X POST https://geolens.example.com/api/config-ops/validate/ \
  -H "Authorization: Bearer $TOKEN"

The response probes storage and cache, plus one entry per enabled OIDC provider keyed by slug:

{
  "storage": { "name": "storage", "status": "ok", "latency_ms": 45.2 },
  "cache": { "name": "cache", "status": "ok", "latency_ms": 1.1 },
  "oidc_providers": {
    "google": { "name": "oidc:google", "status": "ok", "latency_ms": 142.7 },
    "keycloak": { "name": "oidc:keycloak", "status": "error", "latency_ms": 0.0, "error": "Connection refused" }
  }
}

Run validation after any of: (1) adding a new OAuth provider, (2) rotating client secrets, (3) network changes affecting outbound HTTPS to IdP endpoints, (4) certificate renewals on self-hosted IdPs.

Admin overview UI

/admin/overview shows real-time health badges for database, storage, cache, and each enabled OIDC provider, alongside catalog statistics:

• Total datasets and total storage bytes
• Recent additions (last 30 days)
• By-geometry-type breakdown (Point, LineString, Polygon, Raster, etc.)
• By-visibility breakdown (private, internal, restricted, public)
• Users by status (active, deactivated, pending) and total users

The health badges poll /health every 30 seconds; the catalog statistics are computed from a materialized view refreshed hourly. For real-time queue/worker metrics, use Prometheus + Grafana (the badges are intentionally coarse-grained).

Catalog statistics endpoint

For programmatic access to the same statistics:

curl https://geolens.example.com/api/admin/stats \
  -H "Authorization: Bearer $TOKEN"

Returns total datasets, recent additions (30 days), total storage bytes, datasets by geometry type, and datasets by visibility.

For database-level size queries:

docker compose exec db psql -U geolens -d geolens -c "
  SELECT pg_size_pretty(pg_database_size('geolens')) AS db_size;
"

Per-table sizes (largest datasets first):

docker compose exec db psql -U geolens -d geolens -c "
  SELECT table_name,
         pg_size_pretty(pg_total_relation_size('data.' || table_name)) AS size
  FROM catalog.datasets
  ORDER BY pg_total_relation_size('data.' || table_name) DESC;
"

Audit log

Every admin action is recorded in the audit log table. Inspect via the UI at Admin -> Audit Log (filterable by action, user, resource, and date range) or via the API:

# All audit logs
curl https://geolens.example.com/api/admin/audit-logs \
  -H "Authorization: Bearer $TOKEN"

# Filter by action
curl "https://geolens.example.com/api/admin/audit-logs?action=dataset.export" \
  -H "Authorization: Bearer $TOKEN"

# Filter by user and date range
curl "https://geolens.example.com/api/admin/audit-logs?user_id={user_id}&date_from=2024-01-01" \
  -H "Authorization: Bearer $TOKEN"

Available audit actions cover datasets (dataset.view, dataset.export, metadata.edit), collections (collection.create, collection.update, collection.delete), maps (map.create, map.share, map.revoke_share), features (feature.insert, feature.update, feature.delete), embed tokens (embed_token.create, embed_token.revoke), OAuth providers (oauth_provider.create, oauth_provider.update), and config operations (config_import, update, reset, probe_service).

Audit events can be downloaded in bulk — up to 100,000 rows per request — as CSV or JSON, either from the Admin -> Audit Log page or via the API (same filters as the list endpoint):

curl -o audit-export.csv \
  "https://geolens.example.com/api/admin/audit-logs/export/csv?date_from=2026-01-01" \
  -H "Authorization: Bearer {admin_token}"

There is no built-in retention/archival policy (rows are never deleted automatically); for long-term retention, schedule a periodic export or archive the audit table to S3.

Docker logs & debugging

For service-level debugging beyond the metrics endpoint, use Docker Compose log streaming:

# Follow all logs
docker compose logs -f

# Follow specific service logs
docker compose logs -f api
docker compose logs -f db
docker compose logs -f worker

# Last 100 lines
docker compose logs --tail=100 api

For service health (Docker-level, not application-level):

# View all service statuses
docker compose ps

# Check specific service
docker compose ps db
docker compose ps api

Service health here reflects container restart status and entrypoint health checks; it does not exercise the application’s own provider probes. Use /health for application-level connectivity checks; use docker compose ps for “is the container running.”

Tip

For production deployments without SSH access to the host, route Docker logs to a centralized log aggregator (CloudWatch, Loki, Datadog) via the Docker logging driver. The default json-file driver works for local development but rotates aggressively under sustained load.

See also

• OAuth/OIDC setup: the Validate Connectivity probe checks IdP reachability
• Backups & restore: operational health includes verifying backup status
• API authentication: for /api/admin/* endpoint authentication details