Skip to content

Telemetry & observability

In production the backend ships telemetry to Azure Application Insights (ADR 0035) so that "is it up", "why was this slow", and "what failed" have answers without SSH. The wiring lives in backend/app/telemetry.py, and it's built to be invisible in development and tests.

Off by default, on when configured

Everything keys off one setting, APPLICATIONINSIGHTS_CONNECTION_STRING:

  • Unset (local dev, tests, CI) → telemetry is a complete no-op. The Azure SDK isn't even imported, no exporter threads start, nothing phones home.
  • Set (production, by the App Service from the App Insights resource) → the exporter and instrumentation are configured.

The connection string is supplied by the environment and held in Key Vault; it never appears in code or in these docs.

Two entry points, and the ordering that matters

app/main.py calls two functions, and the order is load-bearing:

configure_telemetry()      # BEFORE FastAPI() is constructed
app = FastAPI(...)
instrument_fastapi_app(app)  # AFTER, on the instance
  • configure_telemetry() runs at import time, before the app exists, so each gunicorn worker initialises its own exporter after fork. It's idempotent and lazily imports the Azure SDK only when the connection string is present.
  • instrument_fastapi_app(app) instruments the app instance, not the global FastAPI class. Because main.py binds the FastAPI name at module scope before telemetry configures, the distro's global class-patch would miss it — instrumenting the instance sidesteps that.

What's collected

When enabled:

Signal Source Goes to
HTTP requests FastAPI requests
Database queries psycopg (SQLAlchemy) dependencies
Outbound HTTP httpx (e.g. the Entra JWKS fetch) dependencies
Unhandled exceptions FastAPI exceptions
Logs (WARNING and above) root logger traces

Privacy filtering — built in, not optional

The instrumentation is deliberately constrained so telemetry can't leak user content or credentials:

  • The SSE stream is excluded from request instrumentation. A multi-hour streaming connection would otherwise look like a giant slow/failed request and drown the signal.
  • No request or response bodies, and no headers are captured — so Authorization tokens and task content (titles, descriptions, emails) never enter telemetry.
  • Database telemetry records parameterised query text only — … WHERE project_id = %s, never the bound values. Task content and auth tokens stay out of the query logs.
  • Logs are floored at WARNING for export; INFO stays in the container log only.

Alerts

Alerts are declared in infrastructure-as-code, not clicked together in the portal (ADR 0030) — so they're reviewable in a pull request and can't silently drift. The set includes a request-failure-rate alert and a backup-freshness alert (the daily database export emits a success heartbeat; the alert fires if none has arrived in ~26 hours). A telemetry-cost tripwire emails the operator if Application Insights spend trends past a set monthly figure. The concrete thresholds and targets live in the infrastructure templates and the operator's runbook.