Observability reference v7

AIDB emits OpenTelemetry (OTel) traces, logs, and metrics for its own function calls, background workers, and SQL query execution. For the GUCs that turn this feature on and configure it, see Configuring AIDB parameters — OpenTelemetry tracing.

Schemas

When aidb.otel_client = 'database', AIDB writes traces, logs, and metrics into three tables in the aidb_otel schema:

• aidb_otel.traces — one row per span.
• aidb_otel.logs — one row per log record.
• aidb_otel.metrics — one row per periodic metrics snapshot (a PeriodicReader collection bundling every instrument at that point in time).

All three tables share a common set of columns, plus a few columns specific to each.

Common columns

Every row in all three tables carries these columns:

At least one of payload_json/payload_binary is always populated — see Storage format.

aidb_otel.traces — additional columns

Each row is exactly one span. These identifier and attribute columns are extracted from the span before either payload column is serialized, so they're available regardless of storage format, and querying them never requires parsing the JSON or protobuf payload.

trace_id/span_id/parent_span_id are lowercase hex, the same representation payload_json's own traceId/spanId/parentSpanId fields use — the two match as strings, so you can look up a row by either. (This isn't the same as payload_binary's raw OTLP protobuf bytes fields, which aren't hex or text at all.)

aidb_otel.logs — additional columns

Each row is one log record, with two columns linking it back to the span it was emitted under:

Configuring storage format

The aidb.otel_client_storage_format GUC controls which payload column the database client (one of the aidb.otel_client destinations — see Configuring AIDB parameters) writes to: payload_json, payload_binary, or both — the two columns described under Common columns. Values:

aidb.otel_client_storage_format only affects the database client (aidb.otel_client = 'database'). It has no effect on noop (nothing is written), stdout or grpc (both always emit the full OTLP payload directly, not through these columns), or log, which always writes payload_json regardless of this setting, since a server log line is a human-readable debugging aid, not storage a collector later forwards.

When exporting a row that has both payloads populated (storage_format = both), the exporter worker sends only payload_binary, since it's cheaper to have produced.

Triggering retention cleanup

The cleaner background worker deletes old rows automatically (see Running background workers), but this function lets you trigger a cleanup pass immediately instead of waiting for its next tick.

aidb.run_otel_retention_cleanup()

Manually triggers a full retention cleanup pass, deleting rows older than aidb.otel_cleaner_retention_days from all three aidb_otel tables, in the same bounded batches the cleaner background worker uses. Runs immediately, regardless of aidb.otel_cleaner_enabled — useful for an ops task or a test that shouldn't wait for the worker's own wake-up interval.

Requires the aidb_governance role (see Managing audit access) — aidb_users has no EXECUTE on this function.

Takes no arguments and returns void. Safe to call while the cleaner background worker is running — the two don't compete over the same rows.

SELECT aidb.run_otel_retention_cleanup();

Internally, the database client writes through aidb_otel.store_metrics(), aidb_otel.store_logs(), and aidb_otel.store_traces(). These aren't meant to be called directly — aidb_users has EXECUTE on them (needed for any traced call to work) but no SELECT on the tables they write to.

Running background workers

Two background workers run per database, each spawned only when enabled for that database (see Configuring AIDB parameters for the governing GUCs).

Exporting traces, logs, and metrics

Forwards pending, due rows from aidb_otel.{traces,logs,metrics} to the OTLP/HTTP collector at aidb.otel_exporter_endpoint, on a 15-second tick. payload_json rows are sent as OTLP/HTTP+JSON. payload_binary rows are sent as OTLP/HTTP+protobuf (Content-Type: application/x-protobuf) — a real OTel Collector's HTTP receiver dispatches on that header, so both formats share the same endpoint. Every outbound request goes through AIDB's egress allowlist.

Failed deliveries back off per row (not per tick), so one row's retry schedule never blocks the rest of the batch. After aidb.otel_exporter_max_retries attempts, a row is marked export_status = 'failed' and stops being retried automatically — find these with:

SELECT * FROM aidb_otel.traces WHERE export_status = 'failed';

Cleaning up expired rows

Deletes rows older than aidb.otel_cleaner_retention_days from all three tables, in batches of aidb.otel_cleaner_batch_size, on an hourly tick. Set aidb.otel_cleaner_retention_days = 0 to disable cleanup entirely without stopping the worker itself.

Traced attributes

AIDB attaches attributes to spans, following OpenTelemetry semantic conventions where applicable (gen_ai.*, db.*, error.type), plus custom aidb.* attributes for concepts with no standard equivalent. These attributes are recorded automatically whenever tracing is enabled — you don't set them yourself. They reflect what AIDB observes about each call.

Setting resource attributes

Set once per process, shared by every signal (traces, logs, and metrics):

Setting root-span attribute

Set once per trace, only on its root span:

db.postgresql.session_id uses the format <start_time_hex>.<pid_hex>, matching Postgres's own %c log_line_prefix escape. It's per-connection, so it's set only on each trace's root span — not on every span, and not as a resource attribute — since sharing it with the metrics pipeline would inflate metric cardinality. It's promoted to the indexed pg_session_id column on aidb_otel.traces.

Recording Agent Hub attributes

Set on spans for agent conversations, tasks, and messages:

Recording Tools Hub attributes

Set on spans for tool calls, including those imported from an MCP server:

Recording purpose-enforcement decisions

Every tool SQL execution run under a purpose-resolved role emits exactly one purpose_decision span carrying these attributes:

Tracing SQL queries

Set on any span that executes SQL:

Tracing pipelines, semantic KBs, and models

These cover pipeline CRUD and execution, semantic KB CRUD and search, model inference (OCR, summarization, reranking, embedding, generation), data processors (chunking, PDF/HTML parsing), model maintenance, and volume operations.

These cover pipeline CRUD and execution, semantic KB CRUD and search, model inference (OCR, summarization, reranking, embedding, generation), data processors (chunking, PDF/HTML parsing), model maintenance, and volume operations.

These cover pipeline CRUD and execution, semantic KB CRUD and search, model inference (OCR, summarization, reranking, embedding, generation), data processors (chunking, PDF/HTML parsing), model maintenance, and volume operations.

These cover pipeline CRUD and execution, semantic KB CRUD and search, model inference (OCR, summarization, reranking, embedding, generation), data processors (chunking, PDF/HTML parsing), model maintenance, and volume operations.