Monitoring

GQLDB exposes a layered monitoring surface: logs for events, Prometheus-style metrics for time series, gRPC health checks for liveness, and GQL-visible introspection for state. This page is the index: what each surface gives you, how to enable / scrape it, and which alerts are worth wiring.

Surfaces at a Glance

Production monitoring is rarely a single surface. The default stack is: metrics scraped into Prometheus + Grafana, logs shipped to a search backend (Loki, OpenSearch), health checks driving the load balancer, and SHOW HA STATUS polled on a slower cadence for on-call dashboards.

For a starting alert set built on these surfaces, see Alert-Worthy Signals.

Logs

The fastest path to a problem is usually the log. GQLDB writes structured lines (timestamp, level, subsystem, message) to stderr by default. Configure routing and rotation at startup; see Database Installation → See All Flags for the full flag table.

For ad-hoc local testing, shell redirection (> gqldb.log 2>&1) captures everything (stderr logs, stdout banners, panic traces) into one flat file. For production, set -log-file to a directory and let the rotation flags do the work; pair with a log shipper to push files into your search backend.

Metrics

GQLDB runs an in-process Monitor that aggregates metrics from registered MetricsCollectors on a configurable interval (default 10s). Two collectors are built-in: query and hardware. Metric names are Prometheus-style (ultipagqldb_*) with gauge or counter types. Scrape them via the Prometheus endpoint (see Scraping).

Reading Metrics with SHOW STATS

SHOW STATS returns a snapshot of the collected metrics in-band as a query result — one row per metric, with columns category, metric, and value. Pass a category to filter to a single collector:

GQL
SHOW STATS              -- all metrics
SHOW STATS QUERY        -- query throughput / latency / errors
SHOW STATS HARDWARE     -- heap, GC, goroutines, CPU

Only the registered collectors have data, so QUERY and HARDWARE are the categories that return rows. Use SHOW STATS for ad-hoc checks from a GQL session; use the Prometheus endpoint for continuous monitoring.

Query Metrics

Emitted by the QueryCollector:

Hardware / Runtime Metrics

Emitted by the HardwareCollector:

Hardware samples are cached for 1 s — multiple Snapshot() calls inside a second return the same numbers, avoiding repeated runtime.ReadMemStats cost.

Scraping

The metrics endpoint is served by the gRPC server layer. Point your Prometheus job at the configured endpoint (consult the install or your operator); a default config typically scrapes every 15–30 s.

A snapshot is collected on demand each scrape — the 10s collection ticker is a pacing hint, not a hard cadence. Metric timestamps are server-side, so clock skew between scrapers shows up as a per-metric attribute.

Custom Collectors

Monitor.RegisterCollector(collector MetricsCollector) accepts any type implementing the MetricsCollector interface (Name, Category, Collect). Vendor / customer-deployed plugins can add their own metric family without touching the server build.

Health Checks

Standard gRPC HealthService.Check returns SERVING / NOT_SERVING. In HA mode, the response metadata is extended with:

Compatible with any standard gRPC health probe (k8s readiness/liveness, ELB/NLB target groups, grpc-health-probe). The base SERVING / NOT_SERVING behavior is preserved for tooling that doesn't know about the HA extension.

Use it as liveness (server responds at all). Use compute_ready = true + a sensible lag_bytes ceiling as readiness if you don't want a node to take traffic before its caches are warm.

Alert-Worthy Signals

A starting alert set:

Tune the thresholds to your workload; the rule shapes above are the starting point.