Monitoring

GQLDB exposes a layered monitoring surface: logs for events, Prometheus-style metrics for time series, gRPC health checks for liveness, and GQL-visible introspection for state. This page is the index: what each surface gives you, how to enable / scrape it, and which alerts are worth wiring.

Surfaces at a Glance

Production monitoring is rarely a single surface. The default stack is: metrics scraped into Prometheus + Grafana, logs shipped to a search backend (Loki, OpenSearch), health checks driving the load balancer, and SHOW HA STATUS polled on a slower cadence for on-call dashboards.

Logs

The fastest path to a problem is usually the log. GQLDB writes structured lines (timestamp, level, subsystem, message) to stderr by default. Configure routing and rotation at startup; see Database Installation → Important Flags for the full flag table.

For ad-hoc local testing, shell redirection (> gqldb.log 2>&1) captures everything (stderr logs, stdout banners, panic traces) into one flat file. For production, set -log-file to a directory and let the rotation flags do the work; pair with a log shipper to push files into your search backend.

Memory-Pressure Logs

A separate watermark log fires when heap usage crosses configured thresholds. Lines you'll see (rate-limited to one per 30s):

Log
[MEMORY WATERMARK] heap=<n> MB >= warn threshold=<m> MB (watermark=<w> MB)
[MEMORY PRESSURE ABORT] heap=<n> MB >= abort threshold=<m> MB ... — refusing new aggregations until heap drops
[MEMORY PRESSURE CLEARED] heap=<n> MB below abort threshold=<m> MB — aggregations re-enabled

Enable by setting config.MemoryWatermarkBytes in -config to your process budget; default 0 disables the feature entirely. The abort threshold causes new memory-hungry operators (GROUP BY, ORDER BY, aggregations) to reject with a clean error instead of pushing the host toward the OOM killer. See Memory Pressure below.

Metrics

GQLDB runs an in-process Monitor that aggregates metrics from registered MetricsCollectors on a configurable interval (default 10s). Three collectors are built-in: query, hardware, cache. Metric names are Prometheus-style (ultipagqldb_*) with gauge or counter types.

Query Metrics (QueryCollector)

Hardware / Runtime Metrics (HardwareCollector)

Hardware samples are cached for 1 s — multiple Snapshot() calls inside a second return the same numbers, avoiding repeated runtime.ReadMemStats cost.

Cache Metrics (CacheCollector)

Tracks the AST cache and plan cache:

• Hits / misses / hit rate for each — low hit rate on the plan cache means the workload is parameter-thin (each query is unique), warranting query parameterization on the client side rather than tuning here.
• Entries — current cache size; bounded by the cache's eviction policy.

Scraping

The metrics endpoint is served by the gRPC server layer. Point your Prometheus job at the configured endpoint (consult the install or your operator); a default config typically scrapes every 15–30 s.

A snapshot is collected on demand each scrape — the 10s collection ticker is a pacing hint, not a hard cadence. Metric timestamps are server-side, so clock skew between scrapers shows up as a per-metric attribute.

Custom Collectors

Monitor.RegisterCollector(collector MetricsCollector) accepts any type implementing the MetricsCollector interface (Name, Category, Collect). Vendor / customer-deployed plugins can add their own metric family without touching the server build.

Health Checks

Standard gRPC HealthService.Check returns SERVING / NOT_SERVING. In HA mode, the response metadata is extended with:

Compatible with any standard gRPC health probe (k8s readiness/liveness, ELB/NLB target groups, grpc-health-probe). The base SERVING / NOT_SERVING behavior is preserved for tooling that doesn't know about the HA extension.

Use it as liveness (server responds at all). Use compute_ready = true + a sensible lag_bytes ceiling as readiness if you don't want a node to take traffic before its caches are warm.

HA / Cluster Monitoring

For HA deployments, the cluster-level dashboard should poll a small set of signals:

See Clustering for the full HA admin surface.

Query Monitoring

Real-time visibility into running queries via SHOW QUERIES, TOP QUERIES, and KILL QUERY <id>. These management statements bypass concurrency limits and share a dedicated execution path, so they always respond even when the database is saturated. Required permission: OpManageQuery (admin-tier by default; see Access Control).

Long-running operators (filter, ORDER BY, UNION, k-hop) check the cancellation flag frequently, so KILL QUERY takes effect promptly rather than waiting for the operator to finish.

Full syntax, return columns, and concurrency-slot model: Query Management.

Memory Pressure

Optional watermark monitor that turns hard-to-debug "process suddenly OOMs" into "operators reject cleanly with a known error before that happens."

Set in -config YAML; the flag set doesn't expose these directly. Once enabled:

1. Warn: when heap crosses MemoryWatermarkRatio × MemoryWatermarkBytes, the [MEMORY WATERMARK] log fires (rate-limited to 30 s between emissions).
2. Abort: when heap crosses MemoryAbortRatio × MemoryWatermarkBytes, the [MEMORY PRESSURE ABORT] log fires once and new aggregations / GROUP BY / ORDER BY operators reject with a clean error.
3. Recovery: when heap drops below the abort threshold, [MEMORY PRESSURE CLEARED] logs once and operators re-enable automatically.

Pair with the ultipagqldb_heap_alloc_bytes metric for dashboard-driven alerting — the watermark log is for postmortem context; the metric is what your alert manager reads.

Alert-Worthy Signals

A starting alert set:

Tune the thresholds to your workload; the rule shapes above are the starting point.