Prometheus: scrape, rules, alerts

These are starting points written against the metric catalogue. Adapt the thresholds to your usage. The scrape configuration and both rule groups pass promtool check config and promtool check rules (Prometheus 3.15). Re-check them after you edit them.

Scrape configuration

scrape_configs:
  # Long-lived UI sidecar: knowledge-base gauges, always up.
  - job_name: memors-kb
    scrape_interval: 60s
    static_configs:
      - targets: ["127.0.0.1:9470"]
        labels: { kb: my-project }

  # MCP server: live tool and search metrics; only up while a client session runs.
  - job_name: memors-mcp
    scrape_interval: 15s
    static_configs:
      - targets: ["127.0.0.1:9469"]
        labels: { kb: my-project }

Prometheus must run on the same machine, because both endpoints are loopback-only. A Grafana Alloy or OpenTelemetry Collector agent on the machine can scrape them and forward the samples. That forwarding is your choice; memors-mcp itself never pushes.

Do not alert on up{job="memors-mcp"} == 0. The server exists only while a client session is open, so being down is normal. Alert on up{job="memors-kb"} instead, if you rely on the sidecar.

Recording rules

groups:
  - name: memors-recording
    interval: 1m
    rules:
      - record: memors:tool_calls:rate5m
        expr: sum by (kb, tool, outcome) (rate(memors_mcp_tool_calls_total[5m]))
      - record: memors:tool_error_ratio:rate5m
        expr: |
          sum by (kb) (rate(memors_mcp_tool_calls_total{outcome="error"}[5m]))
          /
          sum by (kb) (rate(memors_mcp_tool_calls_total[5m]))
      - record: memors:tool_latency_seconds:p95_5m
        expr: histogram_quantile(0.95, sum by (kb, tool, le) (rate(memors_mcp_tool_call_duration_seconds_bucket[5m])))
      - record: memors:search_latency_seconds:p95_5m
        expr: histogram_quantile(0.95, sum by (kb, le) (rate(memors_search_duration_seconds_bucket[5m])))
      - record: memors:search_abstain_ratio:rate15m
        expr: |
          sum by (kb) (rate(memors_search_total{outcome="abstain", granularity=~"chunk|document"}[15m]))
          /
          sum by (kb) (rate(memors_search_total{granularity=~"chunk|document"}[15m]))
      - record: memors:arm_latency_seconds:p95_5m
        expr: histogram_quantile(0.95, sum by (kb, arm, le) (rate(memors_search_arm_duration_seconds_bucket[5m])))

The abstain ratio counts passage and document searches only. A fact or page search finds nothing whenever no fact or page matches, and the ask prompt runs a fact search for every question.

Alerts

groups:
  - name: memors-alerts
    rules:
      - alert: MemorsToolErrorsHigh
        expr: memors:tool_error_ratio:rate5m > 0.05
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "memors-mcp {{ $labels.kb }}: more than 5% of tool calls fail"
          description: "Check memors_mcp_tool_errors_total by class and the server's stderr."

      - alert: MemorsSearchSlow
        expr: memors:search_latency_seconds:p95_5m > 2
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "memors-mcp {{ $labels.kb }}: p95 search latency above 2 s"
          description: "Look at memors:arm_latency_seconds:p95_5m to find the slow arm."

      - alert: MemorsSearchDegraded
        expr: sum by (kb, reason) (increase(memors_search_degraded_total[30m])) > 0
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: "memors-mcp {{ $labels.kb }}: searches degraded ({{ $labels.reason }})"
          description: "The embedding model is missing or failing; search is keyword-only."

      - alert: MemorsEmbeddingBacklog
        expr: max by (kb, model) (memors_kb_pending_embeddings) > 0
        for: 2h
        labels: { severity: info }
        annotations:
          summary: "{{ $value }} passages lack vectors for {{ $labels.model }}"
          description: "Run memors-mcp backfill, or start a server session to drain the backlog."

      - alert: MemorsJobsFailed
        expr: max by (kb) (memors_kb_jobs_failed) > 0
        labels: { severity: warning }
        annotations:
          summary: "memors-mcp {{ $labels.kb }}: background jobs failed"

      - alert: MemorsDatabaseGrowth
        expr: delta(memors_kb_db_size_bytes[1d]) > 500e6
        labels: { severity: info }
        annotations:
          summary: "memors-mcp {{ $labels.kb }}: knowledge base grew more than 500 MB in a day"

      - alert: MemorsMergeQueueBacklog
        expr: max by (kb) (memors_kb_merge_review_open) > 50
        for: 1d
        labels: { severity: info }
        annotations:
          summary: "{{ $value }} merge candidates waiting for review (memors-mcp graph merges)"

      - alert: MemorsKBSidecarDown
        expr: up{job="memors-kb"} == 0
        for: 10m
        labels: { severity: info }
        annotations:
          summary: "memors-mcp UI sidecar for {{ $labels.kb }} is not running"

Dashboard queries