Metrics

CCAG exposes operational metrics via two channels:

Prometheus: scrape GET /metrics (requires admin auth)
OTLP: push to any OpenTelemetry-compatible collector via gRPC

Both channels observe the same instruments. Enabling one does not disable the other.

Prometheus Scrape Endpoint

GET /metrics
Authorization: Bearer <admin-token>

Returns metrics in Prometheus text exposition format (text/plain; version=0.0.4).

Scrape Config Example

scrape_configs:
  - job_name: ccag
    scrape_interval: 30s
    scheme: https
    metrics_path: /metrics
    authorization:
      credentials: "<admin-session-token-or-api-key>"
    static_configs:
      - targets: ["ccag.example.com"]

OTLP Export

Set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to enable gRPC metric push:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

Metrics are exported every 60 seconds. The exporter uses gRPC (Tonic). Both Prometheus and OTLP can run simultaneously.

Metric Reference

All metrics use the ccag prefix (dots in instrument names become underscores in Prometheus).

Request Metrics

Instrument	Type	Labels	Description
`ccag.requests.total`	Counter	`model`, `streaming`, `status`	Total proxy requests
`ccag.request.duration_ms`	Histogram	`model`, `streaming`, `status`	Request duration in milliseconds
`ccag.requests.in_flight`	UpDownCounter		Currently in-flight requests

Token Metrics

Instrument	Type	Labels	Description
`ccag.tokens.input`	Counter	`model`	Total input tokens processed
`ccag.tokens.output`	Counter	`model`	Total output tokens generated
`ccag.tokens.cache_read`	Counter	`model`	Cache read input tokens
`ccag.tokens.cache_write`	Counter	`model`	Cache write (creation) tokens

Tool & Search Metrics

Instrument	Type	Labels	Description
`ccag.tool_calls.total`	Counter	`tool`, `type`	Tool calls observed (`type`: `builtin` or `mcp`)
`ccag.web_searches.total`	Counter		Web searches executed via interception

Error & Throttle Metrics

Instrument	Type	Labels	Description
`ccag.errors.total`	Counter	`error_type`, `endpoint_id`	Bedrock/upstream errors
`ccag.bedrock.throttles.total`	Counter	`model`, `endpoint_id`	Bedrock throttling events
`ccag.rate_limits.total`	Counter		Gateway rate limit rejections
`ccag.auth_failures.total`	Counter	`reason`	Authentication failures

Operational Metrics

Instrument	Type	Labels	Description
`ccag.spend_flush_errors.total`	Counter		Spend tracker flush failures
`ccag.spend_records_quarantined.total`	Counter		Spend records dropped after individual insert failed with a data-rejection error (e.g. SQLSTATE 22P05). Non-zero indicates real spend data was discarded; alarm on this.

Grafana Dashboard

Import a basic dashboard by creating a new dashboard and adding these panels:

Request rate: rate(ccag_requests_total[5m])
Error rate: rate(ccag_errors_total[5m])
Throttle rate: rate(ccag_bedrock_throttles_total[5m]) (group by endpoint_id)
In-flight: ccag_requests_in_flight
p99 latency: histogram_quantile(0.99, rate(ccag_request_duration_ms_bucket[5m]))
Token throughput: rate(ccag_tokens_input[5m]) + rate(ccag_tokens_output[5m])

Claude Code AWS Gateway