Monitoring

The API exposes Prometheus metrics at /metrics (text exposition format, no auth). In production the endpoint is restricted to the internal network (192.168.1.0/24 in the reference nginx config).

Observability Stack

Component	Purpose	Notes
Prometheus	Metrics storage and alerting	Rules in `deploy/monitoring/`
Grafana	Dashboards	Includes SLO dashboard (`slo_dashboard.json`)
Loki	Log aggregation	Structured JSON logs; retention policy documented
Tempo	Distributed tracing	OpenTelemetry OTLP export
Alertmanager	Alert routing	Loaded with Prometheus rule files

The full stack is defined in deploy/docker-compose.prod.yml and can also be deployed independently.

SLO

99.9% availability (0.1% error budget per 30-day window), tracked with the multi-window burn-rate approach recommended by Google SRE.

Key Metrics

Metric names exposed by the API (used by the alert rules):

Metric	Type	Description
`http_requests_total`	Counter	Total HTTP requests, labelled by status
`http_request_duration_seconds`	Histogram	Request latency distribution
`blocmarket_bids_total`	Counter	Bids placed
`blocmarket_registrations_total`	Counter	User registrations
`blocmarket_transaction_value_total`	Counter	Cumulative transaction value
`blocmarket_pool_active_connections`	Gauge	Active DB connections
`blocmarket_pool_max_connections`	Gauge	DB pool maximum

SLO Burn-Rate Alerts

Defined in deploy/monitoring/prometheus_rules.yml:

Alert	Condition	Severity
`SLOBurnRateHigh`	1h and 5m error rate above 14.4x burn rate, for 2m	Critical
`SLOBurnRateWarning`	6h and 1h error rate above 6x burn rate, for 15m	Warning
`ErrorBudgetExhausted`	30-day accumulated error rate >= 0.1% for 5m	Critical
`ErrorBudgetLow`	< 20% of the 30-day error budget remains	Warning
`HighLatencyP99`	p99 latency > 2.0s for 5m	Warning
`HighLatencyP95`	p95 latency > 1.0s for 5m	Warning

Business Alerts

Defined in deploy/monitoring/business_alerts.yml — thresholds are starting points; tune against production baselines:

Alert	Condition	Meaning
`BidVolumeAnomaly`	5m bid rate > 10x same window 1h ago, for 10m	Viral event or automated bidding
`TransactionValueSpike`	Cumulative transaction value > 100,000	Verify against expected daily volume; check for fraud
`RegistrationSpike`	> 20 registrations in 5m	Possible bot account creation
`HighErrorRate`	5xx rate > 5% of requests for 5m	Check API logs, DB health, recent deployments
`PoolExhaustion`	DB pool > 90% of maximum for 2m	Investigate slow queries and long transactions

Grafana Dashboards

The SLO dashboard (deploy/monitoring/slo_dashboard.json) tracks error budget consumption, burn rate, and latency percentiles. Import it into Grafana alongside per-service dashboards for the API, PostgreSQL, and Redis.

OpenTelemetry Tracing

Distributed tracing via the observability crate with OTLP export to Tempo. Traces span API requests through database queries, enabling end-to-end latency analysis.

Log Aggregation

Logs flow: Application → Promtail → Loki → Grafana.

Structured JSON logging with no PII in production logs. Key patterns to watch:

Pattern	Severity	Meaning
`ERROR.database.connection`	Critical	DB pool exhausted or PostgreSQL down
`ERROR.redis.connection`	Warning	Redis unavailable, rate limiting disabled
`WARN.*rate_limit`	Info	User hitting rate limit (expected)
`ERROR.*migration`	Critical	Migration failed on startup
`ERROR.*smtp`	Warning	Email delivery failure

Query Loki directly:

curl -G "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={service="api"} |~ "error"'

Performance Targets

Baseline targets measured with k6 (scripts/load/api_load_test.js):

Metric	Target
API health	< 50ms p50, < 100ms p99
Property search	< 200ms p50
Auth (login)	< 100ms p50
Notification list	< 150ms p50
API memory	< 512MB RSS
Concurrent users	100 simultaneous
Throughput	> 500 rps (read)

Run the load test against a running API:

cargo run -p bloc-api
k6 run scripts/load/api_load_test.js

Debugging

# Increase log verbosity
export RUST_LOG=info,bloc_api=debug,bloc_market=debug

# Container deployments
docker compose logs -f api
docker compose logs --tail 100 postgres
docker compose logs --tail 100 redis