Observability & Accountability
The platform that makes a small team's fleet legible: who did what, which accounts look wrong, and what an incident actually cost.
- Role
- Sole author
- Period
- 2026
- Stack
- MongoDB · CloudWatch · WAFv2 · Fastify · React
Problem
A fleet of tenant portals produced plenty of logs and very little answerable truth. Which admin changed that account? Is this login pattern normal? Is that error rate real or is it the same benign auth failure counted a thousand times? Every question meant someone reading raw streams.
Constraints
Utility customer data, so retention and access are not free choices. The dashboard had to be usable during an incident by whoever was responding, not only by its author, and it had to run inside the same cost envelope as everything else on the platform.
Decisions
- The audit tiers are deliberate: the trail that answers 'who changed this' outlives the one that answers 'who looked'.
- Benign authentication failures, account lockouts and revoked tokens are folded into an expected class so the error rate means something.
- Per-pattern batch analysis replaced per-event analysis so the AI spend tracks patterns, not traffic.
Architecture
Capture is universal: a wrapper records calls out to identity, mail, payment and billing providers alike, so an event exists whether the failure was ours or theirs, with payloads deep-redacted on the v2 capture path - tokens, passwords and identifiers. Volume was measured, not estimated: 156,656 events a day at 1.8 KB each, projected to about 35 GB at steady state.
Accountability is tiered. Admin writes record which fields actually changed and are kept for a year; admin reads and impersonated sessions are captured too on the v2 API, attributed to the acting admin, and kept for ninety days.
An account-to-IP fan-out view surfaces anomalies, and an alert fires on the email-reversal pattern that precedes account takeover.
A composite index on tenant, environment, user and timestamp cut the sessions view 43x, and Bedrock spend is attributed to its own client in the cost dashboard so AI cost is visible next to everything else.
Outcome
- Admin writes are attributable fleet-wide; admin reads and impersonation are attributable on the v2 API.
- The error rate reflects real failures, and account anomalies surface as a view instead of a hunch.
- One composite index took the sessions view from 102 seconds to 2.4 - 43x, measured live - and AI spend is attributed rather than absorbed.