Monitoring, three ways
The same requirement - know when something is wrong, before a user tells you - produced three different stacks, because the binding constraint was different every time. What follows is how each was designed and why, at the level of architecture rather than inventory.
One stack per constraint
Select a stack to see how the signal travels through it.
A bare-metal Kubernetes platform running internal backend services for a public-sector programme.
The constraint
The cluster nodes could not spare the memory a conventional Prometheus deployment needs, and the team's notifications lived in Microsoft Teams rather than Slack.
What I built
Built the metrics tier on VictoriaMetrics instead of Prometheus, so the same retention and query surface fits in a fraction of the memory. Alert evaluation runs in vmalert, and a small adapter translates alert payloads into the card format Teams renders natively, because no off-the-shelf receiver produced a readable message.
Why it mattered
Full metric retention and alerting on hardware that could not otherwise have hosted it, with alerts arriving in the channel the team already worked in.
What carries across
The positions that hold regardless of which stack is underneath.
Pick the stack that fits the constraint
The same requirement produced three different designs, because the binding constraint was different each time: memory on one, the absence of an orchestrator on another, billing model on the third. Choosing tools before understanding the constraint is how platforms end up oversized and unaffordable.
Design alerts people do not mute
An alert has to say what broke, who it affects, and what to try first. It has to group by cause so one failure is one notification, suppress the symptoms of an outage that is already firing, and always send a resolution. A channel that only carries bad news gets ignored, and an ignored channel is worse than no channel.
Nothing that matters lives only in a UI
Dashboards, alert rules and collector configuration are provisioned from version control. A dashboard clicked together in a browser is lost the first time the pod is rescheduled, and a rule nobody can review is a rule nobody can trust.
Monitor the signal that actually matters
Usually that is availability and latency. On consumption-billed infrastructure it is spend, and on a fleet with no orchestrator it is whether the collector is even running. The measure of a monitoring stack is whether it catches the failure that would otherwise have gone unnoticed.
Panels must fail honestly
When a datasource has no samples the panel says so. Drawing a flat line at zero turns a dead scrape target into a system that looks healthy, which is the most expensive failure mode a dashboard has.
Bound retention before it bounds you
Retention is capped on both time and size, log labels are kept low-cardinality, and high-volume low-value streams are dropped at source. Observability that grows without a ceiling eventually costs more than the systems it observes.