When Dashboards Lie: Building MCP Tools That Chase Down the Truth

Abstract

Dashboards lie. Not maliciously, structurally. Aggregation hides the tenant on fire. Sampling drops slow requests. The p99 looks fine because 47 users who timed out are a rounding error. Every SRE has lived this: green screen, Slack on fire, hunting across five tools to find what the dashboard refused to show. This is a field report from building MCP tools that do the hunting. The agent does not replace the SRE. It does the grunt work nobody has time for at 3 AM: pulling exemplar logs for the slowest 0.1 percent, correlating a deploy against error rates, checking if the metric was even reporting. 1. Why dashboards lie. Sampling, aggregation, the "aggregate green, individual red" pattern. 2. MCP tool design for truth-seeking. Read-only vs side-effecting split, partial-data schemas, outputs that make the model admit uncertainty instead of hallucinating "all good." 3. Correlation loops that work. Deploy to error rate to exemplar logs to suspected change, not seventeen tabs. 4. Guardrails from production. Prompt injection in logs, cost blow-ups, tools we took back after one bad incident. Attendees leave with patterns for MCP tools that chase down what dashboards will not show.