We Built a Live Dashboard That Monitors Every Piece of Our AI Infrastructure

When you run 35 autonomous agents, the question isn’t whether something will break—it’s whether you’ll know when it does. We built a live inventory dashboard that gives us a real-time health view of every component in our AI infrastructure.

The Problem

As an AI system grows, it becomes impossible to hold its full state in your head. Is the outreach pipeline running? Did the knowledge base get reindexed this week? Is the reply classifier catching warm leads? Without a monitoring layer, you only find out something broke when consequences show up—missed leads, stale data, silent failures.

The Architecture

The dashboard is a single-page application with three layers:

• Registry (data.js) — a curated catalog of every component: 366 rows across 16 categories including daemons, scripts, agents, APIs, credentials, logs, knowledge bases, and MCP servers. This is the source of truth for what should exist.
• Collector (collector.py) — a probe that runs every 5 minutes, checks the live state of every component, and writes results to a state file. Each probe is tailored: processes get checked via launchctl, SQLite databases get queried for record counts and last-updated timestamps, credential files get checked for age and existence.
• Server + SPA (server.py + app.js) — a FastAPI server merges the curated registry with live probe data and serves it to a dark-themed single-page app. The UI shows health pills, last-run times, file sizes, and record counts for every component.

What Gets Monitored

Every category of infrastructure gets its own probe logic:

• Processes: Is the launchd daemon loaded? Running or idle? Last exit code?
• Knowledge bases: How many records? When was it last indexed? File size?
• OAuth credentials: Does the token file exist? How old is it?
• Logs: Does the log file exist? How large is it? When was it last written?
• CLIs: Is the binary installed? Is the auth session active?

Health Classification

Every component gets classified into one of six states: healthy, degraded, broken, unreachable, disabled, or unknown. The classification rules are specific—a process that exited cleanly (code 0) is healthy even if it’s not currently running, because that’s normal for time-triggered daemons. A log file that hasn’t been written to in 7 days is degraded. A credential file that doesn’t exist is broken.

The Key Design Decision

We separated the curated registry (what should exist, with human-written descriptions) from the live state (what does exist, with machine-measured metrics). The curated data never gets overwritten by automation—only humans edit it. The live data gets refreshed every 5 minutes by the collector. This means the dashboard always shows accurate descriptions even when the probe fails, and always shows accurate health even when the descriptions are stale.

Practical Value

The dashboard paid for itself in the first week when we caught a knowledge base indexer that had silently stopped running after a system restart. Without the dashboard, we would have had stale search results for an unknown number of days. With it, the staleness showed up as a yellow “degraded” pill within 5 minutes of the next collection run.

Any organization running more than a handful of automated processes should have something like this. The tools to build it—SQLite, a lightweight web server, a scheduler—cost nothing and are universally available.