Early on, the assistant’s scheduled jobs all reported success while one of them quietly did nothing for days. The process ran, exited cleanly, wrote ok in its log, and delivered no data. I only noticed because the morning brief felt thin.
That taught me a rule I now apply to every automated system: process status tells you the machine ran. It does not tell you the work happened. The two come apart more often than you would like.
So I stopped watching statuses and started watching data age. Every ingestion lane in the system has a registered expectation: mail lands at least every few hours, messages more often. A watcher compares the newest actual data in each lane against that expectation. If a lane goes stale it alarms, however green the process looks.
Second rule: alert paths are tested by firing them for real. I once had an alarm call that failed with an error nobody saw. It stayed broken, silently, for weeks. Since then, no alert counts as existing until I have watched it fire. When I wire a new one I deliberately inject the failure condition and confirm the message arrives on my phone. It feels ceremonial. It has caught real gaps every single time.
A third lesson came from a livelier incident. The container was up, the process list looked healthy, and the application inside had wedged itself out of memory on an unbounded scan. The service is running and the service works are two different sentences. Now a small independent check exercises the actual behaviour from the outside, every few minutes.
None of this is specific to AI. But agentic systems make it worse, because they fail politely. A model that cannot reach its data does not crash. It improvises. The only defence is instrumentation that measures reality instead of intent.
Every company has one: the interface that stopped syncing on a Tuesday and was discovered a week later by an angry customer. The job ran, the monitor stayed green, the data never moved. Watching data age instead of job status catches it the same day: how old is the newest order, the newest booking, the newest sync, compared to what you expect. It is a small change to your monitoring, and the first time it fires you will wonder why it was not there from the start.