Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages

Help for owners

My AI agent is live and nobody is watching how it performs

Short answer

An AI feature does not fail with an error message. It drifts: answers get worse, costs rise, a model gets retired, documents go stale. Someone has to own a small routine: sample real conversations, track cost and failures, keep a set of test questions, and watch the provider's deprecation notices.

  • Written for business owners
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Does this sound like your situation?

  • Nobody on my team has read a real conversation since launch.
  • I only hear about problems when a customer complains.
  • The bill from the AI provider keeps changing and I cannot say why.
  • The developer who built it has moved on.
  • I do not know which AI model it uses or when that model is retired.

Why does this happen?

  1. Quality drift makes no noise

    A wrong answer looks exactly like a right one from the outside. There is no crash, no red alert, only customers who quietly stop trusting it.

  2. Your content changes and the assistant does not

    Prices, policies and products change. If the documents it reads are not refreshed, it keeps giving last season's answers.

  3. Models are retired on a schedule

    Anthropic states that it regularly retires older models, that requests to a retired model fail, that it notifies customers by email and in its documentation, and that a usage export in its console shows which models each key still uses. OpenAI describes the same pattern with Active, Deprecated and Shutdown states. If the email goes to a developer who left, nobody sees it.

  4. Nobody owns cost and usage

    A loop, a long conversation or a sudden spike in traffic can multiply the bill. Without a usage view and a limit, you find out from the invoice.

What can I do about it today?

  1. Read twenty real conversationsAsk the provider or builder for an export and read a mix of short and long ones. Mark each as good, doubtful or wrong.
  2. Find out who receives the provider's emailsCheck which address is on the AI provider account. Make it a shared company address so deprecation and billing notices do not vanish.
  3. Write down which model and which accountName the provider, the model, who owns the account and where the keys are kept. Do not paste the keys into a message or document.
  4. Set a spending limit or alertMost providers let you set a budget or alert in their dashboard. Ask whoever owns the account to set one.
  5. Name one ownerOne person is responsible for looking at conversations and cost every month, even if a developer does the technical work.

When should I bring someone in?

  • The person who built it is unreachable and nobody holds the accounts.
  • The provider has announced a retirement date for the model you use.
  • You find wrong answers and cannot tell how common they are.

How do I handle it?

  1. Take stock of what existsProvider accounts, models, prompts, documents, keys, who gets the emails and what it costs to run.
  2. Build a fixed set of test questionsReal questions with approved answers, run on a schedule so a drop in quality shows up as a number.
  3. Sample live conversationsA small, regular sample is read by a person and scored, with failures added to the test set.
  4. Track cost, errors and slow repliesSimple dashboards and alerts on spend, failed calls and response time, with limits set at the provider.
  5. Prepare for model changesWhen a retirement is announced, the test set is run on the replacement before anything switches, and a rollback is kept.

The price depends on what I find, so I quote in writing after a short discovery call. Tell me what is happening and I reply within 24 hours.

What do owners ask about this?

Is monitoring needed if it seems to work?

Yes. The point is that failure is silent. A seemingly fine assistant can be wrong in a corner of your business for weeks.

How often should someone check?

A light monthly routine is a sensible start, more often right after launch, after content changes or after a model change.

Do I need special software?

Not at the start. A spreadsheet of sampled conversations and the provider's own usage page already tell you a lot.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Found a mistake? Tell me.

For your developer: the technical fixes

Next step

Want this off your desk?

Send me a few lines on where things stand. I reply within 24 hours.