Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages

Help for owners

Every time my developers release an update, something breaks

Short answer

Releases that keep breaking things usually mean the change goes straight to live users with no test copy, no checklist, no quick way back and many changes bundled together. Fix those four in order: a staging copy, a short checklist, smaller releases and a tested rollback. A rewrite is rarely the answer.

  • Written for business owners
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Does this sound like your situation?

  • Every release is followed by a day of urgent fixes.
  • Features that worked yesterday stop working after an update.
  • Releases happen late in the day or on Fridays, with everyone holding their breath.
  • Nobody is sure what exactly changed in the latest release.
  • When something breaks, there is no quick way to go back to the previous version.

Why does this happen?

  1. Changes go to live users untested

    Without a staging copy that looks like production, the first real test is by customers. Differences in data, settings or services only show up live.

  2. Releases are too big

    When weeks of changes ship together, a failure could be any of them. Small releases make the cause obvious and the fix quick.

  3. No checklist or checks after the release

    Without a short list of things to check right after, problems are found by customers hours later.

  4. No way back

    If the previous version and the database cannot be restored quickly, every fault becomes an emergency and a rush fix.

  5. Database changes are the hidden risk

    A change to the data structure cannot always be undone by deploying the old code. These need extra planning.

What can I do about it today?

  1. Write down the last three breakagesWhat broke, how it was found, how long it took to fix and what changed in that release. Patterns show up fast.
  2. Ask for a release note before each releaseA short plain list: what changes, what it could affect, how to go back.
  3. Stop releasing at risky timesChoose a time when the developers are available afterwards and customers are least affected.
  4. Ask four questionsIs there a test copy? Can the last version be restored quickly? Is there a backup taken before the release? Who checks what after it is live?
  5. Keep the list of things that must always workLogin, checkout, booking, whatever pays the bills. These are checked after every release.
  6. Do not demand a rewriteMost release problems come from process, which can be fixed in the current system.

When should I bring someone in?

  • Releases cause outages that cost you customers or money.
  • Nobody on the team knows how the live system is deployed.
  • Database changes are involved and nobody can say how to undo them.

How do I handle it?

  1. Read the last releasesI look at what changed and what broke to find the causes that repeat.
  2. Add a test copyA staging environment that matches production closely enough to catch the usual surprises.
  3. Make the release a scriptThe same steps every time, written down and automated where sensible, with a backup before it runs.
  4. Ship smaller and test the way backChanges are split into smaller releases and the rollback is rehearsed, including for database changes.
  5. Check the money paths after each releaseA short, fixed checklist of the flows that must work, run right after going live.

The price depends on what I find, so I quote in writing after a short discovery call. Tell me what is happening and I reply within 24 hours.

What do owners ask about this?

Do I need automated tests?

They help most on the flows that earn money. I would start with those, not with testing everything.

Will this slow my team down?

Usually the opposite after the first weeks, because less time goes on emergency fixes. I cannot promise a number.

Can a release ever be made risk-free?

No. The aim is to make failures small, visible and quick to undo.

How was this page made?

I drafted this page with AI assistance. It is general guidance for business owners, not advice for your own system. Found a mistake? Tell me.

Next step

Want this off your desk?

Send me a few lines on where things stand. I reply within 24 hours.