Help for owners
My site goes down every time I release an update
Short answer
Releases cause downtime when the old version is stopped before the new one is ready. The fix is to start the new version first, check it works, then switch traffic across. Do not release at peak hours while this is unsolved, and make sure a way back exists.
Does this sound like your situation?
- Customers see errors for a few minutes after every release.
- I only dare to release at night.
- Users get logged out or lose a basket when an update goes out.
- The last update broke the site and rolling back took hours.
- A release means stopping the app and copying files by hand.
Why does this happen?
The old version is stopped before the new one is ready
A simple release stops the app, replaces files and starts it again. Everyone who visits in between sees an error.
Servers can be told to hand over gradually
nginx documents that on a configuration reload the master process checks the new configuration first, and if it fails it continues with the old one. Old worker processes finish serving existing clients before they shut down. That same idea is used for application releases.
Confirmed by nginx.org
Database changes lock the app
A change to a large table that the old and new versions both use must be done in steps, not in one jump.
Technical detail: A statement waited for a lock held by another transaction and gave up (error 1205)
Releases are manual
A person copying files at night makes mistakes. A scripted release is the same every time.
Technical detail: A GitHub Actions SSH deploy step cannot log in to the server
What can I do about it today?
- Release outside your busiest hoursLook at when customers are least active, and keep releases there.
- Tell customers about planned downtimeA short notice on the site is better than an unexplained error.
- Keep the previous versionBefore every release, make sure the old files and a database backup exist, so you can go back.
- Release small changes more oftenSmall releases are easier to undo than a big one every few months.
- Write down the steps a release takes todayEven rough notes help plan an automatic version.
When should I bring someone in?
- Downtime during a release costs you orders.
- Only one person knows how to release.
- Database changes are involved.
How do I handle it?
- Map the release as it isI follow what happens at release time and where the break comes from.
- Start the new version beside the old oneThe new version runs, and is tested, before it takes any traffic.
- Switch traffic when the check passesA health check decides when the switch is made, and the old version keeps serving until then.
- Change the database in two stepsChanges are made compatible with both versions first, and cleaned up later.
- Prove the rollbackI test going back to the previous version so it is not a guess on release day.
The price depends on what I find, so I quote in writing after a short discovery call. Tell me what is happening and I reply within 24 hours.
What do owners ask about this?
Is zero downtime always possible?
Not for every app. Some changes need a short break, and I tell you which before it happens.
Does this need a bigger server?
Often two copies of the app must run briefly, so it may need a little more room. I check it first.
Do I need Kubernetes?
No. A small app can switch versions with simpler tools.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Found a mistake? Tell me.
For your developer: the technical fixes
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transactionA statement waited for a lock held by another transaction and gave up (error 1205)ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remainA GitHub Actions SSH deploy step cannot log in to the server