Undo now, investigate later
Software Update Broke Everything? Roll Back First, Then Find Out Why
Nobody earns a medal for debugging a broken checkout in production while customers watch. The last version worked. Put it back, then find out what went wrong somewhere customers cannot see it.
November 19, 2026 · 10 min read
The short answer
Put the last working version back before you investigate. Hosts such as Vercel can switch production to an earlier deployment in one step, and in git a revert adds commits that cancel the faulty one. Check first whether the release changed the database, because rolling back code does not undo data changes. Then reproduce and fix the failure away from production.
On this page
- 4:55 on a Friday, and the pipeline says success
- Your first hour after a release goes wrong
- What a rollback restores, and what it leaves exactly where it was
- Four ways back to a working version
- When the build itself refuses to finish
- Five habits that turn a bad deploy into a bad week
- Roll it back yourself, or hand us the keys
4:55 on a Friday, and the pipeline says success
The deploy finished green. Every check passed. Ten minutes later the support inbox fills with customers who cannot log in, and the developer who shipped it is already in the parking lot. Somebody suggests a quick fix. Somebody else suggests reading the code to understand what happened. Both suggestions are reasonable, and both are in the wrong order.
Green on the dashboard and red in the checkout is still red.
When a release breaks production, the first job is to stop the damage, and the fastest way to stop it is usually to put back the version that worked an hour ago. Diagnosis comes second, on a copy, with the pressure off. A rollback is not an admission of failure. It is a decision to stop charging customers for your debugging time.
There is one important exception, and the next section puts it third on the list for a reason: if the release changed the database, rolling back the code alone can make things worse. Everything else on this page assumes you checked that first. If your site is a WordPress install and a plugin or theme update broke it, our page on a theme that broke after an update is the closer match.
Your first hour after a release goes wrong
- 1
Match the timing
Compare when the problem started with when the release went out. If they line up, treat the release as the cause until something proves otherwise. Glance at your host's status page too; Vercel's, for example, lists components such as Builds, Functions, CDN and DNS, so a platform problem is easy to rule out.
- 2
Stop the line
Freeze merges and further deploys. Turn off automatic deploys if a second push could land on top of the first. One broken release is a problem; two stacked on each other is a puzzle.
- 3
Ask whether the release touched the data
Did it run a migration, add or rename columns, or start a job that rewrote records? If yes, stop and read our database emergency page before rolling back, because old code against a changed database can fail in new ways.
- 4
Roll back the code
Use your host's rollback feature or redeploy the previous release. If your process deploys from git, revert the faulty commit and ship the revert through the normal pipeline.
- 5
Test the paths that make money
Buy something with a test card, send yourself a message through the contact form, sign in with a customer account, tap the phone number on a phone, and confirm the integrations still fire. A rollback that restores the home page but not checkout is not finished.
- 6
Tell the people who saw it
Staff first, then customers who hit errors. Our page on telling customers about an outage covers what to say and how often.
- 7
Keep the broken release
Do not delete the failed deployment or its branch. It is the evidence, and the fastest route to the real fix.
What a rollback restores, and what it leaves exactly where it was
Vercel's rollback guide, dated July 7, 2026, pitches the feature for "a swift recovery from production incidents, like breaking changes or bugs." The same guide lists several limits that matter in the middle of an incident, summarized below in our words. Other hosts behave differently; this is how one widely used platform documents it as of November 2026.
| After a rollback | What it means for you | |
|---|---|---|
| Your domains | Traffic moves to the chosen earlier deployment at once, without a new build | Visitors see the old version right away |
| Environment variables | The restored version runs on its own earlier build; variables edited in the dashboard during the rollback do not reach it | Changing a setting while rolled back fixes nothing live |
| New commits | Automatic assignment of production domains is switched off | Your fix stays out of production until someone promotes a deployment |
| Databases and outside services | The confirm step shows a reminder about "the changing behavior of external APIs, databases, and CMSes" | In our reasoning, data changes are not undone by a code rollback |
| Which versions | Pro and Enterprise teams may pick any earlier production deployment; Hobby accounts may step back only one | On a Hobby plan, the version before last may be out of reach |

The new commits row trips people up the next morning. The developer pushes a fix, the build succeeds, and the live site does not change, because production stopped following the branch the moment the rollback happened. Vercel's guide says the way back to normal is promoting a deployment, either from the dashboard or with its vercel promote command. Put that on the checklist for the end of every incident.
Four ways back to a working version
On teams that deploy from git, the cleanest undo is usually a revert. The git-scm documentation, read in November 2026, says the command will "revert the changes that the related patches introduce, and record some new commits that record them." It adds commits rather than erasing any. That matters in a crisis: everyone can see what was undone and when, and nobody's local copy breaks because history was rewritten underneath them. As practice, we avoid resetting or force-pushing a shared branch during an incident.
Host rollback
Fastest when your platform supports it. Points traffic at the previous build without a new deploy. Remember to re-enable normal deploys afterward.
Revert and redeploy
Records the undo in git, then ships it through the normal pipeline. Slower than a host rollback, but the history stays honest and the next deploy starts from a known state.
Switch the feature off
If the release put the new code behind a setting or flag, turning it off can stop the damage without touching the deploy at all.
Fix forward
Only when the cause is small, understood, and a rollback is impossible, for example after a database change that the old code cannot read. Ship the smallest change that restores service, nothing more.
When the build itself refuses to finish
A failed build is a different emergency from a bad release, and often a smaller one, because the broken code never reached customers. Vercel's guide to troubleshooting builds, updated September 21, 2026, says a failed build shows its error message on the Deployments page, and it advises running the build on your own machine first to surface problems in your code or its dependencies before they reach the platform.
- Vercel caps a build at 45 minutes as of November 2026; a build that runs longer is canceled, and one that exhausts its memory or disk is stopped as well.
- "A 'module not found' error is a syntax error that will appear at build time," per the same page, often after a dependency changed or a file was renamed.
- Builds can fail with no logs at all when, in Vercel's example, "An invalid vercel.json configuration was committed."
- A build that worked yesterday and fails today usually means something outside your code changed: a dependency version, an environment variable, or a setting.
For the Next.js version of this story, see our page on a Next.js or React site that went down and the deployment not found fix. WordPress has its own failure modes after updates: a PHP update can break old plugins, and a theme, builder or plugin conflict is a fixed-price Fix it now repair at $199, paid online on the matching fix page, started within one business day of payment, with a full refund if we cannot fix it. Our WordPress repair page lists the rest.
Five habits that turn a bad deploy into a bad week
- Debugging in production while customers keep hitting the error, because rolling back felt like giving up.
- Rolling back the code after a migration without asking what the old code will do with the new data.
- Pushing three quick fixes in a row, each one built on a guess about the last.
- Deleting the failed deployment to tidy up, along with the logs that would have explained it.
- Forgetting to re-enable normal deploys after a rollback, then losing a morning wondering why the fix will not go live.
The prevention list is short. Ship small releases so the culprit is obvious. Keep migrations separate from code changes when you can, and make them reversible. Test the money paths on a preview before production. And make sure at least two people know how to roll back, because the person who knows how is always the one at a wedding.
Roll it back yourself, or hand us the keys
Handle it yourself if
- Your host has a rollback button and the release changed no data
- Someone on staff has rolled back this app before
- The failure is clearly tied to one release and the old one worked
- Customers can wait a few hours for the real fix
Call us if
- The release ran a migration and rolling back looks risky
- Nobody knows how the app is deployed or where
- The rollback worked but now new deploys are failing
- The developer who shipped it cannot be reached
Our hours are listed on the contact page, and we answer the same business day. A failed release in custom software or a web app falls under our emergency software repair service. It is priced from $575, depending on complexity, and you get a written estimate before anyone starts; what raises or lowers that figure is laid out on the emergency pricing page. We stabilize first, find the root cause second, then fix it properly. We do not quote a fix time before we have read the code.
If the update broke an integration rather than the app, read API integration failures; if the whole app is down for reasons nobody can name, start with the web app triage page. Longer term, a safer release process is part of our web app development and custom software work. We are DataCram, a studio in San Antonio that has made software for the web since 1999 (about us), and the emergency repair hub maps every other kind of outage.
Questions owners ask in an emergency
Roll back in most cases. It restores the version you know worked, stops the damage and gives you time to diagnose on a copy. Fix forward only when the cause is small and understood and rolling back is impossible, typically because the release changed the database in a way the old code cannot handle. Even then, ship the smallest change that restores service.
Not in the way people hope. The restored deployment runs on its own earlier build and the settings that build was made with. According to Vercel's rollback guide, variables you edit in the project settings during a rollback are not applied to it, so a settings change made while rolled back stays invisible until a newer deployment is promoted to production.
Because the rollback detached production from your branch. Vercel's guide explains that it switches off automatic assignment of production domains after a rollback, so fresh commits build but never go live. Promote a deployment, from the dashboard or with the vercel promote command, to return to the normal flow. Other hosts have their own rules, so check yours.
No. In our reasoning, and as Vercel's rollback screen hints with its reminder about databases and outside services, a code rollback leaves the data as it is. Old code may not understand a changed schema. Check whether the release ran a migration before rolling back, and if it did, plan the database side first.
The git-scm documentation describes it as reverting the changes that earlier commits introduced and recording new commits that do so. Nothing is erased: the bad commit and its undo both stay in the history. That makes it the safer choice on a shared branch during an incident, since nobody's copy of the code is rewritten underneath them.
For custom software and web apps, emergency repair starts from $575, depending on complexity, with a written estimate before work starts. For WordPress, a theme, builder or plugin conflict is a fixed-price Fix it now repair at $199, started within one business day of payment, with a full refund if we cannot fix it.
Keep reading
- Yesterday's build still existsNext.js or React Site Down After a Deploy: Roll Back First, Debug SecondA Next.js or React site broke after a deploy. How Vercel's Instant Rollback works, what it will not undo, and how to read a failed build without guessing.
- Stop writing, then thinkDatabase Down or Damaged? The First Hour Before Anyone Presses RestoreDatabase down or data wrong? Stop the writes, copy what exists, check how far your backups reach, and restore into a new copy, never over the original.
- The website offerEmergency Website Repair: Who to Call When the Site Is Down, and What Happens After You DoWebsite down and no one to call? A same-business-day reply, fixed WordPress repair prices, from $575 for other platforms, and the checks before we sign off.
- The software offerEmergency Software Repair: Triage, Containment and a Written Root Cause for Broken Custom SoftwareCustom software or a web app broken today? Triage, stop the damage, restore, find the root cause, and a written estimate first. From $575, depending on scope.
- The mobile offerEmergency App Repair: An iOS and Android Developer for Crashes, Store Trouble and Expired CertificatesApp crashing, rejected or pulled from the store? Crash triage, store fixes, certificates and the access we need, from $575 with a written estimate first.
- Stabilize, then decideOld Software, No Developer: Rescue a Legacy Application Before You Decide Its FateOld software runs the business and its developer is gone. Stabilize it, back it up, map how it works, then choose repair, rewrite or replace with a calm head.
- Everything in Emergency RepairSee the section
From the Insights desk
- Next.js or PHP? The Better Question Is Who Patches Your SiteApache, NGINX and PHP all shipped urgent fixes between November 11 and 19, 2026. PHP is not the villain, and Next.js has patch days too. The real difference is who installs the fix, how fast, and how much of your site runs code at all.
- The Optimizely Hack Was Not an Outage. It Was a Delivery.On November 16, a flaw in Optimizely's sign-up flow let an attacker slip card skimmers, wallet swappers and iPhone exploits into the testing scripts of 178 organizations, the BBC and Zoom among them. How it happened, what it did, and the fixes, in order.
Get back to the version that worked
Tell us what was released, when it broke and how the app is deployed. We reply the same business day with a plan to stabilize it and a written estimate for the real fix.
Which of these sounds like you? (pick any)
Rather talk now? Call (210) 346-0848.