Software triage
Web App Down: A Triage Order for Custom Software When the Error Page Is All Your Users Can See
A custom web app fails in more ways than a brochure site. There is a database behind it, a deploy pipeline, a login system and a handful of outside services, each with its own bad days. When it goes down, the first question is not how to fix it. It is which of those parts broke.
November 18, 2026 · 10 min read
- 10:02 portal returning 504 on shipment pages, all users
- 10:06 deploys frozen; host and cloud status pages clear
- 10:14 last release 09:40: new shipment search filter
- 10:19 logs copied; first error is a database timeout
- 10:31 rolled back to the 09:12 release, pages load
- 11:05 data checked: no migration ran, records intact
- 11:52 cause: filter query missing an index, fix on a copy
The short answer
When a custom web app goes down, triage in this order: write down the exact error users see, check the status pages of your host and outside services, look at what deployed most recently, read the logs for the first error, then check the database. If a deploy caused it, roll back first and diagnose second. Leave logs and data untouched until the cause is known.
On this page
- The status code on the error page is your first witness
- Sixty minutes on a broken web app
- Six layers that fail, and the deeper guide for each
- Roll back first, then investigate on a copy
- How a two-hour outage becomes a two-day outage
- One honest update beats forty worried emails
- Calling in DataCram: response, price and the access we need
The status code on the error page is your first witness
Before anyone opens a terminal, write down exactly what users see, including the number. The 5xx codes are defined in the IETF's RFC 9110, and each one points at a different layer of the stack. A screenshot from a customer that says only "it's broken" costs ten minutes. A screenshot that says 504 saves them.
| Code | What the standard says | Where we look first |
|---|---|---|
| 500 | The server hit an unexpected condition; MDN calls it a generic error | Application logs, the most recent deploy |
| 502 | A gateway or proxy got an invalid response from the server behind it | The app server or function behind the proxy |
| 503 | The server is temporarily overloaded or down for maintenance | Traffic spikes, capacity, a maintenance switch left on |
| 504 | A gateway or proxy did not get a timely response from upstream | Slow database queries, a stalled outside service |
| 401 or 403 | Credentials missing or invalid, or known and refused | Login provider, expired keys, changed permissions |
Our existing pages on the 500 error, the 502 bad gateway, the 503 service unavailable and the 504 gateway timeout cover each one in depth, mostly for websites. This page is the emergency order of work for software: the app your staff or customers log into, built for your business, where nobody can simply reinstall a theme.
Sixty minutes on a broken web app
- 1
Minute 0: one person in charge, one channel, no deploys
Name who makes decisions and where updates go. Stop all deploys and configuration changes. Write the time the problem started and the time it was noticed; they are rarely the same.
- 2
Minutes 0 to 10: capture the failure exactly
Which pages, which users, which actions, which error code, since when. Does it fail for everyone or only for one account, one browser, one region? Save screenshots with timestamps.
- 3
Minutes 10 to 15: read the status pages you depend on
Your host, your cloud provider, your payment processor, your login provider, your email service. Vercel's status page, for example, lists Builds, Functions, CDN, DNS and TLS Certificates separately. A matching incident changes the whole plan.
- 4
Minutes 15 to 25: check what shipped last
If a deploy, a dependency update or a configuration change went out before the failure, roll back to the last version that worked. Diagnose the bad release afterward, on a copy, without users waiting on you.
- 5
Minutes 25 to 40: save the logs, then read them
Copy the application and server logs from before the failure before anyone restarts anything. Look for the first error, not the loudest one. The loud errors are often a downstream echo of a quiet one.
- 6
Minutes 40 to 60: check the database without writing to it
Is it available, is storage full, are connections maxed out, does recent data look right? If data looks wrong, stop the writes that are changing it. Do not restore a backup over it yet.

Six layers that fail, and the deeper guide for each
Once the first hour narrows the cause to one layer, the work becomes specific. Each row points to the page in this section that covers that layer in full.
| Layer | Typical symptom | First check | Deeper guide |
|---|---|---|---|
| A release | Broke right after a deploy or update | Deploy history, rollback | Failed software update |
| The database | Timeouts and 500s on pages that read data, connection errors | Database status, storage, connections | Database emergency |
| An outside API | One feature fails, with 401, 403 or 429 in the logs | The provider's status page, keys and limits | API integration broken |
| The host or cloud | Everything down at once, provider incident listed | Provider status page and health dashboard | Cloud server outage |
| Payments | Checkouts or renewals failing, orders not recorded | Payment provider status, webhook deliveries | Payment processing failing |
| Domain or certificate | App unreachable by name, browser security warning | Domain expiry, DNS, certificate dates | Domain and DNS emergency |
A word on outside providers. AWS's shared responsibility model says AWS protects the infrastructure, while on EC2 the customer manages the guest operating system, the application and their data. So even when the cloud is healthy, the app can be your problem. And when the cloud is not, the EC2 service level agreement commits to a monthly uptime percentage of at least 99.99% for instances across two or more Availability Zones, as of November 2026, with service credits as the remedy. Our reading: an SLA pays back part of a bill. It does not pay for lost sales.
Roll back first, then investigate on a copy
When a release caused the outage, the fastest safe move is usually to put the old release back. Vercel's Instant Rollback page describes reverting to a previous production deployment for exactly this, and says the domains point back to the selected deployment. Pro and Enterprise teams can roll back to any deployment previously aliased to production; Hobby users can roll back to the immediately previous one, as of November 2026.
Two details catch teams out. Vercel says that after a rollback it turns off auto-assignment of production domains, so new pushes to the production branch will not replace the rolled-back deployment until someone promotes one. And during a rollback, changed environment variables are not applied. On other hosts, git's own revert command records new commits that reverse a faulty one, which keeps the history honest and makes the fix easy to redo once the bug is understood.
How a two-hour outage becomes a two-day outage
The common thread in long outages is not a hard bug. It is evidence destroyed in the first hour by someone trying to help. Here are the usual ways it happens.
- Restarting the servers again and again. It can clear the error for ten minutes and erase the logs that explained it.
- Restoring last night's backup over the live database. A point-in-time restore can reach only as far back as the retention window set before the problem, and a blind restore can overwrite the records that show what went wrong.
- Deleting and recreating a database instance to clear an error. AWS's RDS backup documentation says that if you don't choose to retain automated backups when deleting an instance, all automated backups are deleted with it.
- Shipping a quick fix straight to production with no test. Now there are two unknown changes instead of one.
- Rotating every API key at once because one might be compromised. Google Cloud's API key best practices recommend creating new keys, updating applications to use them, then deleting the old ones, in that order.
- Saying nothing to users for three hours while the team works in silence.
Restarting a broken server is the software version of shaking a vending machine.
On the database specifically, AWS says RDS backup retention can be set between 0 and 35 days, as of November 2026, and that setting it to 0 disables automated backups. Neon's history window, which limits how far back a restore can reach, depends on the plan. Find out what yours is before you need it, not during the outage. The database emergency page covers restores in detail.
One honest update beats forty worried emails
Within the first hour, tell users that you know, what is affected and when you will update them next. Not when it will be fixed, because you do not know yet. Atlassian, in its Statuspage documentation, describes the product as a communication tool for informing users about outages and scheduled maintenance, with email and text notifications to subscribers. Our advice, drawn from that: one posted update that people can subscribe to saves your team from answering the same question forty times. A shared email or a banner in the app does the same job on a smaller budget. The SaaS outage communication page covers wording and cadence.
Calling in DataCram: response, price and the access we need
We respond the same business day, during the hours on our contact page. We triage in the order above, stop the damage, restore service, find the root cause and write it down so it does not happen twice. Software repairs are from $575, depending on complexity, with a written estimate before work starts. Connections to outside APIs and integrations are restored the same day once we have access. The emergency software repair page explains how the work runs, and emergency pricing explains what moves a quote.
Hosting and cloud
A login with permission to read logs, view deployments and roll back, on your host or cloud account.
The code
Access to the repository and a note of which branch is production. If nobody knows, that is the first finding.
The database
Read access first. We ask before writing to anything, and we take a copy before any restore.
The outside services
A list of the payment, email, login and data providers the app calls, and who holds each account.
Call us today if
- The app is down and nobody on staff can read the logs
- A rollback did not fix it, or you have no way to roll back
- Data looks wrong or missing
- The developer who built it is not reachable
Wait, or handle it in-house, if
- Your cloud or host lists a matching incident: nobody can speed that up
- Your own team has a rollback and it worked
- Only one outside feature is down and its provider shows an incident
- The app is internal and can wait for business hours
For the longer view, see web app development and custom software development. If the outage exposed that nobody owns the system, a fractional CTO can take that on. DataCram is a web studio in San Antonio, Texas, building for the web since 1999, working with businesses across the US by phone, video and a shared project page. See our work, the rest of the emergency repair section, or call (210) 346-0848.
Questions owners ask in an emergency
Write down the exact error and when it started, stop all deploys, and check the status pages of your host and outside services. If something shipped just before the failure, roll it back. Then save the logs before anyone restarts anything, and check the database without writing to it.
Roll back when a recent release caused the outage and a known good version exists. It restores service while you diagnose on a copy. Remember that a code rollback does not undo database changes the bad release made, so check the data and confirm the old code can still read it.
No. If AWS, your host or another provider lists a matching incident, the fix is theirs. Use the time to post an update for users and confirm your backups. AWS's EC2 SLA offers service credits when uptime falls below its commitment, as of November 2026, which repays part of a bill, not lost sales.
Save the logs first. A restart can clear the symptom for a while and wipe the evidence of the cause. If you must restart to restore service, copy the logs from before the failure, note the time, and expect the problem to return until the cause is found.
Software repairs are from $575, depending on complexity, with a written estimate before work starts. We respond the same business day, and connections to outside APIs are restored the same day once we have access. The emergency pricing page explains what moves the number.
Usually. We need access to the hosting, the code repository, the database and a list of outside services. If the original developer is gone, we start by securing those accounts, then triage the outage in the same order as any other.
Keep reading
- Stop writing, then thinkDatabase Down or Damaged? The First Hour Before Anyone Presses RestoreDatabase down or data wrong? Stop the writes, copy what exists, check how far your backups reach, and restore into a new copy, never over the original.
- Undo now, investigate laterSoftware Update Broke Everything? Roll Back First, Then Find Out WhyA release broke production. Roll back to the last good version, check what a rollback cannot undo, then diagnose away from the live site. The first hour.
- The website offerEmergency Website Repair: Who to Call When the Site Is Down, and What Happens After You DoWebsite down and no one to call? A same-business-day reply, fixed WordPress repair prices, from $575 for other platforms, and the checks before we sign off.
- The software offerEmergency Software Repair: Triage, Containment and a Written Root Cause for Broken Custom SoftwareCustom software or a web app broken today? Triage, stop the damage, restore, find the root cause, and a written estimate first. From $575, depending on scope.
- The mobile offerEmergency App Repair: An iOS and Android Developer for Crashes, Store Trouble and Expired CertificatesApp crashing, rejected or pulled from the store? Crash triage, store fixes, certificates and the access we need, from $575 with a written estimate first.
- Locked out, all at onceApp Login Not Working: Sort Who Is Locked Out, Check the Keys That Expire, and Fix It Without Locking Out MoreUsers can't log in to your app? Sort who is locked out, check provider status and expired secrets, follow Apple's login rule, and avoid lockout-making fixes.
- Everything in Emergency RepairSee the section
From the Insights desk
- The Optimizely Hack Was Not an Outage. It Was a Delivery.On November 16, a flaw in Optimizely's sign-up flow let an attacker slip card skimmers, wallet swappers and iPhone exploits into the testing scripts of 178 organizations, the BBC and Zoom among them. How it happened, what it did, and the fixes, in order.
- Why Your WordPress Form Gets Spammed — and the Fix That Doesn't Annoy Real CustomersForm spam isn't the problem — it's what the spam does: buried leads, a poisoned CRM, and an inbox you've learned to ignore. Why CAPTCHAs make it worse, and the silent fix that stops bots without costing you real customers.
Get your app back up
Tell us what users see, the error code if there is one, and what changed before it broke. We reply the same business day with a triage plan and a written estimate.
Which of these sounds like you? (pick any)
Rather talk now? Call (210) 346-0848.