Skip to main content

We Had Backups. The Restore Still Failed: How to Make Sure Your Backups Actually Work

Updated 6 min readMarkdown

It started with a small mistake. A developer ran a cleanup script that was meant for a test database. The connection string pointed to production. A few important tables were gone in seconds.

Nobody panicked at first. "We have nightly backups," someone said in the incident channel. "Let's just restore."

Then we opened the backup bucket. The newest backup file was 0 bytes. So was the one before it. The backup job had been reporting "success" every night, because it only checked that the dump command finished. It never checked what the command produced.

In 14+ years of working with infrastructure, this is the lesson I've seen teams learn the hard way more than almost any other: a backup you have never restored is a hope, not a plan.

It's also one of my favourite interview questions: "How do you know your backups actually work?"

Why backups fail silently

Backups usually don't fail loudly. They fail quietly, for boring reasons:

  • Credentials expired or a password was rotated, and the dump command wrote an error instead of data.
  • The disk filled up halfway through, leaving a truncated file.
  • The schema changed, and a new database or table was never added to the backup list.
  • The job "succeeds" on exit code alone. A wrapper script returns 0 even when the real command failed.
  • Retention deleted the only good copy. The last good backup was older than the 7-day retention window.
  • Backups were in the same account as production, so the same mistake, bad script or attacker could delete both.

Every one of these looks green on a dashboard until the day you need it.

Step 1: Restore on a schedule, not only in an emergency

The only real proof a backup works is a successful restore. Make it a routine job, not a fire drill.

A simple weekly restore test for a PostgreSQL database might look like this:

# 1. Get the latest backup
aws s3 cp s3://my-backups/prod/latest.dump /tmp/latest.dump

# 2. Restore into a throwaway database
createdb restore_test
pg_restore --no-owner -d restore_test /tmp/latest.dump

# 3. Run real checks, not just "did it run"
psql -d restore_test -c "SELECT count(*) FROM orders;"
psql -d restore_test -c "SELECT max(created_at) FROM orders;"

# 4. Clean up
dropdb restore_test

The checks in step 3 matter most. Row counts tell you the data is there. The latest timestamp tells you how fresh the backup really is.

On AWS RDS you can do the same idea by restoring a snapshot or a point-in-time copy into a new instance, running your checks, and deleting it:

aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier prod-db \
  --target-db-instance-identifier restore-test \
  --use-latest-restorable-time

Time the restore too. That number is your real recovery time, not the one in the design document.

Step 2: Monitor the backup itself, not the exit code

Add alerts on what the backup produced:

  • Size: alert if today's backup is much smaller than yesterday's, or close to zero.
  • Age: alert if the newest backup is older than expected (for example, more than 26 hours for a nightly job).
  • Restore test result: alert if the weekly restore test fails or takes much longer than usual.

A backup job that fails loudly is fine. A backup job that fails quietly for three weeks is the real danger.

Step 3: Know your RPO and RTO

Two terms every cloud engineer should be able to explain simply:

  • RPO (Recovery Point Objective): how much data can we afford to lose? If you back up once a night, you could lose up to 24 hours of data.
  • RTO (Recovery Time Objective): how long can we be down? If a full restore takes 6 hours, that is your best case.

Ask the business for these numbers, then check whether your setup can actually meet them. If they need minutes, not hours, nightly dumps aren't enough. You need things like point-in-time recovery, a standby replica, or a multi-AZ / multi-region design.

Step 4: Keep one copy out of reach

Follow a simple version of the 3-2-1 idea: more than one copy, on more than one type of storage, with at least one copy somewhere separate.

In the cloud, that usually means:

  • Copy backups to a separate AWS account (or a separate project in GCP/Azure) with limited access.
  • Turn on versioning and Object Lock or deletion protection, so backups can't be deleted early, even by an admin with a bad script.
  • Consider a second region for critical data.

Ransomware and "oops" scripts both love backups that sit right next to production.

Step 5: Write the runbook and practise it

At 2 AM during an incident, nobody wants to work out the restore steps from scratch. Write a short runbook:

  • Where the backups are and who has access
  • The exact restore commands
  • How to point the application at the restored database
  • How to check the data is correct before you announce "we're back"

Then practise it as a game day at least once a quarter, with someone who didn't write the runbook. If they get stuck, the runbook needs fixing.

The interview angle

When I ask "How do you know your backups work?", a strong answer covers:

  • Regular restore tests into a separate environment, with real data checks.
  • Monitoring backup size and age, not just the job's exit code.
  • Clear RPO and RTO, and a design that actually meets them.
  • Isolated, protected copies in another account or region.
  • A written, practised runbook.

The weak answer is "we take snapshots every night." That tells me you take backups. It doesn't tell me you can restore.

The lesson

That day, we got lucky. A read replica was still a few minutes behind and hadn't replayed the bad script yet, so we recovered most of the data from it. Luck is not a strategy. Afterwards we added a weekly automated restore test, size and age alerts, and cross-account copies. The restore test caught a broken backup within its first month.

Nobody cares that you take backups. They care that you can restore. Test the restore before the day you need it.

When did your team last do a real restore test, and did it work the first time?


Let's connect: Free 15-min Cloud & DevOps Interview Guidance Call 👆 link in bio (topmate.io/cloudtechbydevang)

About the author

  • I help engineers learn cloud, DevOps, SRE and AI in simple English, crack interviews and follow clear roadmaps. 14+ years building and running platforms on AWS, GCP, Kubernetes and Terraform. One topic a day.

No comments yet. Be the first to comment.