Your backup is not a backup until you have restored it

This is an English write-up of a post from my Japanese dev diary. Original: https://saas-diary.com/tech-log/backup-restore-drill-automation/ For over a year, my backup job has reported success every single night. Green check, every day, no exceptions. Then I asked myself one question and went cold: "How many times have I actually restored from it?" Zero. Not once. "It was backed up" and "it can be restored" are different states My setup has two paths. One mirrors all source to a private repo. The other packs the things I can never recreate — notes, config, and Android signing keys — into an encrypted bundle and ships it to a private channel every night. Both were green every day. But green only proved the upload finished . It never proved the contents were right, or that the archive could even be opened. Within one month, I had two failures that stayed green the whole time. Failure 1. The collector for signing keys used three hardcoded paths. I kept shipping new apps, so the number of keys kept growing — but the collector didn't. By the time I noticed, 7 of 10 keys were missing from the backup . Five of those apps were live on the store. If my machine had died, I could never have shipped an update for them again. The backup reported success every night through all of it. Failure 2. The mirror push failed 7 days in a row (a large binary hit the host's file-size limit). But the script printed "✅ done" and returned exit code 0 even when one half failed. A failure that isn't visible isn't a failure — it's a time bomb. So I automated a restore drill Once a month, a job now does this: Rebuild the encrypted bundle (without shipping it) Actually decrypt it with the stored passphrase Extract it and count what's inside Check the mirror is not stalled (latest commit timestamp via API) Delete the scratch folder and the generated bundle The encryption is openssl-compatible AES-256-CBC with PBKDF2 (SHA-256, 100k iterations). I deliberately avoided depending on the openssl binary,