CI/CDBattleOpsRazorpaySwiggyAsked in DevOps engineer and SRE rounds
A release included a database migration and it's broken in production. How do you roll back?
The scenario
Version 42 renamed a column. The new code is throwing errors; the old code expects the old column name.
Model answer
- You often can't just roll back the code: the old version expects the old schema.
- Short term: if the migration is reversible and no new data depends on it, run the down migration and redeploy v41. Otherwise fix forward.
- The real answer is preventing this: expand and contract. Add the new column, deploy code that writes both, backfill, switch reads, and only drop the old column in a later release.
- Every migration in a release must be backward-compatible with the previous version of the code.
- Gate destructive migrations behind a separate, manual step.
Common mistakes
- Assuming the code rollback is enough
- Renaming columns in one step