"Data schema changes are a potential problem, but that's a general issue with rollbacks. Backwards compatibility and zero downtime deployments are a way to make sure this will work regardless of the changes."
At at 52 minutes in David Brady asks about irreversible migrations
DAVID: Irreversible migrations, question mark?
[Laughter]
BADRI: Yes?
FLORIAN: Part of the game.
DAVID: Part of the game. How do you do that? So, we talked earlier about the customer name table. I've got a name in there like David A. Black. And I need to make first name be David, last name be Black. But at the reverse migration, I've lost that middle initial. How do we make that be part of the game?
JAMES: Don't do it.
CHUCK: [Laughs]
JAMES: I'm serious. I think that's my answer. That is definitely not a case where you can't split it into, how many negatives does it have in there? Maybe I got lost. You can put that in two fields or you can separate it out or whatever.
DAVID: Okay. I can give a better example. The customer name field contains James Edward Gray II.
[Laughter]
JAMES: [inaudible]
DAVID: I know you have said that you have had systems that think your last name is Edward Gray 2.
JAMES: Right.
DAVID: Right?
JAMES: Yeah.
CHUCK: It's just 2.
DAVID: Now, that's a different problem. That's the problem of getting the name division wrong. But let's say I do actually get it right and now I've got you stored in the system as James Gray, which technically is your dad, right? But aside from the fact that we never should have done that migration in the first place, [laughs] how do you handle that irreversible migration? Do you hang onto the data for six months and then decide to throw it away? Do you just suck it up and go with it and just realize that, "Hey guys, we can't roll this one back. So, everyone, all that panic we haven't had since we went to extreme deployment, it's back."
JAMES: Add a feature flag so you can shut the feature off.
FLORIAN: Yeah. Deploy a second migration a week later or a month later that just removes the column if necessary, but just keep as much in the data around until you're super sure it's not going to be used anymore. And there's just move over, over a period of migrations. And then just wait until you're sure that nobody's going to have any problems with it. So, that's what we've been, when we did that, we've been successful with it.
Obviously, so something that we do as well is just take migrations, put a dump of our database everyday onto a staging system, and run the migrations over them before we deploy just to make sure nothing breaks. And typically we do small migrations there. Nothing really breaks or they're pretty fast to run over that staging system as well. That's really important. But then just don't delete data until you're really, really sure, until everybody asks you, "Why is that field still in there? Didn't we remove that six months ago?"
DAVID: Right.
FLORIAN: When you're at that point, then you can remove it.
BADRI: Yeah.
DAVID: I would also add that a step you have to add going forward, which is that you have to continue to populate that legacy field. So, when somebody comes in and they type in first name James, last name Gray, you have to save first name James, last name Gray, and you have to save customer name as James Gray. So that when you roll back, you've still got your legacy column.
JAMES: Right. Avdi?
BADRI: Or you could separate the reading and the writing and keep a background that breaks the name down. So, you could continue to write the one big name and you could have a job that populates the new columns. And that way, you separate the reading process and the writing process. And I've found that that separation really helps as well.
DAVID: Right.
AVDI: So, basically in summary, to quote Bane, "When Gotham is ashes, then you have my permission to delete the column."