Populate initial data with data migrations and unittest. What's the pattern with remake-migrations ?

[ticosax]

Hi,
We are considering adopting remake-mirations for our projects, but it seems there is a blocker for us.
I'd like to ask how you deal with data migration to populate initial data in the context of unittests.

As explained from the django documentation,
https://docs.djangoproject.com/en/6.0/howto/initial-data/#provide-initial-data-with-migrations,
It is a known pattern to populate initial data with migrations.

It provides few guarantees

• All environments are populated with same data.
• Once data is loaded it will never be updated again.
• Unittests are also expected to find these fixtures.

Now, If we optin for remake-migrations, we are going to loose our data migrations, as advertised, from the history. It is safe to do so for all envs, such dev/staging/production which already have the data, but it will defeat unittest data provisioining that is replaying the migration history every time we recreate the db (observed with pytest-django).

What's the recommended pattern to deal with initial data loading for unittests from existing migration history and use remake-migrations together ?

[browniebroke]

Hi,

Good question. That's not currently supported out of the box, as remake migration re-generate your DDL migration from scratch and delete all the migration files, without any special treatment for files for data migrations. Last time I checked, this was the same with the built-in squashmigrations, if one of the migration in between the ones you squashed were a data migration, it's lost. I vaguely recall Django putting some clues to help you spot them, but it's been a while...

Your best bet at the moment is probably to hook into REMAKE_MIGRATIONS_POST_COMMANDS, and write a command that regenerate your data migration... Kind of hacky though 🫤

I usually steer away from filling my DB with initial data, and use something like factory-boy to add whatever my test need. I also use a similar mechanism to seed a test database for deployment previews environments.

I can see how it may be useful for some config-style tables holding near-static data which is used by most of the app, recreating and deleting them in every test is wasteful. It's just a problem I don't really have, perhaps due to the characteristics of the system I'm working on...

Anyway, it's a bit of a long winded way of saying that this is currently not solved, but I'm open to a solution, as long as it's not too complicated to maintain.

[ticosax]

Thanks for answering. You are right the builtin squashmigrations doesn't address this use case either.

Makes me think, that even if we manage to keep the data migrations, we can no longer guarantee the shape of the table is the same as the one when the data migration was first written. It probably doesn't make sense at all to try to do that.

It's probably a dead end and relying on test fixture built within the test framework is probably the direction we will go for.
Thanks, I'll close the conversation. I don't think it's worth pursuing the idea to keep data migration in the history.