2

Assume I have those Django models:

class Book(models.Model):
    title = models.CharField(max_length=100)

class Author(models.Model):
    name = models.CharField(max_length=100)
    books = models.ManyToManyField(Book)

I already have a production system with several objects and several Author <-> Book connections.

Now I want to switch to:

class Book(models.Model):
    title = models.CharField(max_length=100)

class BookAuthor(models.Model):
    book = models.ForeignKey(Book, on_delete=models.CASCADE)
    author = models.ForeignKey("Author", on_delete=models.CASCADE)
    impact = models.IntegerField(default=1)

    class Meta:
        unique_together = ("book", "author")

class Author(models.Model):
    name = models.CharField(max_length=100)
    books = models.ManyToManyField(Book, through=BookAuthor)

If I do this Migration:

from django.db import migrations

def migrate_author_books(apps, schema_editor):
    Author = apps.get_model('yourappname', 'Author')
    BookAuthor = apps.get_model('yourappname', 'BookAuthor')

    for author in Author.objects.all():
        for book in author.books.all():
            # Create a BookAuthor entry with default impact=1
            BookAuthor.objects.create(author=author, book=book, impact=1)

class Migration(migrations.Migration):

    dependencies = [
        ('yourappname', 'previous_migration_file'),
    ]

    operations = [
        migrations.CreateModel(name="BookAuthor", ...),
        migrations.RunPython(migrate_author_books),
        migrations.RemoveField(model_name="author", name="books"),
        migrations.AddField(model_name="author", name="books", field=models.ManyToManyField(...),
    ]

then the loop for book in author.books.all() will access the new (and empty) BookAuthor table instead of iterating over the existing default table Django created.

How can I make the data migration?

The only way I see is to have two releases:

  1. Just add the new BookAuthor model and fill it with data, but keep the existing one. So introducing a new field and keeping the old one. Also change every single place where author.books is used to author.books_new
  2. Release + migrate on prod
  3. Remove author.books and rename books_new to books. Another migration, another release.

Isn't there a simpler way?

Improve this question
7
  • 1
    You can all do it in one migration file. Although I don't really see why you want that: Django does topological sorting on the migration files, but that happens in O(n). Commented Sep 12, 2024 at 12:02
  • I'm not sure how a concept of a "release" really ties into this. A single commit (that might get pushed into a production system somehow) can of course have multiple migrations, and Django will run them in order. Commented Sep 12, 2024 at 12:05
  • @willeM_VanOnsem When I did it as described above on a test system, all relations were gone. When the migration is executed, then author.books in the code already has the custom throguh model (BookAuthor) which is empty. Instead, it should use author.books from before the code change (the default one) Commented Sep 12, 2024 at 12:08
  • @MartinThoma: well you should not remove the m2m and so on and then perform makemigration. You simply "fold" multiple migrations in one file. But regardless, you don't need to move each migration individually to production. Commented Sep 12, 2024 at 12:09
  • 2
    @MartinThoma: normally you don't run makemigration. This is extremely dangerous. You make the migrations when you develop, and then run these migrations on production. By making migrations on production, these can be different than the ones in development, so the migration table of the dev/prod database are no longer in sync, and you get situations like the one you desribe. Commented Sep 12, 2024 at 12:21

1 Answer 1

Reset to default
5

You don't actually need a data migration at all to add a through table to a many-to-many relation.

When you create a many-to-many relation without a through, Django creates a virtual model for it you behind the scenes:

>>> from xxxx.models import Author
>>> Author.books.through
<class 'xxxx.models.Author_books'>
>>> Author.books.through._meta.db_table
'xxxx_author_books'
>>> Author.books.through._meta.get_fields()
(<django.db.models.fields.BigAutoField: id>, <django.db.models.fields.related.ForeignKey: author>, <django.db.models.fields.related.ForeignKey: book>)
>>> Author.books.through._meta.unique_together
(('author', 'book'),)

Armed with this knowledge, you can create the same table as a real model (nb: I didn't check the exact fields from the virtual through table – you might want more diligence here!).

The important bit, however, is that you will need to set db_table manually to what the virtual through table's name is.

class BookAuthor(models.Model):
    book = models.ForeignKey(Book, on_delete=models.CASCADE)
    author = models.ForeignKey("Author", on_delete=models.CASCADE)

    class Meta:
        unique_together = ("book", "author")
        db_table = "xxxx_author_books"

You'll also need to set the through on the ManyToManyField at this point.

If you create a migration out of this, you will get a CreateModel, but trying to run that migration will understandably fail – you already have a "xxxx_author_books".

You'll need to modify the migration to wrap the operations in this migration in a SeparateDatabaseAndState, so the physical database is not touched – after all, it doesn't need to be touched, since all we did was write out the same table that already was implicitly created:

migrations.SeparateDatabaseAndState(
    database_operations=[],
    state_operations=[
       migrations.CreateModel(...),
       migrations.AlterField(...), 
       ...

Migrating this should go through without a hitch (and without touching the database).

You now have a bona fide through table you add the impact field on, and migrate as usual.

EDIT: I just noted this operation has been described in the manual, too.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.