SwiftData Migrations and the Real World
My ongoing app project, Cartographer, uses SwiftData for its backing store. Before shipping I knew I needed to get a better grasp on how migrations are handled with SwiftData. I have a long roadmap of features I want to be able to add over time and I needed to be sure I have a handle on how my data models could evolve over time.
Cartographer is a native MacOS application using SwiftUI. It allows viewing and editing GPS data. You can plan future trips or review past adventures. I created the app to help generate data for some of my other map based app projects and to reduce my reliability on third party, mostly web based, tools. I’m also focusing on you owning your own data and privacy but that’s for another post.
The two main data models for Cartographer are Locations and Paths. Each also has a relationship to a Waypoint that holds the information necessary to locate the item on the map such as latitude and longitude. A Location will have a single Waypoint while a Path will store a series of Waypoints to trace a route.
Since the earliest iterations of the app, I’ve provided a Group as a way to organize collections of Locations and Paths. Say you are planning a multi-day trip and you want to break it up into several Paths, one for each day, and specific Locations along your route. You can place all these Paths and Locations into a Group to quickly display on the map all the data associated with the Trip you are planning.
In SwiftData terms the Locations and Paths have a Relationship to a Group. A Location (or Path) can belong to one Group and Group can have many Locations. I elected to call my Group objects AnnotationGroup so as to not be confused with a SwiftUI Group. The Location Model looked something like this:
(side note: I had to add Hashable conformance for the Model to stop Xcode from complaining about the Model not conforming to Hashable. Prior to this addition I would have to compile twice in order to get this error to go away so I added this declaration and the problem stopped. Go figure)
This is not the entire model definition and a similar setup was used for Paths. This was fine and worked as expected. Later on in the development process I realized that a one Group per Location/Path relationship was not ideal. What if I want to use a Location in many Groups? I knew I needed to update my models in order to reflect this change.
Since I was using SwiftData, the backing store (where the data lived) was on my Mac and changes to the Model definitions in my code (the Schema) would cause errors upon launching the app. The app did not know what to do with the old data that didn’t match the new Schema. Up until this point if I changed a model I just deleted the data files and started over. The default SwiftData app template will create a ModelContainer for you that stores it’s data at a path something like this:
/Users/<your_user_name>/Library/Containers/<your_apps_bundle_id>/Data/Library/Application Support
Within that directory you will find three files named:
default.store
default.store-shm
default.store-wal
These three files contain your apps data and the configuration needed to work with that data. If you are just beginning your project and frequently updating the models it’s fine to just delete these three files and start fresh any time you make a breaking change.
The default app template using SwiftData will provide you with a typical ModelContainer setup and a Model to use. As you develop your app you will add more Models and in turn add them to the ModelContainer schema. This works fine during development or if the app you are building is not likely to have Schema changes once you are done.
In the real world for more than a tutorial app or a quick throwaway project, you are likely to need to continue to revise your Schema updating the Model definitions and migrating old data into a new Schema. If you’ve done work with a database backend before you know the drill.
Your users will not want to delete their old data and start over each time you update your app. You are going to need to migrate older data into your newer formats.
There are Apple Developer videos that explain working with SwiftData. They allude to Migration strategies but for the most part the content provided is not very deep. These are valuable learning tools to get started with SwiftData and you should certainly review these videos.
Model Your Schema with SwiftData
These videos and associated sample code will get you started working with SwiftData and handling simple data migrations across schemas. There are other tutorials and blog posts out there that more or less repeat the information in these Apple videos. There is nothing wrong with any of these resources except they leave out some details.
In most non-trivial applications it’s up to you as the developer to take the frameworks, documentation and sample code in and come up with your own ways to use them in solving your problems.
Unfortunately when it comes to SwiftData Migrations the documentation is all but non-existent.
There are pages in the Apple Developer Docs that correspond to the required Structs needed to handle Migration but no relevant information is provided beyond the Struct or Method signatures. You will need to review the Videos, etc I’ve mentioned and do your own trial and error to get up to speed. Further, most samples and tutorials are trivial examples and solving real world problems will require trial and error.
I’ve done some trials and made some errors and here is what I’ve learned. I don’t claim any of this is the correct way to do things but it worked for me and may be helpful to others.
In the standard template Xcode uses for apps using SwiftData, a ModelContainer is configured and created for you. This configuration can be tailored a bit to make some changes such as how and where to store the data but you generally are unlikely to need to do much more than add your models to the schema. A typical ModelContainer might be defined as follows:
You can include all your Models in the schema but SwiftData is clever enough to add any related Models automatically. So in my case I can omit specifying Waypoint and AnnotationGroup Models as they are inferred by the Relationships defined in the Location and Path Models. Feel free to include them if you like but it’s not needed. I tend to include all my Models so I don’t need to think about it but I’ve left them out here for brevity.
The default ModelContainer setup defines the Schema including what models are to be stored, how to configure the store and then returns the container. You can create more than one container to store different Models or different configurations for different use cases but this is the default setup the Xcode template provides.
You might create a ModelContainer that is only used in Previews, for example, and you may change the `isStoredInMemoryOnly` setting to true as the Preview data is ephemeral and not required to be persisted.
You then create Classes for each data type or Model you need. Using the @Model macro is enough to tell SwiftData that the Class is a data model. You can then use other SwiftData macros to specify things like Uniqueness and Relationships among the Model data.
You can do all this and have a working SwiftData application without any concern for Migrations. Then one day you realize you need to make a breaking change that will require migrating data from an old format to a shiny new format.
In order to define how your data is transitioned from one schema to another you create versioned Schemas and a Migration Plan. I’ve used the term Schema throughout this article thus far but for the most part it’s just a specification of how your Models are defined from one version to another.
In my app, as I’ve explained earlier, I wanted to change the Relationship between a Location or Path so that each could belong to one or more Groups. This is called a Many to Many relationship. A Location may belong to many Groups and a Group may contain many Locations. It’s easy to define these relationships in your models. Just specify that the relationship is an Array of the desired type. In my case the updated Model looked something like this:
The only change here is that the earlier version defined a Group (singular) Relationship and here the Relationship is to Groups (plural) and those are defined as an Array of Annotation Group.
This would work just fine if I had no existing data. I already created a bunch of Locations and Paths in the old format and I don’t want to recreate all that data to work with this new Schema. If I was to run my app after making this change the app would crash with errors mentioning it didn’t know what to do with a Group, etc. The solution appears simple, I just need to take the old data and for each group a location belonged to insert that group into the new groups and carry on. Simple.
So I will need a new versioned Schema with this updated Model definition. Again, simple.
But wait I don’t even really have a versioned Schema for the original definition. I just used the default Model Container and defined a few models. How do I get from that to this new world of versioned Schemas?
That part isn’t difficult either but like a lot of development work it’s a bunch of cutting and pasting and typing. It goes with the job.
To create a Versioned Schema you define an enum for the schema. In this enum you include the version, what models are covered and the Model definitions themselves.
Wait I have to move my Models into this new enum? Yes. I don’t like i either. My Models contain a bunch more code not shown here and most of it doesn’t directly relate to SwiftData and what it needs.
If you are not terribly concerned with how your code is organized in your project or on disk feel free to skip this next bit. It’s a me problem but perhaps it’s a you problem too?
For example I have many @Transient properties to help with data presentation in the app. If you are not familiar with the @Transient macro, it essentially tells SwiftData you don’t need to be concerned with persisting this (i.e. dynamic properties.) Do I need to copy all that into this new versioned Schema?
It turns out you don’t. You can if you wish but you don’t need to include the entire Model definition within the versioned Schema. You do need to include the base Model definition along with any stored properties and at least one init definition. You can then create an extension of the Model that may include other helpful code. Of course if any of that helpful code will need to be versioned it will need to be in this versioned Schema. I dislike the idea of of all my models being in this single enum so I’ve left a bunch of it defined as extensions of the Model stored in its own file within the project. You may not be as offended by this huge enum definition and if you are not, feel free to copy and paste your entire Model(s) into the enum. If you are in doubt just copy the entire Model into the new VersionedSchema.
A typical VersionedSchema will define the version using a semantic numbering system (line 2) but the driving definition for use of the Schema is actually the enum type itself as we’ll see. You then specify what Models are included in the Schema and define the Models. Yes you have to include a minimum of a complete SwiftData Model definition for each model. If you have 10 Models you will have 10 classes defined within this enum. And yes, you will need to repeat these Model definitions for each version of your Schema. None of this is difficult but it is a code maintenance issue and for those of us who try not to repeat the same code over and over it can be a bit painful. The reasons should be obvious but we don’t have to like it.
In SwiftData Migration terms, I want to Migrate from SchemaV1 to SchemaV2. I will need to define each schema version in a VersionedSchema as shown above.
Once you have the Schemas defined you need to define the Migration Plan.
A SchemaMigrationPlan is just what it sounds like, a plan on how the data is to be migrated from one Schema to another (or more.) A base SchemaMigrationPlan looks something like this:
Again it’s defined as an enum and again the enum type will be important later. You specify which Schemas this Migration Plan is using and what stages to include.
What is a Stage? A stage is a two step process that will be executed when the Migration is run. In the example above, I define migrateV1toV2 as a MigrationStage. You can have more than one stage but this is already confusing enough. Actually, you may think of reasons to have multiple stages. I didn’t use that ability.
In defining a MigrationStage you indicate the starting Schema (fromVersion) and the end Schema (toVersion). Then two different closures will be called when your Stage is executed: willMigrate and didMigrate. Each closure will be provided a modelContext so you can manipulate your models and data as needed. Sounds good.
What does that mean?
WillMigrate is called first and will be provided with the ModelContext for the first schema. This may seem obvious and clearly it is self evident as the documentation for this method is completely empty in Apple’s Documentation. Sigh. You are given your old data in a Model Context. You can then make changes to that data if it makes sense or otherwise get a hold of the old data for use in the new context later. WilMigrate is called, as its name implies, before any migration has occurred.
For example a few of the tutorials make note of adding uniqueness to a property that was not originally unique. Let’s say you have a name field and it’s a String. It’s possible prior to adding the Unique macro that users created non-unique names. When you are provided the context in the willMigrate closure you can then update any names that collide and write new values. If the new Schema still includes the name property this updated data will migrate across just fine with the new values.
Refer to Apple's Developer Videos for a more detailed explanation of this type of migration.
You don’t need to make any changes to any Models or properties that don’t change. My initial thinking was I needed to rewrite each old model into the new model format. You don’t need to do anything of the sort. In an increasingly rare instance these days of “it just works”, it just works. You only need to change things that won’t survive the migration.
What sort of things won’t survive or work with an automatic migration? Well in my example a change in a Relationship from a single to a many type won’t just work. If you recall from way earlier (and kudos for still reading this. Thanks!) I can “just” copy the old Group value and insert that into the new Groups property. Done. Not so fast. I only have the single context provided here and that context is the old Schema context that knows nothing about the fancy new Schema and it’s Groups (many to many) relationship. I can’t just do something like:
newLocation.groups.insert(oldLocation.group)
Though I sure wish I could.
Here is where I will discuss what I did along with another concept that should achieve the same result in a different manner.
As mentioned the willMigrate and didMigrate each receive a different context to work with. Basically they do as they are labeled. Prior to SwiftData performing the initial migration (willMigrate) of all the things it can do automatically you are given access to the old data and can manipulate it or as I said perhaps hold on to portions of it to be reused with the new context in didMigrate.
So I’ll just declare some var to hold on to this data and read those values later. Here is another thing that feels a bit like a code smell. You can’t define any vars in your MigrationPlan or VersionedSchemas so you need to make them global. Not ideal. However, I feel that for the limited lifetime of a migration this is not the worst thing and all “rules” are made to be broken if not bent a little.
I ended up defining a Class to hold the old data and an array of that Class to write from willMigrate and be read from didMigrate.
This simple Class holds all the Locations and Paths that correspond to a Group with a specified ID. Seeing it in use may help explain. (Line 13 begins with a tribute to Modern Swift Concurrency to appease the compiler, praise Jobs.)
Here is a portion of the willMigrate closure that creates a GroupItem for each group. When we iterate over each Location and Path later in the willMigrate closure we insert the Location or Path into the appropriate GroupItem. This happens in willMigrate where we have the original ModelContext (i.e. the old data)
If you’ve followed thus far you can see where this is going. In didMigrate (with the new ModelContext) we then write all the data needed to setup the relationships for Locations and Paths to their associated Groups.
The missing piece to put this all together is telling SwiftData to use this Migration Plan. This is done in your ModelContainer initialization. If you specify a migationPlan it will be executed prior to the container being created (line 2 below.) So all that magic (and tedious data manipulation) we outlined above will happen and your old data will be safe and sound in its new Schema home.
As you add migrations you can continue to extend your migrationPlan. Keep in mind a user may have data from version 1.0 of your app and you have created several new schemas. The Migration Plan will need to assume always beginning with the first version.
This all works as expected. My old data has been migrated and the new data is correct. Each Group still have the correct Locations and Paths and now Locations and Paths can be added to multiple groups.
I was content with this solution. There is too many repetitive definitions of Models and a lot of boilerplate code but in the end the actual working parts are not too bad.
A few hours after I had all this working it dawned on me that this could be done another way.
An alternate Migration Plan could use 3 Schemas. SchemaV1 is the original with only the single relationship Group. SchemaV2 would include BOTH Group and Groups. SchemaV3 would remove the original Group relationship. In SchemaV3 when you were given the ModelContext in willMigrate both properties would exist and you could read from Group and insert into Groups. Then when the migration was completed the original Group field would be dropped leaving only the new Groups.
I believe this would work and perhaps this is what Apple intended and I’m sure they will document some best practices around this any day now.
Is the alternate a better solution? I’m not sure.
Keep in mind all the repetitive code you need to write for each VersionedSchema. Is that better than use of a global var to hold some temporary data? It wasn’t enough for me to redo my migration so it lives on as the version outlined here. Perhaps in another migration I will try the alternate way but I’m really allergic to that much repeated code. We’ll see.
Some additional thoughts:
Don’t start with VersionedSchemas when you’re just beginning your app development. It’s fine to tweak your models, delete the old data and carry on.
It’s a good idea to create Git Branches with the original schema and any newer versions as you work through your migration journey. You are unlikely to get things correct the first few times and being able to load back up the old schema may prove handy.
Backup the data stores (those default.store files) before you start making changes. Then you can always restore them rather than creating test data for your migration.
You will find that your Models now have type names such as SchemaV1.MyModel and in my case I put all these types into a Persistence Struct so I had to reference Persistence.CartographerSchemaV1.Location rather than just Location. The solution is to use a typealias such as:
typealias Location = Persistence.CartographerSchemaV2.Location
Keep in mind that while you are working thru your migrations this may confuse you a bit depending on the context you're working in. The point is once the migrations are all working your code can go back to referring to Location rather than the fully qualified type name. Remember to update your typealias as you add new Schemas.
You can create reverse migrations, to go from V2 to V1 for example, if you have that need. I just kept deleting my data an importing some old format data when I needed to revert during the process.
I’m hardly an expert here. I had to solve a problem and found the existing documentation lacking. Nothing I read was wrong but as is often the case it didn’t provide the full picture. I tried things until I found what worked. I’m not claiming I sat down and wrote the code you see here. It took iteration. It’s complex stuff but when broken down into its parts it’s not too bad.
I’ve joked along the way about Apple and their documentation and those jabs are warranted. The documentation is all but non-existent. Still SwiftData and these Migrations are a good system. Perhaps it can be improved but it does work and gives you tools to do complex things.
If you’ve read this entire brain dump I applaud your efforts. I don’t expect this explanation of SwiftData Migrations to be directly applicable to your apps data models but hopefully it provides more insight into how Migrations work and provides new ways of thinking when it’s time for you to deal with Migrations.
As always if you have questions or comments reach out on Mastodon
Even better, If you are looking for help with your Apple Development Projects then get in touch. I’m available to hire on a remote freelance basis. If you have a full-time gig doing interesting things for the greater good and don’t make people run through a code puzzle gauntlet in your hiring process I would entertain that as well.