core data

How to Set Up Core Data and CloudKit When You Haven’t the Faintest Clue What You’re Doing

Note: This was posted before WWDC 2021, so if major changes were made to Core Data + CloudKit, they aren’t reflected here. This right here is just pure dumpster fire all the way down. Also if you’re an Apple engineer…I’m sorry.

When Apple introduced changes to Core Data + CloudKit integration in 2019, they sold developers on a dead-simple API: add iCloud sync to your Core Data app with “as little as one line of code.” That one line, of course, is simply changing NSPersistentContainer to NSPersistentCloudKitContainer and enabling a few capabilities in the project settings. Boom, done! And in fact, Apple’s “Core Data –> Host in CloudKit” SwiftUI project template does those things for you, so you’re good to go, right?

Turns out, if you want to sync Core Data-backed data between devices and have those changes reflected in your UI in a timely manner, you have some more work to do. To figure out what that work is, you can’t look at Apple’s Core Data templates. You have to look at their sample code.

My SwiftUI app was created before Apple even added a SwiftUI + Core Data project template, so I created a class called “CoreDataStack” that has a shared instance. If you use the template, that becomes a struct called “PersistenceController.” I’m sure the struct is SwiftUI-ier, but the class from Apple’s sample code (which does not use SwiftUI) makes more sense to my brain, so I went with that.

Step 1: Make your container lazy and set some important options

In Apple’s sample code, you’ll notice that within the persistent container’s lazy initialization, two options are set on the container’s description. Include these.

guard let description = container.persistentStoreDescriptions.first else {
        fatalError("###\(#function): Failed to retrieve a persistent store description.")
    }
    description.setOption(true as NSNumber, forKey: NSPersistentHistoryTrackingKey)
    description.setOption(true as NSNumber, forKey: NSPersistentStoreRemoteChangeNotificationPostOptionKey)

If you don’t set the first option, you’ll regret it. Look, I don’t even really understand what it does, I just know that somewhere down the line, you’ll find some dumb way to break sync and then when you finally get it working again, you’ll find that only managed objects created after this key was set will sync properly. Every object created before the NSPersistentHistoryTrackingKey was set will stubbornly refused to sync unless you modify it and re-save it, which is a giant pain in the derrière. I mean, at least that’s what my…uh…friend told me.

The second option is the first step toward receiving notifications when magic cloud stuff happens. You’ll subscribe to that NSPersistentStoreRemoteChangeNotification later, but for now, just make sure that option is set.

Step 2: Stir in some of this stuff that I have a super weak grasp of

After your container loads its persistent stores, but before you return the container itself, these lines are also important:

container.viewContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
    container.viewContext.transactionAuthor = appTransactionAuthorName
       container.viewContext.automaticallyMergesChangesFromParent = true
    do {
        try container.viewContext.setQueryGenerationFrom(.current)
    } catch {
        assertionFailure("###\(#function): Failed to pin viewContext to the current generation:\(error)")
    }

There are several merge policies, and you can read about them in the docs.

Again, I barely understand this stuff. For my purposes, I set “appTransactionAuthorName” to the name of my app’s container, which was simply “YarnBuddy.” From what I kinda understand, setting the transaction author here allows me to later filter for changes that weren’t created by my app on this particular device and act on them.

Now, I’ve always had “automaticallyMergesChangesFromParent” set to true, but what I didn’t realize is that it doesn’t just refresh your view hierarchy immediately when a change occurs. Maybe it should, but for me, it doesn’t. That’s where the remote change notification comes in.

Step 3: Dip your toes into Combine for a hot second and subscribe to notifications

I put this code right before “return container.”

NotificationCenter.default
      .publisher(for: .NSPersistentStoreRemoteChange)
      .sink {
        self.processRemoteStoreChange($0)
      }
      .store(in: &subscriptions)

And somewhere within the class I have declared this variable:

private var subscriptions: Set<AnyCancellable> = []

Make sure you import Combine at the top. I know extremely little about Combine at this point. It’s number one on my list of things to learn, and I plan to start with John Sundell’s “Discover Combine” materials.

We’ll get into what my “processRemoteStoreChange” function does in a minute.

Step 4: Just copy over these blessed code snippets from the sample code

Copy the following from CoreDataStack.swift in Apple’s sample code:

  • the initializer
  • lastHistoryToken variable
  • tokenFile variable
  • historyQueue variable

Also copy over the NSPersistentContainer extension in “CoreData+Convenience.swift.”

Also, my “processRemoteStoreChange” function is identical to the sample code’s “storeRemoteChange” function.

Step 5: Merge new changes into the context

I modified Apple’s “processPersistentHistory” function to look like this:

func processPersistentHistory() {
    let backgroundContext = persistentContainer.newBackgroundContext()
    backgroundContext.performAndWait {

        // Fetch history received from outside the app since the last token
        let historyFetchRequest = NSPersistentHistoryTransaction.fetchRequest!
        historyFetchRequest.predicate = NSPredicate(format: "author != %@", appTransactionAuthorName)
        let request = NSPersistentHistoryChangeRequest.fetchHistory(after: lastHistoryToken)
        request.fetchRequest = historyFetchRequest

        let result = (try? backgroundContext.execute(request)) as? NSPersistentHistoryResult
        guard let transactions = result?.result as? [NSPersistentHistoryTransaction],
              !transactions.isEmpty
            else { return }

        print("transactions = \(transactions)")
        self.mergeChanges(from: transactions)

        // Update the history token using the last transaction.
        lastHistoryToken = transactions.last!.token
    }
}

The “mergeChanges” function looks like this:

private func mergeChanges(from transactions: [NSPersistentHistoryTransaction]) {
        context.perform {
            transactions.forEach { [weak self] transaction in
                guard let self = self, let userInfo = transaction.objectIDNotification().userInfo else { return }
                NSManagedObjectContext.mergeChanges(fromRemoteContextSave: userInfo, into: [self.context])
        }
    }
}

Most of that code was pulled from Stack Overflow. Apple’s code has a bunch of deduplication logic in it that frankly, I’m not emotionally ready to process, so I skipped it.

I’ve seen a few folks say that merging changes like this shouldn’t be necessary. However—and maybe it’s some sort of weird placebo effect—it seemed like changes from my watch synced much more quickly to my phone. Not instantaneous, but a handful of seconds instead of requiring me to sometimes force quit and restart the app (or switch tabs or something) to see changes.

Step 6: Never forget to deploy your updated schema from the CloudKit Dashboard

Honestly, it’s not that I forgot to do this, it’s that I failed to make sure it actually happened. CloudKit threw some weird error at me and told me my development and production schemas were the same, when they were in fact extremely different. I never double-checked, and chaos ensued! Don’t be like me: make sure your schema is deployed.

After launch, remember that you still have to do this every time you change your Core Data model, before you release your update to testers or the App Store. If your production CloudKit schema doesn’t properly correspond to your production Core Data model, syncing is going to break in all kinds of terrifying ways.

Conclusion

I know I probably could have saved myself a lot of frustration if I’d forked over some money for a Ray Wenderlich membership or some other paid tutorials/books related to Core Data. I’m also guessing there’s a much easier way to set everything up so that cloud changes are reflected near-instantaneously. But y’all, I’ve combed the free internet for weeks and this is the best I could come up with. Maybe it’ll help you too.

Data Persistence Dilemma

Sometimes, in order to solve a problem, I have to think through it out loud (or in this case, in writing).

Here’s the sitch: I’m making an app for knitters and crocheters. They need to be able to manage projects (i.e. “Baby blanket,” “Socks for mom,” etc.). In addition to a bunch of metadata about each project, users should be able to add photos and designate one or more images or PDF files as the project’s pattern. A PDF or image of the pattern isn’t required, but including one will allow users to enter a split view where they can view the pattern and operate a row counter at the same time.

A project shouldn’t necessarily “own” its pattern. In other words, a pattern can have multiple projects associated with it (say you want to make the same baby blanket for multiple babies), so as to avoid the needless duplication of the pattern file. A pattern can exist without a project and a project can exist without a pattern, but when linked, the project becomes the child of the pattern.

My user base includes people who may not always have an internet connection. Therefore, all data needs to be stored locally. However, those who do have an internet connection are going to want iCloud sync between devices.

I like Core Data. If I were to set this up in Core Data, without any consideration of iCloud syncing, I’d create Project and Pattern entities, store images and PDFs in the file system, and call it a day.

iCloud syncing is where things get murky for me. Core Data + iCloud is deprecated, and I don’t want to use it. Not only that, I don’t know what to do with the PDFs and images. Storing them as BLOBs in Core Data seems like a bad idea. I understand how to save them to the file system but don’t understand how to sync them via iCloud and also have a reference to them in Core Data. Do I use iCloud Document Storage for them? Do I zip them up somehow (NSFileWrapper??) and use UIDocument? How do I store a reference to them in Core Data (just the file name of the UIDocument, since the file URL is variable?). If users will be adding photos and PDFs at different times, do I use one UIDocument subclass for photos and one for PDFs or do I use a single document and update it with the added information? You can tell I obviously have no idea how this works, and a multitude of Google searches has yet to clear it up for me.

As for the rest of the information in Core Data, I’m thinking of trying to sync it  with CloudKit using something like the open source version of Ensembles or Seam3.

I guess I’m not sure if I’m on the right track and would welcome any advice/feedback. I’d really like to stay away from non-Apple services (like Realm) for the time being. Comments are open!

Waiting for Review Day 2: WatchKit & Core Data

Note: This is the second entry in a series of posts about writing my first iOS app. The app is currently in review, and until it is rejected or approved, I plan to write something every day about what I’ve learned.

In my first post, I talked about how I decided to use Core Data to store the verses in my Bible verse app and how I went about seeding that database. The next thing I had to figure out was how in the world my Watch app and iOS app could access the same Core Data model. I learned from a Make & Build tutorial that I needed to create both an App Group and a custom framework that could be shared between the two apps.

After creating a framework, I added two files to it: my Verse class, and a singleton DataManager class that handles all of the Core Data methods. Singletons are still a little confusing to me, so I pretty much just copied the code from this excellent tutorial video. At first I couldn’t figure out why my apps couldn’t seem to “see” anything in the framework, even after linking to it. Then I realized that I forgot to mark everything in the framework as “public.” /facepalm

My DataManager class has three simple methods: a method to fetch all of the verses in the store and place them in an array, a method to fetch only verses marked as a favorite, and a method to select a random verse from the array. Using one of the methods in my WatchKit app was as easy as saying:

var verse:Verse = DataManager.sharedInstance.getRandomVerse(verses)

Tomorrow, I’ll either talk about my app’s visual style (colors, icon, name, etc.), or about Glances and using Handoff.

Waiting for Review Day 1: Creating a pre-populated Core Data database

Guess what? I finally did it. I submitted my first app to the iOS App Store around 2am this morning. It feels so unbelievably great to have actually finished something (well, as much as any 1.0 can be “finished”). As it’s now waiting to be reviewed by someone from the App Store team, I thought I’d write one post every day during the waiting period, describing the app and what I learned from making it. A quick note before I go any further though: it is a Bible verse app, so if you don’t want to hear about that, this would be a good time to stop reading.

The Idea

My first idea for an app was a sort of non-traditional To-Do list app. It seemed like a neat concept and well-suited for a beginner, but after messing around with it for a few months I realized it wasn’t quite coming out the way I’d hoped. I felt slightly discouraged—that is, until I got my Apple Watch on April 24. Because I am a Christian and my faith is very important to me, one of the first things I did upon receiving my watch was look for an app that would allow me to view Bible verses on my wrist.

As of this writing, there are about a dozen watch apps that show up when you search for “bible verses” in the App Store. They fall into roughly two camps: apps that display daily verses, and apps that allow you to read the entire Bible. I wasn’t really looking for either one of those. What I wanted was an app that would allow me to hit “refresh” and see a new verse whenever I wanted, without having to wait for the next day. At first, I thought I could create the whole thing using just a Glance; however, when I learned that Glances must be static and can’t have any buttons, I set out to build a full-fledged app. According to Xcode, I started the project on April 30.

First Steps: Creating a database

I knew I would need a database of verses to pull from, and that I would need to decide what translation(s) to use. Because I didn’t feel like messing with copyright issues, I decided to build my own database using translations that are in the public domain: the World English Bible and King James Version. I pasted the verses into a Google spreadsheet with columns for the verse text, reference, and translation.

Next I needed to figure out how to get that data into my app. I stumbled upon a Ray Wenderlich tutorial that described how to create a command line utility app for OS X that would basically spit out a pre-filled SQLite database for use with Core Data. I liked the idea of shipping my app with the database already populated, so even though the tutorial was old (iOS 5!), I decided to give it a try. Translating it into Swift was surprisingly easy, but then I hit a problem: the tutorial used a JSON file, and all I had was a spreadsheet.

I feel the need to reiterate that I have no computer science/programming background, and know next to nothing about databases. For all I know, it’s probably ridiculously easy to parse a CSV file. However, my tutorial used JSON, and so I did what any self-respecting woman with over 20 years of Microsoft Office experience would do: I did a mail merge. Yes, I literally mail merged my spreadsheet into a Word document so that it matched proper JSON syntax, and then converted it to plaintext and threw a “.json” at the end. If you’re an experienced programmer and that’s the most ridiculous thing you’ve ever read: you’re welcome. ;-)

Anyway, in case any of you want to use that Ray Wenderlich tutorial with Swift, here is what I did:

  • I pasted the boilerplate Core Data stack into a new Swift class called “DataManager.”
  • After starting a new Xcode project for my iOS app and setting up Core Data, I copied my Core Data class called “Verse” and my .xcdatamodeld file from that project into the command line utility app project.
  • I copied my newly created “Verses.json” file into the utility app project.
  • Note: there are a couple changes you have to make to the Core Data stack code. You can find those changes in the tutorial.

Here is the code for my main.swift:

import Foundation
import CoreData

var dataManager = DataManager()

// Get JSON data
var err:NSError? = nil;
let jsonURL = NSBundle.mainBundle().URLForResource("Verses", withExtension: "json")
let jsonData = NSData(contentsOfURL: jsonURL!)
let verseArray = NSJSONSerialization.JSONObjectWithData(jsonData!, options: NSJSONReadingOptions.AllowFragments, error: &err) as! NSArray

// Seed Core Data
if let managedObjectContext = dataManager.managedObjectContext {
    for item in verseArray {
        let verseText = item.objectForKey("text") as! String
        let verseReference = item.objectForKey("reference") as! String
        let verseTranslation = item.objectForKey("translation") as! String
        
        var verse = NSEntityDescription.insertNewObjectForEntityForName("Verse", inManagedObjectContext: managedObjectContext) as! Verse
        verse.text = verseText
        verse.reference = verseReference
        verse.translation = verseTranslation
        verse.isFavorite = false
        
        var e: NSError?
        if managedObjectContext.save(&e) != true {
            println("insert error: \(e!.localizedDescription)")
        }
    }
}

Tomorrow, I’ll talk about how I set up Core Data so that it could be accessed by my iOS app, Watch extension, and Today View Widget.