The CloudKit nightmare: LiftCoach's diary stopped in June

I wanted to start testing CloudKit sync.

LiftCoach is the app I’m building. It logs workouts, routines, and nutrition on the iPhone, the iPad, and the Apple Watch. It is not on the App Store yet. I run it from TestFlight and I actually use it while I build it: the diary is mine, and so are the bugs.

The iPad showed no nutrition and no workout history. The iPhone had all of it, and that is where I log the day. LiftCoach syncs through iCloud. When one device is missing the diary, the usual guess is that it has not finished downloading. This time it was not downloading. There was nothing to download.

How I got into the iPad #

I plugged it in over USB, unlocked it, and trusted the computer. Reading an app’s files needs Developer Mode, under Settings. The iPad already had it on. On the iPhone I had to turn it on before I could do the same.

From the Mac, devicectl lists the device and copies a file without installing anything over the app:

1xcrun devicectl device copy from \
2  --device <ipad-id> \
3  --domain-type appGroupDataContainer \
4  --domain-identifier group.com.sergiocomeron.LiftApp \
5  --source "Library/Application Support/default.store" \
6  --destination ./default.store

The first place I looked was empty. The app container, under Application Support, had no database. SwiftData, with the default group, does not store it there. It stores it in the app group, group.com.sergiocomeron.LiftApp, at Library/Application Support/default.store. Next to it are default.store-wal and default.store-shm. Copy the big file alone and you leave behind whatever has not been checkpointed yet. All three, or the reading lies.

Then sqlite3 on the copy. No need to open the app or guess from the screen.

The diary tables come from the model: ZFOODLOG, ZWORKOUT, ZWATERLOG, ZCAFFEINELOG, ZRECIPELOG. A COUNT(*) already showed the iPad did not have the diary. It had a handful of rows from spring and early June.

The CloudKit side lives in tables the container creates, not the model:

• ANSCKEVENT is the sync log. Type 0 is setup, type 1 is a download, type 2 is an upload. ZSUCCEEDED, ZERRORDOMAIN, and ZERRORCODE say whether it landed. The dates are Core Data dates, seconds since 1 January 2001. To read them in local time, add 978307200 and pass them through datetime(..., 'unixepoch', 'localtime').
• ANSCKRECORDMETADATA is the queue. ZNEEDSUPLOAD is what still has to go up. ZNEEDSCLOUDDELETE is what still has to be deleted in iCloud.
• ANSCKRECORDZONEMETADATA says whether the zone thinks it is caught up: ZNEEDSIMPORT, and whether a change token is there.

The query I used most was the latest upload:

1SELECT Z_PK, ZSUCCEEDED, ZERRORDOMAIN, ZERRORCODE,
2       datetime(ZENDEDAT + 978307200, 'unixepoch', 'localtime')
3FROM ANSCKEVENT
4WHERE ZCLOUDKITEVENTTYPE = 2
5ORDER BY Z_PK;

That is where the cutoff showed. Uploads that finished cleanly until 18 June, and after that one after another with CKErrorDomain code 2, in one second, with zero records stored.

I could not pull the iPad’s own logs. log collect against the device wants root, and the Mac shell’s log is a different command. I read the other side in the CloudKit console, production environment, filtered to the time of those uploads. The same operation, seen from the server.

Then I copied the store off the iPhone the same way. Same queries, different numbers: the full diary locally, and the same queue stuck.

iCloud had stopped on 18 June #

The phone had the full diary locally. The iCloud copy did not. The last upload that actually landed was on 18 June, at 12:23. After that minute, every attempt lasted about a second, returned an error, and stored nothing.

Everything after that hour had stayed behind: meals, workouts, water, caffeine, routines. Months of use, only on the iPhone. When the iPad looked at iCloud, it received the June slice and considered itself done.

One batch, or none #

LiftCoach stores the diary with SwiftData and syncs it with CloudKit, in the private database. The container does not send rows one by one. It packs them into a batch, and the zone takes the whole batch or it takes nothing. If one record in the batch is invalid, none of them go in.

That is why the failure looked empty. Zero records stored, the diary untouched on the phone, and another identical attempt a moment later. Opening the app did not clear it. The phone sent the same batch and got the same rejection.

In the CloudKit console the status was USER_ERROR and the error was BAD_REQUEST. The record type was missing. In the private database the payload is encrypted, and the log returns the envelope, __pcs_data, not the model name. I knew the server had rejected the request. I did not know which of the thousands of records had broken it.

The schema-deploy documentation says the sentence that matters here: deploying the schema to production copies record types, fields, and indexes, not the records. The path back, from the phone to production, is a different one. And it was closed.

First the log said “delete” #

The first attempt I could read was not a save. The operation was RecordDelete.

The local queue held deletes for rows that no longer existed on the phone. Almost none of them had ever been uploaded. iCloud rejected that delete. Because the batch is all or nothing, not a single meal went up behind it.

Those deletes had never reached iCloud, so the rejection had not deleted anything there. The phone’s diary was still complete. With the app closed, and a separate copy of the store set aside, I cleared the delete marks from the queue. I did not touch the meals or the workouts.

The next time I opened it, the operation was no longer a delete. It was a RecordSave. That was rejected too, in the same second, with the same BAD_REQUEST.

Production did not know the current model #

The model had grown since June. A history of recipes. A meal tagged as before, during, or after a workout. The barcode on a custom food. A set with a completion time, a rest interval, and a duration.

In the development environment, CloudKit creates the type and the field when you save a record that carries them. Production does not. Production changes only when you deploy the schema, and that step does not move records or delete the ones already there.

The first deploy added types that were missing. The iPhone’s save still failed. The sample records had left the optional fields empty, and CloudKit does not add a field the record never sends. The real batch did send them. One unknown field is enough. The whole zone bounces.

I saved another round of sample records, into a temporary store from a development build, not into the diary. This time the workout context, the barcode, and the completed set had values. The deploy dialog was no longer empty. I deployed and opened the iPhone.

In about a minute the upload queue dropped to zero. The meal count and the workout count did not move. What was on the phone was now in iCloud. Nothing that was already there was lost on the way.

The iPad thought June was the whole zone #

The first time the iPad synced, iCloud could only hand it the June slice. A finished workout arrived before its sets. With no sets, the app read it as an abandoned workout and marked it for deletion. Those deletes never left either, because of the same jam. The damage stayed local: the iPad’s copy decided it was caught up. For that copy, the short zone was the entire zone.

Installing an update on top of that copy would not have helped. It would still believe nothing was missing, and it had local changes of its own waiting to upload. I deleted LiftCoach on the iPad only and installed again the TestFlight build the phone was already running. A fresh install does not inherit that copy. It downloaded the whole zone.

Nutrition and history showed up. Saturday’s workout arrived with its sets marked done. I did not uninstall anything on the iPhone. Until the upload finished, that was the only complete copy.

What I check from now on #

A TestFlight build talks to production CloudKit. A development build talks to the development database. Saving a sample record in the second one does not fix the first until you press Deploy Schema Changes. If that dialog comes up empty, it is not doing anything: development and production already match, and the field you need has reached neither.

From now on, when the model grows, the order is this. A sample record with the field filled in, a check that the deploy lists types or fields, the deploy itself, and only then opening the device that holds the diary. A second device that synced while the first one was stuck is not waiting. It is convinced it already has everything.

That was my Sunday. Just so we wouldn’t get bored.