Suggestions for things to address in documentation

[dave256]

I realize the documentation is in the very early stages but here are some questions I have and think may be worth addressing as others may have the same/similar questions:

CloudKit.md

• Designing your schema with synchronization in mind

  • indicate uuid id should not have a default (I believe Brandon indicate on Slack it should not have a default value)
  • perhaps also link to the additional restrictions for Sharing a root record here
  • this sentence: "Note that the id column may never be used in your application code, but it is necessary to facilitate synchronizing to CloudKit." can be interpreted multiple ways. I'm fairly certain by "may never be used" you mean "might not be needed" but it could also be interpreted as you shouldn't use it for anything in your app.

• Record Conflicts

  • I believe this is now out of date as you now do a per column time stamp vs. the entire record

• I look forward to reading the other sections - especially how to write unit tests for this

• I assume the updating triggers comments will be expanded

CloudKitSharing.md

• the milk reminder and weekend tag makes sense
  • and it seems CloudKit only allows a single parent for a child record but then how is it ok to have join tables (multiple foreign keys) if you are just syncing within your own iCloud account (not sharing with another user). I understand that if you delete the weekend tag, you'd want it deleted on both of your devices but what I'm not understanding is that this also seems to violated the single parent for a child record. If that can be worked around (it isn't clear how you are working around that for non-shared records), how is it different from the situation with a shared record with multiple parents where you would want it deleted on both devices?

Other questions

• is it possible to turn on syncing after data has already been added to the SQLite database? if so what precautions or issues do you need to handle? possible scenarios for this:

  • existing app that syncing is being added to (I think there is some mention of this in the docs already)
  • feature is behind an in-app purchase so user buys this feature later

• is it possible to turn off syncing after data has already been added? if so, is the data in CloudKit deleted? possible scenarios for this:

  • user's iCloud storage is full or they just want to free up space
  • free trial of syncing feature and user decides not to purchase
  • can syncing then be turned back on later if user decides to buy the feature or buys more iCloud storage from Apple (i.e., can it be turned on/off multiple times and again, what happens to the data in CloudKit)

• if the data in CloudKit seems corrupted (or syncing doesn't seem to be working) is there a way (an API) so the the app could have a button the user could tap to delete the data in CloudKit and then start over with the existing data in the SQLite database?

• a feature I'm thinking about building into my app is the ability to backup/export part of the database and then later allow it to be imported. For example before Sharing a root record, they back it up (perhaps some sort of SQLite dump file they could email to themselves, Save to Files app, etc.) that way if the person they shared it with corrupts the shared record (deletes data, etc.) they could restore from their backup. I'm assuming the way to restore it would be to delete the shared root record and then import the data back from the SQLite dump file. I'm guessing that wouldn't cause any issues with syncing, but didn't know if "playing back" the dump file (I need to look at what options there are for doing this) would cause problems with the sync engine. this feature could also be useful for exporting/archiving old data that isn't likely needed again but then later they do want to restore the data.

• what happens if the app allows the user to copy/export the entire SQLite file (again email, save to Files), makes changes in the app that updates the SQLite file, and then later wants to restore that SQLite file back to before those changes via the export SQLite file. are there any sync related issues with that?

• could a similar issue happen if the user has backed up their iPhone/iPad to iCloud, makes changes to the SQLite file via app interactions, then loses/breaks their phone before another iCloud backup is done, gets a new phone and restores from the iCloud backup? would the newer data stored in CloudKit sync properly over older SQLite file that was in the iCloud backup or would CloudKit think that data had been deleted since its now not in the SQLite file that was restored and delete it from other devices (such as an iPad)?

[mbrandonw]

Hi @dave256, thanks for this detailed list of things that need to be fixed/thought about more deeply. I can respond to a few things here, and then for a bunch of this I will create tasks for us in our internal tracker.

indicate uuid id should not have a default (I believe Brandon indicate on Slack it should not have a default value)

We are still refining this, but right now the rule is basically: for a new table, you do not need to provide defaults for anything if you don't want. For alterations to tables, you do need to provide defaults, or make the column nullable. This is just forced on us since adding a column will leave many rows with no data for that column, and something has to be done about that.

this sentence: "Note that the id column may never be used in your application code, but it is necessary to facilitate synchronizing to CloudKit." can be interpreted multiple ways. I'm fairly certain by "may never be used" you mean "might not be needed" but it could also be interpreted as you shouldn't use it for anything in your app.

Fair enough, can clarify!

and it seems CloudKit only allows a single parent for a child record but then how is it ok to have join tables (multiple foreign keys) if you are just syncing within your own iCloud account (not sharing with another user). I understand that if you delete the weekend tag, you'd want it deleted on both of your devices but what I'm not understanding is that this also seems to violated the single parent for a child record. If that can be worked around (it isn't clear how you are working around that for non-shared records), how is it different from the situation with a shared record with multiple parents where you would want it deleted on both devices?

Associations are shared in CloudKit via the parent property on CKRecord. If recordA is shared, and there's a recordB with recordB.parent == recordA, then recordB will be shared too. And since each CKRecord has only one parent (as in, there is literally only one property on CKRecord for this), it forces us to have the restriction of shared associations having exactly one foreign key (except for the root, which should have no FK's).

For plain synchronization (no sharing) the parent field doesn't seem to be useful and is not really needed, and that is why it does not inhibit plain, non-sharing sync. You are allowed to synchronize records that have any number of foreign keys amongst your devices, and hence synchronizing many-to-many relationships works just fine (yet they cannot be shared).

is it possible to turn on syncing after data has already been added to the SQLite database? if so what precautions or issues do you need to handle? possible scenarios for this:

• existing app that syncing is being added to (I think there is some mention of this in the docs already)

Yes, if an app is already deployed to the App Store with no sync, and an update is released that adds sync, we will upload all records currently on the device to CloudKit

• feature is behind an in-app purchase so user buys this feature later

This should also be possible. As soon as a SyncEngine is created it will start the upload and synchronization process.

• is it possible to turn off syncing after data has already been added? if so, is the data in CloudKit deleted? possible scenarios for this:

  • user's iCloud storage is full or they just want to free up space
  • free trial of syncing feature and user decides not to purchase
  • can syncing then be turned back on later if user decides to buy the feature or buys more iCloud storage from Apple (i.e., can it be turned on/off multiple times and again, what happens to the data in CloudKit)

We haven't thought about this situation in particular much, but it should be possible. We have an internal tearDownSyncEngine that we use when we detect the user logs out of their iCloud account. It does a bit more than what you would need for "pausing" the sync engine, so we could maybe have a start() and stop() on SyncEngine specifically for this.

• if the data in CloudKit seems corrupted (or syncing doesn't seem to be working) is there a way (an API) so the the app could have a button the user could tap to delete the data in CloudKit and then start over with the existing data in the SQLite database?

Nothing like that right now. Is this only in the context of debugging? Or is there a use case for this in production?

• a feature I'm thinking about building into my app is the ability to backup/export part of the database and then later allow it to be imported. For example before Sharing a root record, they back it up (perhaps some sort of SQLite dump file they could email to themselves, Save to Files app, etc.) that way if the person they shared it with corrupts the shared record (deletes data, etc.) they could restore from their backup. I'm assuming the way to restore it would be to delete the shared root record and then import the data back from the SQLite dump file. I'm guessing that wouldn't cause any issues with syncing, but didn't know if "playing back" the dump file (I need to look at what options there are for doing this) would cause problems with the sync engine. this feature could also be useful for exporting/archiving old data that isn't likely needed again but then later they do want to restore the data.

As long as when you "play back" the dump file by using the database.write that you provide to the SyncEngine, then those changes should be enqueued to the engine and sent to CloudKit automatically.

• what happens if the app allows the user to copy/export the entire SQLite file (again email, save to Files), makes changes in the app that updates the SQLite file, and then later wants to restore that SQLite file back to before those changes via the export SQLite file. are there any sync related issues with that?

Similar to above, as long as you are using the database.write that you provide to SyncEngine to write the imported data it should work. However, if you were to simply overwrite the SQL file on the file system, then the sync engine will not be able to detect any of those changes.

• could a similar issue happen if the user has backed up their iPhone/iPad to iCloud, makes changes to the SQLite file via app interactions, then loses/breaks their phone before another iCloud backup is done, gets a new phone and restores from the iCloud backup? would the newer data stored in CloudKit sync properly over older SQLite file that was in the iCloud backup or would CloudKit think that data had been deleted since its now not in the SQLite file that was restored and delete it from other devices (such as an iPad)?

I believe this would be handled gracefully.

The scenario is:

• Device backs up to iCloud at time=0.
• Device writes new data to database which is sync'd to CloudKit at time=1.
• Device is wiped and iOS re-installed at time=2 before a new backup can be made, which restores the version of the database at time=0.

In this situation, the database restored at time=2 (which is the version of the database from time=0 due to the backup) would have its CKSyncEngine.State also archived at time=0. That means when the sync engine starts up with this old state it knows to fetch everything from CloudKit that changed since the CKSyncEngine.State was archived, including all the changes that were made after the backup.

[dave256]

Brandon, thanks for the detailed response. I think I understand everything in your response although have one follow-up question in my next comment. To answer your question about whether "reseting the data in CloudKit" is for debugging or production, my thought was in production. To the user CloudKit is effectively an opaque box and if something seems to go wrong with it, they can't ask Apple (or me) to fix it so having some way of "starting over" seems like a good idea. I use iCloud for my email and at one point one mailbox folder got corrupted. I contacted Apple and there was nothing they could do so I had to abandon that folder and start a new folder losing the data in the old folder. I was hoping to have some way to allow them to delete everything in CloudKit related to the app (both if they want to clean up storage or something seems to have gone wrong and they want to try starting fresh with the data on device).

[dave256]

One clarification. So for regular syncing the parent record in CloudKit isn't used (i.e., Apple doesn't care what it is - it's only when Sharing a record that Apple looks at the parent record) - is that correct? In my case I think I can get away with only one parent record even though I effectively have two (just won't mark one as a foreign key) because:

1. both tables will be shared because there is a path to the shared root record for both of them
2. my application's code sets those "fake foreign key" fields (i.e., the user can't edit them directly) when the row is inserted so they will be set correctly
3. when a row of a table is deleted, the application code will manually do the cascade delete for the one that is not marked as a foreign key. I am wondering if it makes sense to set up temporary triggers to do the extra cascade delete for the one that is not marked as a foreign key vs. the app logic doing it?

Thus as long as the records end up on the device of both the person who shared and the person with whom it was shared (which they should be given there is a path to the shared root record for both of them), I don't see why everything wouldn't be fine. Or am I missing something?

[mbrandonw]

To answer your question about whether "reseting the data in CloudKit" is for debugging or production, my thought was in production.

Ok, that's good feedback to have. I will add a task for us to discuss it more.

it's only when Sharing a record that Apple looks at the parent record) - is that correct?

That is correct.

3. I am wondering if it makes sense to set up temporary triggers to do the extra cascade delete for the one that is not marked as a foreign key vs. the app logic doing it?

Yeah that should be possible and should work fine.

Thus as long as the records end up on the device of both the person who shared and the person with whom it was shared (which they should be given there is a path to the shared root record for both of them), I don't see why everything wouldn't be fine. Or am I missing something?

It does worry me a little to have a "fake" foreign key, i.e. a column that points to another table, but not explicitly designate it as a foreign key in the schema. We use this FK information quite a bit and so I'm not sure what could go wrong if it is omitted. For example, it is used to topologically sort the tables so that we can make sure to insert parent records before children, as well as delete child records before parents.

But, it may just all work out. I would need to think about it more.

And just to clarify, this is the situation you are in:

graph BT
  B --> A
  C --> A
  D --> B
  D -.-> C

Loading

? Where D has a "fake" foreign key pointing to C?

[mbrandonw]

We discussed this at length today, and as of right now it seems the fake foreign keys could potentially work. I think you would definitely want to create your own trigger to emulate "ON DELETE CASCADE", and it might even be a good idea to install a trigger that prevents the foreign key from ever being changed after insertion by raising an error in an "AFTER UPDATE parentTableID" trigger.

[dave256]

@mbrandonw yes, that is the situation. In my example, A is a Course that has AttendanceDates (B) and Students ( C ) and then D is the join table that records the status (along with some other information such as time stamp, note, etc.) for a given student for a given date (with an explicit foreign key to B and a column with a corresponding key of C. I already added the trigger the other day and verified that seems to work (i.e., I deleted a Student and the trigger deleted all their dates and it seemed to sync properly). I haven't tried it yet with foreign keys enabled using the latest updates you made to Sharing-GRDB-iCloud but I will either this weekend or next week. If this works, I'm fine with it but I'm wondering if there's anything the library could do internally to just set the CloudKit parent record for just one of the foreign keys? I'll also look into adding the other after update trigger to disallow setting the fake foreign key.

And regarding the other question about turning on/off syncing, I can definitely see some people wanting to do that in production. If they're just using the app without any sharing and only have one device (say an iPhone), they'd probably prefer not to "waste" the iCloud storage space. But then maybe they get an iPad later and decide to sync with it, then they'd want to turn it on. Or maybe later they decide to share it with someone, then they might want to turn it on. And then if they later stop sharing, they might want to turn it off to reclaim the iCloud storage space.

[stephencelis]

@dave256 We've added preliminary support for stopping and starting a sync engine here: https://github.com/pointfreeco/sharing-grdb/pull/142

1. If you want to initialize SyncEngine in a stopped state, you can pass startImmediately: false to the initializer.
2. You can invoke try await syncEngine.start() to start the engine.
3. You can invoke syncEngine.isRunning to check if it's running.
4. You can invoke syncEngine.stop() to stop the engine.

There are likely more things to test, but if you'd like to point your app to that branch you can hopefully get your feet wet!

[dave256]

@stephencelis thanks! I'm back teaching full-time but I'll try to find some time this weekend to check out the branch and add a little code to use it.