Simultaneous usage on two or more devices: thoughts

[TheLastProject]

Issue by shapirus
Monday Mar 05, 2018 at 08:37 GMT
Originally opened as brarcher/loyalty-card-locker#220

---

Let's assume I want to keep a cards database shared with my family (say 2 devices for now). Right now, there is one standard possibility:

1. Edit cards on device 1
2. Export on device 1
3. Sync with your favorite file sync software
4. Import on device 2

And then in the reverse order when the database is changed on device 2.

This requires manual actions and in addition the export path is not configurable yet (as described in brarcher/loyalty-card-locker#168) which doesn't allow to use synchronization software which works with folders (you don't want to share the sdcard's entire root where the export .csv file is saved). The latter can be worked around using the "send to" button, though.

A better option would be to sync the actual database. But there are a few issues preventing this:

1. The database path cannot be configured. Current path is /data/data/protect.card_locker which means that whatever software is trying to access that location to sync it to other devices, must run with root privileges. Making this path configurable or creating a copy of the database (updated on every change) in a user-accessible location would solve this.

2. The application must implement the reload-on-database-change function in order to load the changes made on the other device and synced back. Or, if we sync the database's copy in /sdcard, load that one when it's changed.

3. Conflict resolution (when two devices edit the databases while no data sync is happening between them). It is the most difficult part implementing which is probably not worth the trouble. Or maybe not? We can probably simply merge cards that are in /data with the updated version from /sdcard thus keeping added cards from both devices. Deletions and edits have to be additionally handled, though. "Newest change wins" principle? That will need a new "datetime changed" field for each record in the database if it's not there yet.

Then again, cloud sync for those who don't care to set up their own file sync solution? That might be planned for the future. That cloud subscription might (and probably should) be a paid one, earning something in addition to the donations :)

[TheLastProject]

Comment by brarcher
Tuesday Mar 06, 2018 at 03:55 GMT

---

I'll agree that the app currently does not support well the multi-user case outside of the existing backup/restore feature. I've received the request in the past to support some sort of network backup/restore feature (see brarcher/loyalty-card-locker#159). Unfortunately, I turned down the request. One of the design goals of this application is to have minimal permissions, not take user data, and to be free (as in beer and speech). Supporting a network backup option, or charging for such a feature, is a move away from the design goals.

Allowing the export path be configurable would help the multi-user case if one were backing up to a cloud storage backed folder. Besides that, I'm not sure I'm up for implementing any of the other changes you proposed, such as having a database outside of app private storage.

Is there a middle-ground option possible? One of your issues is that the sharing of data with others involves a manual step. Is there a possibility of having a companion app which does access the network and check for card updates, then on an update pushes the change to the loyalty card app? Could adding a card result in auto-writing a backup file to somewhere on storage? If someone were interested in investigating such a path and creating a companion app (if that makes sense) and add modifications to this application to support it I'd be willing to look at the proposed changes.

[TheLastProject]

Comment by shapirus
Tuesday Mar 06, 2018 at 08:03 GMT

---

Let's see what can be done with the least effort and in a way that the modifications do not look out of the place, but be natural and welcomed by the users (and the developer, too!).

On GNU/Linux (and what would be generally called a Unix-way) I would set up the following chain to auto-export the database:

1. An inotify event watcher tool such as incrond to watch for file changes and run commands when they happen.
2. Rsync or syncthing or whatever seems appropriate to handle the changed data.

I am pretty sure it is possible to set up pretty much the same stack on Android, only it will not be at all friendly to the average android user, including the need to have root permissions. The trouble here is with the file change watching task.
Moving this to loyalty card locker will simplify things and will not be alien to the app at all.
What I mean is adding a new feature looking like this:

• In the settings menu: "Enable automatic database backup (or 'export'?)", and when enabled, show a "Backup folder location" suboption;
• In the code: on every database update, export the DB to the location configured above, if enabled.

It doesn't look neither very difficult to implement, nor too heavy on system resources. It will solve brarcher/loyalty-card-locker#168 at the same time. It will then probably make sense to remove the manual "export" function and replace it with a "share/send to..." button.

Users then can use the data inside the export location in any way they like, feed it to their favorite cloud backup software, syncthing and whatnot.

So far so good, to me it looks like a feature that will fit very naturally in the app.

Then comes the import part, which is a different matter in terms of making it nice and clean, especially if we want it to be "auto-". It does not have to be implemented at the same time as the export functionality that I described above, but they can nicely work together when both are ready.

As far as conflict resolution, it can be solved by asking "overwrite or merge?" when the user tries to import a CSV file. Merging algorithm should handle edits and removals properly and generally speaking is not trivial to implement.

As far as the "auto-" part, well, there could be a setting like "watch this folder for data to import" which, when enabled, would offer to import the DB (merging or overwriting) when an updated CSV file is detected there. It will get the job done, but I cannot honestly say that it seems to me very nice and clean the way it is described. Looks more like a workaround or hack that we have to make for not having a better solution.

[TheLastProject]

Comment by brarcher
Monday Mar 12, 2018 at 16:06 GMT

---

The auto-export option does not sound bad. If someone were interested in contributing this feature I could help review the pull request.

As for the import, the application is not running all the time. Probably most of the time it is not running. Having the application periodically wake up and check for updates could be done, but it would involve a background service that would periodically check. Say that it checks once an hour. There will still be the case where one opens the app between updates and there was a change in the data. Should it always attempt to check on load? If so, should automatic updates happen anyway? Is the delay on checking on load acceptable?

As you mentioned it would also involve a different import strategy. Right now the imports are supported if the app is in a clean state (no cards). If there are cards it may not import cleanly depending on conflicts. As with the export, I'll probably not work on it but if someone were interested in implementing it and submitting a pull request I could review it.

[TheLastProject]

So, I've had random thoughts about this while not being able to fall asleep.

The hardest part is dealing with conflicts. That is why I am thinking something like the following:

1. A sync is a directory on a fileshare (CalDav, Dropbox, whatever)
2. Every time a change is made, a new file is created on the fileshare with the timestamp as name, containing instructions like "card added with ID 6", "card 6 field barcode type set to Aztec", etc.
3. To prevent conflicts when both devices add a new card while they are offline, the card IDs in the database will no longer be simply numeric but actually be an UNIX timestamp
4. Each client internally stores every single timestamp synced so that it can tell what is new. If a new timestamp appears in between synced timestamps, syncing is redone from the old timestamp so changes are always applied in the right order

This strategy will provide us with the following:

• Newest change wins: if you edit the note on two different devices before you have internet, the most recent change will end up on both devices
• Reasonable simplicity codewise

Performance could become an issue though... Maybe have a server do all this complex work? Would mean that we lose the ability to just sync on any user cloud storage though...

[Atn-D]

If it can sync with a NextCloud app, it would be perfect!

[Korb]

It seems to me that when developing a mechanism for sharing a set of cards with close people, it is important to take into account that in the periods between synchronizations (no matter how they are implemented), in general, each user can add or remove an arbitrary number of cards. And these collisions will have to be resolved somehow. It's possible that #1394 addresses this issue, but I'm not sure.

Upd. In the absence of a cloud infrastructure for synchronization from the application developer, it would be possible, as an option, to use synchronization via Google Drive. This works in the case of Notepad++, WhatsApp and some Mozilla Firefox browser add-ons (which for some reason cannot sync through the Mozilla cloud infrastructure).