Syncing a local store

I want to make an Offline-first application. I am considering using standard CoreData for that. The collaborative and syncing features are to be handled through Parse.

To keep my CoreData in sync, I expect to

  • need to monitor for changes (LiveQueries)
  • download changed objects at launch (get all objects changed since last sync)
  • upload changes as they happen, retry when entering background/launching the app if it failed

I’ll probably going to use some hashing of the data + recipe.updatedAt to detect conflicts. I’ll adopt a last change wins strategy to start but I’ll investigate a “merge UI”. I don’t expect concurrent edits to happen that often, I only need to handle offline editing/bad network.

My question resides around deletes. I’ll see them with the LiveQuery but only when the app is running. What about deletes that happens when the app isn’t running? I could run a query for all recipes IDs and compare them, delete anything that is missing. I don’t expect users to have more than 5000 records, so in the end this isn’t too expensive but I wonder if there’s a better solution?

I could leave records in place instead of deleting them and mark recipes with a isDelete column but then they’ll be forever there.

Suggestions?

Your ideas sound like good ones to me. Its similar to how my middleware, ParseCareKit

works which keeps the local core data store in sync using vector clocks with Parse in the sample app I posted on the other thread. It’s basically a synchronized distributed database. To get some ideas on how to setup your core data store for something like this, I recommend looking at Apples CareKit (specifically the OCKStore):

2 Likes

Great, I’ll have a look at the source :blush:

For those looking for local data storage in ParseSwift, I recommend coming up with approaches like @pierlux above. If anyone would like to develop a protocol based modular solution for ParseSwift. The links I provided above may provide insight on a approach. Note that this is different from the other SDKs on purpose to provide more flexibility to the developer. You can see a discussion about the limitations of the current local storage approach in some of the other SDKs here and why we currently believe a protocol approach would be better. This would enable “adapter” (if you are familiar with developing for the Parse Server) style local storage for developers to chose. Third parties would then be able to implement their own local storage adapters.

If anyone would like to tackle this problem, have a look here:

and feel free to submit a Issue/PR to start the discussion.

1 Like

There’s even a Swift DotVersionVector that could be useful.

I’ve been looking into it a lot and while the basic cases work, now that I’m approaching real use cases, I’m getting duplicated data.

My setup:
I create a Library (NSManagedObject).
I save the NSManagedObjectContext.
That triggers a NSManagedObjectContextWillSave where I push new objects to ParseServer.
So far so good.

My problem:
I have a LiveQuery that listens for updates to Library. That triggers before my code has time to save the objectId of my Library to CoreData. Therefore it inserts a new (duplicated) object.

Is there a pattern to follow here that I am not seeing in ParseCareKit?

I think I found the difference: ParseCareKit uses client generated UUIDs to differentiate items. I am using the server generated objectId which is not available right away. I think adding that will solve my problem. Edit: using custom object ids definitely helped here!

@cbaker6 I’m getting duplicateValue errors when I try to update an object with custom ids enabled. Is there some special rules I must follow? For example, I learned after quite some debugging that createdAt should not be populated for the first save().

1 Like

createdAt and updatedAt are generated/updated by the server. Other than that I don’t believe there’s anything else that’s needed.

I don’t see the problem you mentioned. I added a playground example which seems to work fine. See my comment here:

As I found out, if the custom objectID is used, then createdAt has to be nil on first save and then on updates it has to be set. Strange that any date being set in createdAt works and is returned in the response also, but is not changed in Parse Server database.

CreatedAt and updatedAt are Parse business logic, you should never try to set those from the client as you have no control over them. If the server returns those fields to you then you should save them locally as createdAt is set when an object is saved to the server and updatedAt when an object is updated. This aforementioned comments should be followed rather customObjectId is enabled or not. If you want to control fields similar to createdAt and updatedAt for your local storage you will need to create separate fields. It sounds like you’ve been trying to change these fields from the client, which is not allowed by Parse in general.

I am not trying to modify it. The issue is that I am not able to update object if the init on struct is only with objectId + custom fields. That returns "trying to add duplicate… as you mentioned in your comment.

So I either have to store createdAt locally also or as a hack to set any date to createdAt while updating - what seems to do the trick, but is of coarse not logical, as you said.

@pierlux, have you implemented that library? What is your experience and would you have any examples on read/writes?

Thank you!

Storing the createdAt date locally or a boolean that’s true if the object has been saved seems reasonable IMO as ideally, your local storage should be in-sync with the server. I believe the way the server and client are currently interacting when customObjectId is enabled logically makes sense. If a client doesn’t believe an object has been saved to the server, it sends it with createdAt == nil (because it believes it was never created on the server). The server will save/create it if the object truly hasn’t been created before (a new objectId), it throws an error otherwise saying the object (objectId) already exists. If the client believes the object has been saved before then it should have a createdAt != nil (usually the original from the server, but if you send the wrong one, it doesn’t matter because the server isn’t going to allow you to update this value anyway) . I’m not sure how you would make this better as it makes sense. The client should ensure their local storage behaves appropriately. If you believe you have an improvement on this, I suggest opening an issue/discussion on the parse-server repo.

Note that objectId was originally designed to be Parse business logic as well. Since the server is now letting you control this business logic, you will need to manage some additional overhead that the server use to take care of for you.

You can always not choose to use a custom objectId and instead store your uuid of your local object in a separate field not called objectId and then you don’t have to manage this overhead. You should decide on what tradeoffs benefit your design the best.

1 Like

I have not implemented it. It’s way too complex for my needs.

May I ask what strategy you picked? Relying on last updatedAt field seems to be problematic when let’s say two persons are updating the same objects, but they change a different field. For example:

  1. User A modifies fieldA = 20 and is in poor connectivity area. He keeps the fieldB = 50 untouched.
  2. Meanwhile a user B sets fieldB = 55, before the User A gets connected
  3. User A then gets signal and rewrites the fields back or if strictly checked by passing a “saving date” inside context the update could get rejected in beforeSave with message that newer object is available. But in that case the update on fieldA is getting lost, as the most recent state wins and that state does not have fieldA updated

I was thinking about having separate updatedAt for each field in nested object as mentioned here.

But I am not sure if that is a reasonable way and would make rather more sense to go for VectorVersions (seems difficult when I have fields not as Strings, just numbers for later ElasticSearch indexing)…