Checking for uniqueness on a 2000 item saveAll function

I have previously asked for Parse.Object.saveAll limits which I am happy to say are really performant. However, I am now having to check for uniqueness in the beforeSafe function.

const saveData = [{ name: 'Test Name'}, .... 1999 more of these];
Parse.Object.saveAll(saveData)

  async beforeSave(req) {
    const sourceId = get(req.object, 'attributes.sourceId');
    const user = req.user;
    if (sourceId) {
      const query = new Parse.Query('Clients');
      query.equalTo('sourceId', sourceId);
      const results = await query.first({ sessionToken: user.getSessionToken() });
      console.log(results);
      if (results) {
        // don't save this item
        return;
      }
    }
}

Worryingly this is going to perform 2000 queries before the saves. I have tried to query all these items and perform the checking before saveAll, however I’ve found that there are some issues with it and we’ve got some duplicates. I fear we are going to have to use a query just before the save…

Does anyone have any optimisation tips on this?

You can continue with your idea of querying all of them with a single query before the saveAll and create a unique index in MongoDB to make sure that you do not have any race condition generating duplicates.

I think it’s race conditions that are causing my issue. How do you create a unique index in Mongo? Have you any docs on that?

You can refer to MongoDB docs to do it directly on Mongo: https://docs.mongodb.com/manual/reference/method/db.collection.createIndex/

Depending where you are hosting, they can also have other alternatives/tools.