Mass update Parse Relations - performance

Client SDKs JavaScript SDK
,
1

Hello :wave:

I have a webapp where a _User can add a Tag to a bunch of Profile. For example, as a User, I have a list of 3577 profiles and I want to bulk update them with the tag “Friend”.

I achieved this with the following snippet but it tooks a lot of time to be executed. I would expect to be done in <5 seconds, is there a way?

Here is my collection attributes, very simple:

Profile :

firstName: string
lastName: string

Tag :

title: string,
color: string,
tagged: relation<Profile>

And code to mass ADD the tag with id Kv6YISgDqq to a list of profiles:

// 1. find profile ids

const pipeline = getConnectionsProfileIdsPipeline(me, filters);
const documents = await new Parse.Query(ParseConnection).aggregate<Document[]>(pipeline); // <- this one executes very fast ~0.2 sec

const ids = documents.map((d) => d.id);

// 2. find the tag, add profiles to the tagged relation and save the tag

const parseTag = ParseTag.createWithoutData("Kv6YISgDqq") as ParseTag;

await parseTag.fetch({ sessionToken });

const tagged = parseTag.get("tagged");

ids.forEach((id) => {
  tagged.add(_createParseProfileWithoutData(id));
});

await parseTag.save(null, { sessionToken }); // <- TAG added [ Friend ] to 3577 profiles. Took: 104.708 secs

I am doing the right thing? How can I boost this request? My database is hosted on mongodb atlas and Parse Server on heroku. Also I ensured the _Join:tagged:Tag table is indexed as shown below.

Thank you very much for your help.

image
image1168×333 43.8 KB

2

NOTE

I have tried to split the ids into smaller chunks. It actually improves performance.

By the way, I don’t understand why, I mean, I don’t understand why making several requests (= http ping/pong + more database opening + insert + closing) is faster that one fat request. It sounds counterintuitive to me.

Here is the code that provide better performance.

But still, I would like the whole action to take <5 seconds. Any suggestion?

const chunkSize = 300; // I tried several values here, 300 looks to offer better performance
const idChunks = [];
for (let i = 0; i < ids.length; i += chunkSize) {
  idChunks.push(ids.slice(i, i + chunkSize));
}

for (const idChunk of idChunks) {
  // Note: if I move these 3 lines up, outside the for loop, the `save` takes more and more time to execute in the next loop :thinking-face:
  const parseTag = ParseTag.createWithoutData("Kv6YISgDqq") as ParseTag;
  await parseTag.fetch({ sessionToken });
  const tagged = parseTag.get("tagged");

  idChunk.forEach((id) => {
    tagged.add(_createParseProfileWithoutData(id));
  });

  await parseTag.save(null, { sessionToken }); // <- TAG added [ Friend ] to 300 profiles. Took: ~1.9 secs
}

// Done. Took: 25.578 secs (instead of 104.708 as mentioned previously!!)
3

The fastest way to do this is probably with a MongoDB script or a native update operation (assuming you’re using MongoDB). These kind of bulk operations are best done by the DB engine itself, especially if you calling them often as part of your operation.

I’m not sure which of the native ops are currently supported by Parse Server, but you may want to take a look at the update docs. You can also update in stages of the aggregation pipeline.

4

Ok thanks. Can I use the Parse Server JS SDK to run native update operation?

I am using Parse.Query.aggregate already because I needed a really custom search. But it’s only designed for querying the data, not for updating, right? (I’m learning mongodb as I go…)

5

If you open the link in my previous comment you can read that:

Starting in MongoDB 4.2, you can use the aggregation pipeline for update operations.

6

That’s a great point, thank you.

I see there are updateMany, updateMany, insertMany and more, but I get confused, are you suggesting there is a way to invoke them using Parse.Query.aggregate?!