Slow Parse Server Performance / Slow saveAll() for Batches of 800

[rob]

Problem:

When I run the code below for one user (one batch of 800 records that should take around 15 seconds to complete), the time instead grows infinitely to over a few minutes per batch until the server must restart:

                // see if sender tracking already exists in db for this user (query in one call)
                let senderTrackingRecords = await new Parse.Query("EmailSenderTracking").equalTo("userId", userId).containedIn("senderId", senderIdArray).grapeFind();

                // create array of sender IDs from "senderTrackingRecords" (to compare later with "senderIdArray")
                let senderTrackRecordSenderIds = [];
                for (let index = 0; index < senderTrackingRecords.length; index++) {
                    const senderId = senderTrackingRecords[index].get("senderId");
                    const existingCount = senderTrackingRecords[index].get("count");
                    senderTrackRecordSenderIds.push(senderId);
                    let currentObj = senderTrackingObjBySenderId[senderId];
                    
                    // update existing records
                    senderTrackingRecords[index].set("lastMessage", currentObj.messageId);
                    senderTrackingRecords[index].set("count", existingCount + currentObj.count);
                }

                if (senderTrackingRecords && senderTrackingRecords.length > 0) await Parse.Object.grapeSaveAll(senderTrackingRecords);

I have the same symptoms as most of the other posts I’ve seen with similar issues:
Parse server performance loss after time

Parse Server performance loss

Fetch large data for reports slow

Fetch large data for reports slow

Saving 3000 objects in cloud code with saveAll takes more than 10 seconds

But unfortunately none of those seem to have a clear solution and I’ve tried enabling directAccess, databaseOptions/enableSchemaHooks, and lots of testing.

This is what the growth looks like on the server when processing one batch of 800 records – it keeps growing.

As a test, I created one function that repeatedly saves 1,000 hard-coded records without any other logic using saveAll():

    const stopTime = Date.now() + 3600000; // 1 hour = 3600000; 30 mins = 1800000;
    while (Date.now() < stopTime) {
        let newSenderTrackingRecords = [];
        for (let index = 0; index < 1000; index++) {
            const Record = Parse.Object.extend("EmailSenderTracking");
            const record = new Record();
            record.set("userId", "XXXXXXXXXX");
            record.set("lastMessage", "XXXXXXXXXX");
            record.set("lastMessageTime", Date.now());
            record.set("senderId", "XXXXXXXXXX");
            record.set("count", 45);
            record.set("mailboxId", "XXXXXXXXXX");
            newSenderTrackingRecords.push(record);
        }
        await Parse.Object.grapeSaveAll(newSenderTrackingRecords);
        console.log("SAVED VIA PARSE");
    }

And it looks like this (same problem with growing time to complete task & crashing server):

Then I saved directly to mongo db via mongoose and voila – it’ll save 1,000 records no problem in less than 1 second repeatedly.

Here’s my Parse config:

const config = {
    databaseURI: databaseUri || 'mongodb://localhost:27017/dev',
    cloud: process.env.CLOUD_CODE_MAIN || __dirname + '/cloud/main.js',
    appId: process.env.APP_ID || 'myAppId',
    masterKey: process.env.MASTER_KEY || '', //Add your master key here. Keep it secret!
    serverURL: process.env.SERVER_URL || 'http://localhost:1337/parse', // Don't forget to change to https if needed
    serverSelectionTimeoutMS: 60000,
    connectTimeoutMS: 60000,
    enableAnonymousUsers: false,
    allowClientClassCreation: false,
    directAccess: true,
    databaseOptions: {
        enableSchemaHooks: true,
    }

Running Parse Server 5.4 and MongoDB 5

Can you all think of something else to test or if there’s another obvious configuration I need?

A workaround I can see is to stop using Parse for all db calls, or to save directly to mongo for 800+ batches, but then it’s a mess to use both Parse & mongo to save to db if I can’t use them interchangeably. I’m aware of the {json:true} trick to return JSON objects from Parse.Query() queries, but then I haven’t figured out how to resolve errors appears due to incompatibilities between Parse/JSON objects in mongo db. Today I saw that there might be an adapter for Parse-to-Mongo object storage, but then I’m not sure if that’s even the best solution.

Appreciate any insights!

[Manuel]

@rob

Parse.Object.grapeSaveAll

What is this custom method and do you observe the same if you use the Parse.Object.saveAll?

Also, could you try to save each object individually instead of saveAll? There may be a leak in the batch logic.

[rob]

It simply shortens our code:

And I have tried save() (sequential calls), but it takes even longer due to waiting on network more. I just tried making concurrent save() calls and it appears promising, but server crash again with other problem so I’ll troubleshoot that more and report back.

[Manuel]

Should be addressed by perf: Increase constructor speed of `Parse.Object.extend` by dblythy · Pull Request #1682 · parse-community/Parse-SDK-JS · GitHub