Node SDK makes network requests

Client SDKs JavaScript SDK
,
1

Hi,

I’ve built a basic catelog app using Parse and it all works beautifully, although I bumped into something the other day that was a bit unexpected…

TD;DR;

The node parse sdk seems to make network requests for each interaction with the system. I guess this makes sense, seeing as it’s the same sdk used in the browser, which will obv cuts down on maintenance and if it works from the browser, my guess is it’s all using the rest api under the hood, yes?

The thing that caught me off guard is that I was getting a bunch of 503s when doing many updates to my Product class using a script on the server (details below if you’re interested). The script uses async await, so the requests are staggered to avoid bringing the server down.

Natrually, none of this made sense at first, as the server should easily handle the requests and it didn’t seem to be under any serious strain.

After looking around for a while, I realised it was my nginx config that was limitting my requests to 20 req/s (something I had set).

Does anyone know if there’s a way to either whitelist the requests coming from the sdk or perhaps have a suggestion/pointer to better practises for doing bulk updates?

Backstory

My app pulls a csv file every 2 min, which contains updates to product stock counts and prices (there are abot 3k products). I reduce load on the server by storing a copy of the file on the server and doing a diff between the incoming file and the stored file. In this way, I only ever touch products that have actually changed.

Having said that, there have been a few occassions where the system got out of sync for whatever reason, which meant that I had to process the whole file to sync the system back up with the file.

I know there is a batch operation feature built into the rest api, so it dawned on me to use this, but I’ve never done batch operations using the sdk…I can’t find any docs that speak to how to do this?

I’d appreciate any thoughts/pointers/suggestions :slight_smile:

Thanks!

2

I’d do that as a Cloud Code Job and use directAccess option set true. This way, your request will go directly to the database. Talking about the batches, you can use Parse.Object.saveAll() to save multiple objects at once. If. you can share the relevant part of your code, I can try to give some more additional ideas.

3

Firstly, I just want to say thank you! You always respond so quickly and so helpfully to people asking for help! This is part of what makes this community so great, so thanks for all the effort!

Jobs! That’s a great idea, I’ve never used Parse Jobs and hadn’t actually thought of it, but it sounds like a great fit for this use case!

In terms of the batch operations… I currently have a crude way of fetching batches of 200 products at a time, combining them into an array, updating them as needed and then saving each separately in a for of loop, so nothing fancy - here’s an edited and abbreviated version of the code:

// a var to gather results to be returned once complete
const results = [];

// I create a map of the retrieved products by sku, 
// assigning the updated product as the value
const existingProductsByCode = {};

for (const group of existingProductDAOgroups) {
  for (const product of group) {
    existingProductsByCode[product.get("sku")] = product;
  }
}

// then update each one individually
for (const product of products) {
  if (existingProductsByCode[product.sku]) {
     const result = await updateExistingProduct(
       existingProductsByCode[product.sku], 
       product
     );
    results.push(result);
  }
}

I found the “saveAll” option you metioned - looks pretty straight forward, thanks!

Where does one set the directAccess option? In the save call? Something like:

Parse.Object.saveAll(objects, { 
  useMasterKey: true, 
  directAccess: true 
});
4

Sorry - quick comment: using await in a loop can result in the operation taking a lot longer than expected as it await each iteration before proceeding to the next (see: here).

A better solution would be to utilize Promise.all, so each iteration can be resolved in parallel.

await Promise.all(products.map(async product => {
  if (existingProductsByCode[product.sku]) {
     const result = await updateExistingProduct(
       existingProductsByCode[product.sku], 
       product
     );
    results.push(result);
  }
});
5

Thanks @dblythy, I am aware of this. I’m deliberately doing this in an attempt to slow the operation down and avoid the 503s I’m getting (I didn’t know about the directAccess option), so nginx gives me 503s when doing large batches of changes.

I need to try to get the directAccess option working and will refactor all of this :slight_smile:

Thanks again, appreciate the willingness and time you’ve taken to help!

6

For those interested, it seems you set the directAccess option on your parse server config.

There are a few interesting things to read around the topic in this (and related) issue

1 Like
7

For those interested, I found these benchmark results related to directAccess…this is brilliant!:

image

8

On another project, I have some rather onerus reports that need to be run and I’ve been hitting some performance issues…

I’ve implemented directAccess and switched to using aggregation queries (the results all go to json, so no need for Parse objects) and wow…I’ve seen some seriously huge gains in performance!

1 Like
9

I’ve recently switched on directAccess for my main app, which uses the multi tenant solution I described here.

Sadly, I’ve not found a way to pass my tenant data along with those queries, so the solution described in that post longer works. I’m not too sure what direction to take yet, but if anyone knows how to pass some context info along to these directAccess calls, please let me know.

For those that don’t know what directAccess is, it allows you to make your db calls without going through the http/network layer. I’ve seen huge performance increases with another project, using this setting.