I wanted to create a topic where the community could share their tips on getting better Parse Server performance. Perhaps posts could include type of infrastructure, workloads and tips for improving performance? Here I’ll share some of our experience:
- configuration: AKS with our own custom helm chart, official docker parse-server images:
** dev/stage/prod namespaces
*** app server - multiple parse-server instances to handle client requests
*** job server - a dedicated parse-server where we run our jobs
- Database: MongoDB Atlas 4.4, tens of millions of objects < 1kb, 1-20k objects per user; our indexes don’t fit memory but we still get great performance
- Usage Patterns: cyclical with pronounced daily/weekly patterns
- Request types: heavy cloud code with multiple queries per request; custom search queries and indexing in cloud code and MongoDB
- Push-notifications: silent as well as scheduled (with daily peaks); custom schedule and dispatch queue in cloud code
- Load balancing
** Using a redis-cache instance is a must for horizontal scaling.
** We route requests from app server instances back to the load-balancer - our heavy cloud code that issues multiple concurrent queries gets properly balanced.
** Using a dedicated parse-server instance as our job server made it easy to monitor/troubleshoot jobs and eliminated job resource usage spikes from affecting the client-serving parse-server instances.
- Push notifications
** Switch from certificate based to API key authenticated notifications - we saw more consistent memeory usage; this also eliminated need for us to manually renew expiring certificates.
** We enabled brotli with gzip fallback on the ingress controller - big savings for our workloads, especially when clients load batches of objects.
** We enabled zlib and snappy compression for the mongodb driver (in the database uri string) with good results.
** We profiled batch size (query limit) and saw 11% traffic decrease as well as faster loads when we went from 100 to 250 limit on the batch load queries.
** Don’t use query skip/limit for paging! MongoDB can’t efficiently use skip - all the skipped keys/documents will be loaded and examined. We page on updatedAt and objectId (with proper indexes created to support that).
** Use query.select() to load only needed fields. When appropariate, create an index so that the query is covered in MongoDB and no documents get loaded.
** We favor locality in indexes to reduce paging (and memory pressure) - a lot of the objects we query have natural grouping by user account and creating compound indexes with the acccount as a prefix key increases locality (less of the index needs to be resident in memory for the query to scan); it will allow us to easily do sharding on the account key in the future.
** We don’t index on _rperm as it is an array field, takes lots of space and most of the time it’s faster for the query planner to just fetch the document and filter. Disclaimer: we create unique role per user account (shared by multiple users) and most workloads either query ‘public’ or ‘account role’ where other keys on the query makes the permissions almost always guaranteed to match (keys examined and docs returned for us is almost always 1:1).
** Be careful with query.include() - this triggers a subsequent query on the objectId field (hits the _id index). In WiredTiger _id index is huge for huge collections and queries on it may cause more paging due to its distribution (depending on your workload, ours queries hit both old and new objects). For our workload where we load both old and new objects (as opposed to mostly newish objects), we get better performance when we create a separate query, constrain on the user account (natural grouping for locality) and force the query to execute on the compound index.
- Client caching
** Super important - our apps are optimized to extensively use the cache and fetch only what is needed. General strategy is have version and changelog; load state from cache and compare with version from server; fetch and apply only changes during client session.
Some performance wishlists/questions:
- It would be nice to be able to add additional conditions to be applied on the query.include() resulting query (as well as hints for index use). One apparent use-case would be sharding - if the collection (that objects are included from) is sharded, we’d need to specify the shard key as condition on the query to make that efficient.
- Does the current mongodb driver in parse-server support zstandard compression setting?