The ongoing file issue between Parse Server, Parse Dashboard, and MongoDB

#1

Hey guys I was hoping for a little help on narrowing down an issue I’ve noticed with parse for a while now. 3 years ago I noticed that removing rows via Parse Dashboard didn’t clean up file data. In MongoDB both fs. files and fs. chunks stayed the same. It wasnt that big of a deal then because we werent doing a bunch of file storage.

For the last few weeks I’ve been working on migrating a clients huge project from Firebase to Parse for an image intensive application. Being primarily an iOS developer now I decided to use Parse Dashboard because its easier to see how my migration went between objects. Its also a little faster to just click a few buttons and clear the entire database and make an adjustment and run it again.Several times now I’ve hit my database size limit even after deleting every object from every table and found that fs. file and fs chunk is not empty and fs. chunk size is massive. Deleting those tables entirely temporarily resolves my issue and I am able to move forward till I hit the size limit again.

Now, I see that the issue was fixed for fs. file via issue #506 for Parse Server back in 2016. I see no issue for fs chunk at all on either Parse Server or Parse Dashboard.

So here’s my questions. I’m trying to figure out possibly what the issue is and who might be responsible for it so I can add it to the correct channel. So, I have a few questions.

  1. What is fs chunk? Should it be cleaned up immediately or is it temporary file data that would have eventually been cleaned up but I am running scripts too quickly for that to happen. In other words am I the issue?

  2. If I’m not the issue then who’s responsible for it? Parse Server or Parse Dashboard. I started out running the latest version of parse dashboard with parse server with the latest version of parse dashboard. After re-discovering the issue I thought perhaps it was a mismatch between versions and installed parse dashboard with the latest version it supports. So its not a version issue. So the next thing I’m trying to figure out is who actually does the deletion. I would assume its Parse Server’s issue and Parse Dashboard merely runs the exact same JavaScript script that Parse Server would run. Or maybe it uses the JS Parse Client deletion script. Which I guess could be an issue with the JavaScript Parse SDK. This is where I’m the most confused.

  3. Is there a workaround? I figured anyone who’s an expert with parse server and its internal workings could figure this out faster than I could. Would nullifying the column prior to deletion remove the file data?

On a unrelated note while looking around in mLabs I noticed that all Parse Relation tables never got cleared either after deleting all data from a class. Think its worth opening a issue for that? I would assume whoevers responsible for question 3 is also responsible for this.

#2

I thought I might add that between StackOverflow and Github I’ve noticed several “Size limit” issues popping up. Most people are given the explanation that it means their database size is too large. I feel that some of these could actually be the same issue I am having.

#3

I guess I should probably just add an issue and hope someone addresses me on github huh?

#4

First of all, it’s important to mention that the default GridFSBucketAdapter is only recommended for development environments. For production, I’d recommend you to use one of the other available file adapters: https://github.com/parse-community/parse-server/#configuring-file-adapters

What is happening in your case: when you create a file in Parse Server, it stores the file in the adapter’s storage (in your case, MongoDB GridFSBucket - this is the responsible for managing the file and fs chunk collections) and only a reference to this file in the class object. When you delete an object (that’s what you are doing when purging the whole class using the Parse Dashboard), the file reference is also deleted, but the file itself is not deleted from the storage. You can also delete the file by doing a specific api call (https://docs.parseplatform.org/rest/guide/#deleting-files) but there is no such option in Parse Dashboard.

Therefore my recommendation to you is to write a script in order to purge the files you need.

There is a long discussion about this in the Github repo: https://github.com/parse-community/parse-server/issues/1023