The ongoing file issue between Parse Server, Parse Dashboard, and MongoDB

Hey guys I was hoping for a little help on narrowing down an issue I’ve noticed with parse for a while now. 3 years ago I noticed that removing rows via Parse Dashboard didn’t clean up file data. In MongoDB both fs. files and fs. chunks stayed the same. It wasnt that big of a deal then because we werent doing a bunch of file storage.

For the last few weeks I’ve been working on migrating a clients huge project from Firebase to Parse for an image intensive application. Being primarily an iOS developer now I decided to use Parse Dashboard because its easier to see how my migration went between objects. Its also a little faster to just click a few buttons and clear the entire database and make an adjustment and run it again.Several times now I’ve hit my database size limit even after deleting every object from every table and found that fs. file and fs chunk is not empty and fs. chunk size is massive. Deleting those tables entirely temporarily resolves my issue and I am able to move forward till I hit the size limit again.

Now, I see that the issue was fixed for fs. file via issue #506 for Parse Server back in 2016. I see no issue for fs chunk at all on either Parse Server or Parse Dashboard.

So here’s my questions. I’m trying to figure out possibly what the issue is and who might be responsible for it so I can add it to the correct channel. So, I have a few questions.

  1. What is fs chunk? Should it be cleaned up immediately or is it temporary file data that would have eventually been cleaned up but I am running scripts too quickly for that to happen. In other words am I the issue?

  2. If I’m not the issue then who’s responsible for it? Parse Server or Parse Dashboard. I started out running the latest version of parse dashboard with parse server with the latest version of parse dashboard. After re-discovering the issue I thought perhaps it was a mismatch between versions and installed parse dashboard with the latest version it supports. So its not a version issue. So the next thing I’m trying to figure out is who actually does the deletion. I would assume its Parse Server’s issue and Parse Dashboard merely runs the exact same JavaScript script that Parse Server would run. Or maybe it uses the JS Parse Client deletion script. Which I guess could be an issue with the JavaScript Parse SDK. This is where I’m the most confused.

  3. Is there a workaround? I figured anyone who’s an expert with parse server and its internal workings could figure this out faster than I could. Would nullifying the column prior to deletion remove the file data?

On a unrelated note while looking around in mLabs I noticed that all Parse Relation tables never got cleared either after deleting all data from a class. Think its worth opening a issue for that? I would assume whoevers responsible for question 3 is also responsible for this.

I thought I might add that between StackOverflow and Github I’ve noticed several “Size limit” issues popping up. Most people are given the explanation that it means their database size is too large. I feel that some of these could actually be the same issue I am having.

I guess I should probably just add an issue and hope someone addresses me on github huh?

First of all, it’s important to mention that the default GridFSBucketAdapter is only recommended for development environments. For production, I’d recommend you to use one of the other available file adapters: https://github.com/parse-community/parse-server/#configuring-file-adapters

What is happening in your case: when you create a file in Parse Server, it stores the file in the adapter’s storage (in your case, MongoDB GridFSBucket - this is the responsible for managing the file and fs chunk collections) and only a reference to this file in the class object. When you delete an object (that’s what you are doing when purging the whole class using the Parse Dashboard), the file reference is also deleted, but the file itself is not deleted from the storage. You can also delete the file by doing a specific api call (https://docs.parseplatform.org/rest/guide/#deleting-files) but there is no such option in Parse Dashboard.

Therefore my recommendation to you is to write a script in order to purge the files you need.

There is a long discussion about this in the Github repo: https://github.com/parse-community/parse-server/issues/1023

1 Like

Thank you so much for the response. Don’t think I would have ever found this discussion had you not pointed it out. Also after checking the rest documentation I went back to check if the iOS and JavaScript documentation has any mention of it. It’s really easy to miss. It’s just kinda tossed in at the end of the File section with no header whatsoever. That might be something to improve in the future.

1 Like

Yes. It makes sense to add some notes in the docs. Would you be willed to send a PR with this?

Would be my pleasure.

I wanted to confirm this part of the documentation is left over from when parse was a service.

You may request a cleanup of unused files in your app’s Settings page.

Figured Id remove this while I was at it.

Yes. I think it should be removed.