Use multiple atomic operations on same field

[CastAsHuman]

I have a field in my Post collection called reactions, which is an array of JSONs. Whenever I want to add a new object to that array, everything works as it should! There are times though that I want to replace an object inside that array with a new one, doing this;

const newReaction = { userId: currentUserId, isDownvote: false };

post.remove("reactions", post.get("reactions")[indexOfReactionToRemove]);
post.add("reactions", newReaction);

The thing is that as soon as it gets to the add call, the process hangs… any idea why?

[davimacedo]

I don’t know if it is related, but I do not recommend using arrays to store lists with more than 20 items on average. It can cause serious performance problems. I’d use relations or joint classes instead.

[CastAsHuman]

I appreciate the comment! Could you explain why 20? It actually is right now a join-class and I thought it would be much of an improvement not to have to query a collection every time the user requests content!

EDIT; Every post has 20 likes on average, although there are others with 100 or more… not many of the daily feed though!

[davimacedo]

20 is an empirical number based on what I’ve seen. If you have few objects with more than 100 items in an array field, maybe some of them with 500 or 1000, it can start causing problems in the whole database. I’d also not use arrays for lists that can grow over time. Example of something that fits well on arrays: post’s categories, tags. Example of something that can cause problems: post’s followers, likes, comments.

[CastAsHuman]

Thanks for responding! Could you clarify what you mean by “it can start causing problems in the whole database”? What kind of problems?

[davimacedo]

There are many articles out there talking about issues with big arrays and objects on MongoDB. Long story short it makes your collection to be fragmented on your storage. Since it is fragmented, any write operation on this collection (not only related to the array field, but actually any field) can start taking a long time. Since write operations take lock of the database, all other concurrent operations can start delaying as well. I’ve seen this happening myself many times.

[CastAsHuman]

Thank you very much for your detailed explanation!!

[lsmilek1]

David,

I would have a follow up questions if I may.

@davimacedo

it makes your collection to be fragmented on your storage

Question #1:

for my better understanding, is is related to array count, or also on the size of fields not being in array? As it is possible to convert some arrays to strings (the ones not being used in query criterias) and decomposed it on the client side (or cloud code)…

For example:

Scenatio #1:

ParseObject.arrayField = [“longString1”,“longString2”,…100 more…,“longString103”]

Scenatio #2:

ParseObject.stringField = “longString1_longString2_…100 more…_longString103”

In both scenarios can the ParseObject have certain and let say not small size - lets say 300kB (as MongoDB allows I thing 16MB per object)… Would the mentioned phenomena occur in both cases?

Question #2:

Since it affects only collection, then for example a Role - where ACL is also an array and can have 100 entries - would be most certainly always affected BUT it is not a big deal as the Role object is not being updated as often as the other objects. The other objects would then stay not affected by large ACL, am I correct?

Question #3:

I also wonder, if there is a difference between having two arrays with 50 items each versus one array of 100 items? Does the issue stacks up within one object?

Thank you!

[davimacedo]

In my point of view:

Question #1: string field will cause the same problem if you keep updating it constantly and it is supposed to grow over time. I’d rather go with a relation or joint class for that.

Question #2: if it is not supposed to change constantly or grow over time, you should be good.

Question #3: if they you will update them constantly and they are supposed to grow over time, you will probably have the same problem.

[lsmilek1]

From your answer (#3) it seems that the fragmentation is dependent more on the object size in kB rather than on the array count. Would you have a recommendation on what the average size of an object in kB should be?

[uzaysan]

Default size limit of documents is 16MB. Few KB of data will be ok I guess.

[lsmilek1]

@davimacedo mentioned array of average size 20, therefore my guess is that fragmentation occurs much sooner than 16MB. For the discussion I assume array of short strings.

[armourer]

Example of something that can cause problems: post’s followers, likes, comments.

Sorry, for hijacking the post, but can you please elaborate what in your opinion is the right way to build the following relations:
I have users, each user can have followers and can follow someone, to simplify:

1. Table of users
2. Table of followers with two columns: userId, userIdToFollow. Both cols are references to Users table

Is it right way to build such schema ?

[davimacedo]

You can either create the joint table yourself as you suggested or use Relation, which is actually a joint table behind the scenes.