Saving byte data to ParseObject

lsmilek1 · July 6, 2021, 10:41am

I am considering to store small amount of data to a parse object to save this extra 30% storage over saving a base64 string. The data size is rather small - short encrypted text for example so using files and their urls would not be optimal - but considering there will be millions of these short datas, the 30% size reduction is appealing.

Do I understand it correctly that I should set the field in Dashboard as Array and then use var data: [UInt8]? in my ParseSwift client? Or there is some other trick?

What I am not sure about is the javaScript’s Number vs swift’s UInt8 conversion and if that at the end will not take more storage than base64string as to my limited knowledge the javascript uses Int32.

davimacedo · July 7, 2021, 4:21pm

Parse Server supports a column type Bytes which I believe would fit your needs. Not sure if Swift SDK has it implemented though.

lsmilek1 · July 7, 2021, 5:16pm

I remember you mentioned it before, but after looking for it in the dashboard I got confused as I have not found it there:

Do I look in a wrong place or is it because of back4app having a version that do not support it yet?

davimacedo · July 9, 2021, 9:12pm

I think it was only implemented in Parse Server and never included as a Parse Dashboard option. Currently you’d need to change the schema via api or send a first binary data using master key.

cbaker6 · July 10, 2021, 4:39pm

@davimacedo thanks for pointing this out. Just added in:

github.com/parse-community/Parse-Swift

Add ParseBytes and ParsePolygon

parse-community:main ← parse-community:types

opened 03:04AM - 10 Jul 21 UTC

cbaker6

+662 -129

Adds two missing Parse types to the Swift SDK: `ParseBytes` and `ParsePolygon`. … - [x] add `ParseBytes` - [x] add `ParsePolygon` - [x] (Breaking change) improve `ParsePolygon` by making the initializer through if a geo point is out of bounds instead of asserting - [x] fix `withinPolygon(key: String, points: [ParseGeoPoint])` to make it send an array of doubles instead of an array of GeoPoint's - [x] add `withinPolygon(key: String, polygon: ParsePolygon)` - [x] add test cases - [x] add playground examples

lsmilek1 · July 10, 2021, 5:23pm

Great! Thanks for quick implementation! I will test it in the next days.

Do you know if the cloud code’s JavaScript will handle Bytes type also? In the JS documentation I found only init of a file from an Array of bytes. Does that mean that I should handle that Bytes field in the cloud code as an array of bytes? If you would have any short example I would highly appreciate it!

cbaker6 · July 10, 2021, 6:10pm

I don’t see Bytes formally in the JS SDK, but not sure it needs it because the Cloud Code will still see the fields when you save, query, etc:

github.com

parse-community/Parse-SDK-JS/blob/ffc523f9cd0fa2dd3fcc9754d5f15c3c1c78314f/integration/test/ParseObjectTest.js#L1365-L1379


it('works with bytes type', done => {
  const object = new TestObject();
  object.set('bytes', { __type: 'Bytes', base64: 'ZnJveW8=' });
  object
    .save()
    .then(() => {
      const query = new Parse.Query(TestObject);
      return query.get(object.id);
    })
    .then(o => {
      assert.equal(o.get('bytes').__type, 'Bytes');
      assert.equal(o.get('bytes').base64, 'ZnJveW8=');
      done();
    });
});

lsmilek1 · July 13, 2021, 3:17pm

Hey Corey, I finally tested to save ParseBytes. I had to remove my “d” column in the dashboard and set the CLP so that a field can be added. The ParseObject now looks as following:

struct Inbox: ParseObject {
    
    //Those are required for Parse Object
    var objectId: String?
    var createdAt: Date?
    var updatedAt: Date?
    var ACL: ParseACL?
    
    /**
     Recipient's objectId
     */
    var rid: String?
    
    /**
     Data (encrypted InboxMessage)
     */
    var d: ParseBytes!
    
    /**
     Salt string for key agreement
     */
    var s: String?
}

After saving the file I can see in the Dashboard that the field “d” was added, but it also show that is being saved as base64 string:

Is this how only the dashboard shows/interprets the field, or is it really saved as nested object JSON with base64 string data? If so, then there is really no advantage compare to saving base64 string directly.

EDIT:

I found in the source code a BytesCoder so I believe it is only Dashboard representation and the data are stored in MongoDB as Binary indeed:

github.com

parse-community/parse-server/blob/1594afec6421bea31e732dff9b21eb099898bc3a/src/Adapters/Storage/Mongo/MongoTransform.js#L1410-L1443


var BytesCoder = {
  base64Pattern: new RegExp('^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$'),
  isBase64Value(object) {
    if (typeof object !== 'string') {
      return false;
    }
    return this.base64Pattern.test(object);
  },


  databaseToJSON(object) {
    let value;
    if (this.isBase64Value(object)) {
      value = object;
    } else {
      value = object.buffer.toString('base64');
    }
    return {
      __type: 'Bytes',
      base64: value,
    };

This file has been truncated. show original

The data are in that case just transferred over the network in base64 format, but with my limited knowledge that is a must and we cannot send binary data in the REST and other SDK APIs, is that correct?

cbaker6 · July 14, 2021, 10:48am

You should make this optional or else any other object that uses Inbox as a pointer will not decode.

I don’t know much about how Byte is used on the server side, but you may be right, @davimacedo can probably confirm for you

lsmilek1 · July 14, 2021, 1:15pm

Thank you for pointing that out, I did correct it. As everything with ParseBytes seems to work well for me, I can confirm that the feature was successfully implemented. Thank you!