JSONValue::parse() - any limits?

[gkujawsk]

Hi guys,
I’m trying to move away from ArduinoJson to the built-in parser for JSON.
When I use relatively short json string the parser works as expected. isValid() returns true and I can iterate over the keys. If I use longer json string (1.5kB) the parser silently fails. isValid() just returns false without any details. Electron does not panics. I’m using 2.0.0 firmware.

I’m 100% percent sure the json string is correct. I verified it using a couple of tools.
I hardcoded the json string (it’s escaped properly) into the code and then memcpy into the local buffer.
so it should be possible use JSONValue::parse safely. Are there any limits related to the built-in parser I should be aware of? Docs does not say what’s the maximum length or tokens number.

{"log":[{"level":"info","handler":"particle","facility":"app"},{"level":"info","handler":"serial","facility":"app"},{"level":"warn","handler":"particle","facility":"system"},{"level":"warn","handler":"particle","facility":"system"}],"ss":{"internal":[{"on":1,"name":"Sys.pwr","pt":20,"mt":200,"u":0},{"on":1,"name":"Sys.bat","pt":60},{"on":1,"name":"Sys.rssi","pt":20,"mt":200,"u":0},{"on":1,"name":"Sys.gps","pt":60},{"on":1,"name":"Sys.mem","pt":20,"mt":200,"u":0},{"on":1,"name":"Sys.tmp","pt":60},{"on":1,"name":"Sys.soc","pt":30,"mt":200,"u":0}],"dinputs":[{"name":"S.chn","u":1,"in":0,"mt":500,"pt":60},{"name":"Brn.flr","u":1,"in":1,"mt":500,"pt":60},{"name":"Pmp.flr","u":1,"in":2,"mt":500,"pt":60},{"name":"G.flr","u":1,"in":3,"mt":500,"pt":60},{"u":1,"mt":500,"pt":60,"in":4,"name":"Door"}],"ainputs":[],"vinputs":[{"u":1,"mt":200,"pt":60,"in":0,"name":"VIRT"},{"u":1,"mt":200,"pt":60,"in":1,"name":"NAZWA"}],"doutputs":[],"mbrtu":[{"len":1,"udev":0,"tfunc":0,"targ":1,"f":3,"sid":9,"addr":3180,"u":1,"mt":1000,"pt":120,"name":"SP","rs":485,"type":"int16","targ2":0}]},"alarms":[{"name":"AWARIA2","timeout":0,"autoEnable":0,"on":{"output":{"state":1,"out":0},"conditions":[{"reg":"Brn.flr","op":"eq","v":"1"}]},"off":{"output":{"state":0,"out":0},"conditions":[{"reg":"Brn.flr","op":"ne","v":"1"}]}},{"timeout":0,"autoEnable":1,"on":{"output":{"state":0,"out":1},"conditions":[{"reg":"VIRT","op":"eq","v":"0"}]},"off":{"output":{"state":1,"out":1},"conditions":[{"reg":"VIRT","op":"ne","v":"0"}]},"name":"TEST"}]}```

[gkujawsk]

After some more digging in the device-os I’m quite sure that the json string is not parsed correctly if there are too many tokens. So far I managed to find that in spark_wiring_json.cpp jsmn_parse returns -128 for more complex/longer json string, while for shorter json string I get the positive number, which allows to finish parsing successfully.

What I don’t get is that according to the signature the jsmn_parse should return jsmnerr_t which is defined as enum type, while the function returns the number of tokens. I’m not the expert but it seems that if the number of tokens is large enough the return value overflows. I would appreciate any comments from more experienced colleagues.

typedef enum {
    /* Not enough tokens were provided */
    JSMN_ERROR_NOMEM = -1,
    /* Invalid character inside JSON string */
    JSMN_ERROR_INVAL = -2,
    /* The string is not a full JSON packet, more bytes expected */
    JSMN_ERROR_PART = -3
} jsmnerr_t;

–
/G

[gkujawsk]

Now I can definitely confirm that if there are more than 128 jsmn tokens to be produced then the json string is not parsed correctly. For 129 tokens the count value return by jsmn_parse is actually -127 0xFFFFFF81 and for 131 tokens -125 0xFFFFFF83 respectively. It looks in fact like the return value is somehow overflowing.

[gkujawsk]

And the last heads up from me regarding the built-in json parser. Setting the number of tokens manually in sparK_wiring_json.cpp:tokenize() and disabling jsmn_parse checks for < 0 allows for happy json parsing…

As I wrote at the beginning I was trying to replace ArduinoJson library with the built-in jsmn lib to save some space on the flash. After all it seems that if you have pretty complex json structure as me the ArduinoJson lib is a way to go. With ArduinoJson my code size is 129,912B while with built-in jsmn it’s overflowing flash 1068.

Thats’a all I wanted to share. And of course don’t use built-in jsmn to parse anything thats is complex enought to require more than 128 tokens. If you do - remember to mess up with spark_wiring_json.cpp.

[deancs]

Nice work, thank-you. I use the built in parser with a relatively small number of tokens but good to know. You should raise a ticket to have this resolved.

[ScruffR]

@gkujawsk, where do you get that massvie JSON string from and do you need all of its content?

Reason for asking is, if you happened to get that sort of JSON via a webhook and wouldn’t need all of its content, you could maybe create a response template that already boils down the original response to a somewhat less overwhelming size that only contains the data needed and along the way cut down on some overhead (e.g. shorten overly verbose key names).

Not only could that bring the response down to a more manageable size it would also help reduce your cellular data consumption.

[gkujawsk]

Hi,
Not that easy to shrink this down. It’s a configuration string for my product and I need every bit of it. And actually this one is not the longest possible My electron is a more or less universal measurement device. It means that I have to have option to configure different measurement subsystems (digital inputs, analog inputs, modbus rtu, modbus tcp, pt100 etc.) depending on where my device is installed. Thats the reasoning behind so big json string.

It works pretty well. Config is delivered by a couple of webhook responses, merged and parsed and stored locally od sd card until refreshed. Unfortunately electron is no longer accommodating new ideas ;-> Moving to 2.0.0 helped just a little bit. I know that others are facing the same limit but it seems that the only way is to introduce second uc.