Repository navigation
dumps return bytes support ? #136
Description
Hello !
I'm looking to switch from simplejson to rapidjson.
I need to have my objects serialized in bytes.
Does rapidjson work internaly in str or in bytes in utf_8 ? If in bytes, to avoid loosing performences with two additionals conversions (one inside rapidjson bytes >str ) and one outside with encode ( my_object_serialized_bytes = dumps (my_object).encode ("utf_8")), I would like to have an option my_object_serialized_bytes = dumps (my_object, to_bytes = True) or something like that. Do you think it would be easy to add it? Dump already accept files open in binary mode.
If yes, it could be great if RawJSON could take bytes too !
Thank you very much!
Activity
[-]dump return bytes support ?[/-][+]dumps return bytes support ?[/+]lelit commentedon May 8, 2020
Uhm, I doubt the encoding/decoding overload would change, even if done internally by RJ... and given that you seem massaging not trivial amount of data, that it is high enough to be measurable.
Do you have some kind of benchmark that would support that?
guilbut commentedon May 11, 2020
Hello !
I need to serialize lot of bytes data in my jsons, for that I use Base 64 encoding .
pybase64.b64encode(my_bytes) return a bytes object with only ascii chars not needing escape in json (no \t \n " etc) . Thus it could be used as it is with juste aditionnal "" if rawJson could take bytes (and better add "" himself if asked) . Making many conversion bytes <-> string for this base64 encoded bytes are very time and memory consuming. With frequently memory error when python's memory is fragmented and python is no more able to find enough contiguous memory avaible for exemple after several hours runing .
I wrote an exemple as simple as i can, with cpu and memory profiling :
dumps_bytes_with_base64_exemple.txt
cpu and memory consumption inside rapidjson are not monitored, but comparaison with orjson wich work inside with bytes and return bytes can give an idea of what could be espected without useless bytes <-> string conversion. Orjson made the choice of returning bytes for that reason :
https://github.com/ijl/orjson/issues/66
Maybe rapidjson.dumps could have a option for returning bytes, or a new fonction called "dumpb" .
For the rawJSON supporting bytes, maybe we could have 2 options : bytes and bytes_needing_quotation_marks in order to avaid a supplementary copy juste for adding quotes.
lelit commentedon May 11, 2020
I will try to understand the impact. But my first reaction to your use case, with due respect, is that JSON is not really the right tool: encoding hundreds Mb of raw data as base64 is not going to fly, and if it were me, I would surely investigate better alternatives.
Anyway, I will report back.