Permalink
Reply
On Friday, 22 December 2023 at 22:33:35 UTC, H. S. Teoh wrote:
>IMNSHO, if I had very large data files to load, I wouldn't use JSON. Precompile the data into a more compact binary form that's already ready to use, and just mmap() it at runtime.
I wondered about that decision as well, especially because this was internal game data that did not have to be user readable.
That's beside the point though; it was a ~10 MB JSON file that took them several minutes to parse. That's really just insane. Turns out it helps if you don't count the length of the entire document for every single value. It also helps if you don't iterate over your entire array of already written values every time you want to insert a new one. :)
In case you didn't know the story, here's a link:
https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/
I think there are several great lessons in there. Rockstar must have noticed how slow the loading is, but apparently just accepted it as a given... for 7+ years. Who needs optimizations on today's great hardware, right? There couldn't possibly be algorithmic problems in something simple like a JSON parser, right?
Second, look at what people suspected as the root cause of the problem, like the P2P architecture. It's funny how speculations about performance problems are always wrong. Only measuring will tell you the truth.
