Reading a huge JSON file without loading it

Back to list2025-10-25
You

A 300 MB JSON array does not fit comfortably in memory. Options?

ChatGPT

Stream it. Either a pull parser like ijson, or, if the structure is a flat array of objects, read in chunks and use json.JSONDecoder().raw_decode to peel one object at a time. Memory then depends on the largest single object, not on the file.

You

What is the most common way people get this wrong?

ChatGPT

Doing it once and never verifying. The setup is the visible part, so it gets attention, and the check that would catch a silent failure never gets written.