A 300 MB JSON array does not fit comfortably in memory. Options

Back to list2024-04-23
You

A 300 MB JSON array does not fit comfortably in memory. Options?

ChatGPT

Stream it. Either a pull parser like ijson, or, if the structure is a flat array of objects, read in chunks and use json.JSONDecoder().raw_decode to peel one object at a time. Memory then depends on the largest single object, not on the file.

You

Does this still hold if I am working alone?

ChatGPT

It gets easier, not harder. A fair share of the difficulty is coordination, and that disappears. What remains is remembering your own decisions, which is what writing them down is for.