How do I structure a file that is both a module and a CLI

Keep the work in functions, put argument parsing in main(), and guard with if __name__ == "__main__": sys.exit(main()). Return codes from main then work for both the shell and the tests.

Not usually, so look at what is inside. A comprehension that calls a method per item pays the attribute lookup every time; hoisting it out (append = out.append) or using a generator when you do not need the list is what actually moves the number.

Stream it. Either a pull parser like ijson, or, if the structure is a flat array of objects, read in chunks and use json.JSONDecoder().raw_decode to peel one object at a time. Memory then depends on the largest single object, not on the file.