Iterating through large file-like objects 1.1

This function provides a simple iterator for the lines of a file-like object. (It also provides a flag for removing/retaining trailing newlines.)

I ran into the problem a number of times that I wanted to read-in a very large file-like object (say a GB or two) line by line. This is a problem for any object that wants to read the entire "file" in at once, unless you have several GB of memory just begging to be used...

In Python 2.3, while file objects now support this sort of single-line iteration, not all file-like objects do. For example, using the for-in construction with sys.stdin still causes sys.stdin to read in everything before the for-loop begins iteration. Using the iterfile function allows you to process sys.stdin iteratively, which may be important if you have a lot of information coming in on standard input.