The standard way to read a file line by line in Python is to open it in a with statement and loop over the file object, as in for line in f:, which reads one line at a time and works for files of any size. Methods such as readlines() and read().splitlines() load the whole file into memory and suit small files only.
with open("data.txt", encoding="utf-8") as f:
for line in f:
print(line.rstrip("\n"))apple
banana
cherryThe methods for small files and the memory-efficient methods for large files are compared side by side, followed by line numbers, blank lines, specific lines, encoding errors, fileinput and a log-parsing example. The file data.txt holds the three lines shown.
1. Methods to Read a Small File
These methods return all lines at once, which is convenient for sorting or indexing but needs memory for the whole file.
| Method | Return value | Keeps the newline? |
|---|---|---|
| f.readlines() | list of str | Yes |
| f.read().splitlines() | list of str | No |
| [line.rstrip(“\n”) for line in f] | list of str | No |
| Path(“data.txt”).read_text().splitlines() | list of str | No |
1.1. Using for Loop (the Default for Any Size)
A for loop over the file object does not load the whole file. Python reads the file in buffered chunks and yields one line per pass, so this method works equally well for small and very large files. Each line keeps its trailing newline, which rstrip(“\n”) removes.
try:
with open("data.txt", encoding="utf-8") as file:
for line in file:
print(line.rstrip("\n"))
except FileNotFoundError:
print("File not found.")We prefer rstrip(“\n”) to strip() when leading spaces matter, such as in indented text or fixed-width data, because strip() also removes them.
1.2. Using readlines()
The readlines() method returns all lines as a list of strings, each one ending with a newline except perhaps the last. A list lets us count, sort or index the lines.
with open("data.txt", encoding="utf-8") as file:
lines = file.readlines() # ['apple\n', 'banana\n', 'cherry\n']
count = len(lines) # 3
last = lines[-1].rstrip("\n") # 'cherry'1.3. Using splitlines() and List Comprehension
The splitlines() method splits the full text without keeping the newlines. A list comprehension over the file does the same and can filter or transform the lines on the way.
from pathlib import Path
lines = Path("data.txt").read_text(encoding="utf-8").splitlines() # ['apple', 'banana', 'cherry']
with open("data.txt", encoding="utf-8") as file:
long_words = [w for line in file if len(w := line.rstrip("\n")) > 5] # ['banana', 'cherry']2. Methods to Read a Large File
For files that do not fit in memory, we process one line at a time and never build a list of all lines.
| Method | When to use |
|---|---|
| for line in f | The default for sequential processing |
| f.readline() | Reading a header or a few lines before the loop |
| A generator function | Reusable pipelines that clean or filter lines |
| itertools.islice() | A range of lines, such as lines 1,000 to 1,010 |
| mmap | Searching bytes in a huge file or random access |
2.1. Using readline() Method
The readline() method returns the next line as a string, including the newline, and returns an empty string at the end of the file. It is useful for reading a header line before looping over the rest.
with open("data.txt", encoding="utf-8") as file:
header = file.readline().rstrip("\n") # 'apple'
rest = [line.rstrip("\n") for line in file] # ['banana', 'cherry']A loop with the walrus operator reads line by line with readline(), as in while (line := file.readline()):. A plain for loop does the same with less code.
2.2. Generator Function
A generator function wraps the reading and cleaning steps and yields one line at a time. The caller decides what to do with each line, and memory use stays constant.
def read_lines(file_path):
with open(file_path, encoding="utf-8") as file:
for line in file:
line = line.strip()
if line and not line.startswith("#"):
yield line
fruits = list(read_lines("data.txt")) # ['apple', 'banana', 'cherry']2.3. Reading Specific Lines with islice()
The itertools.islice() function skips lines without storing them, so reading lines 2 to 3 of a huge file costs little memory.
from itertools import islice
with open("data.txt", encoding="utf-8") as file:
middle = [line.rstrip("\n") for line in islice(file, 1, 3)] # ['banana', 'cherry']2.4. Memory-Mapped Files
The mmap module maps a file into memory, so the operating system loads only the pages we touch. It works with bytes and suits fast searches in large files. Opening the file in “rb” mode with ACCESS_READ is enough for reading, and the last line needs care when the file does not end with a newline.
import mmap
with open("data.txt", "rb") as file, mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as mm:
lines = []
start = 0
while start < len(mm):
end = mm.find(b"\n", start)
if end == -1:
end = len(mm) # last line without a newline
lines.append(mm[start:end].decode("utf-8"))
start = end + 1
# lines = ['apple', 'banana', 'cherry']The with statement closes both the map and the file. The mm.readline() method is a shorter alternative that returns bytes including the newline.
3. Line Numbers, Encodings and Empty Lines
Real files need a few extra touches. The enumerate() function adds line numbers, a check on line.strip() skips blank lines, the encoding argument avoids platform-dependent decoding, and errors=”replace” keeps the loop running when a file has invalid bytes.
with open("data.txt", encoding="utf-8", errors="replace") as file:
numbered = [f"{n}: {line.rstrip()}" for n, line in enumerate(file, start=1) if line.strip()]
# ['1: apple', '2: banana', '3: cherry']Without encoding, Python 3.14 uses the locale encoding, which is UTF-8 on most systems but can differ on Windows, so a file with accented characters can raise UnicodeDecodeError. PEP 686 makes UTF-8 the default from Python 3.15 onward.
3.1. Reading Several Files or Standard Input with fileinput
The fileinput module loops over the lines of several files as one stream, and over standard input when no file is given. Command-line scripts use it to behave like Unix tools.
import fileinput
with fileinput.input(files=["data.txt", "data.txt"], encoding="utf-8") as stream:
total = sum(1 for line in stream) # 6, three lines from each file4. Real-World Example: Counting Errors in a Log File
A server log can grow to gigabytes, so we stream it line by line and keep only the counts. The example counts error lines per module in constant memory.
from collections import Counter
with open("app.log", "w", encoding="utf-8") as f:
f.write("INFO db started\nERROR db timeout\nERROR api 500\nINFO api ok\nERROR db timeout\n")
errors = Counter()
with open("app.log", encoding="utf-8") as log:
for line in log:
if not line.strip():
continue # skip blank lines
level, module, *_ = line.split()
if level == "ERROR":
errors[module] += 1
summary = dict(errors) # {'db': 2, 'api': 1}The same pattern works for CSV files, although the csv module handles quoted fields correctly.
5. Read File Line by Line FAQs
The last line, line counts and line endings raise the remaining questions.
5.1. How do we read the last line of a large file?
A collections.deque(file, maxlen=1) keeps only the last line while streaming. The keep last N items tutorial shows the pattern for the last N lines.
5.2. How do we count the lines of a file?
Opening the file in binary mode and counting the lines in a with block skips text decoding, which helps for large files.
with open("data.txt", "rb") as f:
line_count = sum(1 for _ in f) # 35.3. Why does each line end with \n?
The file object keeps the line ending so that writing the lines back reproduces the file. rstrip(“\n”) or splitlines() removes it.
6. Conclusion
Looping over the file object in a with statement is the right default for reading a file line by line, for small and large files alike. readlines() and splitlines() return lists for small files, islice() reads a range of lines, generators build reusable pipelines and mmap serves byte-level searches, and an explicit encoding keeps the result the same on every platform.
7. References
Happy Learning !!
