Files & JSON

Every variable in a Python program disappears the moment the program ends. Files are how your work survives: a log of what happened, a spreadsheet someone sent you, a settings file, the response saved from a web service. Almost every real program reads data from a file, writes results to one, or does both, so this is one of the first skills that turns small scripts into useful tools.

This chapter starts with the big picture of how Python talks to files, then builds up one skill at a time. You will learn how open() modes and the with statement keep files safe, how to read and write text, and how pathlib treats paths as objects instead of fragile strings. Then come the details that cause most real-world trouble: encodings and UTF-8, the csv module for tabular data, and json for structured data that moves between programs and languages.

By the end you will be able to load a CSV or JSON file, clean it up and save the result. You will handle a missing file with a clear message instead of a crash, and stream a file that is too big for memory one line at a time. You will also recognise the most common file mistakes (forgotten closes, wrong modes, unspecified encodings, hand-built paths) and know the fix for each. The last page is a cheat sheet you can keep open while you work.

Every file operation in Python starts with open(). It does not hand you the contents of the file. It returns a file object, a handle that stands between your code and the operating system. The handle keeps track of three things: the OS file descriptor (the number the operating system uses to identify the open file), a position (where the next read or write happens), and a buffer (a chunk of memory that batches data so Python does not call the OS for every character).

The example below opens a new file for writing and looks at the handle itself. Notice that nothing has been read yet. The handle starts at position 0, and writing moves the position forward.

import os, tempfile

folder = tempfile.mkdtemp()
path = os.path.join(folder, 'notes.txt')

f = open(path, 'w', encoding='utf-8')
print(type(f).__name__)
print(f.mode, f.closed)
print(f.tell())
print(f.write('hello'))
print(f.tell())
f.close()
print(f.closed)

The handle is an object with state: mode, position and a closed flag.

write() returns how many characters it wrote, and tell() reports the position. After close() the handle still exists as a Python object, but closed is True and any further read or write on it raises an error.

The mode string you pass to open() decides what kind of data the handle deals in. Text mode (the default) decodes the bytes on disk into str using an encoding, and translates line endings for you. Binary mode (add a b to the mode) skips all of that and gives you raw bytes exactly as stored.

Here the same file is read both ways. The bytes version shows the b prefix on the result, which is Python's way of telling you it is not a string.

with open(path, 'rb') as f:
    data = f.read()
print(type(f).__name__, data)

with open(path, 'r', encoding='utf-8') as f:
    text = f.read()
print(type(f).__name__, text)

Same file, two views: bytes and str.

Every file handle follows the same life cycle. You open it, you read or write through it, and you close it. Closing matters for two reasons. It hands the file descriptor back to the operating system, which only allows a limited number of open files per process. And it pushes any data still sitting in the buffer out to the disk.

The surprising part is the buffer. When you call write(), the data usually does not reach the file straight away. It waits in memory until the buffer fills up, until you call flush(), or until close() runs. The next example opens the same file with a second handle to prove it. The writer has written text, but a reader sees an empty file until the flush.

w = open(path, 'w', encoding='utf-8')
w.write('saved?')

with open(path, encoding='utf-8') as r:
    print(repr(r.read()))

w.flush()
with open(path, encoding='utf-8') as r:
    print(repr(r.read()))

w.close()

Data written but not flushed is not yet visible in the file.

If the program crashed between the write and the flush, the text would simply be gone. That is how skipping close() can lose data. In practice you rarely call close() by hand. The with statement, covered in the next section, closes the file for you even when an error occurs.

Working with files in Python comes down to three layers. Each later section of this chapter uses one or more of them, so it helps to know which layer solves which problem.

The last example touches all three layers. pathlib builds the path and does the quick read and write, and json converts between a Python dictionary and text. Under the hood pathlib still opens and closes a file handle for you.

from pathlib import Path
import json

p = Path(folder) / 'data.json'
p.write_text(json.dumps({'ok': True}), encoding='utf-8')
print(p.name, p.suffix)
print(json.loads(p.read_text(encoding='utf-8')))

pathlib for the path, json for the format, open() working underneath.

Every call to open() answers one question before anything else: what are you about to do with this file? You answer it with the second argument, the mode. It is a short string of letters, and it decides whether the file must already exist, whether old content survives, and whether you get text or raw bytes.

There are four main modes. Leave the argument out and you get 'r', so open('a.txt') means "read text from a file that must already exist".

Two more letters combine with those. + adds the other direction, so 'r+' can read and write and 'w+' can too. t means text and is the default: you work with str and Python translates line endings and encodes for you. b means binary: you work with bytes and nothing is translated. So 'rb' is read bytes, and 'r' is the same as 'rt'.

The example below runs all four behaviours against a scratch file in a temporary folder. It reads a missing file, creates one with 'w', adds a line with 'a', reads it back, and finally tries 'x' on a file that is now there.

import tempfile, os

folder = tempfile.mkdtemp()
path = os.path.join(folder, 'notes.txt')

try:
    open(path)
except FileNotFoundError:
    print('r: missing file -> FileNotFoundError')

with open(path, 'w') as f:
    f.write('first\n')
with open(path, 'a') as f:
    f.write('second\n')
with open(path) as f:
    print(f.read().splitlines())

try:
    open(path, 'x')
except FileExistsError:
    print('x: file exists -> FileExistsError')

The path variable is reused in later examples.

An open file holds an operating-system resource, and buffered writes may not reach the disk until the file is closed. The old way to be safe was try/finally: open the file, do the work in try, and call f.close() in finally. It works but it is noisy, and it is easy to forget.

The with statement does that job for you. In with open('a.txt') as f: the file is opened, bound to f, and then f.close() is called as soon as the block ends. That includes the case where an exception is raised inside the block. Both versions are shown here, and the second one raises on purpose.

The exception was not swallowed: it still reached the except. What with guarantees is that cleanup runs on the way out. The name g also stays available after the block, and g.closed confirms the file is shut.

This is not a special trick of files. It is the context manager protocol. Any object with an __enter__ method and an __exit__ method can sit after with. Python calls __enter__ first and binds its return value to the name after as. When the block ends, normally or by exception, it calls __exit__. A file object simply has an __exit__ that closes it.

You can watch the protocol with a tiny class of your own. The __exit__ method receives the exception type (or None when all went well). Returning False lets the exception continue to propagate, which is what files do.

The modes differ most in what they do to data that is already in the file. This is where students lose work, so the comparison is worth keeping in your head.

The next example shows all of this on one file. It writes some text, opens it with 'r+' to read and then overwrite the start, appends with 'a', and finally opens with 'w' and writes nothing at all. After r+ and a the content is intact; after w it is empty.

The last check is easy to run yourself. The second close() passes quietly, and only the read fails.

Once open() gives you a file object, you choose how much text to take from it at a time. Python offers four reading methods, and the difference between them is what comes back and how much of the file they consume. The main choice is between one big string, a fixed number of characters, a single line, or a list of every line.

f.read() returns the whole file as one str. f.read(n) returns up to n characters, and fewer if the file runs out first. f.readline() returns one line, and that line keeps its trailing '\n'. f.readlines() returns a list of all the lines, each one still ending in '\n' (except possibly the last, if the file does not end with a newline).

The example below first creates a small file called notes.txt so that you can run it as it stands. Writing is covered properly in a later section, so for now treat that first block as setup. Each read then gets its own with block, so every method starts from the top of a freshly opened file. Printing with repr() makes the invisible '\n' characters visible.

with open('notes.txt', 'w', encoding='utf-8', newline='\n') as f:
    f.write('alpha\nbeta\ngamma\n')

with open('notes.txt', encoding='utf-8') as f:
    print(repr(f.read()))

with open('notes.txt', encoding='utf-8') as f:
    print(repr(f.read(7)))

with open('notes.txt', encoding='utf-8') as f:
    print(repr(f.readline()))
    print(repr(f.readline()))

with open('notes.txt', encoding='utf-8') as f:
    print(f.readlines())

newline='\n' in the setup keeps the file byte-for-byte identical on every operating system

Notice that read(7) stopped in the middle of the second line. It counts characters, not lines, so it happily cuts through a '\n'. Notice also that the two readline() calls returned different lines: each call picks up where the previous one stopped. That behaviour comes from the file position, which is the subject of the next page.

A file object remembers where it is. Think of a cursor that starts at the beginning when you open the file and moves forward with every read. Whatever the cursor has already passed is not returned again. This single idea explains why readline() gives you a new line each time, and why reading a file twice in a row gives a surprising result.

When the cursor reaches the end, you are at EOF (end of file). There is nothing left to read, so read() returns an empty string ''. It does not raise an error. Calling read() a second time on the same open file therefore gives '', because the first call already moved the cursor to the end.

Two methods let you inspect and control the cursor. f.tell() reports the current position, and f.seek(0) jumps back to the start so you can read the content again.

The next example reuses notes.txt from the previous page. Follow the numbers printed by tell(): the file holds 17 characters in total (alpha\n is 6, beta\n is 5 and gamma\n is 6).

The sequence matches the diagram. The first read(6) consumed 'alpha\n' and moved the cursor to 6. The plain read() took everything that remained and moved the cursor to 17. The next read() found nothing and returned ''. After seek(0), readline() returned the first line again.

read() and readlines() both load the entire file into memory before you can use any of it. For a notes file that is perfectly fine. For a log file of several gigabytes it can exhaust your RAM. The alternative is to loop directly over the file object with for line in f:. Python then hands you one line at a time and keeps only that line in memory, so the file can be as large as the disk.

Every line you receive, whichever way you read it, still carries its '\n'. To get clean text, remove it. line.rstrip('\n') removes only trailing newlines, so leading spaces survive. line.strip() removes whitespace from both ends, which is handy for plain words but will also erase meaningful indentation. The example loops over the file, shows the line before and after cleaning, and then builds a clean list with a comprehension. The last lines show why the newline matters in practice.

File reading comes up often in Python interviews because the answers test whether you understand the file position and memory use. Practise saying each answer out loud in one or two sentences, then back it up with a short snippet.

Once a file is open in a writing mode, the main tool is f.write(s). It takes one str, puts it into the file at the current position, and hands back the number of characters it wrote. It adds nothing of its own: no space, no separator, and no newline. If you want your text on separate lines, the '\n' is your job.

The example below makes a scratch folder, writes three pieces, and keeps each return value. Then it reads the file back with repr() so the invisible newline characters show up as \n.

import tempfile
from pathlib import Path

folder = Path(tempfile.mkdtemp())
path = folder / 'notes.txt'

with open(path, 'w', encoding='utf-8') as f:
    a = f.write('alpha')
    b = f.write('beta\n')
    c = f.write('gamma\n')

print(a, b, c)
print(repr(path.read_text(encoding='utf-8')))

write() returns a count and never adds a newline

alpha and beta ran together because the first write had no newline. The counts are 5, 5 and 6 because 'beta\n' is four letters plus the newline, and 'gamma\n' is five plus the newline. The return value is counted in characters, not bytes, so a non-ASCII character still counts as one.

f.writelines(items) takes any iterable of strings and writes them one after another. Despite its name, it also adds no newlines. It is only a shortcut for a loop of write() calls. Include the '\n' in each item, or add it as you generate them.

lines = ['red', 'green', 'blue']

with open(path, 'w', encoding='utf-8') as f:
    f.writelines(lines)
print(repr(path.read_text(encoding='utf-8')))

with open(path, 'w', encoding='utf-8') as f:
    f.writelines(line + '\n' for line in lines)
print(repr(path.read_text(encoding='utf-8')))

writelines() glues items together unless they carry their own newline

If you would rather not manage newlines, send print() to the file with its file= argument. print('text', file=f) writes the text and then a newline, exactly as it would on screen. It also converts non-strings for you, and you can still use sep= and end=.

write() is stricter. It accepts only a str. Handing it a number raises TypeError, so convert first with str(). The example mixes all three styles in one file.

with open(path, 'w', encoding='utf-8') as f:
    print('first line', file=f)
    print('total:', 42, file=f)
    f.write(str(42))
    f.write('\n')
    try:
        f.write(42)
    except TypeError as err:
        print(type(err).__name__)

print(path.read_text(encoding='utf-8'), end='')

print adds the newline; write needs a str

The TypeError line comes first only because it is printed to the screen while the file text is read back at the end. The failed f.write(42) wrote nothing, so the file holds just the three lines before it.

Everything so far used mode 'w', which empties the file when it opens. To add to the end of an existing file instead, open it with 'a' (append). The file is created if it does not exist, and every write lands after the current content.

with open(path, 'w', encoding='utf-8') as f:
    f.write('one\n')

with open(path, 'a', encoding='utf-8') as f:
    f.write('two\n')
    print('three', file=f)

print(path.read_text(encoding='utf-8'), end='')

'a' keeps what is there and writes after it

Writing to a disk one tiny piece at a time would be slow, so Python collects your text in a buffer in memory and hands it over in larger chunks. A write() call that returns successfully therefore does not mean the text has reached the file yet.

f.flush() pushes the buffer to the operating system on demand. Leaving the with block does that and also closes the file. The next example checks the file size from outside before and after a flush.

f = open(path, 'w', encoding='utf-8')
f.write('saved?')
print(path.stat().st_size)
f.flush()
print(path.stat().st_size)
f.close()

the text only reaches the file after flush

Before the flush the file was still empty, even though write() had already returned 6. If the program had been killed at that moment, those six characters would have been lost. Buffered data lives only in your process, and a crash or power cut gives it no chance to be written.

Until now a path has been a plain string such as 'data/in/a.txt'. That works, but a string knows nothing about folders, extensions or separators. The pathlib module wraps a path in a Path object that does know. You build one by starting from Path('data') and joining pieces with the / operator, and Python picks the separator for the operating system it runs on: a backslash on Windows, a forward slash on Linux and macOS.

At least one side of the / must be a Path. Path('data') / 'in' works, and so does 'data' / Path('a.txt'), because the Path object handles the operator from either side. Two plain strings, 'data' / 'a.txt', raise a TypeError, since strings do not support /. In practice you create one Path at the start and keep joining from it.

Once you have a Path you can read its pieces as properties. .name is the last part, .stem is the name without its final suffix, .suffix is that final extension including the dot, .parent is the folder that holds it, and .parts is a tuple of every component. The method .with_suffix('.csv') returns a new Path with the extension swapped. It does not rename anything on disk. The example prints .as_posix() so the output uses forward slashes on every machine.

from pathlib import Path

p = Path('data') / 'in' / 'a.txt'
print(p.as_posix())
print(p.name)
print(p.stem)
print(p.suffix)
print(p.parent.as_posix())
print(p.parts)
print(p.with_suffix('.csv').as_posix())

print(('data' / Path('a.txt')).as_posix())
try:
    'data' / 'a.txt'
except TypeError:
    print('TypeError')

Build a path, read its pieces, and see which operands the / operator accepts

Building a Path never touches the disk, so the folders in it do not need to exist yet. Questions about the disk are separate methods. .exists() says whether anything is there, .is_file() and .is_dir() say which kind it is, and .resolve() turns a relative path into an absolute one, following any .. parts and links. The next page uses the first three.

A Path can read and write its own file. p.write_text(s, encoding='utf-8') opens the file, writes the string and closes it in a single call. p.read_text(encoding='utf-8') opens, reads everything and closes. There are no handles to forget about, so no with block is needed. Always pass encoding='utf-8' yourself, as in the earlier sections, so the result does not depend on the machine's default.

For raw bytes there is a matching pair. p.write_bytes(b) and p.read_bytes() skip text decoding entirely, which is right for images, archives and anything that is not text. When you need a real file object, for example to loop over lines, p.open() is a drop-in for open(p). It takes the same mode and encoding arguments and works in a with block.

The example below runs inside a temporary folder that disappears afterwards, so you can run it without leaving files behind. It writes one accented word, checks the file with .exists(), .is_file() and .is_dir(), then reads it back as text and as bytes. Printing with ascii() keeps the output plain even on a terminal that cannot show accents.

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    base = Path(tmp)
    note = base / 'note.txt'
    print(note.exists())
    note.write_text('h\u00e9llo', encoding='utf-8')
    print(note.exists(), note.is_file(), note.is_dir())
    print(ascii(note.read_text(encoding='utf-8')))
    print(note.read_bytes())

    blob = base / 'blob.bin'
    blob.write_bytes(b'\x00\x01\x02')
    print(blob.read_bytes())

    with note.open(encoding='utf-8') as f:
        print(len(f.read()))

Text helpers, byte helpers and p.open() on a temporary file

The bytes line shows why the encoding matters. The single character é is stored as two bytes, \xc3\xa9, in UTF-8, yet the text read back has five characters. read_text did the decoding for you, and it can only decode correctly if you told it the right encoding.

The same object also manages directories. p.mkdir(parents=True, exist_ok=True) creates a folder safely. parents=True creates any missing folders along the way, so reports/2026 works even when reports does not exist yet. exist_ok=True stops Python raising an error when the folder is already there, so running the line twice is harmless. Without those two flags, mkdir fails with FileNotFoundError for a missing parent and FileExistsError for an existing folder.

To look inside a folder, p.iterdir() yields every child, files and folders alike, in no promised order. p.glob('*.csv') yields only the children matching a pattern. p.rglob('*.py') does the same but searches recursively through every subfolder. All three give Path objects, not strings, and they produce results lazily, so wrap them in sorted(...) or list(...) when you need a stable order. The methods p.unlink() and p.rename(new) delete a file and move or rename it.

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    root = Path(tmp)
    out = root / 'reports' / '2026'
    out.mkdir(parents=True, exist_ok=True)
    out.mkdir(parents=True, exist_ok=True)

    (out / 'sales.csv').write_text('a,b', encoding='utf-8')
    (out / 'costs.csv').write_text('c,d', encoding='utf-8')
    (out / 'notes.txt').write_text('x', encoding='utf-8')
    (out / 'deep.py').write_text('', encoding='utf-8')
    (root / 'tool.py').write_text('', encoding='utf-8')

    print(sorted(c.name for c in out.iterdir()))
    print(sorted(c.name for c in out.glob('*.csv')))
    print(sorted(c.name for c in root.rglob('*.py')))

    (out / 'notes.txt').rename(out / 'notes.md')
    (out / 'costs.csv').unlink()
    print(sorted(c.name for c in out.iterdir()))

Create nested folders, list and search them, then rename and delete

Notice that the second mkdir call did nothing and raised no error. glob('*.csv') found only the two CSV files in out, while rglob('*.py') also reached deep.py inside the nested folders from the root. After the rename and the delete, three files remain.

Older code builds paths from strings. os.path.join(a, b) joins pieces with the right separator, and plain concatenation such as 'data' + '/' + 'in' hard-codes one. Both work on strings, so every extra step needs another function: os.path.basename, os.path.splitext, os.path.exists, and so on, all wrapped around the same string. pathlib is object-based: the path carries its own methods, you chain them left to right, and the code reads like a description of what you want.

Windows paths bring a second trap. In an ordinary string literal the backslash starts an escape sequence, so 'C:\new\table.txt' contains a real newline after C: and a real tab before able. The path is silently wrong. Prefix the literal with r to make a raw string, or let Path build the path for you. The example uses PureWindowsPath, which formats paths the Windows way on any machine, so its output is the same everywhere.

from pathlib import PureWindowsPath

bad = 'C:\new\table.txt'
good = r'C:\new\table.txt'
print('\n' in bad, '\t' in bad, len(bad), len(good))

win = PureWindowsPath(good)
print(win.name, win.parent)
print(PureWindowsPath(r'C:\data') / 'in' / 'a.txt')

Escape characters in a plain Windows path string

The plain string is two characters shorter, 14 against 16, because \n and \t each collapsed into a single control character. The raw string kept every backslash.

Why prefer pathlib over os.path? Because a Path is an object that bundles the path with its operations, the / operator handles separators for any OS, and the same code reads, writes, searches and renames files without a pile of string functions. What does p.stem return for 'a.tar.gz'? Only a.tar, because .stem strips just the last suffix and .suffix is .gz. The full list of extensions is in .suffixes. How do you create nested folders safely? With mkdir(parents=True, exist_ok=True). What does Path('.') resolve to? The current working directory as an absolute path, the same as Path.cwd(). The example checks the last three.

from pathlib import Path

q = Path('backup/a.tar.gz')
print(q.stem)
print(q.suffix)
print(q.suffixes)

print(Path('.'))
print(Path('.').resolve() == Path.cwd())

Answers to the interview questions, checked in code

A file on disk holds only bytes, which are numbers from 0 to 255. Python's str holds text, which is a sequence of characters. An encoding is the rule that turns one into the other. str.encode() goes from text to bytes, and bytes.decode() goes back. When you call open() in text mode, Python does this conversion for you on every read and write.

UTF-8 is the encoding you should use almost everywhere. It is variable length: each character takes between one and four bytes. Plain ASCII characters take 1 byte, most accented Latin letters take 2, many other scripts and symbols such as the euro sign take 3, and emoji take 4. For example, '\u00e9'.encode('utf-8') gives b'\xc3\xa9', two bytes for one character.

The example below encodes four characters and prints how many bytes each one needs. It writes the characters as escapes and prints them with ascii(), so the output is plain ASCII and looks the same on every machine.

samples = ['A', '\u00e9', '\u20ac', '\U0001f600']
for ch in samples:
    data = ch.encode('utf-8')
    print(ascii(ch), len(data), data)

One character, one to four bytes

If you call open() without encoding=, Python picks the platform default, which comes from locale.getpreferredencoding(). On many Windows machines that is cp1252, while on Linux and macOS it is usually UTF-8. The same script can therefore read the same file in two different ways on two different computers, and nothing warns you.

The next example writes a file as UTF-8 and then reads it twice, once with the right encoding and once with cp1252. The wrong read does not fail. It quietly produces mojibake, which is garbled text, because every byte of the two-byte letter is read as its own character.

import tempfile
from pathlib import Path

path = Path(tempfile.gettempdir()) / 'enc_demo.txt'
path.write_text('caf\u00e9', encoding='utf-8')

print(path.read_bytes())
print(ascii(path.read_text(encoding='utf-8')))
print(ascii(path.read_text(encoding='cp1252')))

Same bytes, two readings (shown with ascii() so the output is plain ASCII)

The bytes on disk are the same in all three lines. The second line is the correct word. The third line has five characters instead of four, because \xc3 and \xa9 were treated as two separate cp1252 characters. The fix is one short habit: always write encoding='utf-8' in every open(), read_text() and write_text() call.

A UnicodeDecodeError means the bytes in the file are not valid for the encoding you asked for. It is common with files saved by older Windows programs, which often use cp1252 or latin-1. You have three options: make the decoder forgiving, accept some loss, or find the real encoding.

Here the bytes come from a Latin-1 file. UTF-8 refuses them, and each option then gives a different result. The printed text uses ascii() so the replacement character shows up as the escape \ufffd.

data = b'caf\xe9 au lait'

try:
    data.decode('utf-8')
except UnicodeDecodeError as e:
    print(e)

print(ascii(data.decode('utf-8', errors='replace')))
print(ascii(data.decode('utf-8', errors='ignore')))
print(ascii(data.decode('latin-1')))

One byte, four outcomes

Only the last line recovers the real word. Prefer finding the true encoding over replace or ignore, and use ignore only when losing data is acceptable.

A BOM (byte order mark) is a few special bytes at the very start of a file. In UTF-8 it is EF BB BF. Excel on Windows looks for it to recognise a CSV as UTF-8, so a file without it may show accented letters wrongly. The codec 'utf-8-sig' writes the BOM when writing and removes it when reading. Plain 'utf-8' leaves it in as a stray character \ufeff.

path.write_text('caf\u00e9', encoding='utf-8-sig')
print(path.read_bytes())
print(ascii(path.read_text(encoding='utf-8')))
print(ascii(path.read_text(encoding='utf-8-sig')))

Reuses path from the earlier example

Line endings are the last encoding-related detail. In text mode, Python turns each '\n' you write into the operating system's line ending, which is \r\n on Windows. The csv module writes its own line endings, so you pass newline='' to stop Python translating them a second time and producing blank rows. The example sets newline explicitly so the result is the same on every system.

with open(path, 'w', encoding='utf-8', newline='') as f:
    f.write('a\nb\n')
print(path.read_bytes())

with open(path, 'w', encoding='utf-8', newline='\r\n') as f:
    f.write('a\nb\n')
print(path.read_bytes())

newline='' writes \n untouched; newline='\r\n' converts it

A CSV file looks like plain text with commas, so it is tempting to read it with line.split(','). That works until a field contains a comma itself, such as the name Smith, John. The csv module exists to handle exactly these cases: it knows the quoting rules, so you get back the fields the file's author meant.

csv.writer(f) wraps an open file. Call writerow(row) to write one list, or writerows(rows) to write many lists at once. When a value contains the delimiter, a quote or a newline, the writer wraps it in double quotes for you.

import csv
import tempfile
from pathlib import Path

folder = Path(tempfile.mkdtemp())
path = folder / "people.csv"

rows = [
    ["name", "city", "age"],
    ["Smith, John", "Leeds", "34"],
    ["Ana Ruiz", "Lima", "29"],
]

with open(path, "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(rows[0])
    writer.writerows(rows[1:])

print(path.read_text(encoding="utf-8"), end="")

The comma inside Smith, John is protected by quotes

Reading is the mirror image. csv.reader(f) is an iterator: each step gives you one row as a list of strings. Because it understands the quotes, "Smith, John" comes back as a single field, while a plain split(',') tears it in two.

with open(path, newline="", encoding="utf-8") as f:
    for row in csv.reader(f):
        print(row)

line = '"Smith, John",Leeds,34'
print(line.split(","))

reader keeps the name whole; split breaks it

Counting positions gets tedious. csv.DictReader(f) takes the first row as the header and yields each later row as a dict, so you write row['name'] instead of row[0]. For writing, csv.DictWriter(f, fieldnames=[...]) needs the column names up front, and you must call writeheader() before the first writerow(dict), or the header line never appears.

fields = ["name", "city", "age"]
with open(path, "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=fields)
    writer.writeheader()
    writer.writerow({"name": "Smith, John", "city": "Leeds", "age": 34})
    writer.writerow({"name": "Ana Ruiz", "city": "Lima", "age": 29})

total = 0
with open(path, newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        print(row["name"], type(row["age"]).__name__)
        total += int(row["age"])
print("total age:", total)

Values go in as numbers but come back as str

One habit goes with every CSV file you open: pass newline='' (and encoding='utf-8'). The csv module handles line endings itself. If Python also translates them, writing on Windows can add a blank line between rows, and quoted fields that contain a newline can be split wrongly when read back.

import io

old = "name,age\nSmith,34\n"
new = "age,name\n34,Smith\n"
for text in (old, new):
    first = list(csv.reader(io.StringIO(text)))[1]
    named = next(csv.DictReader(io.StringIO(text)))
    print("reader:", first[0], "| DictReader:", named["name"])

tsv = "a\tb\n1\t2\n"
for row in csv.reader(io.StringIO(tsv), delimiter="\t"):
    print(row)

The same code after the columns swap, then a tab-separated file

After the columns swap, positional access quietly returns the age while DictReader still finds the name. For tab-separated files (TSV), pass delimiter='\t'. The csv module reads one row at a time and needs no install, so it suits light, streaming work. For heavy analysis such as grouping, joining or statistics, reach for pandas.

JSON is the format most programs use to exchange structured data: settings files, API responses, saved game state. Python's standard json module converts between Python objects and JSON text. It gives you four functions, and the only choice is where the text lives: in a file object or in a string.

Start with the string functions, because you can see their result directly. dumps turns a dictionary into JSON text, and loads turns that text back into an equal dictionary. Notice that Python's True and None appear in the text as true and null.

import json

data = {"city": "Zurich", "ratings": [5, 4], "open": True, "owner": None}
text = json.dumps(data)
print(text)
print(json.loads(text) == data)

Round trip through a string

The file versions do the same job but take an open file as the second argument (for dump) or the only argument (for load). Open JSON files in text mode and pass encoding="utf-8", as you learned in the encodings section. The example below uses a temporary directory so it leaves nothing behind.

from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as tmp:
    path = Path(tmp) / "city.json"
    with open(path, "w", encoding="utf-8") as f:
        json.dump(data, f)
    with open(path, encoding="utf-8") as f:
        loaded = json.load(f)
print(loaded["ratings"])

dump writes to a file, load reads it back

JSON has only a handful of types, and each one maps onto a Python type. When you load a document, you get the Python side of this table. When you dump, Python values are converted the other way.

doc = json.loads('{"a": 1, "b": 2.5, "c": true, "d": null, "e": [1, 2], "f": "x"}')
print({k: type(v).__name__ for k, v in doc.items()})

Loading gives you the Python types from the table

Python has many types that JSON does not. A tuple is accepted but is written as an array, so it comes back as a list: JSON has no way to say "this array was a tuple". Other types, such as set, datetime and your own classes, are not accepted at all.

The first branch is the easy one: basic types are written exactly as the table shows. Only values that fall through to the second question need attention. The next example shows both outcomes: the tuple survives as a list, and the set raises a TypeError.

print(json.dumps((1, 2)))
print(json.loads(json.dumps((1, 2))))

try:
    json.dumps({1, 2})
except TypeError as err:
    print(err)

A tuple turns into a list; a set is rejected

You have two ways out. Convert the value yourself before dumping (for example sorted(my_set) gives a list), or pass default=str. The default function is called for every value JSON cannot handle, and its return value is written instead. With str, a date becomes its text form.

from datetime import date

record = {"day": date(2026, 1, 5), "tags": sorted({"b", "a"})}
print(json.dumps(record, default=str))

default=str handles the date; the set was converted first

By default dump writes everything on one line with no spacing choices, which is compact but hard to read. Three options fix that. indent sets how many spaces to indent nested levels, which spreads the document over many lines. sort_keys writes dictionary keys in alphabetical order, which makes files stable and easy to compare. ensure_ascii controls non-ASCII characters: the default True escapes them as \uXXXX, while False writes them as they are.

The first example uses indent and sort_keys. The name contains a ü, and with the default ensure_ascii=True it is written as an escape. That keeps this printed output pure ASCII.

data = {"name": "Zürich", "ratings": [5, 4], "a": 1}
print(json.dumps(data, indent=2, sort_keys=True))

indent=2 and sort_keys=True

Escapes are valid JSON, but a person opening the file sees Z\u00fcrich. With ensure_ascii=False the file holds the real character, so you must open it with encoding="utf-8" when writing. To see the difference without depending on your terminal's encoding, the next example prints the encoded bytes: ü becomes the two UTF-8 bytes \xc3\xbc.

raw = json.dumps({"name": "Zürich"}, ensure_ascii=False)
print(raw.encode("utf-8"))

ensure_ascii=False keeps the character itself

A common way to combine all three for a file you want people to read is json.dump(data, f, indent=2, ensure_ascii=False, sort_keys=True).

There is one more quirk. JSON object keys are always strings, so Python dictionary keys of other types are converted to strings on the way out. They stay strings when you load the file again.

print(json.loads(json.dumps({1: "a"})))

The int key 1 comes back as '1'

A JSON file holds one document. To read it, json.load must parse the whole thing and build the full object in memory. That is fine for settings or a small export. It is a poor fit for logs or event data that grows forever, because you cannot append one more record to a valid JSON array without rewriting the closing bracket.

JSON Lines (often saved as .jsonl) solves this by writing one complete JSON object per line. To add a record you append a line. To read, you walk the file line by line and call json.loads on each line, so only one record is in memory at a time.

with TemporaryDirectory() as tmp:
    path = Path(tmp) / "events.jsonl"
    with open(path, "w", encoding="utf-8") as f:
        for i in range(1, 4):
            f.write(json.dumps({"id": i, "ok": i != 2}) + "\n")

    good = 0
    with open(path, encoding="utf-8") as f:
        for line in f:
            record = json.loads(line)
            if record["ok"]:
                good += 1
print(good)

Write one object per line, then stream it back

Each line is written with json.dumps (never with indent, since a record must fit on one line) and ends with \n. Reading is the same line-by-line loop you used for text files.

Files live outside your program, so they can fail in ways your code cannot control. A path may be wrong, a folder may be locked, or a file may hold bytes that are not valid text. Python reports each of these problems as a specific exception. If you learn the names, you can catch exactly the problem you can fix and let the rest crash loudly.

Four of the common ones belong to one family. FileNotFoundError, PermissionError, IsADirectoryError and FileExistsError are all subclasses of OSError. Writing except OSError therefore catches every one of them. The two content errors, UnicodeDecodeError and json.JSONDecodeError, are different. They come from bad data, not from the operating system, and both are kinds of ValueError.

You can check this family tree yourself. The program below asks Python for the parent of FileNotFoundError and tests each class against OSError.

Next, here is each error raised on purpose. A temporary folder keeps the demo safe to run anywhere. The program prints only the exception name, which is the part you will write in your own except clauses.

The basic way to handle a missing file is to put the with open(...) block inside a try and name the exception you expect. If the file is there, the try body runs normally. If not, Python jumps straight to the except clause and your program carries on.

There is a catch that surprises many learners. A relative path such as "note.txt" is not resolved against your script's folder. Python resolves it against the current working directory, which is the folder your terminal was in when you started the program. The same script with the same path can succeed in one terminal and fail in another.

The program below makes the effect visible. It builds two folders and puts note.txt in only one of them. It then changes the working directory and calls the same function twice.

To make a path independent of the terminal, anchor it to the script itself. The built-in __file__ holds the location of the running script, so Path(__file__).parent / "data.txt" always points at data.txt beside the script, no matter where you launched it. Use Path(__file__).resolve().parent if you also want an absolute path.

There are two styles for dealing with a file that might not exist. LBYL means Look Before You Leap. You check first with if p.exists(): and open only when the answer is yes. EAFP means Easier to Ask Forgiveness than Permission. You simply try to open the file and handle the exception if it fails. Python programmers prefer EAFP, and for files there is a concrete reason.

The weakness of LBYL is a race condition. Between the moment you ask whether the file exists and the moment you open it, another program, a cleanup job or the user can delete it. The check gave a true answer that went stale. The program below imitates that gap by deleting the file right after the check.

Since you need the try anyway, the exists() check adds nothing. It only gives a false feeling of safety. Catch the narrowest exception you actually know how to handle. FileNotFoundError says exactly what you mean, while a wide net such as except Exception also swallows problems you never planned for. If you want both a specific and a general handler, put the specific one first, because Python uses the first clause that matches.

Catching the error is only half the job. You also need to decide what a missing file means for your program. There are three honest answers, and the right one depends on the file's role.

A default on missing is right for optional files such as a config. A first run has no config.json yet, so the function returns an empty dictionary. A create on missing is right when you are about to write. The file will not exist yet, and neither might its folder, so call p.parent.mkdir(parents=True, exist_ok=True) before writing. When the file is truly required, you can do nothing useful locally. Log and re-raise with a bare raise, which keeps the original traceback for whoever can handle it.

Look at how the pieces fit. load_config returns {} on the first call, because the folder does not exist yet. save_config creates the app folder and the file in one step. read_report logs the problem and then re-raises, so the caller still sees the original FileNotFoundError.

A file object is an iterator over its lines. When you write for line in f:, Python reads a block of bytes into an internal buffer, hands you one line at a time, and refills the buffer when it runs dry. Lines you have already processed are dropped. Memory use therefore stays roughly constant whether the file is 1 MB or 100 GB. That is O(1) memory and O(n) time, because every line is visited once.

The opposite habit is to pull everything in at once. f.read() returns one giant string and f.readlines() returns a list holding every line, so both cost O(file size) memory. On a multi-gigabyte file that can exhaust RAM, slow the machine to a crawl, or crash the program with a MemoryError.

To try the examples below, the first one builds a small log file in a temporary folder. Later examples reuse the names work and log from it.

import tempfile
from pathlib import Path

work = Path(tempfile.mkdtemp())
log = work / "app.log"
with open(log, "w", encoding="utf-8") as f:
    for i in range(1, 11):
        level = "ERROR" if i % 4 == 0 else "INFO"
        f.write(f"{i:03d} {level} request {i}\n")

count = 0
with open(log, encoding="utf-8") as f:
    for line in f:
        count += 1
print(count)

Iterating counts lines without ever holding more than one of them

For contrast, here is what readlines() does with the same file. It builds a list with every line in it. That is harmless for ten lines and dangerous for ten million.

Because the loop yields one line at a time, a generator expression can count matches directly from the file. sum(1 for line in f if 'ERROR' in line) adds one for every matching line and never builds a list.

When you need the matching lines themselves, resist the urge to collect them in a list. Write each result to a second file the moment you find it. Memory stays flat on the output side too, and if the program dies halfway you still have everything written so far. Opening the source with errors='replace' is a good habit for messy logs, because a few bad bytes then turn into a replacement character instead of stopping the whole run.

errors_file = work / "errors.txt"
written = 0
with open(log, encoding="utf-8", errors="replace") as src, open(errors_file, "w", encoding="utf-8") as out:
    for line in src:
        if "ERROR" in line:
            out.write(line)
            written += 1
print(written)
print(errors_file.read_text(encoding="utf-8"), end="")

Read one line, test it, write it, move on

Here is what errors='replace' does in practice. The file below contains a byte, 0xff, that is not valid UTF-8. With the default strict setting the loop would raise UnicodeDecodeError. With replace the bad byte becomes the character U+FFFD and the loop carries on.

Not every file is made of lines. A video, an archive or a database dump may have no newlines at all, or one enormous line. For binary or unstructured data, read fixed-size chunks instead. The walrus operator makes the loop compact: while chunk := f.read(1024*1024): process(chunk) reads up to 1 MiB, stops when read returns an empty value at end of file, and keeps memory bounded by the chunk size. The example uses a small 4096-byte chunk so the output is short.

blob = work / "data.bin"
blob.write_bytes(b"x" * 10_000)
total = 0
with open(blob, "rb") as f:
    while chunk := f.read(4096):
        total += len(chunk)
        print(len(chunk))
print(total)

The last chunk is simply whatever is left

Several processing steps can be chained without building any intermediate list by writing each step as a generator function that yields lines. A generator produces its next value only when the loop asks for it, so a line travels through the whole pipeline before the next line is even read from disk.

def clean_lines(f):
    for line in f:
        line = line.strip()
        if line:
            yield line

def only_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

with open(log, encoding="utf-8") as f:
    for line in only_errors(clean_lines(f)):
        print(line)

Each function is small and testable, and none of them builds a list

Streaming is not always better. For a small file, reading everything at once is shorter and just as fast. The comparison below shows when each approach is the right tool.

Two formats stream naturally. The csv reader is itself an iterator, so you can loop over it row by row. A file with one JSON object per line (often called JSON Lines) can be read with json.loads on each line. In contrast, a single whole-file json.load must read and parse the entire document before it returns anything, so it does not stream.

import csv
import json

events = work / "events.jsonl"
events.write_text('{"user": "a", "ms": 120}\n{"user": "b", "ms": 340}\n{"user": "a", "ms": 80}\n', encoding="utf-8")
total_ms = 0
with open(events, encoding="utf-8") as f:
    for line in f:
        total_ms += json.loads(line)["ms"]
print(total_ms)

sales = work / "sales.csv"
sales.write_text("item,qty\npen,3\nbook,5\npen,4\n", encoding="utf-8")
with open(sales, newline="", encoding="utf-8") as f:
    print(sum(int(row["qty"]) for row in csv.DictReader(f)))

Both loops hold only one record at a time

Most file bugs are not exotic. They come from a handful of habits that look fine until the day they cost you data. The first and most painful is opening a file with 'w' when you meant to keep what was in it. Mode 'w' truncates the file to zero bytes the moment open() returns, before you write a single character. Even if your program crashes on the very next line, the old contents are already gone.

The fix is to pick the mode that matches your intent. Use 'a' to add to the end, and use 'x' when overwriting would be a bug, so Python refuses instead of silently destroying data. The example below keeps a note, appends to it, shows 'x' refusing, and then shows what a careless 'w' does.

Sometimes you really do need to rewrite a whole file, for example saving a settings file. Writing straight over the original is risky because a crash halfway leaves a half-written file. The safe pattern is to write the complete new content to a temporary file next to the original, close it, and only then swap it into place with Path.replace. The swap is a single rename, so readers see either the whole old file or the whole new one, never a mixture.

If the program dies while writing the temp file, the original is still intact and you only have a stray .tmp file to delete. After a successful run no temp file is left, because replace moved it over the original.

The next group of mistakes is about things you forgot to say. Each one works on your machine most of the time, which is exactly why they survive until someone else runs your code.

The csv module handles line endings itself, so the file object must not translate them as well. With newline='' the writer's \r\n goes to disk exactly as written. Without it, Windows turns that into \r\r\n, which shows up as an empty row in spreadsheets.

An open file keeps a position, like a cursor. read() moves it to the end, so a second read() starts at the end and finds nothing. It returns '' rather than raising an error, which makes this bug quiet and confusing. You can move the cursor back with seek(0), or simply open the file again.

The same thing happens when you loop over a file twice with for line in f. The second loop sees no lines. Either call seek(0) between them, or read the lines into a list once if the file is small enough.

Files hold text, and Python does not guess what that text means. Three conversions trip people up again and again.

• Every value read by the csv module is a str, even when it looks like a number. '10' + '5' joins the strings into '105'. Convert with int() or float() first.
• write() accepts only str. Passing an int raises TypeError, so convert with str(n) or an f-string.
• JSON object keys are always strings. A dict with key 1 comes back from a dump and load round trip with key '1', so data[1] fails with KeyError.

There is also the newline. When you iterate over a file, each line keeps its trailing '\n'. Calling line.strip() on a last line that has no newline is harmless: it simply returns the same text. Forgetting to strip is the real danger, because the stray newline stays in your data and makes comparisons fail. The last lines of the example below show a line that looks equal to 'yes' but is not until it is stripped.

The last four mistakes are about structure. Building paths by gluing strings with + or backslashes breaks on other operating systems and invites typos. Path objects join with /, work on every platform, and give you .name, .suffix and .parent for free. A second trap is relying on the working directory: a relative path like 'data.txt' means whatever folder the program was started from. Anchor your paths to the script instead, for example Path(__file__).resolve().parent / 'data.txt'.

Loading a huge file with read() pulls every byte into memory at once. A file object is already an iterator over lines, so a for loop holds one line at a time. Finally, wrapping a whole block in except Exception hides real bugs such as typos and wrong types behind a vague message. Catch only the error you expect, like FileNotFoundError, and keep the try block small.

Every file operation starts with the mode you pass to open(). The mode decides whether the file may be read, written, created or erased, so choosing it carelessly is the quickest way to lose data. The table below is the whole vocabulary; modes combine, so rb means read bytes and w+ means write and read.

The habit to memorise is one line: with open(p, 'r', encoding='utf-8') as f:. The with block closes the file for you even when an error happens, and the explicit encoding stops your program from depending on the machine it runs on. Leave the encoding out only for binary mode.

Reading has four styles. f.read() returns everything as one string, f.readline() returns one line including its newline, f.readlines() returns a list of all lines, and looping with for line in f hands you one line at a time lazily. Writing has three: f.write(str) writes one string and does not add a newline, f.writelines(list) writes each string in a list without adding newlines either, and print(x, file=f) writes like print does, newline included.

A file keeps a position, and each read moves it forward. That is why a second f.read() gives an empty string, and why f.seek(0) is the way to start over. The example writes a file three ways, then reads it every way.

pathlib treats a path as an object instead of a string. The / operator joins parts, so Path(a) / b works the same on Windows and Linux. A path can read and write its own file with .read_text() and .write_text(), create folders with .mkdir(parents=True, exist_ok=True), list matches with .glob(), and tell you its parts through .suffix and .stem.

For CSV, open the file with newline='' so the csv module controls line endings itself, then use csv.reader (rows as lists), csv.DictReader (rows as dicts keyed by the header), csv.writer or csv.DictWriter. Everything read from a CSV is a str, even numbers. For JSON, json.load(f) and json.dump(obj, f, indent=2, ensure_ascii=False) work on open files, while json.loads and json.dumps work on strings. Setting ensure_ascii=False keeps accented letters readable instead of escaping them.

Python prefers EAFP: try the operation and handle the failure, rather than checking first and racing against the file system. Catch FileNotFoundError for a missing file and the broader OSError for permissions, folders and disk problems. Put the specific handler first. For big files, never call read() on the whole thing: iterate over lines, or read fixed-size chunks with a loop, so memory stays flat.

Interviewers like this trio because each part tests a different habit. Start with the function: read the CSV with DictReader, then save the rows as JSON. The version below also answers the second question, because it never writes straight onto the destination.

The third question is about diagnosis. When a read fails, work through five checks in order, and most failures show up in the first two.

Work out each answer before you open it. These questions ask you to predict what code does or to find the bug, not to recite definitions.

• Open files with with open(p, encoding='utf-8') so they close even when an error is raised, and so the text decodes the same on every machine.
• Mode 'w' wipes the file the moment it opens; use 'a' to add, 'x' to refuse to overwrite.
• Reading moves the position forward, so a second read() returns '' until you seek(0); iterate with for line in f to stream big files.
• write() takes only str and adds no newline; use pathlib (Path(a) / b, read_text, write_text) to build and handle paths.
• CSV needs newline='' and gives back only strings; JSON turns int keys into strings and tuples into lists.
• Catch the narrowest exception (FileNotFoundError, or OSError for the family) instead of checking first or using a bare except.