Handbooks / Pandas & Data Analysis / Chapter 1

DataFrames and Indexing

7 pages · ~4 min✓ Reviewed

Start here. Next up: Cleaning Data.

Pandas & Data AnalysisPage 1

Series and DataFrame Mental Model

1. Series

1-D labelled array: values + index
One dtype for the whole Series

2. DataFrame

2-D table
Each column is a Series
All columns share one row index

python

import pandas as pd
df = pd.DataFrame({'name': ['A','B'], 'age': [30, 25]})

dict keys become column names

Attribute	Gives
`df.shape`	(rows, cols)
`df.dtypes`	type per column
`df.columns`	column labels
`df.index`	row labels

3. Alignment

s1 + s2 matches by label, not position
Label in only one Series gives NaN

Alignment

Missing labels never raise an error; they give NaN.

Pandas & Data AnalysisPage 2

Loading and Inspecting Data

1. Loading a CSV

pd.read_csv('file.csv') returns a DataFrame
sep=';' sets the delimiter; usecols=[...] keeps only those columns
dtype={...} forces types; parse_dates=['date'] makes datetimes
nrows=1000 reads just the first 1000 rows

python

df = pd.read_csv(
    'file.csv', sep=';', usecols=['id', 'date', 'city'],
    dtype={'id': 'int64'}, parse_dates=['date'],
    nrows=1000)

2. Inspecting It

headlook
infotypes, nulls
describespread
isna().sum()gaps

Call	Shows	Note
`head(n)` / `tail(n)`	first / last n rows	default 5
`info()`	dtypes, non-null, memory	non-null < rows = missing
`describe()`	count, mean, std, min, quartiles, max	numeric only
`describe(include='all')`	adds non-numeric columns	text is `str` in pandas 3
`isna().sum()`	missing values	per column
`df['col'].nunique()`	distinct values	a count
`df['col'].value_counts()`	count per value	most common first

3. Interview Point

How do you spot missing values? info() non-null below row count, or isna().sum()
Which columns does describe() cover? Numeric only; include='all' adds text
How do you load only part of a big CSV? usecols and nrows

Pandas & Data AnalysisPage 3

Selecting Columns, loc vs iloc

1. Picking Columns

df['col'] → a Series
df[['a','b']] (a list) → a DataFrame
Double brackets keep the 2-D shape

2. loc vs iloc

	loc	iloc
Selects by	label	integer position
Slice end	inclusive	exclusive
`df[...][0:2]` rows	`df.loc[0:2]` → 3	`df.iloc[0:2]` → 2

Same pattern for both: [rows, cols]
Examples below: label, position, then all rows with a column range

python

df.loc[5, 'age']       # label
df.iloc[0, 1]          # position
df.loc[:, 'a':'c']     # all rows, cols a to c

3. Assigning

Do: df.loc[df['age'] > 30, 'senior'] = True
Avoid: df[df['age'] > 30]['senior'] = True

Chained indexing

df[...]['col'] = x never changes df. It writes to a temporary copy. Use one loc call.

Pandas & Data AnalysisPage 4

Boolean Filtering and Index Control

1. Boolean Mask

df[df['age'] > 30] builds a boolean Series mask
Keeps only the rows where the mask is True

2. Combine and Test

Op	Meaning
`&`	and
`\|`	or
`~`	not

Wrap each condition in parentheses
With &, both conditions must hold
isin([...]) tests membership
str.contains('an') filters text

python

df[(df.age > 30) & (df.city == 'NYC')]
df[df['city'].isin(['NYC','LA'])]
df[df['name'].str.contains('an')]

3. Index Control

set_index('id') makes a column the index
df.loc[101] then looks up by that label
reset_index(drop=True) restores 0..n-1 after filtering or sorting

python

df = df.set_index('id')
df.loc[101]
df = df.reset_index(drop=True)

Remember

set_index returns a new DataFrame. reset_index returns a new DataFrame; without drop=True the old index becomes a column.

Pandas & Data AnalysisPage 5

Common Mistakes

1. Filter mistakes

and / or on a Series raises ==truth value is ambiguous
Use &, |, ~ instead, with parentheses around each condition
df.age > 30 & df.x == 1 runs in the wrong order and fails

python

df[(df.age > 30) & (df.x == 1)]

2. Writes that vanish

Mistake	Result	Fix
`df[df.a > 1]['b'] = 0`	df unchanged, `ChainedAssignmentError`	`df.loc[mask, 'b'] = 0`
`set_index`, `reset_index`, `drop`	new object, change lost	`df = df.set_index('id')`

3. Index after filtering

The filtered rows keep their original labels
loc uses labels, iloc uses positions: different rows

python

import pandas as pd
df = pd.DataFrame({'a': [1, 5, 9]})
sub = df[df.a > 1]
print(sub.loc[1, 'a'], sub.iloc[1, 0])
sub = sub.reset_index(drop=True)
print(sub.loc[1, 'a'])

output

5 9
9

Remember

Need positions after a filter? reset_index(drop=True). Always assign the result.

Pandas & Data AnalysisPage 6

Pandas Basics Cheat Sheet

1. Load and Peek

Call	Shows
`pd.read_csv(path)`	loads the CSV
`df.head()`	first 5 rows
`df.info()`	dtypes, non-null counts
`df.describe()`	numeric columns only
`describe(include='all')`	adds text (str) columns
`df.shape`	(rows, cols)

2. Columns and Rows

Form	Returns / picks by	Slice end
`df['a']`	Series
`df[['a','b']]`	DataFrame
`df.loc[rows, cols]`	label or mask	inclusive
`df.iloc[r, c]`	position	exclusive

df.loc[0:2] gives 3 rows
df.iloc[0:2] gives 2 rows

3. Filter and Index

python

df[(df.age > 30) & (df.city == 'NYC')]
df[df['city'].isin(['NYC', 'LA'])]
df[~df['city'].isin(['LA'])]

parentheses around each condition; ~ means not

set_index('col') and reset_index(drop=True) return a new DataFrame
Assign it: df = df.set_index('id')
Reset after filtering to get 0..n-1 back

Chained assignment

df[mask]['b'] = 0 never changes df and warns (ChainedAssignmentError). Use df.loc[mask, 'b'] = 0.

Pandas & Data AnalysisPage 7

Check yourself

A DataFrame has index labels [10, 11, 12, 13]. How many rows do df.loc[11:13] and df.iloc[1:3] return?

loc is end-inclusive: 3 rows (11, 12, 13)
iloc is end-exclusive: 2 rows (labels 11 and 12)

Why does df[df.age > 30 & df.x == 1] fail, and what is the fix?

& binds tighter than > and ==, so the parts are combined in the wrong order
Wrap each condition: df[(df.age > 30) & (df.x == 1)]

Ages are [20, 40, 50]. After df2 = df[df.age > 30], what do df2.iloc[0] and df2.loc[0] do?

iloc[0] returns the first remaining row, labelled 1
loc[0] raises KeyError: label 0 was filtered out
Use reset_index(drop=True) if you need positions as labels