Handbooks / Pandas & Data Analysis / Chapter 5

Time Series

7 pages · ~4 min✓ Reviewed

Pandas & Data AnalysisPage 1

Datetime Mental Model

1. Three Types, One Index

Type	Is	dtype
`Timestamp`	one point in time	`datetime64[unit]`
`DatetimeIndex`	index of Timestamps	`datetime64[unit]`
`Timedelta`	difference of two	`timedelta64[unit]`

Unit: ns in pandas 2, us inferred in pandas 3
Time series work = a DatetimeIndex, sorted and unique-ish
It unlocks slicing, resample and rolling('7D')

2. Strings and NaT

Dates arrive as strings: str dtype in pandas 3, object before
Parse first: nothing time-aware works on string columns
NaT is the NaN of dates
NaT propagates through arithmetic; most aggregations skip it

python

df['date'] = pd.to_datetime(df['date'])

parse before anything else

3. Mixing Naive and Aware

Operation on naive vs aware	Result
naive `==` aware	False, no error
naive `<` `>` aware	TypeError
naive `-` aware	TypeError

Check first

Naive has no tz; aware carries one. Run df.dtypes and df.index.tz before any time work.

Pandas & Data AnalysisPage 2

Parsing Dates and Building a DatetimeIndex

1. Parsing Basics

pd.to_datetime(s) parses strings, ints, lists → datetime64
Series in → Series; list in → DatetimeIndex
Unparsed date strings stay str dtype

Code	Meaning
`%Y %m %d`	year, month, day
`%H %M %S`	hour, minute, second
`%b` `%A`	month name, weekday

2. Arguments That Matter

Argument	Effect
`format='%Y-%m-%d'`	faster; no day/month guessing
`dayfirst=True`	03/04/2024 = 3 April; default March 4
`errors='coerce'`	bad value → NaT; default `raise` stops
`unit='s'` / `'ms'`	epoch integers → datetimes
`utc=True`	mixed offsets → one UTC column

3. Building the Index

parse_dates + index_col builds it in one step
Or parse first, then set_index('date')

python

df = pd.read_csv('s.csv', parse_dates=['date'], index_col='date')
# or, step by step
df['date'] = pd.to_datetime(df['date'])
df = df.set_index('date')

Coerce hides bad data

Count the NaT rows: df['date'].isna().sum()

Pandas & Data AnalysisPage 3

Date Ranges and Slicing by Time

1. Building a range

pd.date_range(start, end, freq='D') builds a fixed-frequency index
Give any two of start, end, periods
end is inclusive when given
pd.date_range('2024-01-01', periods=5, freq='W') makes exactly 5 weekly stamps

2. Frequency aliases

Alias	Step
`D`	day
`B`	business day
`W`	week
`ME`	month end
`MS`	month start
`h`	hour
`min`	minute

Old aliases

'M' and 'H' were removed in pandas 3: they raise ValueError. Use ME and h.

3. Slicing and grouping keys

python

df.loc['2024']
df.loc['2024-03']
df.loc['2024-03-01':'2024-03-15']

sorted DatetimeIndex; both ends inclusive

Tool	Gives
`idx.year`, `idx.month`	grouping keys
`idx.dayofweek`	Mon=0 … Sun=6
`idx.day_name()`	'Monday' …
`df.between_time('09:00','17:00')`	rows by clock time

Pandas & Data AnalysisPage 4

Resample, Rolling, Shift and pct_change

1. Resample

df.resample('ME').sum() is groupby on time bins
Needs a DatetimeIndex, or on='col'
Pick the aggregation: sum, mean, last, ohlc

Direction	Result	Then
Down (daily to monthly)	Aggregates	Choose agg
Up (daily to hourly)	NaN rows	`.ffill()` `.bfill()` `.interpolate()`

2. Rolling windows

Call	Window
`rolling(7).mean()`	7 rows
`rolling('7D')`	7 calendar days, needs DatetimeIndex
`expanding().mean()`	grows from the start
`ewm(span=7).mean()`	weights recent values more

min_periods=1: first window-1 rows are not NaN

3. Lag, lead, change

python

s.shift(1)  # lag: values down a row
s.shift(-1)  # lead: next value up
s.shift(freq='D')  # moves index labels
s.pct_change(periods=7)  # vs 7 rows back
s.diff()  # absolute change

s.pct_change() == s / s.shift(1) - 1, first row NaN

Pandas & Data AnalysisPage 5

Time Zones and Common Mistakes

1. Localize vs Convert

Call	On	Effect
`tz_localize('Asia/Kolkata')`	naive	attaches zone, wall-clock same
`tz_convert('UTC')`	aware	new zone, same instant
`tz_localize(None)`	aware	strips the zone

tz_convert on naive data raises TypeError
tz_localize on aware data raises too

python

ts = pd.to_datetime(df['ts'])
ts = ts.dt.tz_localize('Asia/Kolkata')
ts = ts.dt.tz_convert('UTC')

localize first, convert after

2. UTC and Daylight Saving

Store and compute in UTC; local time only for display
DST gap (missing hour): tz_localize(..., nonexistent='shift_forward')
DST repeat (ambiguous hour): tz_localize(..., ambiguous='infer')

3. Silent Bugs

shift or rolling(n) on an unsorted index: wrong, silently
rolling('7D') on an unsorted index raises; sort_index() first
shift(1) across gaps moves rows, not days: asfreq('D') first
Naive column vs aware Timestamp: < > raise, == gives False
Dates left as str strings: parse with to_datetime

Check first

Look at df.dtypes and df.index.tz before any time series work.

Pandas & Data AnalysisPage 6

Time Series Cheat Sheet

1. Parse and Build

pd.to_datetime(s, format=..., errors='coerce', utc=True)
read_csv(parse_dates=..., index_col=...)
pd.date_range(start, periods=n, freq='D')
Slice df.loc['2024-03']; by clock time df.between_time(a, b)

2. Aggregate and Windows

Goal	Call
Monthly bins	`df.resample('ME').agg({'sales':'sum','price':'mean'})`
Upsample	`resample(...)` then `.ffill()`
7 rows	`rolling(7).mean()`
7 calendar days	`rolling('7D').sum()`
Running max	`expanding().max()`
Recent-weighted	`ewm(span=7).mean()`

3. Change and Zones

Call	Gives
`shift(1)`	lag
`shift(-1)`	lead
`pct_change()`	ratio
`diff()`	difference

tz_localize('Asia/Kolkata')naive to aware
tz_convert('UTC')same instant

Before any resample

Run sort_index() and check df.dtypes. Row-based shift and rolling go wrong silently on unsorted data.

Pandas & Data AnalysisPage 7

Check yourself

s has daily values 1, 2, 4 on Jan 1, Jan 2 and Jan 4. What does s.shift(1) put on Jan 4?

2, the value from Jan 2
shift moves rows, not days
Call asfreq('D') first to expose the Jan 3 gap

pd.to_datetime('03/04/2024') is meant to be 3 April. Which date do you get, and what is the fix?

March 4, because month comes first by default
Pass dayfirst=True or an explicit format='%d/%m/%Y'

pd.Timestamp('2024-03-01 09:00').tz_convert('UTC') fails. Why, and how do you fix it?

The timestamp is naive, so tz_convert raises TypeError
Run tz_localize('Asia/Kolkata') first, then tz_convert('UTC')
Result: 03:30 UTC

ts.tz_localize('Asia/Kolkata').tz_convert('UTC')