Handbooks / CI/CD & DevOps / Chapter 4

Deployment Strategies

7 pages · ~5 min✓ Reviewed

Builds on Infrastructure as Code. Next up: Monitoring and Rollback.

CI/CD & DevOpsPage 1

Deployment Strategy Mental Model

1. Deploy vs Release

Deploy = code lands on servers
Release = users can see it
Feature flags and traffic shifting split the two

2. Zero-Downtime Needs Three Things

Health checkgates traffic
Graceful shutdowndrain in-flight requests
Side by sideold and new versions coexist

3. Strategy Trade-offs

Trade-offs: cost, speed, rollback time, blast radius
Blast radius = share of users hit by a bad release

Strategy (safer ↓)	Blast radius	Cost
Recreate	100%	1x
Rolling	growing share	1x+
Blue-green	0% until switch	~2x, the most
Canary	1-5%	small extra slice

4. Common Mistake

Green CI is not safe

Safety comes from health checks, metrics gates and a tested rollback path.

CI/CD & DevOpsPage 2

Recreate and Rolling Updates

1. Recreate

Stop all old instances, then start the new ones
Downtime, but never two versions at once
Suits breaking schema changes and dev environments

2. Rolling Update

Replace instances in small batches
maxSurge: extra pods allowed
maxUnavailable: pods allowed down
maxUnavailable: 0 keeps full capacity
Readiness probe: no traffic to unready pods
preStop sleep + terminationGracePeriodSeconds drain connections

Start v2 podmaxSurge
Readiness passesgets traffic
Drain v1 podpreStop + grace
Stop v1 podnext batch

3. Commands

bash

kubectl set image deploy/app app=img:v2
kubectl rollout status deploy/app
kubectl rollout undo deploy/app
kubectl rollout history deploy/app

v1 and v2 overlap

During a rolling update both versions serve traffic together. Incompatible APIs, schemas or cache formats break requests mid-rollout.

CI/CD & DevOpsPage 3

Blue-Green and Canary Releases

1. Blue-Green

Two full environments: blue is live, green is idle
Deploy and test green, then flip router, load balancer or Service selector
Rollback = flip back, no redeploy
Cost: ~2x infrastructure during switch
Shared DB must fit both versions; only the app tier is duplicated

Deploy green
Test green
Flip router
Flip backrollback

2. Canary

1%metrics gate
5%metrics gate
25%metrics gate
100%

Each step: compare error rate, latency p99, business metrics vs baseline
Breach → automatic abort and rollback

Tool	Role
Istio VirtualService	weighted routing
ALB weighted target groups	weighted routing
Argo Rollouts, Flagger	automated analysis, promotion

3. Common Mistakes

Too little canary traffic or no automated metrics gate proves nothing
Sticky sessions pin users to one version
Stale blue kept running long after cutover

Remember

Canary needs a metrics gate; blue-green needs a DB both versions accept.

CI/CD & DevOpsPage 4

Feature Flags and Dark Launches

1. Flag Basics

if flags.isEnabled('new-checkout', user) = runtime switch
Decouples deploy from release
Flag off = rollback, no redeploy

Type	Life	Example
Release	short	new checkout
Ops kill switch	long	disable payments
Experiment	A/B	button colour
Permission	long	admin tools

2. Dark Launch and Ramp

Deployrolling, flag off
Dark launchshadow traffic
Compareold vs new
Ramp1%, 10%, 50%, 100%

Targeting: percentage, user segment, internal users first
Evaluate a stable user hash so each user stays consistent

3. Flag Debt

Old release flags leave dead branches and untested combos
No owner or expiry date
No safe default when the flag service is down

Remember

Remove release flags after 100%. Give each an owner and expiry; default to the safe state.

CI/CD & DevOpsPage 5

Zero-Downtime Database Migrations

1. Safe Change Flow

Schema must stay backward compatible with running code
Old and new app versions share the same database mid-rollout

Expandadd nullable column
Dual-writewrite both, read old
Backfillin batches
Switch readsto new column
Contractdrop old, later release

Rename a column = the same five steps; a direct RENAME breaks old code instantly

2. Avoid Long Locks

Postgres: CREATE INDEX CONCURRENTLY
Backfill in small batches
Set lock_timeout
MySQL: gh-ost or pt-online-schema-change

sql

SET lock_timeout = '2s';
CREATE INDEX CONCURRENTLY idx_users_email ON users (email);

3. Common Mistake

Order matters

DROP COLUMN, NOT NULL without default or a type change in the same release as the code leaves rollback no schema. Ship additive migrations first and separately; drop or narrow columns only in a later release, once no running code uses them.

CI/CD & DevOpsPage 6

Zero-Downtime Deploy Cheat Sheet

1. Strategy Picker

Strategy	Downtime	Cost	Rollback
Recreate	yes	1x	redeploy
Rolling	none	1x+	slow
Blue-green	none	2x	instant flip
Canary	none	small extra	shift weights back

2. Rolling and Canary Checklists

Rolling: readiness probe gates traffic
maxUnavailable: 0 keeps full capacity
preStop drain + terminationGracePeriodSeconds
kubectl rollout status to watch, rollout undo to revert

Canary: start at 1-5% of traffic
Compare error rate and p99 with baseline
Automate abort, then promote in steps

3. Flags and Migrations

Flag: default off, percentage ramp, kill switch
Owner and expiry; remove after rollout

Expand
Dual-write
Backfill
Switch reads
Contractlater release

CREATE INDEX CONCURRENTLY, batched backfill, lock_timeout

Destructive DDL

Never ship DROP COLUMN with the code change. Contract in a later release.

CI/CD & DevOpsPage 7

Check yourself

A rolling update ships v2 after running RENAME COLUMN name TO full_name. What happens mid-rollout?

v1 pods still query name, so they fail at once
Safe way: add full_name, dual-write, backfill, drop name in a later release

A canary gets 1% of 100 req/min. After 5 minutes it shows zero errors. Promote it?

No: only about 5 requests reached it
That is too few to show an error rate; gate on error rate and p99 with enough traffic

After a blue-green cutover, the release also ran DROP COLUMN old_col. Flipping back to blue now fails. Why?

The database is shared, and blue still needs old_col
Drop columns in a later release, once no running code uses them