Text Processing

7 pages · ~5 min✓ Reviewed

Builds on Filesystem and Permissions. Next up: Shell Scripting.

LinuxPage 1

Streams, Pipes and Redirection Model

1. Streams and Pipes

stdin = fd 0, stdout = fd 1, stderr = fd 2
A filter reads stdin and writes stdout
a | b wires a's stdout to b's stdin
Both run at the same time
Only stdout travels; stderr still hits the terminal

2. Redirection

Operator	Effect
`>`	truncate and write a file
`>>`	append to a file
`<`	feed a file to stdin
`2>file`	stderr to file
`2>&1`	merge stderr into stdout
`&>file`	both streams to file (bash)

bash

cmd >f 2>&1   # works: both in f
cmd 2>&1 >f   # stderr stays on screen

order matters

Left to right

2>&1 copies where stdout points at that moment.

3. Gotcha

sort f > f empties f before sort reads it

bash

sort -o f f
sort f > tmp && mv tmp f

safe: -o or a temp file

LinuxPage 2

Searching with grep and Regex Basics

1. grep Basics

grep PATTERN file prints matching lines

Flag	Effect
`-i`	ignore case
`-v`	invert: non-matching lines
`-n`	line numbers
`-c`	count matching lines
`-r`	recursive
`-l`	filenames only

2. Regex Building Blocks

Token	Matches
`^` / `$`	start / end of line
`.`	any one character
`*`	zero or more of the previous item
`[abc]`	one of a, b, c
`[^abc]`	none of a, b, c
`[0-9]`, `[[:digit:]]`	a digit (POSIX class)

\d-style shortcuts work only with -P
Plain grep (BRE) needs \+ \? \| 
grep -E takes + ? | ( ) unescaped: use it by default

3. Gotchas

-F literal text, -w whole words, -o only the match

bash

grep -E 'ab+|cd' f
grep -F 'a.b' f
grep -w 'cat' f
grep -o '[0-9]*' f

Quote the pattern

Unquoted * is expanded by the shell before grep sees it.

LinuxPage 3

Slicing and Counting: cut, sort, uniq, wc, head, tail

1. Pick and order

cut -d, -f1,3 picks fields 1 and 3, comma-delimited
cut -c1-5 picks characters 1 to 5
Default cut delimiter is a TAB

sort flag	Effect
`-n`	numeric
`-r`	reverse
`-k2`	by field 2
`-t,`	set delimiter
`-u`	unique lines
none	lexical: 10 before 2

2. Collapse and count

uniq collapses only adjacent duplicates, so run sort | uniq
uniq -c counts; uniq -d shows only duplicates

Command	Shows
`wc -l`	lines
`wc -w`	words
`wc -c`	bytes
`head -n 5`	first 5 lines
`tail -n 5`	last 5 lines
`tail -n +2`	skips the header
`tail -f`	follows a growing file

3. Top-N idiom

cutIP column
sortgroup equal
uniq -ccount
sort -nrbiggest first
headtop lines

Top-N

cut -d' ' -f1 access.log | sort | uniq -c | sort -nr | head Skip the first sort and uniq counts only runs.

LinuxPage 4

Transforming Text with sed and awk

1. sed: Line Edits

s/old/new/g replaces every match on each line
No g: only the first match per line changes
Any delimiter works: s|/usr|/opt|

sed option	Effect
`-n '5,10p'`	prints only lines 5-10
`/re/d`	deletes matching lines
`-i`	edits the file in place
`-i.bak`	GNU: keeps a backup

2. awk: pattern { action }

Runs per line: pattern selects, { action } runs
$1..$NF fields, $0 whole line, NR line number
-F, sets the field separator
END block runs once, after the last line

bash

awk '$3 > 100 {print $1}' f
awk '{s+=$2} END{print s}' f

filter then print; sum a column

3. Choosing the Tool

Tool	Use for
`cut`	simple column pulls
`sed`	line edits, substitution
`awk`	per-field logic, conditions, arithmetic

Remember

Forget g and only the first match per line changes. -i rewrites the file: use -i.bak.

LinuxPage 5

Building Pipelines with xargs and Avoiding Mistakes

1. What xargs Does

xargs turns stdin items into arguments; it splits on blanks and newlines
Needed because many commands ignore stdin
grep -l TODO *.c | xargs wc -l

2. Safe Input and Flags

my file.log becomes two arguments
Fix: NUL-separated input with -0

bash

find . -name '*.log' -print0 | xargs -0 rm

-print0 pairs with -0

Flag	Effect
`-n1`	one argument per call
`-I{}`	places the argument anywhere: `xargs -I{} cp {} /backup`
`-P4`	runs 4 jobs in parallel

3. Exit Status and Mistakes

Pipeline status is that of the last command only
set -o pipefail makes a failing stage fail the pipeline

Useless cat file | grep x: use grep x file
uniq without sort
Unquoted patterns
xargs rm without previewing

Preview first

Run ... | xargs echo rm, check the list, then drop echo.

LinuxPage 6

Pipelines Cheat Sheet

1. Redirection

Op	Effect
`>`	truncate to file
`>>`	append
`<`	file to stdin
`2>`	stderr to file
`2>&1`	stderr into stdout
`cmd \| tee f`	write file, pass on

2. grep and regex

-i case, -v invert, -n line no., -c count, -r recurse, -l names
-E extended, -F literal, -w whole word, -o only match
Regex: ^ $ . * + ? [] [^] () |

3. Columns, sed, awk

Task	Command
Field	`cut -d, -f2`
Sort	`sort -t, -k2 -nr`
Count	`sort \| uniq -c`
Lines	`wc -l`
First	`head -n 5`
Skip header	`tail -n +2`
Replace	`sed 's/a/b/g'`
Range	`sed -n '3,5p'`
Delete	`sed '/x/d'`
Column	`awk -F, '{print $1}'`
Filter	`awk '$2>10'`
Sum	`awk '{s+=$1} END{print s}'`

4. Recipes

Top-N, safe delete (preview with echo), count matches

bash

sort f | uniq -c | sort -nr | head
find . -print0 | xargs -0 echo rm
grep -c TODO f

Gotcha

uniq needs sorted input; xargs splits on blanks, so use -0.

LinuxPage 7

Check yourself

You run sort data.txt > data.txt. What is in data.txt afterwards?

Empty: the shell truncates the file before sort reads it
Fix: sort -o data.txt data.txt

What does printf '10\n2\n10\n' | sort | uniq -c report?

Lexical sort gives 10, 10, 2, so the duplicates are adjacent
Result: 10 counted 2 times, 2 counted 1 time

With cmd 2>&1 >f, where do stdout and stderr go?

2>&1 runs first, so stderr copies the terminal
Then stdout goes to f: stdout in f, stderr on screen
Use cmd >f 2>&1 to capture both