sort / uniq / comm Cheatsheet

Basic Sort

The sort command orders text lines, defaulting to lexicographic order.

### Basic Operations

``yaml # Sort lexicographically sort file.txt

# Sort numerically sort -n numbers.txt

# Reverse order sort -r file.txt

# Numeric reverse sort -rn numbers.txt

# Case-insensitive sort sort -f file.txt

# Ignore leading blanks sort -b file.txt `

### Check Sort Status

`yaml # Check if already sorted (no output = sorted) sort -c file.txt

# Silent check (exit code indicates status, 0=sorted) sort -C file.txt `

Advanced Sort

### Field-Based Sorting

`yaml # Sort by the 2nd column (tab/space delimited) sort -k2 data.txt

# Sort by the 3rd column numerically sort -k3n data.txt

# Sort by 2nd, then 3rd numerically sort -k2,2 -k3n data.txt

# Specify column delimiter (colon) sort -t: -k3n /etc/passwd

# Keep header row when sorting (head -1 data.txt && tail -n+2 data.txt | sort -k2n) > sorted.txt `

### Human-Readable and Version Sorting

`yaml # Human-readable numeric sort (K/M/G suffixes) sort -h sizes.txt

# Version number sort sort -V versions.txt

# Mixed sort sort -h -k2 size_report.txt `

### Merge and Output Control

`yaml # Merge already sorted files sort -m sorted1.txt sorted2.txt

# Sort and deduplicate sort -u file.txt

# Write to file instead of stdout sort file.txt -o sorted.txt

# Specify buffer size sort -S 1G large.txt

# Specify temporary directory sort -T /tmp large.txt

# Random sorting sort --random-sort file.txt `

Deduplication (uniq)

The uniq command removes or counts consecutive duplicate lines (usually used after sort).

### Basic Dedup

`yaml # Remove consecutive duplicates (usually sort first) sort file.txt | uniq

# Inline dedup (equivalent to sort -u) sort -u file.txt

# Count occurrences of each line sort file.txt | uniq -c

# Show only duplicate lines sort file.txt | uniq -d

# Show only unique lines (non-repeating) sort file.txt | uniq -u `

### Advanced Dedup

`yaml # Case-insensitive dedup sort file.txt | uniq -i

# Compare only first N characters sort file.txt | uniq -w 5

# Skip first N characters sort data.txt | uniq -s 3

# Count with case-insensitivity sort -f file.txt | uniq -ic

# Count and sort by frequency (descending) sort file.txt | uniq -c | sort -rn `

Set Operations (comm)

The comm command compares two sorted files line by line.

### Intersection and Difference

`yaml # Show all three columns (file1-only | file2-only | common) comm sorted1.txt sorted2.txt

# Intersection: lines common to both comm -12 file1.txt file2.txt

# file1-only lines (difference A-B) comm -23 file1.txt file2.txt

# file2-only lines (difference B-A) comm -13 file1.txt file2.txt `

### Set Operation Combinations

`yaml # Symmetric difference (A∪B - A∩B) comm -3 file1.txt file2.txt | tr -d '\t'

# Union (all unique lines) cat file1.txt file2.txt | sort -u

# Check if A is a subset of B # If A-B is empty then A⊆B [ -z "$(comm -23 A.txt B.txt)" ] && echo "A is a subset of B" `

Pipeline Patterns

sort, uniq, and comm are frequently combined with other commands in powerful text processing pipelines.

### Log Analysis

`yaml # Count Nginx access IPs (descending) awk '{print $1}' access.log | sort | uniq -c | sort -rn

# Count HTTP status code distribution awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Find top 10 most active IPs awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10 `

### File Dedup and Sorting

`yaml # Merge multiple files and deduplicate cat file1.txt file2.txt file3.txt | sort -u > combined.txt

# Find common and different lines between two files sort file1.txt > a.sorted sort file2.txt > b.sorted comm a.sorted b.sorted `

### Utility Commands

`yaml # Shuffle lines randomly shuf file.txt

# Topological sort (dependency ordering) tsort depend.txt

# Count word frequency tr ' ' '\n' < file.txt | sort | uniq -c | sort -rn

# Count unique lines in a file cat file.txt | sort | uniq | wc -l ``