Text Processing with Bash and CLI Tools
Sort, deduplicate, and analyze text files with sort, uniq, awk, and sed in the terminal.
Published:
Tags: text processing Bash, bash sort uniq awk, Linux text manipulation
Text Processing with Bash and CLI Tools The Unix toolkit — , , , , , and — can process millions of lines of text in seconds without installing anything. These composable utilities follow the Unix philosophy: each does one thing well, and you chain them with pipes to build powerful one-liners. --- Why CLI Tools Beat GUI Editors for Bulk Text Work When a file has 100,000 lines and you need to deduplicate, sort, and extract a column, opening it in a text editor is a bad idea. CLI tools: Process files larger than RAM using streaming Compose with pipes into reproducible, scriptable pipelines Run on any Linux/macOS server without a desktop environment Operate in milliseconds on files that would freeze a GUI A five-command pipeline that took thirty seconds to write can save hours of manual…
Frequently Asked Questions
How do I sort lines in a file with Bash?
Run `sort filename.txt` for alphabetical order or `sort -n filename.txt` for numeric order. Add `-r` to reverse the result. Use `-u` to sort and deduplicate in one step.
How do I remove duplicate lines with uniq?
`uniq` removes consecutive duplicate lines, so always sort first: `sort file.txt | uniq`. Use `uniq -c` to prefix each line with its occurrence count and `uniq -d` to print only the duplicates.
How do I count word frequency with awk?
Use `awk '{for(i=1;i<=NF;i++) freq[$i]++} END {for(w in freq) print freq[w], w}' file.txt | sort -rn` to list every word alongside its count, sorted from most to least frequent.
How do I add line numbers with nl?
Run `nl filename.txt` for default line numbers, or `nl -ba filename.txt` to number all lines including blank ones. The `cat -n` command also works: `cat -n filename.txt`.
How do I find and replace text with sed?
Use `sed 's/old/new/g' file.txt` to replace all occurrences of 'old' with 'new'. Add the `-i` flag to edit the file in place: `sed -i 's/old/new/g' file.txt`. On macOS, `-i ''` is required.
All articles · theproductguy.in