Shell

Shell Scripts Are Slow Because You Fork Too Much

By 柒号华仔 · Aug 29, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Many developers abandon shell scripts for Python at the first sign of slowness, but the gap is often just bad habits — unnecessary forks and serial execution. Fixing those patterns keeps deployment scripts, log processors, and CI pipelines fast without adding a language dependency.

Summary

The core law of shell performance is that fork is expensive — every external command invocation spawns a child process, and in a loop of thousands of iterations, those milliseconds add up to seconds. Bash built-ins like `$(<file)`, `${#arr[@]}`, `[[ ]]`, and variable expansion avoid fork entirely and run 20x to 100x faster than their external counterparts.

For large datasets, awk and sed remain exceptions: a single awk call processes an entire file in C, often beating a bash loop by an order of magnitude. The real multiplier, though, is parallelization. Using `&` with `wait`, `xargs -P`, or GNU parallel turns independent tasks into concurrent work, delivering 5x to 8x speedups on multi-core machines.

A worked example processing 1000 log files drops from 50 seconds to 6.25 seconds — mostly from parallelism, with smaller gains from eliminating `cat | grep` chains and hoisting repeated work out of loops. The lesson: profile with `time`, `strace`, or `perf` before optimizing, and reach for built-ins and parallelism before rewriting in another language.

Takeaways

— Every external command call forks a child process; the overhead is milliseconds, but loops magnify it into seconds.

— Replace `cat file | grep` with `grep file` to halve the process count and double speed.

— Use `$(<file)` instead of `$(cat file)` to read a file into a variable without forking — a 20x difference.

— For small field splits, bash `read` beats awk; for large data, awk's single C-coded invocation beats bash loops by 40x or more.

— Simple string operations belong in bash variable expansion (`${var/old/new}`), not sed — a 50x gap.

— Use `[[ ]]` for conditionals instead of `[ ]`; the latter forks a test process, costing 60x more in loops.

— Bash built-ins cover string length (`${#s}`), array length (`${#arr[@]}`), substring extraction (`${s:0:5}`), and arithmetic (`$((a+b))`) — all fork-free.

— Cache command results outside loops: one `date` call before a 1000-file loop saves 5 seconds.

— Replace O(N*M) grep-in-a-loop membership checks with an associative array lookup for a 10x speedup.

— Parallelize independent tasks with `&`/`wait`, `xargs -P`, or GNU parallel; CPU-bound tasks scale with core count, I/O-bound tasks cap at 10–50 concurrent processes.

— Profile first with `time` (wall-clock vs CPU), `strace` (system call tracing), or `perf` (CPU hotspots) before optimizing.

— A 1000-file error-counting script went from 50s to 6.25s: removing `cat` saved 10s, parallelism on 8 cores saved the rest.

Conclusions

The fork-is-slow rule explains why shell performance advice often feels contradictory: awk is an external command but wins on large data because it pays the fork cost once, not per line.

Parallelism, not micro-optimization, delivers the largest speedup in the case study — single-point fixes like dropping `cat` netted only 20%, while `&` + `wait` cut runtime by 8x.

The `read` vs awk trade-off has a concrete threshold: under roughly 1000 lines, bash's no-fork `read` wins; above that, awk's C implementation dominates despite the fork cost.

Many 'slow shell' complaints are really slow I/O patterns — reading a file 1000 times instead of once, or re-running `date` inside a loop — not inherent language limitations.

Concepts & terms

fork

The system call that creates a child process. In bash, every external command invocation triggers a fork plus exec, loading dynamic libraries and initializing stdio — typically a few milliseconds per call.

bash built-in

A command implemented inside bash itself (cd, echo, read, [[, declare, let, etc.) that runs without spawning a child process, avoiding fork overhead entirely.

$(<file)

Bash syntax that reads a file's content directly into a variable without invoking cat or any external command, eliminating a fork.

xargs -P

An xargs flag that runs a specified number of parallel processes, turning a serial batch of independent tasks into concurrent execution without manual &/wait scripting.

GNU parallel

A superset of xargs that adds progress bars, result collection, and cross-platform consistency for parallel job execution.

associative array

A bash 4+ data structure (declare -A) that maps keys to values, enabling O(1) membership lookups as a fast replacement for repeated grep searches inside loops.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗