← Back to the summary

Shell Scripts Are Slow Because You Fork Too Much

The previous articles covered shell's language features, error handling, modularity, testing, signal handling, and then spent three articles on common commands. Once a script can run, be maintained, and go live, many people feel it's enough — but the last hurdle to push a script from "it runs" to professional is performance.

Many people misunderstand shell performance. One view is "shell just runs commands, how fast can it be?"; another is "if performance is an issue, switch to Python." Both are wrong. The performance gap between shell scripts and Python isn't that dramatic; often the difference lies in how you write it. Turning a "slow shell script" into a "fast shell script" is often just a matter of changing a few habits.

This article systematically covers the mindset and specific techniques for shell performance optimization. After reading, you should be able to: keep a mental alarm when writing scripts, know which patterns are performance traps, and know how to use awk, bash built-ins, parallelization, and other means to accelerate.

1. The Fundamental Law of Shell Performance

Before diving into specific techniques, establish one understanding: the root cause of shell performance problems is fork.

What is fork? Every time you call an external command, the shell forks a child process to execute it. The overhead of fork itself isn't large (microsecond level), but in bash, fork also requires an exec system call, loading dynamic libraries, and initializing stdio — the whole process usually takes several milliseconds.

This leads to the first law of shell performance: avoid unnecessary forks. An overhead of a few milliseconds doesn't seem like much, but if a script has thousands of iterations and forks once per iteration — several seconds pass.

How big is the real difference? A simple example: counting lines in a file.

# Method 1: wc
time wc -l bigfile.txt
# 0.005s

# Method 2: loop with read
time while read line; do ((count++)); done < bigfile.txt
# 0.8s

For the same file, wc is 160 times faster than a bash loop. That's the gap between fork and no fork.

Once you understand "fork is slow," all the following techniques revolve around it.

2. Avoiding Unnecessary Forks

2.1 Never Use `cat file | grep`

This is the most classic "anti-pattern" in the shell world:

# Anti-pattern: cat + pipe + grep
time cat file.txt | grep "pattern" > result.txt
# 0.020s

# Correct: grep reads the file directly
time grep "pattern" file.txt > result.txt
# 0.010s

The anti-pattern creates two processes (cat + grep); the correct way creates only one (grep). Performance doubles.

Similar anti-patterns:

# Anti-pattern: cat multiple files + grep
time cat file1 file2 file3 | grep "pattern"
# 0.030s

# Correct: grep accepts multiple files
time grep "pattern" file1 file2 file3
# 0.012s

If a command can read the file directly, don't use cat.

2.2 Use `$(<file)` Instead of `$(cat file)`

Reading file content into a variable:

# Method 1: cat + command substitution
content=$(cat file.txt)        # forks cat
# 0.020s

# Method 2: bash built-in
content=$(<file.txt)           # no fork
# 0.001s

$(<file) is bash's "read file into variable" syntax; it calls no external command. Performance difference is 20x.

This technique is especially useful when you need to read an entire file into a variable for processing.

2.3 Use `read` Instead of `awk` for Field Extraction

If you're just splitting a few columns by a delimiter, using read is faster than awk:

# Method 1: awk
time awk -F: '{print $1}' /etc/passwd > users.txt
# 0.015s

# Method 2: read + bash built-in
time while IFS=: read -r user _; do echo "$user"; done < /etc/passwd > users.txt
# 0.030s

Method 2 is actually slower? That's because the overhead of the bash loop exceeds the cost of a single awk invocation. awk is written in C; processing millions of lines is faster than a bash loop.

So the real conclusion from this example is: use awk for large data, use read for small data. This trade-off was covered in the earlier strings article; it's emphasized again here.

2.4 Use Bash Built-ins for String Processing Instead of sed

For simple string operations, bash built-ins are much faster than sed:

# Method 1: sed
time echo "hello world" | sed 's/world/shell/'
# 0.005s

# Method 2: bash variable expansion
time str="hello world"; str="${str/world/shell}"; echo "$str"
# 0.0001s

Performance difference is 50x. For simple string replacement, removing prefixes/suffixes, bash variable expansion is always faster than sed.

Use sed/awk only for complex regex matching. Rule of thumb: if it can be solved with ${var//pattern/replacement}, don't use echo "$var" | sed.

2.5 Use `${#arr[@]}` for Array Length Instead of wc

# Method 1: wc
time echo "${arr[@]}" | wc -w
# 0.020s

# Method 2: bash built-in
time echo "${#arr[@]}"
# 0.0001s

${#arr[@]} is bash's built-in array length — no process fork.

2.6 Use `[[ ]]` for Testing Instead of `[ ]`

# Method 1: [ ] (test command)
time for i in {1..1000}; do [ "$i" -gt 500 ] && true; done
# 0.300s

# Method 2: [[ ]] (bash built-in)
time for i in {1..1000}; do [[ "$i" -gt 500 ]] && true; done
# 0.005s

[ ] actually forks a test process; [[ ]] is a bash built-in keyword. 1000 iterations show a 60x difference.

Use [[ ]] daily; use [ ] only when POSIX compatibility is required.

3. Bash Built-ins vs External Commands

Shell commands fall into two categories: bash built-ins and external commands.

Built-ins: implemented by bash itself, no fork. cd, echo, read, test, printf, declare, set, export, let, local are all built-ins.
External commands: standalone executable files; every invocation forks. ls, cat, grep, sed, awk, wc, sort are all external.

Principle: use built-ins whenever possible. Here's a common built-in substitution table:

Operation	External Command (slow)	Bash Built-in (fast)	Performance Gap
String length	`echo "$s" \| wc -c`	`${#s}`	100x
Array length	`echo "${arr[@]}" \| wc -w`	`${#arr[@]}`	100x
Substring	`echo "$s" \| cut -c1-5`	`${s:0:5}`	50x
String replace	`echo "$s" \| sed 's/a/b/'`	`${s/a/b}`	50x
Arithmetic	`expr $a + $b`	`$((a + b))`	20x
Condition test	`[ "$a" -gt 0 ]`	`[[ "$a" -gt 0 ]]`	60x
Read file	`$(cat file)`	`$(<file)`	20x
File exists	`[ -f file ]`	`[[ -f file ]]`	50x

Memorize this table; prioritize built-ins when writing scripts.

4. awk/sed Are Performance Accelerators

Earlier we said "bash built-ins are faster than external commands," but awk is an exception.

A single awk invocation can accomplish what a bash loop plus multiple command calls does. For large data, awk is almost always an order of magnitude faster than a bash loop.

4.1 awk Replaces Loops

# Method 1: bash loop counting
time while read line; do ((count++)); done < bigfile.txt
# 2.5s

# Method 2: awk
time awk 'END{print NR}' bigfile.txt
# 0.05s

awk runs dozens of times faster. This is the dividend of awk being written in C + processing the entire file in one invocation.

Another example — summing by column:

# Method 1: awk
time awk '{sum += $3} END {print sum}' data.txt > result.txt
# Method 2: pure bash
time sum=0; while read -r a b c rest; do ((sum += c)); done < data.txt; echo $sum > result.txt
# Method 1: 0.05s
# Method 2: 2.0s

awk is 40 times faster.

4.2 When to Use awk

awk is suitable for:

Processing large files (MB level and above)
Column-based data processing
Statistics (counting, summing, averaging)
Complex field operations

awk is not suitable for:

Simple field splitting (read is enough for small files)
Complex logic decisions (bash is clearer)
Scenarios requiring many other command calls

Generally, under 1000 lines a bash loop can handle; above 1000 lines use awk.

4.3 What sed Is Suitable For

sed is suitable for "batch line-by-line modification" — s/old/new/g, d deletion, p printing. For pattern-matching replacement, sed is much faster than a bash loop.

# Method 1: bash loop + string replacement
time while read line; do echo "${line/old/new}"; done < file.txt
# Method 2: sed
time sed 's/old/new/g' file.txt
# Method 1: 2.0s
# Method 2: 0.05s

5. Avoiding Repeated Work Inside Loops

Often a script is slow not because a single operation is slow, but because small overhead inside a loop is magnified a thousand times.

5.1 Cache Frequently Used Command Results

# Anti-pattern: recalculate every iteration
for file in *.log; do
    today=$(date +%Y%m%d)
    count=$(wc -l < "$file")
    echo "$file: $count lines, today: $today"
done

# Optimized: hoist outside the loop
today=$(date +%Y%m%d)
for file in *.log; do
    count=$(wc -l < "$file")
    echo "$file: $count lines, today: $today"
done

One date call takes 0.005s; 1000 files means 5s. Hoisting it outside saves 5s.

Similar cases:

Read config file once; don't re-read in every function
Use variables for constant strings; don't concatenate every time
Parse paths once; don't cd + pwd every time

5.2 Use Associative Arrays Instead of grep

Checking "whether a value is in a list":

# Method 1: grep every time
time for name in "${names[@]}"; do
    if grep -q "^$name$" allowed.txt; then
        echo "$name allowed"
    fi
done
# 5s

# Method 2: read allowed into an associative array first
time declare -A allowed
while read -r line; do allowed[$line]=1; done < allowed.txt
for name in "${names[@]}"; do
    if [[ -n "${allowed[$name]}" ]]; then
        echo "$name allowed"
    fi
done
# 0.5s

O(N*M) becomes O(N+M); performance difference is 10x.

5.3 Reduce File I/O

File reads and writes are relatively slow operations. Read once if possible; don't read multiple times:

# Anti-pattern: read file every iteration
for id in "${ids[@]}"; do
    name=$(grep "^$id:" /etc/passwd | cut -d: -f5)
    echo "$id: $name"
done

# Optimized: read the entire file once
declare -A names
while IFS=: read -r id _ _ _ name _; do
    names[$id]=$name
done < /etc/passwd

for id in "${ids[@]}"; do
    echo "$id: ${names[$id]:-unknown}"
done

Reading a file once is much faster than reading it 1000 times.

6. Parallelization

If a script has independent tasks — processing multiple files, calling multiple APIs, querying multiple databases — parallelization can fully utilize a multi-core machine's performance.

6.1 & + wait: Basic Parallelism

# Serial: 5 tasks, 1 second each
time for host in web1 web2 web3 web4 web5; do
    ssh "$host" uptime
done
# 5s

# Parallel: 5 tasks concurrently
time for host in web1 web2 web3 web4 web5; do
    ssh "$host" uptime &
done
wait
# 1s

& puts a task in the background; wait waits for all background tasks to finish. 5x speedup.

6.2 xargs -P: More Controllable Parallelism

xargs' -P parameter specifies the number of parallel processes, more flexible than hand-written &:

# 8 processes in parallel
time find . -name "*.jpg" | xargs -P 8 -I {} convert {} -resize 50% {}.small.jpg
# Serial: 40s
# Parallel (8 cores): 5s

8x speedup. xargs -P is suitable for "batch independent tasks," each task processing one file.

6.3 GNU parallel: The Most Powerful Parallelism

GNU parallel is a superset of xargs, supporting parallelism, progress display, and result merging.

# Installation
apt install parallel    # Debian/Ubuntu
yum install parallel    # CentOS

# Parallel gzip all .log files
time find . -name "*.log" | parallel gzip {}
# 8x speedup

# Show progress
find . -name "*.log" | parallel --progress gzip {}

# Collect results to files
find . -name "*.log" | parallel --result output/ gzip {}

parallel vs xargs -P:

xargs is simple, lightweight, built-in
parallel has more features, shows progress, better cross-platform consistency

For daily use, xargs is enough; for complex scenarios, use parallel.

6.4 A Few Pitfalls of Parallelization

Pitfall 1: Output will be interleaved. Multiple processes outputting to stdout simultaneously will interleave results. Solution: each task outputs to a separate file, then merge at the end:

for host in web1 web2 web3; do
    ssh "$host" uptime > "/tmp/uptime_$host.txt" &
done
wait
cat /tmp/uptime_*.txt
rm /tmp/uptime_*.txt

Pitfall 2: Too many processes actually slows things down. 100 concurrent processes max out the CPU and congest disk I/O. Rule of thumb: CPU-bound use N=core count; I/O-bound use N=10-50.

Pitfall 3: Chaotic logs. Multiple tasks writing to the same log file simultaneously will produce garbled output. Separate log per task.

7. Performance Analysis Tools

Before optimizing, you need to know "where it's slow." This section covers a few common performance analysis tools.

7.1 time: Basic Timing

time ./script.sh
# real    0m5.123s    # actual elapsed time
# user    0m3.456s    # user-mode CPU time
# sys     0m0.234s    # kernel-mode CPU time

real is actual wall-clock wait time; user + sys is CPU time consumed. If real is much larger than user + sys, the script is waiting on I/O (network/disk).

7.2 strace: Tracing System Calls

# See what system calls the script makes
strace -c ./script.sh

# See specific calls
strace -e trace=open,read,write ./script.sh

strace tells you what the script is waiting on — too many forks, disk I/O, or network.

7.3 perf: CPU Performance Analysis

# Record CPU events
perf record ./script.sh
perf report

perf tells you which functions CPU time is spent on. For shell scripts, mainly look at the call counts of external commands like awk and grep.

7.4 Bash's Built-in time

# bash's built-in time, higher precision
TIMEFORMAT='real %3R, user %3U, sys %3S'
time ./script.sh

This time is a shell keyword; it won't fork /usr/bin/time.

8. Case Study: Accelerating a Slow Script

Let's look at a real slow-script case and see how to optimize step by step.

Original version (processing 1000 log files, counting errors):

#!/bin/bash
for file in *.log; do
    count=$(cat "$file" | grep "ERROR" | wc -l)
    echo "$file: $count errors"
done

Performance: 1000 files, 0.05s per file, total 50s.

Optimization Step 1: Remove cat

for file in *.log; do
    count=$(grep -c "ERROR" "$file")    # grep -c counts directly
    echo "$file: $count errors"
done

Performance: 0.04s per file, total 40s, 1.25x speedup.

Optimization Step 2: Avoid repeated cat inside loop

total=0
for file in *.log; do
    count=$(grep -c "ERROR" "$file")
    echo "$file: $count errors"
    total=$((total + count))
done
echo "Total: $total"

Performance: basically unchanged, but code is clearer.

Optimization Step 3: Parallelization

total=0
for file in *.log; do
    (
        count=$(grep -c "ERROR" "$file")
        echo "$file: $count errors" >> /tmp/results.txt
        echo "$count"
    ) &
done > /tmp/counts.txt

# Wait for all tasks to finish
wait

# Aggregate
total=$(awk '{s+=$1} END {print s}' /tmp/counts.txt)
echo "Total: $total"
rm -f /tmp/results.txt /tmp/counts.txt

Performance: 8-core machine, 6.25s, 8x speedup.

Optimization Step 4: Change approach, process entire flow with awk

# One awk invocation per file
total=0
for file in *.log; do
    count=$(awk '/ERROR/{c++} END{print c+0}' "$file")
    total=$((total + count))
    echo "$file: $count errors"
done
echo "Total: $total"

Performance: 0.05s per file, total 50s. awk processing one file at a time offers no advantage.

The real optimization: batch processing

# Merge all logs and process once
cat *.log | awk '/ERROR/{count++} END{print "Total:", count+0}'

But this loses per-file statistics — so keeping per-file counting + parallelization is the best solution.

Final performance comparison:

Original: 50s
Remove cat: 40s
Parallel: 6.25s

Total 8x speedup. This is the real effect of shell performance optimization, mainly driven by parallelization; single-point optimizations yield limited gains.

9. Summary

Shell performance optimization is a detail-oriented craft. The core ideas are:

Avoid unnecessary forks: use built-ins instead of external commands whenever possible
Prioritize bash built-ins: variable expansion, [[ ]], ${#arr[@]} are all cheap
awk is suitable for large data: one invocation processes the entire file
Parallelization is the biggest speedup: CPU-bound tasks benefit from core-count parallelism
Profile first, then optimize: use time/strace/perf to find bottlenecks

Shell performance problems are usually not because the shell language is slow, but because the usage is wrong. Many slow scripts become 10x faster just by changing how they're written.

With performance optimization covered, the entire shell programming series is now fairly complete. From language features to engineering practices, from common commands to performance tuning, the core content needed to master shell programming has been broadly covered.