跪拜 Guibai
← Back to the summary

Shell Scripts Don't Fail Gracefully by Default — Here's How to Make Them

"Quick Mastery of Shell" — Error Handling and Logging in Shell Programming

In the scripts we discussed in previous articles, there was an implicit assumption: all commands will succeed, all data will be clean, and all dependencies will be present. This assumption works fine on your own laptop, but when placed in a production environment — hanging in crontab, written into a CI/CD pipeline, or running on a user's machine — a script must know what to do when things go wrong.

That is the problem this article aims to solve: turning a script from something that can run into something that is reliable. We will talk about three things — exit codes, error handling, and logging. Together, these three determine whether a script's behavior when problems occur is a silent failure or a clear, controllable process.

This article's style is a bit different from the previous ones. Before, we were continuously adding capabilities; this time, we are constraining capabilities. You will see many rules about what not to do and how you should write things instead. Rules sound boring, but the root cause of 90% of shell script problems in production incidents is that these rules were not followed.

1. Exit Codes

Before discussing error handling, we must first clarify the fundamental concept of exit codes. It is the only language shell processes use to communicate success and failure to each other, and all error handling is built upon it.

1.1 What is an Exit Code

Every command, after execution, returns a number — an exit code. 0 indicates success, and any other value indicates failure. It sounds simple, but that "any other value" is the key point: the shell does not mandate the meaning of specific numbers, only stipulating that 0 = success, and everything else means something went wrong.

#!/bin/bash
ls /tmp/does_not_exist
echo "Exit code: $?"

Output:

ls: cannot access '/tmp/does_not_exist': No such file or directory
Exit code: 2

$? is the exit code of the last command. This is the most common way to query what happened with the previous command. Note that $? is a "transient state" — once you execute another command, it gets overwritten. Therefore, querying $? must be done immediately after the target command:

# Correct way
ls /tmp/not_here
result=$?
echo "$result"   # Correct

# Wrong way
result=$?
ls /tmp/not_here
echo "$result"   # Always 0 (the exit code of echo itself)

In the second approach, $? actually captures the exit code of the command before ls, and echo's exit code is always 0. This is one of the most common mistakes shell beginners make.

1.2 Common Exit Code Conventions

Although the shell does not enforce them, the industry has a set of conventional semantics:

Exit Code Meaning
0 Success
1 General error (many commands use it to mean "something went wrong")
2 Command argument error (bash builtins typically use this)
126 Command found but cannot execute (no execute permission)
127 Command not found
128+N Process terminated by signal N (130 = Ctrl+C, SIGINT)
130 Exited via Ctrl+C (128 + 2)
137 Killed by SIGKILL (128 + 9)
143 Terminated by SIGTERM (128 + 15), common with docker stop

Just remember a few common ones: 1 = general error, 2 = argument error, 126 = not executable, 127 = command not found, 130 = Ctrl+C. For the rest, rely on the semantics that $? carries.

1.3 The exit Command

When writing a script, if you want the script itself to return a specific exit code, use exit:

#!/bin/bash
if [ ! -f "$1" ]
then
    echo "Error: File $1 does not exist" >&2
    exit 1
fi

# Normal processing
process "$1"
exit 0

A few key points:

1.4 Sending Error Messages to the Right Place

A detail often overlooked: a script's error messages should be output to stderr, not stdout. The difference between the two is:

echo "Normal output"           # stdout
echo "Error message" >&2        # stderr

>&2 is a file descriptor redirection, sending the output to file descriptor 2 (which is stderr). The benefit of doing this is: when a user redirects the script's output to a file, error messages can still be displayed on the terminal:

$ ./script.sh > output.log
Error: Missing arguments!           # Still displayed on the terminal
$ cat output.log
Normal output                 # Normal output is in the file

The >&2 syntax might look a bit odd, but any diagnostic information, error messages, or warning messages should use >&2. This is the third iron rule of shell programming — the first two iron rules being "add double quotes" and "${arr[@]}" with quotes.

2. The set Command

Anyone who writes shell scripts has likely been burned by silent failures — the script finishes running without errors, but the result is wrong. Several options of the set command are used to add guardrails to scripts, and it is strongly recommended to enable them at the beginning of every script.

2.1 set -e: Exit on Error

#!/bin/bash
set -e

cd /nonexistent_dir
echo "This line will not execute"

set -e tells the shell: if any command returns a non-zero exit code, exit the script immediately (unless in contexts like if/while/&&/||, which are inherently meant to check exit codes).

This option is somewhat controversial. Some say adding set -e makes writing scripts troublesome, requiring manual error handling everywhere, but for production scripts, set -e is mandatory — it minimizes the cost of forgetting to check for errors.

2.2 set -u: Error on Undefined Variables

#!/bin/bash
set -u

echo "$undefined_var"

The default behavior is to output an empty line; with -u enabled, accessing an undefined variable will directly cause an error and exit. This one is even more worth enabling than -e, because it can expose spelling mistakes.

#!/bin/bash
set -u

user_nmae="Zhang San"   # Note: it's nmae, not name
echo "$user_name"  # Without -u: outputs empty; With -u: throws an error

Without -u, this kind of typo would cause the script to silently continue running with an empty value, hiding the bug deeply.

2.3 set -o pipefail: Don't Mask Pipeline Failures

This is the easiest pitfall among the three options. Look at this example:

#!/bin/bash
set -e

ls /nonexistent | sort
echo "Pipeline execution finished"

Without pipefail, ls fails, but the pipeline's exit code is taken from sort's exit code (0), so set -e doesn't detect it, and the script continues executing — but ls's error message is completely swallowed.

With pipefail enabled:

#!/bin/bash
set -e
set -o pipefail

ls /nonexistent | sort   # Immediately errors out and exits

pipefail makes the pipeline's exit code the last non-zero one among all commands. This way, any failure within any step of the pipeline will be caught.

2.4 Enabling All Three Together: The Standard

The standard opening for a production script:

#!/bin/bash
set -euo pipefail

This single line condenses three safety nets: exit on error, error on undefined variables, and don't mask pipeline failures. Almost all reliable shell scripts will have this line enabled.

But there are a few edge cases to be aware of:

First, set -e does not take effect after certain commands. For example, commands following command || true will not trigger an exit even if they fail, because the || has already "digested" the failure.

set -e
false || true
echo "Will execute"   # Normal
false || echo "Failed but digested"
echo "Will execute"   # Normal

Second, function return values do not trigger set -e. Look at this counter-intuitive example:

#!/bin/bash
set -e

func() {
    return 1
}

func           # Function returns 1, but the script does not exit
echo "This line will execute"

func's exit code is 1, but set -e does not treat the function's last command as a whole for judgment. To make a function trigger set -e, either explicitly check it when calling:

func || { echo "func failed" >&2; exit 1; }

Or have the function explicitly declare it when returning:

func() {
    [ "$1" -gt 0 ] || return 1
    # ...
}

Third, it also doesn't take effect inside if conditions. This is by design — if is inherently for judging success or failure; if set -e applied here too, if would be unusable.

2.5 set -x and set -v: For Debugging

#!/bin/bash
set -x

name="Zhang San"
echo "$name"

Output:

+ name='Zhang San'
+ echo 'Zhang San'
Zhang San

set -x prints the expanded form of each command before executing it, making it a nuclear weapon for debugging shell scripts. Add it during daily development to troubleshoot problems, and remove it when releasing.

set -v is more primitive than -x — it prints the original lines of the script (without expanding variables). Generally, -x is sufficient.

3. Error Handling Patterns

set -euo pipefail solves the basic protection problem, but there are always scenarios in script logic where "simply failing" isn't acceptable — like "should we continue if a file is not found" or "should we rollback if a step fails". These require explicit error handling patterns.

3.1 || and && Short-circuiting

The shell provides || (or) and && (and) to perform logic "based on the result of the previous command":

mkdir -p /tmp/work || { echo "Failed to create directory" >&2; exit 1; }
grep "pattern" file.txt && echo "Found" || echo "Not found"

cmd1 && cmd2 means "execute cmd2 only if cmd1 succeeds", and cmd1 || cmd2 means "execute cmd2 only if cmd1 fails". This is the most concise way in shell to write "success/failure branches".

When following || with multiple commands, remember to group them with { ... } (note the spaces after { and before }, and the ; at the end):

mkdir -p /tmp/work || { echo "Creation failed" >&2; exit 1; }

3.2 Defensive vs. Fail-Fast

Two styles are endlessly debated in the shell community:

Defensive (don't crash on problems, keep going):

#!/bin/bash
process() {
    for file in *.txt
    do
        if [ -f "$file" ]
        then
            if ! gzip "$file"
            then
                echo "Failed to compress $file, skipping" >&2
                continue
            fi
        fi
    done
}

Fail-Fast (exit immediately on problems, don't leave hidden dangers):

#!/bin/bash
set -e

for file in *.txt
do
    [ -f "$file" ] || continue
    gzip "$file"   # Failure will immediately exit the entire script
done

Which is better? It depends on the scenario. For batch processing tasks ("do X to 100 files, do as many as possible"), use defensive style; for transactional tasks ("deploy code, either succeed or rollback"), use fail-fast.

Practical experience: Fail fast if you can. Many silent bugs are caused by defensive handling swallowing errors.

3.3 Error Handling Function

If a script has multiple places that need to print an error and exit, extract it into a function:

#!/bin/bash
set -euo pipefail

# Error handling
die() {
    echo "[ERROR] $*" >&2
    exit 1
}

# Usage
[ "$#" -ge 1 ] || die "Usage: $0 <argument>"
[ -f "$1" ] || die "File does not exist: $1"

die is the conventionally agreed-upon function name for error-exiting in shell scripts. Used together with set -e, each || die "..." is an explicit failure check.

3.4 Cleanup and Rollback

Some operations have side effects (creating temporary files, modifying configurations, starting processes) and need cleanup upon failure:

#!/bin/bash
set -e

work_dir=$(mktemp -d)
trap "rm -rf '$work_dir'" EXIT

cd "$work_dir"

# Processing logic
download_file > data.txt
process < data.txt > result.txt
cp result.txt /final/path/

trap ... EXIT means "execute this command when the script exits (regardless of success or failure)". This is the standard pattern for cleaning up temporary resources, detailed in the next section.

More complex scenarios require transactional rollback — recording the side effects of each step and reversing them upon failure:

#!/bin/bash
set -e

# Record operations that need rollback
rollback=()

register_rollback() {
    rollback+=("$*")
}

rollback_all() {
    for cmd in "${rollback[@]}"
    do
        eval "$cmd" || echo "Rollback failed: $cmd" >&2
    done
}

trap rollback_all EXIT

# Business logic
create_user "alice"
register_rollback "delete_user 'alice'"

grant_permission "alice" "admin"
register_rollback "revoke_permission 'alice' 'admin'"

echo "User created successfully"

This "register_rollback + trap" pattern is common in database migration and configuration change scripts.

4. Trap Exits

trap is one of the most underrated commands in shell. It allows you to execute specified code when the script receives a signal or exits — this is the foundation for graceful exit and resource cleanup.

4.1 Basic Syntax

trap 'commands' SIGNAL...

4.2 Three Most Common Scenarios

Scenario 1: Catching Ctrl+C

#!/bin/bash
trap 'echo "Received Ctrl+C, cleaning up..."; cleanup; exit 130' INT

cleanup() {
    rm -f /tmp/workfile
}

# Simulate work
while true
do
    echo "Working..."
    sleep 1
done

Pressing Ctrl+C triggers SIGINT, and trap receives it, cleans up resources, and exits. exit 130 is the standard exit code for "interrupted by Ctrl+C" (128 + 2).

Scenario 2: Cleaning Up Temporary Files

#!/bin/bash
set -e

tmpfile=$(mktemp)
trap "rm -f '$tmpfile'" EXIT

# Do things with tmpfile
echo "Important data" > "$tmpfile"

# Simulate an error
[ 1 -eq 1 ] || true   # Placeholder

# Regardless of success or failure, the EXIT trap will clean up

trap ... EXIT is the standard pattern for "execute cleanup no matter how the script exits". mktemp paired with trap "rm -f" EXIT is the most classic cleanup pattern in shell scripts.

Scenario 3: Ignoring Certain Signals

#!/bin/bash
trap '' HUP    # Ignore SIGHUP
trap '' TERM   # Ignore SIGTERM
trap '' INT    # Ignore Ctrl+C

trap '' SIGNAL means "ignore this signal" — act as if the signal was never received. This usage occasionally appears in daemon processes, but in the vast majority of cases, do not ignore SIGTERM — this is the standard means for your ops colleagues to kill your process; ignoring it will prevent the service from shutting down normally.

4.3 The Magic of the EXIT Trap

EXIT is not a real signal; it is a "pseudo-signal" provided by bash — triggered when the script exits (whether it ends normally, exits via set -e, or is terminated by a signal).

#!/bin/bash
set -e

start_time=$(date +%s)
trap 'echo "Script took $(( $(date +%s) - start_time )) seconds"' EXIT

# Business logic
sleep 2
echo "Done"

Output:

Done
Script took 2 seconds

The EXIT trap, combined with set -e, performs cleanup regardless of success or failure — temporary file cleanup, time elapsed statistics, status reporting can all use it.

4.4 Commands Inside Traps

The command for trap is a string that gets re-parsed and executed by the shell when the signal is received. This means the choice between single and double quotes is critical:

#!/bin/bash
file="/tmp/workfile"

# Single quotes: deferred expansion, calculated when trap triggers
trap 'rm -f "$file"' EXIT   # Correct

# Double quotes: immediate expansion, hardcodes an empty string in the trap
trap "rm -f '$file'" EXIT   # Also works, but variable changes inside the trap won't be reflected

The single-quoted version's $file is expanded when the trap triggers, getting the latest value; the double-quoted version is expanded when the script loads. If the commands inside a trap need to use variables, use single quotes.

4.5 Practical Example: An "Interruptible" Batch Process

#!/bin/bash
set -euo pipefail

total=0
done=0
failed=0

cleanup() {
    echo ""
    echo "===== Interrupt/Exit Summary =====" >&2
    echo "Processed: $done / $total, Failed: $failed" >&2
    exit 130
}

# Call cleanup on Ctrl+C
trap cleanup INT
# Print summary on normal exit
trap 'echo "===== Complete ====="; echo "Processed: $done / $total, Failed: $failed"' EXIT

files=(*.log)
total=${#files[@]}

for file in "${files[@]}"
do
    if grep -q "ERROR" "$file"
    then
        if ! gzip "$file"
        then
            ((failed++)) || true
        fi
    fi
    ((done++))
    echo "Progress: $done/$total"
done

Note ((failed++)) || true — under set -e, if failed was previously 0 and increments to 1, ((...)) returns 0 (because the result is non-zero, bash treats it as "success"), but conversely, if failed was previously 1 and increments to 2, ((...)) returns a non-zero value, triggering set -e to exit. This is a counter-intuitive bash feature; adding || true is a safety measure.

5. Logging

Error handling solves the problem of what to do when things go wrong; logging solves the problem of how to investigate when things go wrong. The two are only complete when paired together.

5.1 Why Logging

Newcomers often think: "I wrote the script myself, I run it myself, why add logging?" But the truth about production scripts is:

Logs are the black box of a script. Without logs after a problem occurs, troubleshooting is just guesswork.

5.2 Basic Requirements for Logging

A qualified log entry should at least answer these questions:

The simplest version:

[2024-01-15 10:23:45] [INFO] Started processing file input.log
[2024-01-15 10:23:46] [ERROR] Parsing failed: Line 12 has incorrect format
[2024-01-15 10:23:46] [INFO] Processing complete: 95 succeeded, 5 failed

5.3 Writing a Logging Function

#!/bin/bash
set -euo pipefail

LOG_LEVEL=${LOG_LEVEL:-INFO}
LOG_FILE=${LOG_FILE:-}

log() {
    local level=$1
    shift
    local timestamp
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    # Filter low-level logs
    case $level in
        DEBUG) [[ "$LOG_LEVEL" =~ DEBUG ]] || return 0 ;;
        INFO)  [[ "$LOG_LEVEL" =~ (DEBUG|INFO) ]] || return 0 ;;
        WARN)  [[ "$LOG_LEVEL" =~ (DEBUG|INFO|WARN) ]] || return 0 ;;
    esac

    local msg="[$timestamp] [$level] $*"
    if [[ -n "$LOG_FILE" ]]
    then
        echo "$msg" | tee -a "$LOG_FILE" >&2
    else
        echo "$msg" >&2
    fi
}

# Usage
log INFO  "Starting processing"
log WARN  "Configuration file does not exist, using defaults"
log ERROR "Database connection failed"
log DEBUG "Internal variable: $internal_var"   # Not displayed by default

A few design points:

5.4 Debug Logging

Debug logs are key to troubleshooting problems, but you don't want them too noisy in a production environment. A common practice is to use BASH_SOURCE and LINENO to automatically add location information:

#!/bin/bash
set -euo pipefail

debug() {
    [[ "${DEBUG:-0}" == "1" ]] || return 0
    local timestamp
    timestamp=$(date '+%H:%M:%S.%3N')
    echo "[$timestamp] [DEBUG] ${BASH_SOURCE[1]##*/}:${BASH_LINENO[0]} $*" >&2
}

# Usage
debug "Entering process function, argument: $1"

process() {
    debug "Processing $1"
    # ...
}

Output (with DEBUG=1):

[10:23:45.123] [DEBUG] script.sh:14 Entering process function, argument: input.log
[10:23:45.124] [DEBUG] script.sh:19 Processing input.log

BASH_SOURCE[1] and BASH_LINENO[0] are the "call stack" maintained internally by bash; the subscript 1 indicates "the file that called the current function" and "that line". This trick is very useful for debugging in large scripts.

5.5 Using the logger Command to Write System Logs

Linux comes with the logger command, which can write messages to syslog:

#!/bin/bash
log() {
    local level=$1
    shift
    logger -t "myscript" -p "user.$level" "$*"
}

# Usage
log INFO "Starting processing"
log ERROR "Database connection failed"

The output will go to /var/log/syslog (Debian/Ubuntu) or /var/log/messages (CentOS/RHEL), and can be viewed using journalctl -t myscript. Suitable for long-running service scripts, not suitable for one-off tasks.

5.6 Log Rotation

Writing too many logs can fill up the disk. Log files for production scripts need rotation — periodic cutting, compression, and cleanup.

If using logger to go through syslog, rotation is handled by the system's logrotate. If writing files yourself, you can use a logrotate configuration file:

# /etc/logrotate.d/myscript
/var/log/myscript.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    postrotate
        # Notify script to reopen log file
        kill -HUP $(cat /var/run/myscript.pid 2>/dev/null) 2>/dev/null || true
    endscript
}

Or handle it within the script itself (using SIGHUP to trigger reopening the log file). Everyday scripts won't need this level of complexity; just knowing it exists is enough.

6. Complete Practical Example

In this section, we'll see how to harden a data processing script to production grade, observing how error handling and logging are implemented in a real project.

Original Version:

#!/bin/bash
declare -A total_amount
declare -A order_count

while IFS=',' read -r order_id product quantity price date
do
    [[ "$order_id" == "order_id" ]] && continue
    qty=${quantity:-0}
    prc=${price:-0}
    [[ ! "$qty" =~ ^[0-9]+$ ]] && continue
    [[ ! "$prc" =~ ^[0-9]+$ ]] && continue
    amount=$((qty * prc))
    total_amount[$product]=$(( ${total_amount[$product]:-0} + amount ))
    ((order_count[$product]++))
done < sales.csv

echo "product,total_amount,order_count,avg_amount"
for product in "${!total_amount[@]}"
do
    total=${total_amount[$product]}
    count=${order_count[$product]}
    avg=$((total / count))
    echo "$product,$total,$count,$avg"
done | sort -t',' -k2 -rn

Hardened Version:

#!/bin/bash
#
# analyze_sales.sh - Analyze sales data
# Usage: analyze_sales.sh <input CSV> [output CSV]
#

set -euo pipefail

# ============================================================
# Configuration
# ============================================================
SCRIPT_NAME=$(basename "$0")
LOG_LEVEL=${LOG_LEVEL:-INFO}
LOG_FILE=${LOG_FILE:-}

# ============================================================
# Logging Function
# ============================================================
log() {
    local level=$1
    shift
    local timestamp
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    # Level filtering
    case $level in
        DEBUG) [[ "$LOG_LEVEL" =~ DEBUG ]] || return 0 ;;
        INFO)  [[ "$LOG_LEVEL" =~ (DEBUG|INFO) ]] || return 0 ;;
        WARN)  [[ "$LOG_LEVEL" =~ (DEBUG|INFO|WARN) ]] || return 0 ;;
    esac

    local msg="[$timestamp] [$level] $SCRIPT_NAME: $*"
    if [[ -n "$LOG_FILE" ]]
    then
        echo "$msg" | tee -a "$LOG_FILE" >&2
    else
        echo "$msg" >&2
    fi
}

die() {
    log ERROR "$*"
    exit 1
}

# ============================================================
# Argument Validation
# ============================================================
[ "$#" -ge 1 ] || die "Usage: $SCRIPT_NAME <input CSV> [output CSV]"
INPUT=$1
OUTPUT=${2:-}

[ -f "$INPUT" ] || die "Input file does not exist: $INPUT"
[ -r "$INPUT" ] || die "Input file is not readable: $INPUT"

log INFO "Starting analysis of $INPUT"

# ============================================================
# Temporary Files
# ============================================================
work_dir=$(mktemp -d)
trap "rm -rf '$work_dir'" EXIT

# ============================================================
# Business Logic
# ============================================================
declare -A total_amount
declare -A order_count
declare -A invalid_rows
total_rows=0
skipped_rows=0

while IFS=',' read -r order_id product quantity price date
do
    ((total_rows++)) || true

    # Skip header
    [[ "$order_id" == "order_id" ]] && continue

    # Field validation
    if [[ -z "$product" ]]
    then
        invalid_rows[$total_rows]="Missing product"
        ((skipped_rows++)) || true
        continue
    fi

    if [[ ! "$quantity" =~ ^[0-9]+$ ]] || [[ ! "$price" =~ ^[0-9]+$ ]]
    then
        invalid_rows[$total_rows]="quantity/price non-numeric: $quantity/$price"
        ((skipped_rows++)) || true
        continue
    fi

    # Aggregation
    amount=$((quantity * price))
    total_amount[$product]=$(( ${total_amount[$product]:-0} + amount ))
    ((order_count[$product]++)) || true
done < "$INPUT"

# ============================================================
# Output
# ============================================================
output_file="$work_dir/result.csv"
exec 3> "$output_file"   # Open file descriptor 3 pointing to output

echo "product,total_amount,order_count,avg_amount" >&3

for product in "${!total_amount[@]}"
do
    total=${total_amount[$product]}
    count=${order_count[$product]}
    avg=$((total / count))
    echo "$product,$total,$count,$avg" >&3
done | sort -t',' -k2 -rn

exec 3>&-   # Close file descriptor

# ============================================================
# Wrap-up
# ============================================================
if [[ -n "$OUTPUT" ]]
then
    cp "$output_file" "$OUTPUT"
    log INFO "Results written to: $OUTPUT"
else
    cat "$output_file"
fi

log INFO "Processing complete: Total rows $total_rows, Skipped $skipped_rows, Successful ${#total_amount[@]} products"

if [[ ${#invalid_rows[@]} -gt 0 ]]
then
    log WARN "${#invalid_rows[@]} rows have issues:"
    for row in "${!invalid_rows[@]}"
    do
        log WARN "  Row $row: ${invalid_rows[$row]}"
    done
fi

Comparing the two versions, the hardened one adds these capabilities:

How to use it:

# Normal usage: results output to stdout
./analyze_sales.sh sales.csv

# Output to file + detailed logging
LOG_LEVEL=DEBUG LOG_FILE=/var/log/analyze.log ./analyze_sales.sh sales.csv result.csv

In contrast, what would the unhardened version do?

This is the gap between "can run" and "reliable".

7. Summary

Error handling and logging are the keys to turning a script from a toy into a tool. Let's review the key points:

In this article, we've completed the robustness of our scripts. Next, we'll continue down this path of engineering — talking about how to modularize shell scripts: extracting common functions into library files, how to write scripts that can be sourced, and how to handle multi-script projects. When a script grows from a few hundred lines to a few thousand, whether you can organize it well is the difference between being able to write it and being able to maintain it.

The examples in this article were tested in GNU bash 4.3+ environments. BASH_SOURCE and BASH_LINENO are supported since bash 3.0+. The logger command requires a syslog service; for containerized deployments, it is recommended to use file logging instead. date '+%3N' (milliseconds) requires GNU date.