Telling a Computer to Do Things · will keleher

writing
all posts
about.md
rss.xml

Telling a Computer to Do Things
For the first few years of my career, I didn’t know how to tell my computer to do things. I could kick off a few commands from the terminal – run those tests, install that dependency, start that container, ssh to that machine – but I was limited to running simple commands one at a time. My terminal was the world’s worst GUI, and I thought that the shell was the way to start programs that didn’t have application wrappers.

--command--
mocha
git
brew install
npm install
rm

At a fundamental level, I wasn’t able to tell my computer to accomplish anything that involved logic or stitching together multiple programs like:

Do one thing and then do another
If a command fails, log out an error message
Kick off a program when the computer starts
Run two commands at the same time
Loop through all of the files in a directory and take an action on each one
Use the output of one command as the input for another

In theory, I could have used NodeJS to write those sorts of programs. In practice, I never did. This was partially mindset: I wasn’t used to thinking about the programs I used on the command line as things that I could control. And the rest of it was a lack of skill: I didn’t know the programs that I was using well enough to integrate them into a script that I’d written.
I kept trying to learn the shell though, and I slowly got to the point where I could muddle my way through scripts like this one that had basic logic:
set +e
npm install
status_code=$?
if [[ "$status_code" != "0" ]]; then
echo "Something went wrong with your npm install. Check your ~/.npmrc to make sure it's authed to our registry."
exit 1
fi
set -e
Depending on your familiarity with shell scripting, You might be gibbering right now. Sorry. (If you’re not bleeding from the eyes yet, here’s why you should be: 1)
Even though the commands and scripts I wrote had problems, it was transformational for me; I had the sudden ability to tell my computer to stitch together existing programs to accomplish my goals. It felt a little bit like the change that came from learning to program in the first place.
Over the course of years, I slowly started to learn more shell tools and figure out shell syntax. I’d often learn a new tool or pattern and then have a moment of pain when I realized how much easier a past problem would have been to solve if I hadn’t used a hammer to solve a problem that needed a drill.
Over that time, I’ve worked with a ton of (incredibly strong!) engineers who didn’t spend as much time learning the shell, and instead relied on GUIs to do things like run tests, manage git, talk to databases, and do day-to-day tasks. Relying on a GUI works well until you want to accomplish something that the GUI wasn’t set up to handle, and I think it’s easy to slip into a mindset where you’re limited to what the GUI is capable of. I’ve seen skilled engineers spend a ton of effort because they didn’t know how to do things like use while to keep running a command or write a for loop to do the same operation on every file in a directory.
I think that same GUI-focus can be a problem when it comes to editing and maintaining shell scripts. I’d wager most companies have a decent amount of essential logic to build, deploy, validate code, and test in languages like Bash or Zsh. If you’re not comfortable with the language your tooling is built in, then you won’t be able to easily read or improve it. You might be able to tell the remote servers that your code runs on how to behave but not be able to tell the computer that you work on how to do things like run linting and tests in parallel – that’s a bummer!
Let’s pause to take a quick detour: Why the heck are so many of these scripts end up written in languages that aren’t the main ones the team uses? I think it’s often more ergonomic to write a script that stitches together commands in a language that’s been designed to be easy to stitch together commands. Let’s take a super simple example of running a test until it fails: while pnpm exec mocha ./pathToFile.test.ts; do true; done. There are obvious things to critique about this syntax, but let’s take a look at what it looks like in NodeJS:
const { execSync } = require("child_process");
while (true) {
try {
execSync(`pnpm exec mocha ./pathToFile.test.ts`, { stdio: "inherit" });
} catch (err) {
console.error("failed", err);
break;
}
}
There are a lot of rough edges and gotchas here, and I personally think the shell is easier! I don’t need to worry about creating a file, requiring child_process, or setting { stdio: "inherit" } to see output. And this is a pretty simple example that doesn’t even stitch together multiple programs with a pipe, capture any output, or use a temporary file!

💡

This doesn’t mean that you need to resign yourself to writing in Bash or Zsh or any similar language! For teams that know JavaScript well, one tool I’ve enjoyed is zx. I think it can make these scripts pretty ergonomic to write and maintain.
Ruby and Python are both easier than NodeJS is, but I think there are plenty of languages out there that require even more ceremony to write a quick little script like this.

I’m certainly not arguing that shell scripts will always be easier for build scripts! When problems are simple enough that you’re just stitching two programs together, a tool like Bash or Zsh feels pretty ergonomic. As soon as you want more sophisticated logic and data types, you’ll want to choose a language that makes it easy to represent (and test!) more sophisticated logic and data types.
I suspect that many engineers who gripe about build scripts being written in a shell language haven’t actually tried converting them to a different language. Aside from the syntax (potentially) being more complicated, a huge part of "learning to write shell scripts" isn’t actually syntactical. If you convert a script that stitches together commands but don’t actually know how the commands you’re stitching together behave, the resulting script is likely to be similarly impenetrable.
Knowing the shell – being able to tell a computer to do things – depends a ton on knowledge of the programs that do the things that you want to accomplish! I’d argue that knowing the shell is 20% syntax and 80% having a good toolbox:

If you know fzf, you can build quick utilities with interactive fuzzy-searching. (Example: git checkout $(git branch --sort=-committerdate | fzf) will let you fuzzy-choose a branch.)
If you know tldr or eg, you can pull up usage examples for any other command in this list
If you know rsync, you can copy changed files on to a faster remote server to run something heavy and slow
If you know xargs, you can build up commands incrementally and parallelize work
If you know sed -i or ast-grep, you can quickly rewrite complicated patterns across a bunch of files
If you know direnv, you can make sure the right environment variables are set for everyone who runs commands in a codebase.
If you know duckdb, you can write SQL to query CSV and JSON files locally as part of a larger script.
If you know gh, you can build scripts to check on your PRs and open up new PRs from the cli.
If you know ngrok, you can quickly serve a local port to test something out on a different machine.

Each additional program you learn expands your capabilities more because each new tool can be used with every other tool you already know.
I can’t stress enough that I’m the furthest thing in the world from a shell scripting expert, and I’m terrible compared to people who know it deeply,2 but I’ve still gotten a lot of value out of knowing enough shell syntax to stitch together programs and knowing enough programs that I actually want to stick together.
Telling your computer to do things is great!

This script was written with good intentions, but it was overly complicated because I didn’t know that if's main mode is to take commands. [[ (or test) is just a special command. This means that our if check can just be ! npm install.

With this approach, there’s no need to set +e to allow commands to fail. (set +e and then setting it back with set -e is to avoid the shell-script as a whole failing because a command failed – Unofficial Bash Strict Mode)
It also means we don’t need to capture the exit code with $?

if ! npm install; then
echo "Something went wrong with your npm install. Check your ~/.npmrc to make sure it's authed to our registry."
exit 1
fi
 ↩︎

From an HN comment on a previous blog post: "This post […] means I no longer wonder why 99% of shell scripts I come across look inept." ↩︎

The author details a journey of learning how to instruct a computer to perform complex tasks, initially encountering limitations when restricted to executing simple commands individually from the terminal. Early experience focused on running discrete commands, but the author recognized the need to accomplish tasks involving logic, sequential execution, error handling, parallel operations, and chaining the output of one program as input for another. Although theoretical knowledge suggested languages like NodeJS could handle this, the practical realization was that mastering shell scripting provided a more direct and ergonomic way to stitch together existing programs to achieve goals.

The transition to shell scripting was transformative, offering the ability to reason about programs that operate as sequences of commands. This experience highlighted a dichotomy: reliance on Graphical User Interfaces (GUIs) limits capabilities to what the interface is designed to handle, whereas deep understanding of the command-line environment unlocks capabilities beyond GUI limitations. Skilled engineers often find themselves constrained when the GUI does not facilitate necessary operations, such as running tests in parallel or managing complex file operations.

The text explores the relationship between scripting languages and the operating system environment. While shell scripting is highly effective for sequencing commands, the author notes that complex logic and sophisticated data manipulation might be better suited for languages designed for those purposes. The perceived advantage of shell scripting is its ergononomics in stitching commands together, but the author suggests that true capability stems not solely from syntactic mastery but from knowing the underlying programs and tools.

The ability to effectively use the shell is argued to be more dependent on possessing a robust "toolbox" of tools than on memorizing specific syntax. The value of knowing the shell is derived from familiarity with powerful utilities such as fzf for interactive searching, rsync for efficient file transfer, xargs for parallelization, sed or ast-grep for pattern manipulation across files, and duckdb for local data querying. Each learned tool expands functional capability by allowing for combinations with existing knowledge.

Furthermore, the text offers an example of how structural awareness within shell scripting can lead to more concise and robust code, demonstrating that the structure of the command execution itself can simplify error handling and logic, potentially negating the need for explicit verbosity like setting options to handle failures before performing checks. Ultimately, while shell scripting is highly valuable for system automation, the most significant advantage lies in understanding the operational context and the ecosystem of programs that the shell hooks into.