pkgsrc Is Cool

Home
Posts
Links
Mirrors
About
RSS

pkgsrc Is Cool
Published: 2021-12-23 22:17:43 -0800 PST
Last updated: 2022-04-24 21:06:24 -0700 PDT
I think I saw something about pkgsrc pop up randomly in my
Mastodon feed a few months ago. Before that, I had never heard of
it. And now…well now I’m currently writing these words in a version
of Emacs that I built from source with pkgsrc.
Pkgsrc is NetBSD’s package manager, but thanks to NetBSD’s portability
philosophy, pkgsrc is fully supported as a package manager on other
Unix-like operating systems like Linux, Mac, and Solaris. It supports
installing binary packages, but the reason I was interested in it is
that pkgsrc supports building packages from source as well.
For some context on other similar projects, pkgsrc was originally
forked from FreeBSD’s Ports collection decages ago, so for people
familiar with the other BSD’s, this is a ports collection that works
on other platforms. Even better, pkgsrc and NetBSD get the terminology
right and correctly call these things packages instead of ports (and
leave “port” as the term for cpu architecture support). Other
similar-ish projects include MacPorts, Gentoo’s emerge, GNU Guix, and
Ravenports. I probably missed some, but you get the point. There are
multiple options out there if you’re interested. I’m not going to
compare/contrast them because honestly the only other one I’ve ever
used was MacPorts like 10 years ago when I didn’t know what I was
doing.
Why did I use it?
I’ve always disliked needing to install a new package to get some
program to run. The main reason why is that the new package could pull
in other packages and they all end up polluting my system folders like
/bin and /lib with extra stuff. Half the time I delete the program
I was testing out anyway, but then I’m stuck with this extra stuff in
my base system. I never really even stopped to think that there might
be way to separate out the “base system” from the other crap I’ve
installed along the way.
But this is exactly what NetBSD does. NetBSD’s base system for /bin
and such is really only for the core NetBSD system code like the stuff
you can build from their source code repo. Anything else is supposed
to be installed somewhere else. By default, pkgsrc will install stuff
into /usr/pkg where it created its own subdirectory tree of
/usr/pkg/bin and /usr/pkg/lib and so on. This way that specfic
pkg directory is like a self-contained userland separate from the
base operating system. I’m actually using pkgsrc on Debian rather than
NetBSD itself, but it works the same way.
Even better, pkgsrc lets you use it in unprivileged mode and install
stuff locally in your user’s home directory like ~/pkg. This can
give your user their own separate custom userland from other
users. Depending on your use case this may not be what you want, but I
found it particularly cool that I could run almost a completely
separate system just for my user. Because I didn’t want to need
sudo/doas to install packages, I am actually using pkgsrc in
unprivileged mode with ~/pkg as my base directory.
So that was a long explanation to this answer. Why did I start using
pkgsrc? Because I can install packages locally in my home directory, I
can build those packages from source, and I know this system will work
on any operating system I will ever run into (and I’ve used some weird
ones considering how relatively young I am).
How it works
Pkgsrc isn’t really a package repository itself. It is simply a
collection of Makefiles that define how to build each package as well
as the dependencies needed for a package. Each package’s Makefile
defines where to download the source code for the package, how to
apply any necessary patches, and then how to build it. This actually
answered a long-standing question for me: why do GitHub “releases”
build a tarball/zip snapshot of the source code? Well, systems like
pkgsrc will use that tarball to download the source code for a
specific version without needing to clone a full repository (or even
needing the source control software installed at all).
Pkgsrc reaches out to various websites like GitHub and Sourceforge (or
wherever else a project’s source code is hosted) to download the
specified version of a package to build it. This is highly
decentralized because pkgsrc does not rely on a single repository
server, instead relying on each project’s own servers. (As a random
side note, this is similar to how the Go programming language handles
dependencies). This could lead to the potential problem where a
single project’s website is down so you can’t build their
software. However, the Makefiles also specify alternative distfile
mirrors for where to download the source code. New specific
terminology for me here was a “distfile” which is that source code
tarball snapshot for a specific version of the package.
Tinfoil Hat Time
One potentially “irrational” requirement of mine was that I wanted to
be able to install software from my own network in case the internet
ever goes down. Not just my internet, but THE internet. And I know
that the internet as whole will probably never really go down, but I
do think it may get to the point where the internet is fragmented into
completely different silos because of government censorship.
In a more practical manner, I also wanted to be able to install
software from my local network to save bandwidth when updating
multiple computers. This is why I host some mirrors on my server.
One of those mirrors is the pkgsrc distfile mirror. I was able to use
pkgsrc with the MASTER_SITE_OVERRIDE option to tell it to use my own
distfile mirror. This allows me to build all of the supported pkgsrc
packages offline because the distfile mirror has already reached out
and downloaded all of the source code for every project (or rather,
I’m mirroring from a mirror that has already done that).
I realize this is a weird almost “prepper’ doomsday mentality, but I
really wanted to be able to install software locally on my own
network. Pkgsrc gave me the solution to be able to build the software
from source for any operating system on any cpu architecture that I
could find and do it all from my own server.
More Articles
After searching for more information on pkgsrc, I found these other cool links:

Pkgsrc.org
The pkgsrc guide (there’s also a PDF version here)
NetBSD wiki page about pkgsrc
Code Ghar: Essential pkgsrc - The Missing Mini Handbook
Rubenerd - Using NetBSD’s pkgsrc everywhere I can
NASA even uses it apparently
There is a subreddit for it

Followup
To keep this from ballooning into one gigantic article, I’ll keep this
one as simply why I think pkgsrc is cool and why I wanted to use
it. I’ll follow up with another article about how exactly I’m using
it.
I should probably also write another article about using Clang as the
default pkgsrc compiler since I messed around with that until I
finally got it to work. Their documentation about it is a bit
scattered and misleading.
I did end up writing more posts about pkgsrc!

pkgsrc with Clang
How I use pkgsrc

Git - Mastodon
www.wisellama.rocks by Wisellama is marked with CC0 1.0

Pkgsrc functions as a package manager originating from NetBSD's portability philosophy, extending its capabilities to support installation on various Unix-like operating systems such as Linux, macOS, and Solaris. It supports the installation of binary packages but is particularly valued because it facilitates building software packages directly from their source code. Historically, pkgsrc was forked from the FreeBSD Ports collection, and the author notes the correct terminology, using "packages" instead of "ports," while reserving the term "port" for CPU architecture support, positioning it alongside other similar projects like MacPorts, Gentoo's emerge, GNU Guix, and Ravenports.

The motivation for utilizing pkgsrc stems from a desire to manage system installations without cluttering the base operating system directories, such as /bin and /lib, which often suffer from extraneous files installed by newer packages. Pkgsrc achieves this separation by installing packages into a dedicated location, typically within /usr/pkg, creating a self-contained userland separate from the core operating system. Furthermore, the system allows for advanced use in an unprivileged mode where packages can be installed locally within the user’s home directory, such as ~/pkg. This capability allows a user to establish a separate custom userland environment, offering a degree of isolation without necessitating elevated privileges like sudo or doas.

The core mechanism of pkgsrc does not involve maintaining a centralized package repository; instead, it operates as a collection of Makefiles that define the necessary instructions for building each package, including specifying dependencies. These Makefiles dictate where to download the source code, how to apply necessary patches, and the steps required for compilation. Pkgsrc automatically sources the required package data by reaching out to various hosting sites, such as GitHub or Sourceforge, to retrieve the specified version of the package source code. This architecture promotes decentralization, as it relies on each project's own servers rather than a single repository, although the Makefiles also permit the definition of alternative distfile mirrors for source downloading.

The author explored the utility of pkgsrc for offline operation, driven by a desire to install software locally and mitigate reliance on the internet. To achieve this, the author utilized a local mirror for the pkgsrc distfile repository, employing the MASTER_SITE_OVERRIDE option to direct pkgsrc to use this local mirror. This setup enables the ability to build all supported packages offline, as the necessary source code snapshots have already been downloaded, providing a means to install software from a local network environment. This process provided a solution for building software from source across different operating systems and CPU architectures directly from a local server.