Package Issues - A Postmortem

Over the past few weeks, many Nanobox users have run into issues with packages that have prevented successful builds and, in some cases, hindered their ability to deploy. These issues have been wide-spread and, for many, have raised red flags of instability. One of our core values at Nanobox is complete transparency. This article provides a history of the issue, information about what went wrong, what has been done about it, and what we're doing to prevent it from happening again.

TL;DR: Package issues have been fixed. If you aren't interested in the details and just want the fix, skip to here.

What Happened

Nanobox maintains its own custom-compiled package library from which packages can be downloaded, installed, and included in both your development and production environments. This is done to 1) Keep packages light by removing any unnecessary cruft and 2) Ensure the binaries are safe and reduce the potential attack plane. These custom packages are managed and installed using pkgsrc.

In an effort to keep our packages up to date, we merged in "security" updates from source repositories. Unfortunately, these updates are the source of the issues many have been dealing with. At first we weren't sure exactly how wide-spread the effects were, but it soon became apparent – it affected many of the most commonly used packages in the Nanobox pkgsrc repository.

Sources of the problems:

  1. pkgin, the binary package manager for pkgsrc, kind of sucks at resolving required dependency versions. Depending on the order in which packages are loaded, you'll get different dependency versions and ultimately, version conflicts and/or broken packages. The updated packages came with updated dependency trees and pkgin's lackluster dependency resolution wreaked all sorts of havoc.
  2. Packages were simply broken by the updates. The hardest part about this is that packages would still compile. Issues wouldn't manifest until packages were installed and/or used.

Fixing Packages One by One

At first, we addressed specific packages as issues were reported. In most cases, we'd fix one package only to reveal issues with others. Even after packages were fixed, if someone had already downloaded a bad package, it was cached and there was no simple way to clear it out other than manually (nanobox run --debug > rm -f /path/to/bad-package). It was a major headache and one you shouldn't have to deal with.

Ultimately we realized we were only applying bandaids to a gaping, festering wound. This process ate up a huge chunk of our engineering team's time and wasted a lot of yours. For that, we are incredibly sorry.

What We've Done

As of today, we have rebuilt our entire pkgsrc repository with updated packages and correctly resolved dependencies. Also, the newest build image automatically clears your package cache if the packages aren't pulled from the latest pkgsrc.

What You Should Do

To transition over to the new pkgsrc repository, do the following:

Make Sure Your Package Versions Still Exist

The old pkgsrc repo was full of minor and patch versions for different packages due to natural, organic growth of both packages and the package repo. When rebuilding the pkgsrc repository, many minor and patch versions were removed.

If you specify packages' minor and patch versions in either extra_packages or dev_packages, make sure those packages still exist in the new pkgsrc repository:

If the specific version you're using is no longer available, update the package entries in your boxfile.yml to match those available in the new pkgsrc repo.

Check Runtime Versions

The removed minor and patch versions also affect what language versions are available. Runtime lists have been udpated in guides and engine documentation.

Note: This is especially true for Node.js. The list of available Node.js runtimes has been minimized to essentially one minor version per major version in most cases.

Popular Engine Runtime Lists:
Elixir | Go | Node.js | PHP | Python | Ruby | Scala

Update All The Things!

Once you've made sure all packages and runtimes listed in your boxfile.yml are still available in the new pkgsrc, run the following:

# Update to the latest version of Nanobox Desktop
nanobox-update

# Update the build image to pull from the new pkgsrc repo
nanobox update-images

# Run a new build
nanobox build

Note: Running a new build must be done for each individual project.

Ensure packages load correctly during the build. If you still run into issues, please report them immediately in the Nanobox Slack channel or to help@nanobox.io.

Preventing This From Happening Again

The core of the issue stemmed from updating core dependencies without rebuilding all packages. Moving forward, we will never update core dependencies without rebuilding our entire pkgsrc repository. Build images will be updated to pull from the most recent pkgsrc repo and will clear the package cache if cached images are old.

We are also going to reduce the scope of packages compiled and maintained by Nanobox. There are a handful of custom Nanobox packages on which engines depend that we will always custom-compile, but the list is small. This will free up a lot of time for our package maintainer(s) and allow us to focus efforts on building out new functionality listed on our roadmap.

To handle other packages, we will introduce the ability to pull from other sources. We're still in the process of architecting how it will all work, but essentially, you won't have to ask us to compile packages you need.

Last of all, we are writing a much more robust test suite for packages and dependencies. We never want you to have to deal with something like this again.

Thank You

Thank you so much to everyone who has reported issues and stuck with us through the past few weeks. We are so sorry for the headaches it's caused and look forward to the coming changes. We have been and always will be committed to providing the best possible experience to developers as you build, deploy, and conquer the world.

We are constantly looking at ways to improve our service. If you have any feedback or questions, feel free to drop into the Nanobox Slack channel (you can request an invitation here), send an email to help@nanobox.io, or give us a shoutout on Twitter (@nanobox_io).

Posted in Postmortem