Backstory

Rally Health’s offerings are powered by a fleet of microservices. At the time of writing, we have roughly 80 separate services.

Developing and testing a stand-alone service (with, say, a database dependency) is easy: install the dependency locally, bang on the keyboard, results!

Unfortunately developing services that depend on other services is harder. To do it properly you need to stand up a spider web of services, starting with the services your service depends on, then those services’ dependencies, around and around until you are running the entire system (or get lucky and find some working subset).

Rally Health℠ is working on an in-house solution by building a pair of tools: Maestro and Roost. Maestro is a CLI written in Go that wraps and extends Docker Compose to work within our tool-chain. Roost is a service written in Scala that acts as a facade for the config data Maestro needs.

What Happened

Maestro is Rally Health’s first project using Go. We started at the ground floor, following the typically recommended guides [1] [2] [3], and we looked for advice from the existing gophers [1] [2] [3] [4] [5] [6] [7]. Maestro integrated with Docker Compose by depending on the docker/libcompose and docker/docker projects. (Docker writes a majority of its projects using Go and that is one of the main reasons we decided to use Go.)

Progress was going great… until a routine make clean unexpectedly failed. A short investigation later and we learned that in April 2017 Docker was renamed to Moby. We were not the only ones surprised by this change [1] [2] [3] [4].

One of the interesting design choices of Go is that imports can reference github.com projects. It’s an interesting feature until someone moves a repository to a new owner and name. Which is exactly what Docker did when they moved docker/docker to `moby/moby’.

Problem

The docker/libcompose code that Maestro depends on contains many import "github.com/docker/docker" statements. Unfortunately docker/libcompose’s imports were not updated after the rename. So when we attempted to replace Maestro’s dependency from docker/docker to moby/moby everything fell apart – docker/libcompose does not recognize moby/moby types, and there is no way for docker/libcompose to know that they are the same types with a different package name.

Go’s import design and the docker-to-moby rename wrecked our development setup:

  • Updating our dependencies failed (docker/docker no longer exists)
  • CI completely broke (i.e. the worker can’t download the dependencies)
  • New developers could not create a working development environment
  • Homebrew install failed (because it compiles Maestro locally)

Our investigation of the rename and dependency design identified three places in our development setup that needed changes.

Go Get

The standard dependency tool – go get – has some interesting design decisions:

  • Dependencies are source downloads; there is no JAR or RPM equivalent in Go.
  • Dependencies are downloaded from the original repositories; there is no artifact repository.
  • A peculiar strategy to download from whichever branch matches your locally installed version of Go, e.g. branch “go1” if you have that version installed.
  • The current HEAD of the branch is downloaded – not a specific commit or tag.
  • Dependencies are stored in the GOPATH.
  • Dependencies are organized by name without any branch, commit, or tag identifier.

In short, the lack of an easy-to-backup package or immutable version identifier makes it difficult to ensure that your change occurs only when you explicitly request an update.

Note: A more powerful dependency management tool is currently under development but it is still in alpha.

GOPATH

The GOPATH is a shared directory that holds your dependencies, and each dependency lives in a directory (matching its name) without a version number (or equivalent identifier).

Let’s say you have two Go projects, Project A and Project B, and they have a shared dependency, lib-z, that is stored in GOPATH. That means that Project A and Project B will always use the exact same code when compiling against lib-z. You can’t make Project A depend on lib-z-v1 and Project B depend on lib-z-v2 because there is no version identifier in GOPATH. You need to stay on top of updates to all projects that depend on lib-z since you can’t reference an older version.

Before you start choosing your throwing stone, let me point out that GOPATH is quite similar in design to Python’s PYTHONUSERBASE and Ruby’s GEM PATHS. The difference is that they have solved their shared packages problem with tools such as virtualenv, rbenv, and rvm. Unfortunately, Go seems not to have an official or de-facto solution for this problem yet.

Vendoring

Go 1.5 introduced a technique to help mitigate the limitations described in the last section: vendoring. In short, every Go project can store a copy of its dependencies in a /vendor directory instead of GOPATH. Real-world experience has suggested that it’s only a good solution for projects that aren’t libraries; unfortunately that is not always followed in practice, e.g. docker/libcompose.

What if lib-z contains a /vendor dependency that you wish to use in your project? For example, lib-z depends on awesome (and fake) pony4Go library, and you’d also like to use pony4Go. It would be nice to just reference the pony4Go that lib-z depends on, rather than trying to add your own dependency on pony4Go and trying to make sure they are the same version. Unfortunately, you can’t simply import that dependency directly – e.g. import “lib-z/vendor/pony4Go” – you’ll get a compiler error. Why? The vendoring proposal says “Code in vendor directories is not subject to import path checking.”

Solution

We’ve identified the three areas we need to improve. How do we improve them?

Problem Solution
Go Get Different Dependency Tool
GOPATH Isolate per workspace
Vendoring Move /vendor/* to /src

Go Get Alternative

After a quick look at the offerings we settled on using gvt which seemed to us to be the simplest dependency management tool available at the time. We can specify a specific branch, tag, or commit to download, it downloads into the local /vendor directory, and the details of what are downloaded will be stored in a /vendor/manifest.json.

Using gvt improves our isolation and moves us closer to repeatable builds. Unfortunately it is not enough. While it is extremely unlikely that the source code identified by a tag or commit would change, it is possible that the repository itself moves or is no longer accessible. So we go one step further: we commit our /vendor directory directly to our code base.

As a part of our builds, now we use what is committed and we only use gvt when we are adding/updating a dependency.

Isolation

Most of the Go tools assume you are using the standard workspace directory layout, and the GOPATH points to the root of the workspace. We do not want to rock that boat – so we make separate workspaces for each project and set the GOPATH to the current workspace directory when using Go tools:

projectA/                           # GOPATH / Workspace
    .git/                           # Git repository metadata
    src/
        myOrg/
            projectA/               # project directory
                hello/
                    hello.go        # command source
                vendor/             # vendor directory
projectB/                           # GOPATH / Workspace
    .git/                           # Git repository metadata
    src/
        myOrg/
            projectB/               # project directory
                world/
                    world.go        # command source
                vendor/             # vendor directory

It’s the same workspace layout, just nested. The most significant difference is that the Git repository is now the workspace directory.

We use make to build Maestro so it’s easy to change the GOPATH for the Go tools by adding export GOPATH:=$(shell pwd) to our makefile. With that change, all Go tools that run from our makefile will use the project-specific GOPATH.

To be clear, we still have a shared GOPATH. It’s just used for binary tools (like richgo) that are not strict dependencies for our projects.

There is another reason we settled on this design: during our efforts to get Maestro working again we learned that go build looks at both GOPATH and the local /vendor directory when searching for dependencies. If there is a conflict – the same dependency in both but different source code – compilation will fail. Apparently, your /vendor does not override the contents of GOPATH. (At least not consistently; this was a bit of a heisenbug for me.) Setting the GOPATH to the workspace avoids the possibility of adding or updating a dependency in the shared GOPATH that would break Maestro.

There are several existing tools that aim to do the same thing as our chosen solution (goenv, VenGO, virgo, etc). However, when this hit, we were deep into meatball surgery mode. We consider our approach – isolating GOPATH – an MVP that made us resilient to upstream dependency changes. We are planning a proper evaluation of existing tools once Maestro reaches it’s own MVP status.

De-vendoring

In the previous section, I glossed over the first part of our solution: using gvt to download dependencies to /vendor. Go does not support /vendor at the root of the workspace; that only works within the project’s directory.

Maestro’s source code lives in /src, and we have .gitignore rules to only source control Maestro’s code. We use gvt to download to /vendor (as described before). What’s new is that we add a pre-build rule (e.g. something like this) to our makefile that copies the dependencies from /vendor to /src.

Before running the pre-build makefile rule:

projectA/                           # GOPATH / Workspace
    src/
        myOrg/                      # project directory
            projectA/
    vendor/                         # vendor directory
        otherOrg/
            lib-z/                  # vendor-ed dependency

After running the pre-build makefile rule:

projectA/                           # GOPATH / Workspace
    src/
        myOrg/                      # project directory
            projectA/
        otherOrg/                   # un-vendor-ed dependency
            lib-z/
    vendor/                         # empty vendor directory

This copy step has two interesting and useful side-effects:

First, it makes trying out a new dependency a little easier. You can try out a new dependency using go get (which will download into /src) without fear of mistakenly committing the code or changing /vendor/manifest.json. When you are sure you want to add the dependency, you’ll re-add the dependency using gvt and specify the current commit or tag.

Second, having a /vendor allows you to have temporarily conflicting dependencies when building unrelated binaries. For example, Maestro uses golint which depends on the standard context library. But docker/libcompose depends on golang.org/x/net/context. They are not identical, so one or the other fails to compile.

This is where having a /vendor directory can help you build those binary dependencies ahead of time without problems:

  1. Clear /src of external dependencies (they are safely stored in /vendor)
  2. Download and install golint using go get
  3. Clear /src again
  4. Copy your dependencies from /vendor

Maestro’s most important dependency is docker/libcompose which has its own vendor-ed docker/docker. To really use docker/libcompose, Maestro needs to be able to create instances of various docker/docker types and pass them to various functions. As described earlier we can’t import the vendor-ed docker/docker (compiler error). If you copy docker/docker out of docker/libcompose/vendor to /src, Go will not consider them the same (docker/libcompose will use /vendor first).

Copying doesn’t work but moving does: moving docker/docker to /src forces docker/libcompose to use the same code. So after we copy the dependencies from /vendor to /src we move any nested vendor directories to /src, including any nested /vendor directories.

Before running the flatten makefile rule:

projectA/                           # GOPATH / Workspace
    src/
        myOrg/                      # project directory
            projectA/
        otherOrg/                   # un-vendor-ed dependency
            lib-z/
                vendor/
                    diffOrg/
                        lib-x/      # nested un-vendor-ed dependency

After running the flatten makefile rule:

projectA/                           # GOPATH / Workspace
    src/
        myOrg/                      # project directory
            projectA/
        otherOrg/                   # un-vendor-ed dependency
            lib-z/
                vendor/             # empty vendor directory
        diffOrg/
            lib-x/                  # un-nested un-vendor-ed dependency

Conclusion

Rally Health has fully committed to a microservice architecture which has many positives but brings with it new complexities: developers need to be able to run multiple services at once. That can be difficult, so we need to build tooling to make that easier – like Maestro.

We chose a new language, mostly to reduce integration friction with Docker Compose: the Go language. Rally Health is new to Go, both the language and its dependency management strategy. We stumbled into a gap caused by those realities and a significant disruptive event outside of our control: docker/docker renaming to moby/moby.

We identified three (3) problems and devised three (3) solutions:

Problem Solution
go get provides limited control Use an alternate dependency tool – gvt
GOPATH is shared and does not support versioned dependencies Isolate GOPATH to each project
/vendor doesn’t work well for us Use /vendor for storing dependencies
  Use /src for using dependencies

These techniques isolate Maestro from any future upstream chaos. We’re happy with the solution, it works for us, and we’ll see where it takes us.

I hope this post will help you learn a bit more about Go’s dependency ecosystem, and maybe help you with any similar problems you might be facing. I do not attempt (nor do I have the experience) to claim our stumbling blocks as truly fundamental faults with Go. We are still happily using Go.