Rally Health’s offerings are powered by a fleet of microservices. At the time of writing, we have roughly 80 separate services.
Developing and testing a stand-alone service (with, say, a database dependency) is easy: install the dependency locally, bang on the keyboard, results!
Unfortunately developing services that depend on other services is harder. To do it properly you need to stand up a spider web of services, starting with the services your service depends on, then those services’ dependencies, around and around until you are running the entire system (or get lucky and find some working subset).
Rally Health℠ is working on an in-house solution by building a pair of tools: Maestro and Roost. Maestro is a CLI written in Go that wraps and extends Docker Compose to work within our tool-chain. Roost is a service written in Scala that acts as a facade for the config data Maestro needs.
Maestro is Rally Health’s first project using Go. We started at the ground floor, following the typically recommended guides   , and we looked for advice from the existing gophers       . Maestro integrated with Docker Compose by depending on the
docker/docker projects. (Docker writes a majority of its projects using Go and that is one of the main reasons we decided to use Go.)
Progress was going great… until a routine
make clean unexpectedly failed. A short investigation later and we learned that in April 2017 Docker was renamed to Moby. We were not the only ones surprised by this change    .
One of the interesting design choices of Go is that imports can reference github.com projects. It’s an interesting feature until someone moves a repository to a new owner and name. Which is exactly what Docker did when they moved
docker/docker to `moby/moby’.
docker/libcompose code that Maestro depends on contains many
import "github.com/docker/docker" statements. Unfortunately
docker/libcompose’s imports were not updated after the rename. So when we attempted to replace Maestro’s dependency from
moby/moby everything fell apart –
docker/libcompose does not recognize
moby/moby types, and there is no way for
docker/libcompose to know that they are the same types with a different package name.
Go’s import design and the docker-to-moby rename wrecked our development setup:
- Updating our dependencies failed (
docker/dockerno longer exists)
- CI completely broke (i.e. the worker can’t download the dependencies)
- New developers could not create a working development environment
- Homebrew install failed (because it compiles Maestro locally)
Our investigation of the rename and dependency design identified three places in our development setup that needed changes.
The standard dependency tool –
go get – has some interesting design decisions:
- Dependencies are source downloads; there is no JAR or RPM equivalent in Go.
- Dependencies are downloaded from the original repositories; there is no artifact repository.
- A peculiar strategy to download from whichever branch matches your locally installed version of Go, e.g. branch “go1” if you have that version installed.
- The current
HEADof the branch is downloaded – not a specific commit or tag.
- Dependencies are stored in the
- Dependencies are organized by name without any branch, commit, or tag identifier.
In short, the lack of an easy-to-backup package or immutable version identifier makes it difficult to ensure that your change occurs only when you explicitly request an update.
Note: A more powerful dependency management tool is currently under development but it is still in alpha.
GOPATH is a shared directory that holds your dependencies, and each dependency lives in a directory (matching its name) without a version number (or equivalent identifier).
Let’s say you have two Go projects,
Project A and
Project B, and they have a shared dependency,
lib-z, that is stored in
GOPATH. That means that
Project A and
Project B will always use the exact same code when compiling against
lib-z. You can’t make
Project A depend on
Project B depend on
lib-z-v2 because there is no version identifier in
GOPATH. You need to stay on top of updates to all projects that depend on
lib-z since you can’t reference an older version.
Before you start choosing your throwing stone, let me point out that
GOPATH is quite similar in design to Python’s
PYTHONUSERBASE and Ruby’s
GEM PATHS. The difference is that they have solved their shared packages problem with tools such as
rvm. Unfortunately, Go seems not to have an official or de-facto solution for this problem yet.
Go 1.5 introduced a technique to help mitigate the limitations described in the last section: vendoring. In short, every Go project can store a copy of its dependencies in a
/vendor directory instead of
GOPATH. Real-world experience has suggested that it’s only a good solution for projects that aren’t libraries; unfortunately that is not always followed in practice, e.g.
lib-z contains a
/vendor dependency that you wish to use in your project? For example,
lib-z depends on awesome (and fake)
pony4Go library, and you’d also like to use
pony4Go. It would be nice to just reference the
lib-z depends on, rather than trying to add your own dependency on
pony4Go and trying to make sure they are the same version. Unfortunately, you can’t simply import that dependency directly – e.g.
import “lib-z/vendor/pony4Go” – you’ll get a compiler error. Why? The vendoring proposal says “Code in vendor directories is not subject to import path checking.”
We’ve identified the three areas we need to improve. How do we improve them?
|Go Get||Different Dependency Tool|
||Isolate per workspace|
Go Get Alternative
After a quick look at the offerings we settled on using
gvt which seemed to us to be the simplest dependency management tool available at the time. We can specify a specific branch, tag, or commit to download, it downloads into the local
/vendor directory, and the details of what are downloaded will be stored in a
gvt improves our isolation and moves us closer to repeatable builds. Unfortunately it is not enough. While it is extremely unlikely that the source code identified by a tag or commit would change, it is possible that the repository itself moves or is no longer accessible. So we go one step further: we commit our
/vendor directory directly to our code base.
As a part of our builds, now we use what is committed and we only use
gvt when we are adding/updating a dependency.
Most of the Go tools assume you are using the standard workspace directory layout, and the
GOPATH points to the root of the workspace. We do not want to rock that boat – so we make separate workspaces for each project and set the
GOPATH to the current workspace directory when using Go tools:
projectA/ # GOPATH / Workspace .git/ # Git repository metadata src/ myOrg/ projectA/ # project directory hello/ hello.go # command source vendor/ # vendor directory projectB/ # GOPATH / Workspace .git/ # Git repository metadata src/ myOrg/ projectB/ # project directory world/ world.go # command source vendor/ # vendor directory
It’s the same workspace layout, just nested. The most significant difference is that the Git repository is now the workspace directory.
make to build Maestro so it’s easy to change the
GOPATH for the Go tools by adding
export GOPATH:=$(shell pwd) to our
makefile. With that change, all Go tools that run from our
makefile will use the project-specific
To be clear, we still have a shared
GOPATH. It’s just used for binary tools (like
richgo) that are not strict dependencies for our projects.
There is another reason we settled on this design: during our efforts to get Maestro working again we learned that
go build looks at both
GOPATH and the local
/vendor directory when searching for dependencies. If there is a conflict – the same dependency in both but different source code – compilation will fail. Apparently, your
/vendor does not override the contents of
GOPATH. (At least not consistently; this was a bit of a
heisenbug for me.) Setting the
GOPATH to the workspace avoids the possibility of adding or updating a dependency in the shared
GOPATH that would break Maestro.
There are several existing tools that aim to do the same thing as our chosen solution (
virgo, etc). However, when this hit, we were deep into meatball surgery mode. We consider our approach – isolating
GOPATH – an MVP that made us resilient to upstream dependency changes. We are planning a proper evaluation of existing tools once Maestro reaches it’s own MVP status.
In the previous section, I glossed over the first part of our solution: using
gvt to download dependencies to
/vendor. Go does not support
/vendor at the root of the workspace; that only works within the project’s directory.
Maestro’s source code lives in
/src, and we have
.gitignore rules to only source control Maestro’s code. We use
gvt to download to
/vendor (as described before). What’s new is that we add a pre-build rule (e.g. something like this) to our
makefile that copies the dependencies from
Before running the pre-build
projectA/ # GOPATH / Workspace src/ myOrg/ # project directory projectA/ vendor/ # vendor directory otherOrg/ lib-z/ # vendor-ed dependency
After running the pre-build makefile rule:
projectA/ # GOPATH / Workspace src/ myOrg/ # project directory projectA/ otherOrg/ # un-vendor-ed dependency lib-z/ vendor/ # empty vendor directory
This copy step has two interesting and useful side-effects:
First, it makes trying out a new dependency a little easier. You can try out a new dependency using
go get (which will download into
/src) without fear of mistakenly committing the code or changing
/vendor/manifest.json. When you are sure you want to add the dependency, you’ll re-add the dependency using
gvt and specify the current commit or tag.
Second, having a
/vendor allows you to have temporarily conflicting dependencies when building unrelated binaries. For example, Maestro uses
golint which depends on the standard
context library. But
docker/libcompose depends on
golang.org/x/net/context. They are not identical, so one or the other fails to compile.
This is where having a /vendor directory can help you build those binary dependencies ahead of time without problems:
/srcof external dependencies (they are safely stored in
- Download and install
- Copy your dependencies from
Maestro’s most important dependency is
docker/libcompose which has its own
docker/docker. To really use
docker/libcompose, Maestro needs to be able to create instances of various
docker/docker types and pass them to various functions. As described earlier we can’t import the vendor-ed
docker/docker (compiler error). If you copy
docker/docker out of
/src, Go will not consider them the same (
docker/libcompose will use
Copying doesn’t work but moving does: moving
docker/libcompose to use the same code. So after we copy the dependencies from
/src we move any nested vendor directories to
/src, including any nested
Before running the flatten
projectA/ # GOPATH / Workspace src/ myOrg/ # project directory projectA/ otherOrg/ # un-vendor-ed dependency lib-z/ vendor/ diffOrg/ lib-x/ # nested un-vendor-ed dependency
After running the flatten
projectA/ # GOPATH / Workspace src/ myOrg/ # project directory projectA/ otherOrg/ # un-vendor-ed dependency lib-z/ vendor/ # empty vendor directory diffOrg/ lib-x/ # un-nested un-vendor-ed dependency
Rally Health has fully committed to a microservice architecture which has many positives but brings with it new complexities: developers need to be able to run multiple services at once. That can be difficult, so we need to build tooling to make that easier – like Maestro.
We chose a new language, mostly to reduce integration friction with Docker Compose: the Go language. Rally Health is new to Go, both the language and its dependency management strategy. We stumbled into a gap caused by those realities and a significant disruptive event outside of our control:
docker/docker renaming to
We identified three (3) problems and devised three (3) solutions:
||Use an alternate dependency tool –
These techniques isolate Maestro from any future upstream chaos. We’re happy with the solution, it works for us, and we’ll see where it takes us.
I hope this post will help you learn a bit more about Go’s dependency ecosystem, and maybe help you with any similar problems you might be facing. I do not attempt (nor do I have the experience) to claim our stumbling blocks as truly fundamental faults with Go. We are still happily using Go.