Rally Choice builds its software on TeamCity, a continuous integration server from JetBrains. At a high level, this includes a single TeamCity server which hosts build configurations and orchestrates one to many TeamCity build agents that build and package our software.
Due to painful lessons learned from years of heavily customized, snowflake (see also: flaky) build agents on our previous continuous integration solution, when we switched over to TeamCity, we started from the ground up with heavily automated, immutable infrastructure in mind. Skipping some specifics, we use an Auto Scaling Group for our build agents which takes care of spinning up new agent instances when one goes down.
Our first pain point with TeamCity came with that last detail. The build agents automatically register themselves with the TeamCity server, but TeamCity does not automatically “authorize” the agent. In TeamCity parlance, “authorization” involves assigning one of your agent licenses from your organization’s pool to a build agent. Only when an agent is authorized can it start building TeamCity projects. This authorization step can be done one of three ways:
The primary way of authorizing an agent is through the web UI. Since our agents are EC2 Spot Instances, they could spin up and down multiple times a day so this manual step was not an option for us.
TeamCity Web API
We could potentially use the TeamCity Web API to register our agents in the startup script of the agent instances (which is how we install the TeamCity agent service). However, it can take more than a minute for the agent to register with TeamCity after the agent is started and we can’t authorize it until that’s complete. So far, we’ve tried to avoid polling in our startup scripts if there’s an event-driven option. This lessens non-determinism as well as optimizes the total start up time. Additionally, we would still need a solution for freeing up licences when agent instances are terminated. That brings us to:
Custom TeamCity Plugin
TeamCity exposes a rich and easy to learn extension framework for custom plugins. One of the main programming models for TeamCity plugins is hooking into many different TeamCity events, such as server startup, build completion, and to our particular interest: agent registration and deregistration.
Developing the Plugin
The “Hello World” sample TeamCity plugin provides examples for most of the extension types (agent event hooks, server event hooks, custom views). Because of this, I opted to minimize my effort in project setup and began my plugin by gutting the “Hello World” plugin project and customize the parts I needed.
The sample plugin is found in the TeamCity install directory (not the data directory) at
$TEAMCITY_INSTALL_DIR/devPackage/samplePlugin-src.zip. After extracting the project, I removed all the
.jsp pages from
/resources (since I wasn’t planning on a UI component) and deleted and removed all references to everything in
src/jetbrains/sample except the
TeamCityLoggingListener. This leaves us with the following:
Here we see four significant points that will keep us from having to learn a great deal up front about the TeamCity plugin infrastructure.
- The custom
TeamCityLoggingListenerclass inherits from a
BuildServerAdapter, which is part of the plugin SDK
- The class intects a reference to
SBuildServer, which is used to register/wire up our
- The class demonstrates a number of listener hooks (the
@Overridenmethods). This includes two that sound interesting to us:
- Each agent hook has a reference to a
SBuildAgent, which one can guess attaches behavior to the agent instance
Since I only care about
agentUnregistered, I remove the rest of the methods and helpers (as well as some now-unnecessary fields). After some documentation browsing, I determine that
SBuildAgent.setAuthorized is excatly what I need. Incorporating those changes (as well as an appropriate rename), I end up with this:
After building this, deploying the plugin zip to
$TEAM_CITY_DATA_DIRECTORY/plugins, and restarting the TeamCity server, newly registered agents were automatically authorized, and the agents that I forcibly took offline immediately unauthorized themselves and freed up their license.
With the Agent Auto Authorize plugin, we are no longer burdened with manual intervention whenever our agents die, go offline due to Spot Instance prices fluctuations, or when we deploy new agents. With what essentially amounts to two lines of original code, I don’t hope to win Team City Plugin Contest, but I certainly did learn enough about the plugin development flow to potentially create more meaningful plugins for our team in the future.
See here for the full source (which adds configuration in case you don’t always want to automatically authorize).