Recently I attended the CTO Summit Conference in San Francisco. One of the speakers talked about how many engineering leaders are trying to find information with regards to “how to scale a startup and there is not enough out there.” Most posts focus either at the very early stages of a startup or at the size of 1000+ engineering teams. I have decided to document and share some perspectives of my own odyssey from both a management and technology point of view. This is the first installment of a two-post series and it will focus on management challenges.
When I started at Rally Health, the engineering team (Software Engineering, QA, DevOps) was roughly 30 people. It grew to 200 in 3 years. The objective of this post is to summarize lessons learned, best practices, and key takeaways from this experience. UnitedHealth Group acquired majority interest in Rally Health in February 2014. Since then as an engineering organization, we managed to scale the organization, successfully deliver across all product lines, and integrate our platform into UHG’s technology stack. (Please note that Rally Health is an independent entity and partners with multiple payers besides UHG.)
Here is the high level summary of the approach and some learnings.
Gain the respect of the engineering team
Although a team of around 30 people sounds small, it in fact has its challenges. Engineers are used to working in a certain way, there are already individuals that are considered role models, there is already a set of processes that the teams are following, and key technology decisions including the overall tech stack have already been made. Gaining the respect of the engineering team is key to a leader’s success. At this particular step, the best approach is to understand the people, identify key players, listen a lot, understand the development process, and most importantly discover the top three things that the team is proud of. As a leader, you have to build on these key dimensions if you are planning to make any changes to increase your probability of success. It is extra-critical to do so when the team is distributed. In our case, we had engineering in three locations across the US (three different time zones). A final suggestion here is to take time to bond with your team outside of work (team building events do work!)
Understand the architecture and debt
My main assumption is that folks who would take this job are somewhat technical, otherwise they would not enter at a startup at this early phase. To not be able to interact with the engineers at their level, develop relationships, and offer situational advice will eventually alienate you as a leader. A good way to gain that respect is to understand the architecture of what is already built, the roadmap for the next 6 months (if there is one from a feature and tech investments perspective), as well as the debt that each feature team has. At an early stage like this, internal documentation is usually scarce so the best approach is to utilize whiteboard meetings and even fill the documentation gap with personal contributions. The more you can help and integrate with the team, the better.
High level quantitative analysis
Once the above step is completed, the next best action is to do a quick quantitative analysis at a very high level. Think of these interview questions we all had to go through: “how many manholes exist in the US”, “how many piano tuners exist in the US”, “how many gas stations exist in the US.” In other words, given the problem your company’s vision is trying to solve, make some assumptions and predictions on how much traffic you will have, assuming total domination of the space. Next step is to take these numbers and apply them to your technology stack. If you find any red flags, you have to add them to your debt (if not already present) and make a decision around WHEN you should address them. Anything beyond 12 months will need reprioritization so focus on the short term horizon.
An example in my experience was the fact that we were using sticky sessions on our web tier which would effectively kill any auto-scaling from a UX perspective (i.e., no matter how many new machines you add, the user with the sticky session to a server that is thrashing will experience performance degradation or unresponsiveness). That was a time bomb that took 12 months to get rid of! Another example was the fact that all the code was assuming no traffic was coming into the platform during a “code release”. Effectively we were lacking defensive programming techniques or proper versioning.
Hiring with quality
I think that everyone is aware of this challenge. It is important to identify early the folks you want to bet on internally (always good morale boost to first promote from within) and then focus on strategic hires that will multiply your ability to execute and scale. Setting a clear interview process is a no brainer here; defining roles on the interviews, and identifying who qualifies to interview for each role is key. By the way, in hyper growth mode, you will make mistakes. That is unavoidable. There no way around it! What you can do as a leader is identify the bad hires and help them transition somewhere else internally or externally.
Optimize the processes for the engineering team
A great way to gain insights around the team’s inefficiencies is to immerse yourself in the software development lifecycle or set up your laptop for local development. More to add to your debt list :). Another source of wisdom is to get the team to think about what slows them down. Some things are easy to address, some will take time. That said, at that early stage, low hanging fruit exists everywhere and your past experiences can help creating a happier team very fast (easy wins)! Partnerships with project management and product will be crucial at this phase.
Set the direction
One of the first things that I would recommend is setting the vision and mission of the engineering team, documenting it and taking time to communicate it to the entire company. Additionally, I would recommend defining your key behaviors and tenets and demonstrate them on a regular basis as a leader. For instance, ours were:
- Agility and speed
- Put Rally Health and your team above yourself
- Data- and ROI-driven decisions
- Engineering Mission and Vision
- Accountability & ownership
Set the bar high
At an early stage in a company, most likely the culture is not ROI/metrics driven. There is a natural gravitation to adopt the technology that is “cool and interesting” without formal tech evaluation, comparison, or prototyping. That is normal so please embrace it. However, the key is to start managing the team with goals and key results per quarter and metrics. It will take a while but given that the numbers do not lie, it will make it easier to “prove’ where things are trending well vs. not. I am not suggesting a paralysis from metrics tracking. The idea is to pick key metrics for each goal, and for each piece of technology that is critical. Keep the list small and get the team to track it and evaluate it. Next, define expectations for the team per job level and function. That will help folks trying to rise to the expectations because you set boundaries for them. It’s clean, transparent, and clear to everyone.
Reward with true meritocracy
Once the above is set, it makes reviews easy. You can evaluate your own performance first and foremost as well as the engineering organization’s performance. Meritocracy is a key and has to be demonstrated early.. It is important to set an expectation that people can have good quarters and bad quarters. It is normal as things change; they change or try new roles as the business grows. What you want to avoid is setting a tone that if something is awarded once, it will constantly be awarded. Needless to say that you have to have a running list of your top 10% stuck in your memory. These folks are your multipliers. Keep them engaged, challenged, and happy.
Mentor everyone who reaches out
In the early phase, everyone knows everyone and works together. Invest in people, it will pay off later. It will create strong relationships and level up the whole company. As the company grows in size, you will have less and less opportunity to do that so seizing the opportunity early is critical!
Feature team model
We operated heavily on the feature team model. The feature team consists of product, design, QA, project management, software engineers, DevOps. They work together, they deliver together, they go through the product lifecycle together. It is important for team bonding, forging relationships and keeping people feeling as one uniform group. It leads to higher engagement and job satisfaction.
Future tech investments
Last but not least, interaction with the product and the business teams has to generate a list of future investments. In a small startup, focusing on the tactical execution comes easy. At the same time, you have to invest time understanding the space, what comes next, and how that maps to the existing tech. You are the bridge between the business and engineering, and the more educated you are on the architecture, the better decisions you will make. During this process, transparency with your engineering team is key.
Everything written here is not a checklist. It is a set of best practices that need to happen, and you need to re-adjust them every 2-4 quarters. As the business evolves, the team grows; whatever you set up 6-9 months ago may need optimizations, changes, or replacement. Your role is to be as proactive as possible and reduce the surprise factor from your engineering teams. At some point, there will be a management layer (or layers) underneath that will take some of that on and drive it.
As a leader, you are also human and 99% of the time everyone around you will forget that. So the key thing to remember is that if you do not take care of yourself, no one will. Here are some experiences you will most likely have in your journey to scale:
- Leadership is a lonely place. At your level, folks will not come to you often to tell you good job or see how you are doing. They come with problems and issues.
- Not everyone will like you, you will have to make unpopular decisions at some point.
- Get comfortable being uncomfortable.
- When you see a problem, call it out no matter where it is. It will impact you eventually.
- Risk is your new friend. In a startup, risk is everywhere; technology, people. You can minimize it, but generally, you will have to make some bold steps.
- Problems will happen, expect it! As long as a problem does not happen repeatedly, a mistake is a learning opportunity,
- Communicate and be transparent. The team and everyone around you will appreciate it. Provide context and a framework rather than raw unfiltered information. The latter is randomizing.
- Manage changes actively. We are a very adaptable species. However, we dislike change and therefore change management is something that will need to happen often in a hyper growth environment.
- Get comfortable dealing with 100K feet and 10 feet multiple times per day. In a startup you can go from a strategy meeting to a meeting where the async framework for distributed transactions is being discussed. You have to be able to switch level of detail multiple times per day. As the team scales, this is less and less common. However, at the early stages, you will need to develop that ability.
Hope the community finds the above helpful. Leading through hyper growth is hard because it requires us to change quickly and adapt. Good luck and have fun during the journey!