Mature Software Engineering Using Visual Studio Team Services

Background

A little over a year ago, a friend of mine finally convinced me to join Microsoft to work on a software team in the Cloud+Enterprise division (Azure). The team was in a startup / growth phase, and one of the selling points was that I could come in and facilitate the maturation of the team’s engineering practices to allow for rapid, consistent delivery of software with minimal overhead. We implemented techniques such as effective unit testing, continuous integration, fully automated deployment of all applications, isolated testing environments, and efficient work tracking that allowed us to iterate quickly and produce huge amounts of value for our customers in a short period of time. Visual Studio Team Services (“VSTS”) ended up as our cornerstone tool that allowed all of this.

Why VSTS?

VSTS was not the first tool on my list to use in this process. While at my previous employer, we fine tuned our engineering processes using a wide array of tools such as LeanKit, Trello, GitHub, TeamCity, Artifactory, Octopus Deploy, Chef, and Confluence. I had used these tools for many years and was happy with them, so my first choice was to continue with what I knew and start using these tools to fill those gaps in my new team. However, after attending Microsoft’s internal 1ES (One Engineering System) Day conference I was intrigued by the capabilities of VSTS after hearing a talk by Donovan Brown and also hearing how the VSTS team uses VSTS to build VSTS (yes, that’s a little ‘meta’). Given all this I wanted to try using VSTS as the “one tool to rule them all”. Looking back, that was a great decision.

Source Control

In my opinion, the first part of any mature software engineering organization is effective source control. I was lucky that the friend who convinced me to join the team had already set up good source control using a hosted Git repository in VSTS. I had always thought of VSTS as the old TFS (Team Foundation Server), and about the source control that tool offered; I was happily surprised Git is the default option in VSTS now, and to Microsoft’s continual commitment to Git as a source control system in VSTS.

Using Git, this allowed us to start using the GitFlow branching strategy and start defining how we work on code. We use the standard flow of feature branches where no one should ever commit directly to master or develop. The biggest thing that we introduced was no one should merge their feature branch directly into develop (or master). Instead, we use the Pull Request feature of VSTS so that any changes are reviewed by at least one other person on the team. This feature has allowed great conversation / feedback around our code, and the integration with work items makes it easy to see why the changes were made and how they fit into the bigger picture.

Continuous Integration

One of the main principles of agile development is the usage of Continuous Integration in the engineering process. After we ensured that all of our code was in their proper Git repositories in our VSTS project, we then set up continuous integration builds for each one. These builds are triggered by a change on any branch, and consist of at least the following tasks. If any step fails, then the build fails. Common tasks in all of our builds are:

1. Check for credentials in source control / code
2. Restore nuget / node / bower packages
3. Build the entire solution
4. Run Unit Tests* (We also implemented a huge library of unit tests, but that is worthy of a separate blog post)
5. Package the code (library, web service, windows service, Azure function, etc) as a deployable item
6. Attach the deployable package as a build artifact

Using VSTS, setting up these build tasks were trivial since nearly all of them are available by default. However, thanks to the VSTS extension marketplace we were able to implement tasks who were not available by default and have been happy with the available options since. If there was a few not available, we built our own using Powershell. Compared to my past experience, this was much easier to set up in comparison to TeamCity and the availability of extensions in the marketplace is a clear differentiation.

An important feature of VSTS we learned about when configuring our builds is the ability to use on-premises build agents for executing builds and releases. Some of our projects required access to internal nuget feeds (available only on the Microsoft corporate network) to restore packages, which the hosted build agents were not able to access. Using standard instructions we created a build agent on a bare metal machine on the corporate network, and then on Azure Virtual machines hosted in an Express Route subscription. This was simple to setup and we were able to bootstrap our build agents with a single Powershell script. It was not apparent at first, but using ‘on prem’ build agents allowed us to speed up the build process since we are able to scale up / down the Azure virtual machines as needed. The feedback loop from git push to build pass/fail is now down to 5 minutes or less for all of our builds, thanks to VSTS and Azure.

Deployment

Leading up to our usage of VSTS, the team had manually deployed code via copy / paste to all the different machines, and manual updates of configuration files, certificates, etc (for each service being deployed!). Once all the projects were building in VSTS, executing good tests as part of the build, and packaging up their ‘deployable artifacts’, my next target was to automate the releases so they were a single button click. Until this time, VSTS was just another source control / CI tool but Release Management is what made me a true believer in it’s potential.

To start the process, I partnered with the person on my team who was manually executing the releases. We took an agile approach using the following process:

1. Fully understand the existing manual process for deployment
2. Build a Release Management definition for each top level application / service being released that met this process
3. Deploy to a test environment
4. Deploy to staging, and then to production.

We performed this process for a total of 21 applications / services deployed to both ‘bare metal’ and cloud resources. The work we put into automating our builds and the ‘on-premises’ agents paid large dividends in this action since we were able to leverage many of the techniques and resources to deploy our code. VSTS made this even easier since the Build / Release interfaces are very similar, have a large amount of documentation / help online and gave us great instrumentation to work with when the deployments failed.

Some examples of the types of applications we deploy are ASP.NET MVC web applications hosted in IIS, Azure Application Services, SQL Server database Migrations, console applications, and Windows services. Each of these follow the source control / build process noted above so that any change in production has been through the same process and rigorously tested. Important features of our Release Definitions / deployment agent setup are:

1. When executing a release in each environment, we first deploy to either a new folder or Azure App Service instance. This is where all the configuration is performed, and when that process is complete we switch the website / service to it. This has the added benefits of minimal downtime, if a deployment fails then production is never affected, and we have the previous version to point back to in case of rollback.
2. When setting up deployment agents on the machines being deployed to, we ensured that the agent is running under an account that is locked down. We grant the service account the agent is running under ‘Just in Time’ admin privileges as part of the release process to ensure it only can perform these features only when an actual release is being performed.
3. Releases only deploy code, they do not create or modify the hosting tools for those (i.e. Releases do not make an IIS website / Windows Service / Azure App Service, they only re-point the site at the new folder or resource that was just deployed)

In addition to deploying to different environments, we also built common development tasks into VSTS Release Management to automate part of the development workflow. Items such as scrubbing / configuring / restoring a database, building Active Directory domain controllers, and provisioning our development / test environments have all been built into VSTS Releases. This has allowed the team to follow and enforce a workflow where all changes are first deployed to a test environment, then to staging, and then to production. By the time something is deployed to production it will be at least the third time this process will have occurred, thereby reducing the chance for deployment failures.

After all this work automating our deployments, my team is nearing the dream of Continuous Delivery. We are able to deploy at any time of day now, routinely perform many releases to production each week, and can iterate much quicker and with more confidence than before.

Work Management

This item probably should have been first, but work management has been key to managing our engineering process. Thankfully, this process was already setup in my team when I joined and working extremely well. We use the typical epic / feature / story breakdown in VSTS work management, with stories being the unit of work that an engineer accepts and works on. We extensively use the backlog board with swim lanes and columns to easily visualize bottlenecks and work in progress (WIP). Our current board setup follows a Kanban approach with the following lanes:

1. Backlog: Team backlog where requested items are stored.
2. On Deck: Work items that are queued to be worked on in a prioritized top to bottom list.
3. Active: Items that are currently being worked on.
4. Blocked: Items that either can not be started or have been started and are now blocked. Any work items in this column are an indicator that some work needs to be done to unblock.
5. Dev Complete: Work items are moved here after the feature has been reviewed via pull request, demoed to the team, and released to the test environment. Work items in this lane or the following lanes have the capability of moving to Blocked or Active if issues are found.
6. Test: Features who are currently being tested in the test environment
7. UAT: Features who have passed automated or manual testing and are deployed to the pre-production (staging) environment.
8. Closed: Feature is released to production.

The biggest feature that we are using is the tagging of work items (usually User Stories) in a git commit message. By starting the message with the work item number, VSTS automatically correlates a git commit with that work item (ex: git commit -m ‘#5459 Adding API tests’). This then flows through to Pull Requests, Builds, and eventually Releases so that for a given release, we can see what changes were made, by whom, the related build, who reviewed / approved the pull request, and which story / epic / feature that it was related to. Having this visibility built in and easily accessible has been invaluable many times over.

Instrumentation / Dashboards

Being able to visualize items in the engineering pipeline is a key aspect to managing a functional engineering team. After implementing our process using VSTS, we found that the dashboard widgets available give good insight and help show were items are at. Being focused so much on builds and releases, we created a dashboard that shows all our software releases, and which version is currently in a specific environment:

Dashboard such as this, and those we have created in the Azure Portal using Application Insights have been extremely beneficial in determining the health of our applications.

Summary

Mature software engineering principles are not only tools and processes. My team and I have also implemented a great unit testing paradigm, and operational processes such as software showcasing and using our telemetry to drive decisions on what work should be done next. Visual Studio Team Services has given us all the tools we need to accomplish these, with the result being that we can now release features quicker, with more confidence, and easily than ever before.