Containerized Build Agent
I’ve found myself with about a week between jobs. I wanted to take this time to recreate, document, and publish some of the things I’ve used containers for to make local development easier and not dependent on anything outside the local machine. I am a big proponent of pursuing a scenario where local development is as easy as Clone, Build, Run and deployment to tiers other than local is similarly as easy. As it turns out these two things are commonly one and the same.
This exercise is as much for me as it is for you. I want to have recreated the things I’ve done while working for someone else so that I have a copy that I own; the code I have produced while on the clock does not belong to me and, if I want to reuse something I’ve already done, I either need to recreate my own copy, commit theft, or rewrite the code for someone else to own once again. I want to have fully explored some more options to verify to myself that the ones I pick in the future are actually the ones I want to pick. Bonus: I can publish what I know and share it with you so that you can use it or build off of it.
My experience and examples here are with Visual Studio, Docker Desktop, and Windows 10. That doesn’t mean you can’t use these strategies with some other IDE and/or container tool. These all happen to fit together without too much work and I primarily develop .net code in a windows environment.
Articles in this series:
Containers for Local Stacks: how to use Docker Compose to create a local development environment
Containerized Build Agent (this story): how to set up a Docker container image that can be used as a CI environment in Azure DevOps
Emulating Cloud Services with Containers: how to use a container to run or emulate cloud services locally without needing to install additional programs or services and without needing to pay to use the real thing before you’re ready to deploy
Integration Testing with Docker Dependencies: learn how to use Docker from within an automated testing framework to spin up integration test dependencies and dispose of them when tests have completed
Why would you want to use a container for your Continuous Integration build agent? There are several reasons.
You get an environment that is specific to a single project without having to dedicate an entire machine (virtual or otherwise) to it. No more tool version conflicts or being held back to a specific version by sharing an environment!
You can have the definition of the environment in source control right along side the application it is meant to build. Much like a unit test can be documentation of the behavior being tested with a built-in report of whether that documentation is correct or not you will get a coded documentation of what environment and build dependencies your code has and, if CI builds are working, you know that the documentation is accurate. No additional effort needs to be put in to document required software. You may still find you want to put in that effort since a CI build likely won’t include, for example, a recommended IDE. You can also version it alongside the application as dependencies are added, removed or updated!
Since it is in a container you can the advantages of containers for your build agent as well. You can add or remove agents fairly easily and they can be hosted anywhere that has access to your Azure DevOps instance. There is even a planned feature to enable and support container agents running from a Kubernetes cluster (https://dev.azure.com/mseng/AzureDevOpsRoadmap/_workitems/edit/1705289) which should make scaling based on load that much easier! You can easily spin up a CI environment on a local machine too… let’s say you run into one of the problems that I would wager almost anyone working with automated builds has run into: it worked locally and not in CI but you can’t find any differences to explain it. You can run the container locally, mount a volume containing your code (or download the code inside the container from the source, if you wish, more closely emulating the CI cycle) and run the build tools as they are in CI right on your machine. Coupled with VSCode remote development this can make figuring out CI errors much easier than the typical guess, check, commit, wait cycle. If you really need a UI into your container (perhaps for a specific build tool or running an IDE from inside where VSCode remote development will not suffice) you could even run graphical applications inside the container and see them outside using XWindows (assuming you are using Linux containers, there are XWindow solutions for Windows and Linux hosts).
If you already know how to build a docker image then you don’t need me to get you going. Microsoft provides documentation for both Windows and Linux containers here https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops. I can offer a working example, putting together the pieces they provide with some example build dependencies (.net 5 SDK, specifically), and an explanation of what’s happening. Most of the specifics here are simply applications of what Microsoft has already provided.
If you’re still with me I’m going to take this opportunity to use a dockerfile definition of a DevOps build agent to discuss how to build a dockerfile. Really it boils down to a series of shell commands and docker directives that saves the machine state after each command to be restored into a running container instance later. If you can script creating a machine in the shell language of the OS you’re going to use in your Docker image you can make a Docker file (maybe with a little bit of extra Googling).
Here is a public repository in Azure DevOps containing the code for this article: DockerContinuousIntegration. It contains a dockerfile and the start.sh script provided by Microsoft. The readme tells you how to build and run the image. I’ll walk through the steps here as well.
The first step is to get a Personal Access Toke (PAT) in order to authenticate the agent with Azure DevOps in order to be able to download the packages and add itself to the build pool. The user who owns this token must have access to manage the agent pool(s) for the project.
Next up is to create the dockerfile. This file is what defines what will be available to your build agent once it starts up i.e. what is installed on it.
FROM ubuntuENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyesRUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl4 \
libunwind8 \
netcat \
libssl1.0 \
wgetRUN wget https://packages.microsoft.com/config/ubuntu/20.10/packages-microsoft-prod.deb -O packages-microsoft-prod.deb \
&& dpkg -i packages-microsoft-prod.deb
RUN apt-get update; \
apt-get install -y apt-transport-https && \
apt-get update && \
apt-get install -y dotnet-sdk-5.0WORKDIR /azpCOPY ./start.sh .
RUN chmod +x start.shCMD ["./start.sh"]
Step by step, what this example is doing:
1: we start from the ubuntu image as our base container (defaulting to latest by not specifying otherwise)
2: we configure that base image to make it easier to run non-interactively
3: we update apt-get’s package repositories and install a series of programs
4: we add the microsoft package repository to apt-get’s known repositories
5: we update apt-get again (now that it knows about microsoft’s sources) and install the .net 5 SDK
6: we set the working directory of the container
7: we copy the start.sh file (provided by microsoft in the link above) into the container (make sure you’ve saved it with Linux line endings if you’re working on Windows!)
8: we set the file permissions to executable
9: we set start.sh to be the thing that is run when the container starts
CMD and ENTRYPOINT are similar but distinct. CMD is intended to be easy to change, override, or interact with after the build is complete and ENTRYPOINT is intended to be more difficult, when you don’t intend interaction (it is still possible to override if needed).
You might have noticed that there are a lot of line concatenations (a line ending in \ with more command on the next line) and command chaining (command1 && command2). This is done to preserve readability of the dockerfile with vertical spacing but to avoid running lots of commands. Every time a command is executed during a build docker will add a layer to the resulting image. This can alter the final container size and build time by changing how big steps are and how many layers there are.
Lets take a brief aside to look at another dockerfile generated for me by Visual Studio in Containers for Local Stacks to see an example of a multistage build, a way to control the container size. You’ll notice a lot more FROM statements in the below file and they include AS <name> in them. This is how a “multi stage build” works. You can find more on this from Docker’s documentation. In short, it is a way to control the final size of the container by separating parts of the build into stages, not keeping everything from a prior stage. You don’t necessarily need to make stages build off each other, you could decide to, in an example like this, build an application in an image that has an SDK installed, and produce image eventually based off a smaller base image that only includes a runtime, copying the build artifacts from the stage with the SDK into the final stage which includes only the runtime.
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/sdk:5.0-buster-slim AS build
WORKDIR /src
COPY ["DockerComposeLocalStack/DockerComposeLocalStack.csproj", "DockerComposeLocalStack/"]
RUN dotnet restore "DockerComposeLocalStack/DockerComposeLocalStack.csproj"
COPY . .
WORKDIR "/src/DockerComposeLocalStack"
RUN dotnet build "DockerComposeLocalStack.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "DockerComposeLocalStack.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "DockerComposeLocalStack.dll"]Now that we have our dockerfile we can build it and run it. From the same directory as the docker file run this command docker build -t dockeragent:latest . which will go through the steps you’ve provided to Docker on how to create the image. You’ll see some output of things like downloading the packages installed from apt. The part after “-t “ determines the name and tag of the image. You’ll notice in my screenshot that some layers say “CACHED” and that many layers take 0.0s to execute. I’ve built this image before so it was built from the cached layers. This build (from scratch) does not take 1.0s! You’ll also see more output for things getting downloaded as apt-get fetches binaries.
Once that is done run docker run -e AZP_URL=<Azure DevOps instance> -e AZP_TOKEN=<PAT token> -e AZP_AGENT_NAME=mydockeragent dockeragent:latest filling in the URL to your Azure DevOps instance (e.g. https://chrdyks.visualstudio.com/ for me) and your personal access token we created earlier. If all goes according to plan you’ll get a series of messages from start.sh saying that the agent is being downloaded, starting up, connecting, etc. Once it says that the agent is waiting for jobs you should be able to see it in the agent pool! If that agent pool receives any jobs your agent might be assigned to carry them out depending on the job commands and the agent capabilities found when scanning during startup. If you don’t supply a pool name start.sh will default to the “Default” pool (you can provide a pool name with an environment variable same as the url and token). The PAT in the below screen shot has been revoked already 😉.
You may find you want to rearrange or change some of the parts of how this works once you’ve got it going. I did the first time I set this up. For the purpose here I thought it would be a good idea to mostly stick to the current version of the Microsoft directions. Without changing anything you should be able to revoke the PAT you generated once your container has started and, if you decide to scale up agent instances, generate a new one which only needs to remain valid until the new container instance(s) start. This could be important since providing the token to the container in some ways can cause it to be able to be extracted (extracting a revoked token is useless) and rearranging the scripts in certain ways will cause you to need to not revoke the token (extracting an active token is NOT useless). Providing the token to the build step as an argument, for example, would require the token to remain active and would also embed the token into the image in a way that it could be extracted and read back from someone who has obtained a copy of the image.
Now for one additional fun piece: if your CI environment depends on Docker you can, in fact, install and run Docker inside a container 🤯! You will have to change the run command to give the container more permissions, running with the -privileged options… which is not always wise. You should probably read the DockerHub page for Docker In Docker (a container set up specifically to run Docker inside a container) to get a better idea of what could go wrong and why you might not want to do this before going ahead. If you still want to go on you should now have the tools to get it working. I have once, so I can tell you that it is possible, but I haven’t take the time to recreate it to share as an example. I used this to set up a CI environment that used Docker to spin up integration test dependencies instead of relying on the real thing, outside the CI environment, which I plan to write as a future story in this series.
