Azure DevOps is a great tool and consists of a lot of tools. Today I’ll be talking about the Release Pipelines. More specifically, I’ll investigate the possibilities of locking down the pipeline agents.
Visually, the pipelines agents work as follows:
These pipeline agents are registered using a Powershell script that is generated when you create a new deployment group in Azure Devops. The problem with this Powershell command is that it registers the pipeline agent as a service running as NT AUTHORITY. Something we should try to avoid, right?
Let’s see what we can do about it.
In my case, these were the requiremnts: * Able to use deployment groups using the autogenerated registration script * Able to use the IIS web app manage and IIS web app deploy task * Able to set environment variables using Powershell * Able to schedule tasks using Powershell
The goal was to lock down the permissions of the Azure Pipeline Agent to the minimum required to execute the above listed tasks. I started Googling, pretty sure I would find a way to achieve this goal. I didn’t. It’s surprising to not find an answer about this. It seems like the principle of least privilege does not apply anymore in a devops world.
In the remainder of this post, I’ll go over three steps to lock down your agent: * Inventorize the permissions your agent requires * Create a devopsagent user and assign it the required permissions * Lower the permissions of the Pipelines service to ‘Local Service’ and have it activate the devopsagent user upon each release
The last step is perhaps the most important one. You could skip over the first two and really just lower the permissions of your pipeline agent to NT AuthorityService. Step 3 will give you a way to elevate your agent when you need it: during release time.
I tried to make an overview of the permissions that a pipeline agent really requires, given my requirements. Don’t start applying them just yet, there’s a script at the end to do this all for you.
In the script, I create a ‘devopsagent’ user and assign it the permissions listed above. I also assign the Local Service user the rights to start and stop the pipelines service since we need that in step 3.
You should create three new stages in your pipeline: * elevate agent permissions, which will ask for a password at release time * drop agent permissions * abandon release The abandonment is required to avoid risking that someone simply ‘redeploys’ your elevation stage without having to enter the elevation password.
By default, I will drop the user to Local Service but you could change that by playing with the pipelineLowPrivUser_name variable.
This stage will assume the pipeline agent is currently running as local service, and will elevate it to the user defined in the pipelineElevatedUser_name variable. Three tasks are defined in this stage: * the elevate permissions task * a simple sleep task to give the agent service time to restart and reconnect * the elevation verification task
Visually, your stage should look like this:
This stage will drop the permissions of the elevated pipeline agent to a user of your choosing, by default the local service user. Again, three tasks are defined: * the drop permissions task * a simple sleep task to give the agent service time to restart and reconnect * the drop verification task
Visually, your stage should look like this: <img class=“leftalign src=”docimages/2019-07-23-18-56-26.png">
This stage will abandon the release by calling the Azure Devops API. Make sure to grant this deployment group task access to the OAuth Access Token. This stage will contain one task: * the abandon release task
Visually, this stage should look like this:
A pipeline agent creating a HTTPS binding MUST have administrator privileges. There is no way to lock that down. I personally made the choice to not use HTTPs bindings in the release, and to manually add them using https://github.com/PKISharp/win-acme after the first release. These bindings will not be overwritten and you can keep on doing releases without having to ever worry about them again.
Alternatively, you could simply follow step 3 as explained above and elevate the pipeline user to an administrative user. Then, drop the permissions again after the release has finished.
The logs in the Azure release pipeline are pretty obvious sometimes, and more interestingly contain the commands that the agent is executing. Should you run into a failure because of some other requirement, I recommend you do the following to troubleshoot it:
Then, you should start filtering the procmon output. One obvious thing is to filter for all ‘ACCESS_DENIED’ entries in the ‘Result’ column. Another good way to filter is to filter based on the process. If your command copied in step 2 starts with‘appcmd.exe’ for example, you can simply filter for that process name.
Keep safe, and if you need help contact me at michael(dot)boeynaems(at)portasecura(dot)com!