To give the developers and testers additional faith that their proposed changes work successfully, it is advisable to build and test their code changes in the Pull Request before it is approved and merged into the Master branch.

In order to achieve this we have setup a PR Build in Azure DevOps which creates a new Docker image using the code in a PR, uploads the image to the Docker Repo, runs the image in Azure Container Instances and creates the required Azure DNS records in order for you to communicate with it publicly and test your changes on a running app.

The whole process follows a number of steps which I will go into more details below.

Pull Request Build Integration Workflow

As you can see from the above diagram there are 9 x steps in the workflow for creating a Docker image per PR per country. 1. Developer checks in code they have updated.

git add .
git commit -m "Some code changes to make the platform more awesome"
git push -u origin RemoteBranch

1. A Pull Request is created in Azure DevOps for the new code.
   
   

   2A. The repositories target branch has a number of policies set on the master and develop branches.

   • Require a minimum number of reviewers.
   • Check for linked work items.
   • Check for comment resolution.

   • Build validation. This ensures the listed build completes successfully before allowing a merge to target branch. The build added here, will be the build that creates a Docker Image and runs it per Pull Request.
     
     2B. Setting the branch policy. - Click on the drop repo menu and then select Manage Repositories
     
     

   • Find the branch your want to set the policy for and select Branch Policies button.
     
     
     

   • Set all the required policies for your target branch here, including the PR Build Validation:
     
     

   • Now when we finish creating the PR the PR Build will be triggered.

   • You can see the status of all policies you have set from within your PR here.
     
     
     

2. PR Build is now triggered and running the build definition you attached in the policy.
   
   

   • Build Internals:

     1. Each project uses different languages, code etc and requires a different tools, to compile and test. So you’ll need to setup the build as per your coding requirments. After all those steps are completed and the code is compiled, tested etc, the resulting output can be added to a Docker Image for testing.
        
        
        
        

     2. Here we are using a combination of PowerShell scripts and Azure DevOps custom tasks to achieve the desired output being a running a container instance.

        • Step 1. Fetch Docker Repo (We don’t want to store the necessary scripts for this in every repo, so we pull it in at build time if necessary) using this VSTS Custom Task Git Repository Downloader
        • Step 2. Build PR Docker Image This is done using a PowerShell script within the Docker Repo. (Quick overview of its actions simplified, not the full code)
          

        # Get PR Number only
        $pullRequestNumber = $pullRequestName -replace '[^0-9]', ''
        # We do this to know which country specific containers to create, our Backlog items have a custom Tag field we need to query
        $countries = Get-CountriesRelatedToPullRequest -pullRequestNumber $pullRequestNumber -vstsToken $vstsToken
        # If we have something to do, continue
        if ($countries.count -gt 0) 
        {
            $rootPath = Get-Location
            # Create new Dockerfile with PR ID as Label
            # The FROM command is getting a prepared baseline image with all the requirements to run this project
            # Instruction to ADD the compiled code from the build
            $dockerFileContent =
            @"
            FROM dockerregistry-on.azurecr.io/software:core12 `r`n 
            ADD / /software `r`n 
            LABEL pullrequest=`"$($pullRequestName)`" 
        "@
            # The new Dockerfile is placed in the Staging directory along with the compiled code
            New-Item -Path "$($stagingDirectory)\Dockerfile" -type file -force -value $dockerFileContent
            # Tage Image with BuildID
            $tagImage = "dockerregistry-on.azurecr.io/software:$($buildid)"
            docker login dockerregistry-on.azurecr.io -u "$ServicePrincipalID" -p "$ServicePrinicpalKey";
            # Build new baseline image with project code
            docker build -t $tagImage .
            # Push image to Docker Repo
            docker push $tagImage
        
            # Now we need to build an image per country tag, so start with another new Dockerfile
            # This time adding in the required Web.config file
            $imageName = "dockerregistry-on.azurecr.io/software:$($buildid)"
            $dockerCountryContent =
            @"
            FROM $($imageName) `r`n
            ADD /Web.config /software/Web.config `r`n  
            RUN powershell -NoProfile -Command ""Import-Module WebAdministration; Stop-WebSite 'ASPNET'; Start-WebSite 'ASPNET'""  `r`n 
            LABEL pullrequest=`"$pullRequestName`"
        "@
        #These following three steps perform a transformation on the Web.config file by importing the file as XML transforming some key value pairs to the country specific varients, then building and pushing the required docker images.
        if ($countries.Contains("UK")) {
        Build-ImageForCountry -country "UK"
            }
        if ($countries.Contains("AU")) {
        Build-ImageForCountry -country "AU"
            }
        if ($countries.Contains("US")) {
        Build-ImageForCountry -country "US"
            }
        }

        • Step 3. Run docker images in Azure ACI This is again done using a PowerShell script within the Docker Repo. (Quick overview of its actions simplified, not the actual code)

        # Get PR Number only
        $pullRequestNumber = $pullRequestName -replace '[^0-9]', ''
        # We do this to know which country specific containers to create, our Backlog items have a custom Tag field we need to query
        $countries = Get-CountriesRelatedToPullRequest -pullRequestNumber $pullRequestNumber -vstsToken $vstsToken
        $shouldRunImages = $countries.count -gt 0
        if ($shouldRunImages) {
        # Logging in to AZ Service to create containers
        az login --service-principal -u="$ServicePrincipalID" -p="$ServicePrinicpalKey" --tenant="$env:tenantId"
        # Selecting Dev Infrastructure Subscription
        az account set --subscription "Dev-Infrastructure"
        if ($countries.Contains("UK")) {
        #The following step is how to populate a variable in VSTS Environment, so the preceeding VSTS Custom Task for creating DRecords can check           its existence and evaluate whether to run or not.
        Write-Host "##vso[task.setvariable variable=UK-Container]true"
        # Creating UK Container
        az container create --resource-group PrContainers --name software-docker-$pullRequestNumber-uk --imadockerregistry-on.azurecr.io/           software:$buildid`UK --os-type Windows --cpu 4 --memory 4 --registry-login-servdockerregistry-on.azurecr.io         --registry-username=$registryUserName --registry-password="$registryPassword" --ip-address publ--dns-name-label        software-docker-$pullRequestNumber-uk --ports 443 --no-wait
            }
        }

        • Step 4. Create Azure DNS Record For this step, we have created a custom VSTS extension in PowerShell utilising the AzureRm.Dns Module.
          
          In Step 3. we created a VSTS Variable called UK-Container this variable is used for the custom condition of this task. Because we may want to create DNS records for 3 x countries, or maybe just one.
          
          
          

        • Step 5. Post notification message to Slack This is also achieved using a PowerShell script which has a number of custom modules to communicate with the VSTS API, which get the work items associated with the Pull Request and construct a custom message to post to a channel from a VSTS variable.
          
          

          End of Workflow One

          You now should have a docker image built using a VSTS hosted agent running in Azure ACI with an Azure Public DNS record and a notification to tell the Developers / Testers its ready for testing.

Pull Request Completed / Abandoned Workflow

Once the code has been checked and the image tested, the Pull Request will either be Completed or Abandoned. This will trigger VSTS to fire a service hook to an Azure function which deletes the ACI and Azure DNS Records, last step is a Slack Notification to tell the Developers / Testers it has now been removed.

1. For this step we need an Azure PowerShell function to clean-up ACI and DNS Records. The function runs the following PowerShell script (Not actual code).

#Set session variables from application settings
$azureApplicationId = $env:ApplicationId
$azureDnsSubscriptionId = $env:DnsSubscriptionId
$azureAciSubscriptionId = $env:AciSubscriptionId
$azurePassword = $env:Password
$azureTenantId = $env:TenantId

$secureAzurePassword = ConvertTo-SecureString -String $azurePassword -AsPlainText -Force

# Step 1: Get what we need from the inbound request...
$requestBody = Get-Content $req -Raw | ConvertFrom-Json

$requestStatus = $requestBody.resource.status
$sourceRepo = $requestBody.resource.repository.name
$pullRequestId = $requestBody.resource.pullRequestId
$pullRequestUrl = $requestBody.resource.url

# Step 2: Process...
$allowedStatuses = $( "completed", "abandoned" )
if ($requestStatus -inotin $allowedStatuses) { return }

$aciInstancePrefix = Get-AciInstanceNameFromRepo -Repository $sourceRepo
if ($aciInstancePrefix -eq "software") {
    $aciInstanceName = "$($aciInstancePrefix)-docker-$($pullRequestId)-*"
}
else {
    $aciInstanceName = "$($aciInstancePrefix)-pr-$($pullRequestId)"
}

Import-Module Azure
Import-Module TlsFunctions
Import-Module AzureFunctions
Import-Module AciCleanupFunctions

Add-Tls12Support

Login-ToAzureRmServicePrincipal -TenantId $azureTenantId -ApplicationId $azureApplicationId -Password $secureAzurePassword

$resourceGroup = "PrContainers"
Select-AzureRmSubscription -SubscriptionId $azureAciSubscriptionId

Write-Output "Checking for all ACI Instances in ResourceGroup: $resourceGroup"
$resources = Get-AciInstancesLikeContainerName -ResourceGroup $resourceGroup -aciInstanceName $aciInstanceName
Write-Output ($resources| Format-Table | Out-String)
Write-Output "Resource Count is $($resources.count) "

if ($($resources.count) -eq "0") {
    Write-Output "Couldn't find any ACI instances"
    return
}

foreach ($resource in $resources) {
    Write-Output "ResourceName: $($resource.name) and DNSName: $($resource.DnsName) is about to be cleaned up"

    Remove-AciInstance -SubscriptionId $azureAciSubscriptionId -ResourceGroup $resourceGroup -InstanceName $resource.Name
    Remove-AzureRmDnsCname -SubscriptionId $azureDnsSubscriptionId -ResourceGroup "DNS" -ZoneName "TestZone.com" -HostName $resource.DnsName
}

try {
    # Step 3: Notify message to Slack that the container was cleaned up
    $slackMessage = "Containers for pull request <$($pullRequestUrl)|$($pullRequestId)> has been cleaned up and is no longer accessible"
    $slackInfo = Get-SlackInformationForRepository -Repository $sourceRepo

    Invoke-SendMessageToSlack -SlackToken $slackInfo.Token -SlackChannel $slackInfo.Channel -Message $slackMessage
}
catch {
    # Explicitly do nothing
}

1. To create a Service Hook in Azure Devops, go to https://YourCompany.visualstudio.com/Work/_settings/serviceHooks
   

Create a Service Hook of type Web Hook which is triggered by the event Pull Request Updated


Set the destination URL to your custom Azure function which cleans up Azure Container Instances and Azure DNS Records

1. When you Complete or Abandon your Pull Request, the Azure DevOps Service Hook will be fired. This will send the payload to your custom Azure function. The function will perform the actions.

   • Service Hook Fired
   • Azure Function Consumed Payload
   • Clean-up ACI instances related to PR
   • Remove DNS Records related to PR

   • Notify Slack Container and DNS records removed.
     

     Images older than 7 x days clean-up Workflow

     Similar to the last function, however this one is to delete aged ACI instances, if the trigger did not work, or someone left a PR open for a long time, there is no point leaving the container running, if they want to create it again, all that needs to be done is running another PR Build.

This is triggered by a schedule, ran daily on an Azure function which deletes the ACI and Azure DNS Records, last step is a Slack Notification to tell the Developers / Testers it has now been removed.

1. For this step we need an Azure PowerShell function to clean-up ACI and DNS Records. The function runs the following PowerShell script.

$env:PSModulePath = $env:PSModulePath + ";D:\home\site\wwwroot\Modules"

#Set session variables from application settings
Import-Module TlsFunctions
Import-Module AzureFunctions
Import-Module SlackFunctions
Import-Module AciCleanupFunctions

$azureApplicationId = $env:ApplicationId
$azureDnsSubscriptionId = $env:DnsSubscriptionId
$azureAciSubscriptionId = $env:AciSubscriptionId
$azurePassword = $env:Password
$azureTenantId = $env:TenantId

$aciResourceGroupName = "PrContainers"
$dnsResourceGroupName = "DNS"

Add-Tls12Support

# Step 1: Login Azure
$secureAzurePassword = ConvertTo-SecureString -String $azurePassword -AsPlainText -Force
Login-ToAzureRmServicePrincipal -TenantId $azureTenantId -ApplicationId $azureApplicationId -Password $secureAzurePassword
$restToken = Get-RestApiToken -TenantId $azureTenantId -ApplicationId $azureApplicationId -Password $azurePassword

# Step 2: Start Process...
Write-Output "Checking for all ACI Instances in ResourceGroup: $aciResourceGroupName"
$aciInstances = Get-AciInstancesWithCreationDate -SubscriptionId $azureAciSubscriptionId -ResourceGroup $aciResourceGroupName -Token $restToken
Write-Output ($aciInstances | Format-Table | Out-String)

$cutOff = [DateTime]::UtcNow.AddDays(-7)
Write-Output "CutOff Date for deletion is $cutOff"

$agedInstances = $aciInstances | Where-Object { $_.CreationDate -lt $cutOff }

if ($agedInstances.Count -gt 0) {
    Write-Output "Found the following instances that require removing:"
    Write-Output ($agedInstances | Format-Table | Out-String)
}

foreach ($aciInstance in $agedInstances) {
    Write-Output "Cleaning up instance $($aciInstance.Name)."

    if ([string]::IsNullOrWhiteSpace($aciInstance.DnsName)) {
        Write-Output "No DNS tag set up for this container, will not clean it up"
        continue
    }
# Clean ACI Instance
    Remove-AciInstance -SubscriptionId $azureAciSubscriptionId -ResourceGroup $aciResourceGroupName -InstanceName $aciInstance.Name
# Clean DNS Records
    Remove-AzureRmDnsCname -SubscriptionId $azureDnsSubscriptionId -ResourceGroup $dnsResourceGroupName -ZoneName "TestZone.com" -HostName $aciInstance.DnsName

    try {
        $repo = Get-RepositoryFromAciInstanceName -InstanceName $aciInstance.Name

        $slackMessage = "Container instance '$($aciInstance.Name)' has been cleaned up and is no longer accessible"
        $slackInfo = Get-SlackInformationForRepository -Repository $repo

        Invoke-SendMessageToSlack -SlackToken $slackInfo.Token -SlackChannel $slackInfo.Channel -Message $slackMessage
    }
    catch {
        # Explicitly do nothing if we don't recognise the container that the ACI instance belonged to.
    }
}

1. The above script performs the following actions.
   • Scheduled Time Reached
   • Azure Function Fired
   • Clean-up ACI instances older than 7 x days
   • Remove DNS Records related to instance
   • Notify Slack Container and DNS records removed.