Using Workload Identity Federation In Azure Pipelines For AWS IAM Integration
Introduction
Azure Pipelines is a product in the
Azure DevOps suite that provides the ability to create and run pipelines of almost any sort in Azure. Much like AWS's CodePipeline
service, it offloads the operational toil of a hosted solutions like Jenkins to the cloud service provider, while at the same time
allowing a flexible amount of workload hosting to be done under a customer's control.
Recently, a Rearc customer found themselves desiring to use an Azure Pipeline to deploy Infrastructure as Code in Terraform against
various AWS accounts. The current state of the art for this is to create AWS IAM users, generate API keys, and to store those API keys
in what Azure Pipelines refers to as a Connection. This is not ideal, for a few reasons; API keys are subject to leakage (which is a
disconcerting prospect when the keys are attached to a high-permissions deployment user), API keys should be rotated regularly (which
adds operational toil), and an AWS IAM user must be generated for every target AWS account (or complicated role assumptions must be
done).
Enter Microsoft Entra Workload Identity Federation
(WIF). This tool was originally created to make it more ergonomic to use Microsoft Entra ID service principals from various external
service scenarios. The architectural pattern is for an Azure service to provide an OpenID Connect Protocol (OIDC) Java Web Token (JWT).
Microsoft Entra ID is pre-configured to recognize the OIDC issuer as valid for a particular Service Principal. The client application
can then parlay its OIDC token in to Service Principal access.
For this use-case, however, we will use a WIF OIDC token given to Azure Pipeline jobs with the AWS sts:AssumeRoleWithWebIdentity
API call to provide our job with short-lived AWS API tokens. This will rather neatly reduce the attack surface while at the same time
eliminating the operational toil around API key maintenance.
Setup
Azure Pipeline app registration
To start with, an Azure Pipeline app registration with WIF needs to be created. The Microsoft-provided instructions
will suffice for this. You will want to (in step 4) select a "Subscription" scope for this registration. Make a note of the name you use for the
service connection. It is wifconnectionname in our examples below, and needs to be specified appropriately in a task definition to be used.
Azure Pipeline creation
It is necessary, to discover the OIDC Issuer URL which is needed for the AWS IAM Identity Provider setup, to get the GUID of the Azure DevOps organization. This can be
somewhat tricky from Azure APIs, and we discovered that the easiest way by far was to emit the SYSTEM_COLLECTIONID environment variable in a shell task in a test
pipeline.
However, if you want to avoid that, an alternative was to go into the Azure console and browse resource groups. Each Azure DevOps Organization will have a resource group
named similarly to VisualStudioOnline-0123456789ABCDEF0123456789ABCDEF. The 0123456789ABCDEF0123456789ABCDEF part of the resource group name is the UUID. You will
need to normalize it into the UUID form like 01234567-89ab-cdef-0123-456789abcdef.
AWS IAM Identity Provider setup
Now, it is necessary to create an Identity Provider in AWS IAM. This can be done from the console, but the command line is simple enough. Assuming the SYSTEM_COLLECTIONID
environment variable is set to the Azure DevOps Organization UUID (as discovered in the previous step):
It is now time to setup an IAM role for your Azure Pipeline tasks to assume. You can setup a role with very limited attached permissions for testing, although you
will need significantly higher permissions to usefully deploy Infrastructure as Code in your full pipeline.
From the AWS IAM console, you can easily create a role with the "Web Identity" trusted entity type, which will allow you to select your IAM Identity Provider created
previously from a drop-down menu. If you wish to allow an existing role to use the newly created web identity provider, you will need to attach a trust policy
similar to (replace 01234567-89ab-cdef-0123-456789abcdef with your Azure DevOps Organization UUID from above):
At this point, it is time to test functionality. You can use the awsoidcsetup.sh and pipeline.yaml
files we have provided to create a test pipeline in Azure Pipelines and validate that you can successfully run aws sts get-caller-identity in a task.
OIDC fingerprint refresher
Some months after deploying this solution successfully, we discovered one day without warning that our pipeline had stopped working. Investigation revealed
that Microsoft periodically changes the fingerprint of their OIDC endpoint, leading to AWS to fail authentication. Thankfully there was a aws-oidc-provider-refresher
project that can easily update all OIDC fingerprints in IAM Identity Providers on a schedule as an AWS Lambda function. You can clone the GitHub repository
and the authors have included a "cloudformation/aws-oidc-provider-refresher.yaml" template that can be easily deployed to perform updates.
However, in July AWS made refreshing OIDC fingerprints unnecessary.
If you have the time frame to validate this federation solution, we would recommend holding off on the OIDC fingerprint refresher unless and until it was needed.
A sample repository
maintained by Microsoft with concrete examples on using WIF and Terraform within Azure. This isn't directly the pattern used by us (as we needed Terraform to access
AWS), but still provided useful context.
This Python project refreshes OIDC fingerprints
in IAM identity providers, and provides a convenient CloudFormation template for AWS Lambda-based periodic refresh.