Using Workload Identity Federation In Azure Pipelines For AWS IAM Integration

Introduction

Azure Pipelines is a product in the Azure DevOps suite that provides the ability to create and run pipelines of almost any sort in Azure. Much like AWS's CodePipeline service, it offloads the operational toil of a hosted solutions like Jenkins to the cloud service provider, while at the same time allowing a flexible amount of workload hosting to be done under a customer's control.

Recently, a Rearc customer found themselves desiring to use an Azure Pipeline to deploy Infrastructure as Code in Terraform against various AWS accounts. The current state of the art for this is to create AWS IAM users, generate API keys, and to store those API keys in what Azure Pipelines refers to as a Connection. This is not ideal, for a few reasons; API keys are subject to leakage (which is a disconcerting prospect when the keys are attached to a high-permissions deployment user), API keys should be rotated regularly (which adds operational toil), and an AWS IAM user must be generated for every target AWS account (or complicated role assumptions must be done).

Enter Microsoft Entra Workload Identity Federation (WIF). This tool was originally created to make it more ergonomic to use Microsoft Entra ID service principals from various external service scenarios. The architectural pattern is for an Azure service to provide an OpenID Connect Protocol (OIDC) Java Web Token (JWT). Microsoft Entra ID is pre-configured to recognize the OIDC issuer as valid for a particular Service Principal. The client application can then parlay its OIDC token in to Service Principal access.

For this use-case, however, we will use a WIF OIDC token given to Azure Pipeline jobs with the AWS sts:AssumeRoleWithWebIdentity API call to provide our job with short-lived AWS API tokens. This will rather neatly reduce the attack surface while at the same time eliminating the operational toil around API key maintenance.

Setup

Azure Pipeline app registration

To start with, an Azure Pipeline app registration with WIF needs to be created. The Microsoft-provided instructions will suffice for this. You will want to (in step 4) select a "Subscription" scope for this registration. Make a note of the name you use for the service connection. It is wifconnectionname in our examples below, and needs to be specified appropriately in a task definition to be used.

Azure Pipeline creation

It is necessary, to discover the OIDC Issuer URL which is needed for the AWS IAM Identity Provider setup, to get the GUID of the Azure DevOps organization. This can be somewhat tricky from Azure APIs, and we discovered that the easiest way by far was to emit the SYSTEM_COLLECTIONID environment variable in a shell task in a test pipeline.

However, if you want to avoid that, an alternative was to go into the Azure console and browse resource groups. Each Azure DevOps Organization will have a resource group named similarly to VisualStudioOnline-0123456789ABCDEF0123456789ABCDEF. The 0123456789ABCDEF0123456789ABCDEF part of the resource group name is the UUID. You will need to normalize it into the UUID form like 01234567-89ab-cdef-0123-456789abcdef.

AWS IAM Identity Provider setup

Now, it is necessary to create an Identity Provider in AWS IAM. This can be done from the console, but the command line is simple enough. Assuming the SYSTEM_COLLECTIONID environment variable is set to the Azure DevOps Organization UUID (as discovered in the previous step):

    aws create-open-id-connect-provider --url "https://vstoken.dev.azure.com/${SYSTEM_COLLECTIONID}" --client-id-list "api://AzureADTokenExchange"

AWS IAM Role setup

It is now time to setup an IAM role for your Azure Pipeline tasks to assume. You can setup a role with very limited attached permissions for testing, although you will need significantly higher permissions to usefully deploy Infrastructure as Code in your full pipeline.

From the AWS IAM console, you can easily create a role with the "Web Identity" trusted entity type, which will allow you to select your IAM Identity Provider created previously from a drop-down menu. If you wish to allow an existing role to use the newly created web identity provider, you will need to attach a trust policy similar to (replace 01234567-89ab-cdef-0123-456789abcdef with your Azure DevOps Organization UUID from above):

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": "arn:aws:iam::012345678901:oidc-provider/vstoken.dev.azure.com/01234567-89ab-cdef-0123-456789abcdef"
                },
                "Action": "sts:AssumeRoleWithWebIdentity"
            }
        ]
    }

Testing

At this point, it is time to test functionality. You can use the aws_oidc_setup.sh and pipeline.yaml files we have provided to create a test pipeline in Azure Pipelines and validate that you can successfully run aws sts get-caller-identity in a task.

OIDC fingerprint refresher

Some months after deploying this solution successfully, we discovered one day without warning that our pipeline had stopped working. Investigation revealed that Microsoft periodically changes the fingerprint of their OIDC endpoint, leading to AWS to fail authentication. Thankfully there was a aws-oidc-provider-refresher project that can easily update all OIDC fingerprints in IAM Identity Providers on a schedule as an AWS Lambda function. You can clone the GitHub repository and the authors have included a "cloudformation/aws-oidc-provider-refresher.yaml" template that can be easily deployed to perform updates.

However, in July AWS made refreshing OIDC fingerprints unnecessary. If you have the time frame to validate this federation solution, we would recommend holding off on the OIDC fingerprint refresher unless and until it was needed.

References

1. https://pipe.how/get-oidctoken/

   • This blog article by Emanuel Palm set us on the path of getting an OIDC token in an Azure Pipeline job.

2. https://devblogs.microsoft.com/devops/introduction-to-azure-devops-workload-identity-federation-oidc-with-terraform/

   • This Microsoft devblog provided details on what the Issuer URL (Provider URL in IAM) and Audience should look like for WIF OIDC.

3. https://github.com/Azure-Samples/azure-devops-terraform-oidc-ci-cd

   • A sample repository maintained by Microsoft with concrete examples on using WIF and Terraform within Azure. This isn't directly the pattern used by us (as we needed Terraform to access AWS), but still provided useful context.

4. https://pypi.org/project/aws-oidc-provider-refresher/

   • This Python project refreshes OIDC fingerprints in IAM identity providers, and provides a convenient CloudFormation template for AWS Lambda-based periodic refresh.

5. https://aws.amazon.com/about-aws/whats-new/2024/07/aws-identity-access-management-open-id-connect-identity-providers/ - AWS announcement about no longer needing to update OIDC fingerprints.

6. aws_oidc_setup.sh

   • This shell script will retrieve temporary credentials from STS using the OIDC provider from a Service Connection in an Azure Pipeline task.

7. pipeline.yaml

   • Demonstration pipeline definition that shows how to use the aws_oidc_setup.sh script to get AWS credentials and use them in further tasks.