How to Keep Your Certificates from Expiring...Automatically

How do you prove to someone who you are? You show them an identification document, such as a driver’s license or passport, issued to you by a government agency. These documents contain your name, date of birth, and personal information like eye and hair color that verifies your identity. On the Internet, web servers and browser clients prove their identities using X.509 certificates issued by Certificate Authorities (CAs)——trusted entities responsible for validating the identity of certificate applicants and signing their certificates. These certificates also have identifying details like domain names and postal addresses.

You can see SSL/TLS X.509 certificates when you click the lock icon in a web browser's address bar, but their usage extends far beyond public-facing websites. Internal enterprise systems also rely on certificates to authenticate the identity of servers to their clients (or one internal service to another). Like identification documents that must be renewed before they expire, certificates have expiration dates and require timely renewal.

Enterprises often use a combination of public and private cloud platforms and services to address diverse business needs. However, to ensure interoperability and streamline cross-platform support, many organizations centralize critical identity capabilities, such as Public Key Infrastructure (PKI). A common approach is implementing a Private Certificate Authority (Private CA), which facilitates the issuance and lifecycle management of certificates for internal applications. Numerous solutions are available for deploying a Private CA, ranging from cloud-based options like AWS Certificate Manager Private CA to on-premises systems such as Microsoft Active Directory Certificate Services and hybrid tools like HashiCorp Vault.

While the management of the overall CA may be centralized within an organization, the responsibility for obtaining and renewing certificates is often distributed across business units and delegated to DevOps teams. This approach aligns with the principles of DevOps, which emphasize collaboration, automation, and shared responsibility. However, it also introduces challenges, such as the risk of a missed renewal leading to service outages or security incidents. Relying on manual processes, such as tracking expiration dates via email reminders or spreadsheets, is not only time-consuming but also error-prone.

The solution? Automating certificate lifecycle management and embedding it into existing workflows. CI/CD pipelines, integral to DevOps processes, can include scripts written in Python or other languages to monitor and renew expiring certificates. This automation ensures reliability, reduces human error, and enables teams to focus on higher-value tasks.

The following provides a sample solution for automating the renewal of certificates issued by an on-premises CA (using the open source EJBCA product) for services hosted in AWS. This approach can be tailored to other cloud providers and PKI solutions based on your organization's needs. You can find a copy of this codebase in our GitHub repository.

Determining Which Certificates are Expiring

AWS Certificate Manager (ACM) stores the TLS certificates for our services, and supports seamless retrieval for the Elastic Load Balancing, Amazon CloudFront, and AWS API Gateway services among others. We can query ACM using the AWS Boto3 SDK to retrieve our stored certificates and identify any that are expiring within a certain number of days.

def get_certificates_expiring_soon(client, days=60):
    try:
        logger.info(f"Getting certificates expiring in the next {days} days")
        response = client.list_certificates(
            Includes={
                'keyTypes': [
                    'RSA_1024', 'RSA_2048', 'RSA_3072', 'RSA_4096',
                    'EC_prime256v1', 'EC_secp384r1', 'EC_secp521r1'
                ]
            }
        )
        certificates = response['CertificateSummaryList']
        expiring_soon = []
        for cert in certificates:
            not_after = cert['NotAfter']
            is_imported = cert['Type'] == 'IMPORTED'
            if is_imported and not_after < datetime.now(timezone.utc) + timedelta(days=days):
                expiring_soon.append(cert)
        logger.success(
            f"Found {len(expiring_soon)} certificates expiring in the next {days} days")
        return expiring_soon
    except Exception as e:
        logger.error(f"Error getting certificates expiring soon: {e}")
        return False

Generate Signing Key and Certificate Signing Request

For each certificate identified as expiring, the next step is to generate a new private key and create a Certificate Signing Request (CSR). This CSR must be submitted to the CA to obtain a new certificate. The corresponding private key is used to sign the CSR and later to install the newly issued certificate.

A CSR must contain the following information:

Common Name (CN): The fully qualified domain name (such as "www.rearc.io") for the requested certificate.
Organization (O): The legal name of the organization.
Organizational Unit (OU): The division of the organization handling the certificate.
City/Locality (L): The city where the organization is located.
State/Province (ST): The state or province where the organization is located.
Country (C): The two-letter ISO code for the country where the organization is located.
Email Address: An email address for contact purposes (optional but recommended).
Public Key: The public key corresponding to the private key used to sign the CSR.

def generate_openssl_key(bitsize):
    try:
        logger.info(f"Generating key with {bitsize} bits")
        key = rsa.generate_private_key(
            public_exponent=65537,
            key_size=bitsize,
        )
        pemkey = key.private_bytes(
            encoding=serialization.Encoding.PEM,
            format=serialization.PrivateFormat.TraditionalOpenSSL,
            encryption_algorithm=serialization.NoEncryption()
        )
        return pemkey
    except Exception as e:
        logger.error("Error generating key: {e}")
        return False

def generate_csr(dnsname, pemkey):
    try:
        logger.info(f"Generating CSR for {dnsname}")
        key = serialization.load_pem_private_key(pemkey, password=None)
        csr = x509.CertificateSigningRequestBuilder().subject_name(x509.Name([
            x509.NameAttribute(NameOID.COUNTRY_NAME, u"US"),
            x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, u"New York"),
            x509.NameAttribute(NameOID.LOCALITY_NAME, u"New York"),
            x509.NameAttribute(NameOID.ORGANIZATION_NAME, u"Rearc"),
            x509.NameAttribute(NameOID.ORGANIZATIONAL_UNIT_NAME, u"Delivery"),
            x509.NameAttribute(NameOID.COMMON_NAME, dnsname),
        ])).sign(key, hashes.SHA256())
        pemcsr = csr.public_bytes(serialization.Encoding.PEM)
        return pemcsr
    except Exception as e:
        logger.error(f"Error generating CSR for {dnsname}: {e}")
        return False

Submit CSR to Certificate Authority

For the purposes of this exercise, we are utilizing EJBCA, an open source PKI solution, to serve as our Private CA. The CA validates the information provided in the CSR and issues a new certificate. The CA also returns its own certificates (root and intermediate) that are required to complete the certificate chain and used to verify the certificate's authenticity.

def submit_csr_to_ca(dnsname, csr):

    # EJBCA REST API endpoint for certificate request
    url = 'https://<ejbca-server>/ejbca/ejbca-rest-api/v1/certificate/pkcs10enroll'

    # Data to be sent in the POST request
    data = {
        "certificate_request": csr.decode('utf-8'),
        "certificate_profile_name": "ENDUSER",
        "end_entity_profile_name": "User",
        "certificate_authority_name": "ManagementCA",
        "username": "User",
        "password": "abc123",
        "include_chain": True,
        "email": "useremail@domain.com",
        "response_format": "DER"
    }

    headers = {
        'Content-type': 'application/json',
    }

    try:
        logger.info(f"Submitting CSR for {dnsname}")
        response = requests.post(url, data=json.dumps(data), headers=headers)
        response.raise_for_status()

        # Get the certificate and chain from the response
        response_data = response.json()

        # Get the issued certificate
        certificate = response_data['certificate']

        # Get the certificate chain associated with the CA
        chain = response_data['certificate_chain']

        logger.success(f"Certificate issued for {dnsname}")
        return certificate, chain

    except Exception as e:
        logger.error(f"Error submitting CSR for {dnsname}: {e}")
        return False

Upload Certificate to ACM

Once we have the new certificate, the final step is to upload it to ACM. We use our boto3 client to import the certificate, private key, and certificate chain. We also specify the expiring certificate's ARN (which we obtained earlier) to overwrite the existing certificate in-place.

def upload_cert_to_acm(client, arn, cert, chain, private_key):
    try:
        logger.info(f"Uploading certificate to ACM for {arn}")
        import_response = client.import_certificate(
            CertificateArn=arn,
            Certificate=format_certificate(cert),
            PrivateKey=private_key,
            CertificateChain=format_certificate(chain),
        )
        logger.success(f"Certificate uploaded to ACM for {arn}")
        return True
    except Exception as e:
        logger.error(f"Error uploading certificate to ACM: {
                     e.with_traceback(tb=e._traceback_)}")
        return False

Takeaway

Just as an expired driver's license can prevent legal driving and result in penalties, expired digital certificates can cause service disruptions and security vulnerabilities. By automating certificate management within the DevOps ecosystem, enterprises can ensure seamless operations and protect their systems from the risks of expired credentials.