Release to customer account access controls

Overview

This document describes the way remote commands are executed from the Release platform to deploy infrastructure and code into customer accounts in Amazon Web Services (AWS).

Components

The following components of AWS services are used by the Release platform to connect to customer accounts:

• IAM

• CloudFormation

• CloudTrail

• Lambda

Diagram

Authentication with customer account

Discussion

Role creation and integration with Release Delivery

Here's a high-level overview of the role-creation process:

1. The end user logs in to the Release console and clicks the AWS integration tile. Role permissions are detailed below.

2. The end user is directed to the AWS console where they are prompted to accept and install a CloudFormation template.

   • The template creates a role with the minimum permissions required to create necessary services and infrastructure components.

   • The created role has an "External ID" and source-trust relationship back to the Release account.

   • Combining the External ID and the credentials from the trust relationship establishes a secret, safe way to authenticate and authorize both parties.

3. The CloudFormation template completes successfully, returning a webhook to the Release console confirming the AWS Resource Name for the role that was created.

4. All interactions and actions are logged in the AWS account by CloudTrail.

Ongoing access

Once the role is created, this is the process followed when Release executes commands on the customer's account:

1. Software running in the Release environment acquires a local role credential in the Release production account.

2. The software requests an AWS Security Token Service (AWS STS) AssumeRole credential for the remote customer role.

   • The IAM role is confirmed by AWS on the customer side by verifying the source account and External ID supplied during creation.

   • The IAM role is confirmed by AWS to have enough permissions to execute the command.

   • AWS creates a temporary set of credentials (between 15 minutes and several hours, depending on the configuration).

3. AWS returns a token that is used to execute API calls in the remote account.

   • If the credentials are about to expire, the software can renew the credentials by asking for a new set.

4. The software uses the temporary credentials to execute commands against the customer account.

5. All transactions, identities, and API calls are logged in CloudTrail in the respective account.

Removing access

The customer can remove Release remote access to their account at any time with any of the following unilateral actions (see below for specific details on restricting the role):

1. Deleting or restricting the policy attached to the role.

2. Deleting the IAM role.

3. Deleting the CloudFormation template (which deletes the role and any other artifacts of the installation).

4. Denying and revoking any tokens that were issued to the IAM role.

Keep in mind that these actions are irreversible and would require repeating the AWS integration steps for Release to regain access.

Recommendations

Release strongly encourages our customers to follow these best practices to maintain a good security posture, protecting Release and our customers:

• The hosting account should be a subaccount in the AWS organization to maintain strict boundaries between customer subaccounts (for example, to separate production, vendor, and pre-production accounts).

• The hosting account should be created and used only by Release for better visibility, auditing, reporting, and billing purposes (for example, to capture unblended costs for hosting customer infrastructure in their own account).

• The hosting account can be granted cross-account access to very specific services on an as-needed basis with minimum permissions (for example, allowing access to buckets or databases in a production account).

Technical details

The following section describes the general definitions above in more technical detail for the benefit of developers or security engineers.

Wherever possible, the policies are narrowed down to the bare minimum required for creating, updating, and deploying workloads in your AWS account. Removing or restricting even one of the permissions in the list can break the functionality of the Release platform, so please speak to us before taking any action on the policies and roles in your account.

Minimum-access posture

Release takes a posture of only requesting the minimum permissions required and requesting access to an account that is dedicated to the hosting of Release environments. This is validated by testing and verification where we remove access permissions and observe any failures to operate our platform in a separate account. This verification shows that no permissions we have requested can be removed safely without impacting the functionality of the platform.

Initial CloudFormation permissions

The initial installation requires permissions to install and run a CloudFormation stack. The CloudFormation stack creates a role, some inline and managed policies, and a lambda (to send a webhook back to Release). We recommend installing the CloudFormation stack with administrative permissions (due to iam:createrole and other permissions required). The minimum set of permissions required include:

• CloudFormation (cloudformation:* applied to *):

  • Allows the user to install and run and possibly update the CloudFormation stack required for initial account setup. You can view the AWS stack before applying it in the console during the integration setup.

• IAM (most permissions here are only available to administrators):

  • iam:createrole, iam:attachrolepolicy, iam:passrole, or iam:putrolepolicy applied to arn:aws:iam::*:role/release/* allows the user to create a role that is used for all cross-account access to build and deploy infrastructure.

  • iam:createpolicy applied to arn:aws:iam::*:policy/release/* allows the user to create policies that are used for all cross-account access to build and deploy infrastructure.

  • iam:get* or iam:list* applied to * for validation and checking permissions.

• Lambda (lambda:createfunction, lambda:get*, lambda:list*, lambda:invokefunction applied to *):

  • The CloudFormation template creates a Lambda and Lambda execution role to send a webhook back to Release. The webhook registers the created role ARN and confirms the External ID sent back from the role to verify the request is authentic. The Lambda name is autogenerated by the stack, so it is difficult to know the Lambda ARN to restrict permissions in advance.

Cross-account role details

The cross-account role has a name similar to arn:aws:iam::<YourAWSAccount>:role/releasehub-integration-ConsoleRole-<UUID>. This "integration role" allows Release to perform various operations on your behalf.

The following policies grant the cross-account role permissions:

• PowerUserAccess:

  • Grants permissions to create various infrastructure needed to support your cluster and workloads, including but not limited to EC2 instances and networking, S3 buckets, DynamoDB tables, EKS clusters, CloudFront Distributions, Application and Elastic Load Balancers, KMS keys, SSM secrets and services, EFS file systems, CloudFormation.

  • DOES NOT include permissions for billing, IAM, and other services.

• IAM inline permissions:

  • arn:aws:iam::*:instance-profile/release/* allows Release to create and store permissions for EC2 worker node policies so that workloads have access to various resources needed. You can customize the access granted, for example, access to an S3 bucket from the website. Namespaced to /release/ for auditing and separation purposes.

  • arn:aws:iam::*:policy/release/* allows Release to create and read policies for AWS services, workloads, jobs, and so on. Namespaced to /release/ for auditing and separation purposes.

  • arn:aws:iam::*:role/release/* allows Release to create and assume roles for service accounts or workloads to perform tasks and gain access to resources needed. You can customize the access granted, for example, a batch job processor that needs access to SQS. Namespaced to /release/ for auditing and separation purposes.

  • arn:aws:iam::*:user/release/* allows Release to create and manage users to access the AWS account for coordinated troubleshooting. These would only be used in cases where an agreed-upon and coordinated effort exists between Release and the customer. Namespaced to /release/ for auditing and separation purposes.

  • arn:aws:iam::*:instance-profile/eksctl-* allows Release to create and store permissions for EC2 instances that are attached to the EKS clusters using the open source tool eksctl. Only applies to EKS-attached instances.

  • arn:aws:iam::*:role/eksctl-* allows Release to create and store roles used by eksctl to allow operations related to EKS workers, nodes, and workloads. Only applies to EKS-created instances and roles. Managed by CloudFormation templates.

  • arn:aws:iam::*:oidc-provider/* allows Release to create and access OIDC providers that allow services and service accounts access to various services and roles in AWS. You can customize the access granted, for example, access to Secrets Manager stores from a pod running in EKS.

  • arn:aws:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup is a very specific service role that is needed when creating, updating, and accessing EKS clusters in the customer account.

Restricting access

In the normal course of events, there is no need to restrict the access Release requests to a separate account running in your AWS organization, given the restrictions and limitations of the policies above. However, there may be occasions when you require more fine-grained permissions than specified above. For example:

• Restricting access to services in a particular region or regions.

• Restricting access to certain rare permissions, such as DELETE or DECRYPT.

• Restricting access to fine-grained naming conventions, such as sensitive buckets referred to directly by name for exclusion.

• Restricting access to specific rare services, such as machine learning.

• Restricting fine-grained access to some secrets or KMS keys.

To activate these restrictions, coordinate with Release to set AWS permissions boundaries. The Release team can verify the boundary permissions, identify risks to proper functionality, and (if approved) test and verify the correctness of the boundary permissions with the customer workloads.

The permissions boundaries can be created before the integration is applied and can be optionally specified in the CloudFormation template. However, it is often better to add permissions boundaries after the initial installation steps are completed and confirmed to prevent unforeseen issues causing unnecessary intervention, debugging, and cleanup efforts. In other words, we recommend that customers get their applications deployed, running, and tested before restricting access using permissions boundaries to ensure no permissions-related issues emerge before the application is fully functional.