Automating S3 bucket compliance check & remediation with AWS Config

Posted on | In , , ,

In the last post, we explored the most common S3 security breaches and how to remediate them (if you missed it, you can read it here).

Better than knowing how to protect your data is to automate the entire process, so whenever a similar situation arises, you can rest assured that the problem will be automatically fixed.

Today, I’ll present how to automate the monitoring and remediation for the four recommended bucket configurations mentioned in the last post:

  • Bucket versioning
  • Block public read/write access
  • Bucket logging
  • Server-side encryption

By the end of this post, you will know how to automate the compliance check of your S3 buckets and automatically remediate them back to safety.

Short on time? Here’s the gist with the full CloudFormation implementation of the solution

AWS Config

The first step to automate your S3 bucket configuration is monitoring. We need to have a way to continuously monitor the state of our buckets and compare them against our defined rules.

AWS provides a service exactly for this purpose: AWS Config.

AWS Config allows us to monitor and audit not only S3 buckets, but many other resource types. Resources are monitored by a recorder, that checks their states periodically and compares them against our defined rules. These rules inform AWS Config about the desired state of the resource and, in case it diverges, the resource is reported as non-compliant.

AWS Config provides a set of managed rules that can be used to monitor common scenarios. For example, the managed rule s3-bucket-server-side-encryption-enabled can be used to verify if SSE (server-side encryption) is enabled in a S3 bucket. Here’s the full list of managed rules provided by AWS Config.

Whenever a non-complying resource is discovered, AWS Config can execute an action in response to the event. Among the supported actions, the one that we will use in today’s experiment is the auto-remediation.

AWS Config diagram (source: AWS)

AWS System Manager (SSM)

AWS Config itself doesn’t know how to remediate a resource. This task is delegated to AWS System Manager. It provides a set of runbooks that can be executed to modify a resource. For example, the runbook AWS-EnableS3BucketEncryption can be executed to enable SSE in an S3 bucket. A list of all SSM managed runbooks can be found here.

Hands-on

AWS provides a detailed article on how to set up everything using the AWS console.

Today, we will focus on the CloudFormation template to automate the entire process.

AWS Config Recorder

As mentioned before, AWS Config works by using a recorder to periodically check the state of your AWS resources.

Before we move ahead with the recorder configuration, we need to provision storage for the resource states. AWS Config stores these states in an S3 bucket. So, let’s create the bucket:

#
# S3 bucket used by AWS Config Recorder to record resources state
#
RecorderBucket:
  Type: AWS::S3::Bucket

Next, we need an IAM role granting AWS Config permissions to create objects in the recorder bucket above. For this, we need first a service role for AWS Config and an IAM role granting the necessary permissions and allowing AWS Config to assume this role:

#
# Service role for AWS Config
#
ServiceRole:
  Type: AWS::IAM::ServiceLinkedRole
  Properties:
    AWSServiceName: config.amazonaws.com

#
# IAM role for AWS Config Recorder to interact with the S3 recorder bucket
#
RecorderRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        # Allow AWS Config to assume this role
        - Effect: Allow
          Principal:
            Service:
              - config.amazonaws.com
          Action:
            - sts:AssumeRole
    ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWS_ConfigRole
    Policies:
      - PolicyName: S3Policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            # Grant permissions to put objects in the bucket
            - Effect: Allow
              Action:
                - s3:PutObject
                - s3:PutObjectAcl
              Resource: !Sub
                - ${BucketArn}/AWSLogs/${AccountId}/Config/*
                - BucketArn: !GetAtt [ RecorderBucket, Arn ]
                  AccountId: !Ref AWS::AccountId
            - Effect: Allow
              Action:
                - s3:GetBucketAcl
              Resource: !GetAtt [ RecorderBucket, Arn ]

Next, we need a remediation role. This role will grant SSM permissions to modify the non-compliant buckets and bring them back to the expected state:

#
# IAM role used by AWS Config auto-remediation to change the configuration on S3 buckets
#
RemediationRole:
  Type: AWS::IAM::Role
  Properties:
    # Assume role policy, granting SSM permissions to assume this role
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ssm.amazonaws.com
          Action:
            - sts:AssumeRole
      
    # AWS managed policies, granting permissions for SSM Automation Role operation and full
    # access to our S3 buckets
    ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AmazonSSMAutomationRole
      - arn:aws:iam::aws:policy/AmazonS3FullAccess

Finally, let’s create our AWS Config Recorder. This recorder will be restricted to only S3 buckets:

#
# AWS Config Recorder configuration
#
RecorderConfiguration:
  Type: AWS::Config::ConfigurationRecorder
  Properties:
    Name: S3BucketRecorderConfig
    RecordingGroup:
      AllSupported: false
      IncludeGlobalResourceTypes: false
      ResourceTypes:
        - AWS::S3::Bucket # Limit the recorder scope to only S3 buckets
    RoleARN: !GetAtt [ RecorderRole, Arn ]

Recorder set up, let’s now move to our rules.

Rule #1: Bucket versioning

This rule will check for the S3 bucket versioning setup. Buckets with versioning disabled will be considered non-compliant. This compliance check is done using the AWS Config managed rule s3-bucket-versioning-enabled.

#
# AWS Config rule for bucket versioning
#
BucketVersioningRule:
  Type: AWS::Config::ConfigRule
  DependsOn: RecorderConfiguration
  Properties:
    ConfigRuleName: BucketVersioningRule
    Description: "Rule to enable versioning on S3 buckets"
    Scope:
      ComplianceResourceTypes:
        - AWS::S3::Bucket
    Source:
      Owner: AWS
      SourceIdentifier: S3_BUCKET_VERSIONING_ENABLED

The result is a new rule in AWS Config console, BucketVersioningRule:

New rule listed on AWS Config Dashboard

Now, we need to inform AWS Config how to remediate violations of this rule. We will use for this the SSM runbook AWS-ConfigureS3BucketVersioning.

We need to provide some parameters for this runbook:

  • AutomationAssumeRole: the role assumed by AWS SSM to perform the modifications in the non-compliant resource
  • BucketName: the name of the non-compliant bucket
  • VersioningState: the expected versioning state after the runbook is executed
#
# AWS Config auto-remediation for bucket versioning
#
BucketVersioningRemediation:
  Type: AWS::Config::RemediationConfiguration
  Properties:
    Automatic: true # Automatically executes this remediation when a non-compliant bucket is found
    MaximumAutomaticAttempts: 5
    RetryAttemptSeconds: 30
    ConfigRuleName: !Ref BucketVersioningRule
    TargetId: AWS-ConfigureS3BucketVersioning # The AWS SSM automation runbook to execute
    TargetType: SSM_DOCUMENT
    Parameters:
      AutomationAssumeRole:
        StaticValue:
          Values:
            - !GetAtt [ RemediationRole, Arn ]
      BucketName:
        ResourceValue:
          Value: RESOURCE_ID # This will fill the "BucketName" attribute with the bucket name provided by the AWS Config rule
      VersioningState:
        StaticValue:
          Values:
            - Enabled

Applying our template so far we get the following remediation defined in the AWS Config console:

Versioning remediation listed on AWS Config dashboard

Rule #2: Bucket logging

This rule will check whether logging is enabled on a bucket or not. In case it’s disabled, this resource is marked as non-compliant. This check will be performed by the managed rule s3-bucket-logging-enabled.

#
# AWS Config rule for bucket logging
#
BucketLoggingRule:
  Type: AWS::Config::ConfigRule
  DependsOn: RecorderConfiguration
  Properties:
    ConfigRuleName: BucketLoggingRule
    Description: "Rule to enable logging on S3 buckets"
    Scope:
      ComplianceResourceTypes:
        - AWS::S3::Bucket
    Source:
      Owner: AWS
      SourceIdentifier: S3_BUCKET_LOGGING_ENABLED

The new rule should be listed in the AWS Config console with the name BucketLoggingRule:

Logging rule on AWS Config Dashboard

For the remediation, we will be using the managed runbook AWS-ConfigureS3BucketLogging.

For this runbook we provide the following parameters:

  • AutomationAssumeRole: the role assumed by AWS SSM to perform the modifications in the non-compliant resource
  • BucketName: the name of the non-compliant bucket
  • GranteeType: the type of grantee for the logging bucket. In this example, we will be using Group
  • GranteeUri: the URI of the grantee. We will be using the AWS S3 group Log Delivery, which is a managed group recognized by S3 to deliver access logs to a bucket. The group URL is http://acs.amazonaws.com/groups/s3/LogDelivery
  • GrantedPermission: the level of permission granted for the grantee. In our case, we give FULL_CONTROL
  • TargetBucket: the bucket where the logs will be stored

Additionally, we need to create a bucket that will be used to store the access logs for the other buckets. So, our Cloudformation snippet for the remediation is:

#
# S3 bucket used to log other S3 buckets access
#
LoggingBucket:
  Type: AWS::S3::Bucket
  Properties:
    AccessControl: LogDeliveryWrite # Grant the S3 Log Delivery group write permission on this bucket

#
# AWS Config auto-remediation for bucket logging
#
BucketLoggingRemediation:
  Type: AWS::Config::RemediationConfiguration
  Properties:
    Automatic: true
    MaximumAutomaticAttempts: 5
    RetryAttemptSeconds: 30
    ConfigRuleName: !Ref BucketLoggingRule
    TargetId: AWS-ConfigureS3BucketLogging
    TargetType: SSM_DOCUMENT
    Parameters:
      AutomationAssumeRole:
        StaticValue:
          Values:
            - !GetAtt [ RemediationRole, Arn ]
      BucketName:
        ResourceValue:
          Value: RESOURCE_ID # This will fill the "BucketName" attribute with the bucket name provided by the AWS Config rule
      GrantedPermission:
        StaticValue:
          Values:
            - FULL_CONTROL
      GranteeType:
        StaticValue:
          Values:
            - Group
      GranteeUri:
        StaticValue:
          Values:
            - http://acs.amazonaws.com/groups/s3/LogDelivery
      TargetBucket:
        StaticValue:
          Values:
            - !Ref LoggingBucket

Applying this template, we get our second remediation created:

Logging remediation created

Rule #3: Bucket Public read access

Our third AWS Config rule will be responsible to check for buckets with public read access enabled. If so, the bucket is marked as non-compliant.

For this, we will be using the managed rule s3-bucket-public-read-prohibited.

#
# AWS Config rule for blocking public read
#
BucketPublicReadProhibitedRule:
  Type: AWS::Config::ConfigRule
  DependsOn: RecorderConfiguration
  Properties:
    ConfigRuleName: BucketPublicReadProhibitedRule
    Scope:
      ComplianceResourceTypes:
        - AWS::S3::Bucket
    Source:
      Owner: AWS
      SourceIdentifier: S3_BUCKET_PUBLIC_READ_PROHIBITED

Public read rule created

Since AWS System Manager doesn’t provide dedicated remediation for read and write access, we will be using the runbook
AWS-DisableS3BucketPublicReadWrite, which disables both read and write for public accessible buckets. This is fine for our use case since we want both public read and write access blocked but, in case you want to have more control over this (like restricting only writing access, for example), make sure to check the section about custom remediation.

The following parameters are provided for the runbook:

  • AutomationAssumeRole: the role assumed by AWS SSM to perform the modifications in the non-compliant resource
  • BucketName: the name of the non-compliant bucket
#
# AWS Config auto-remediation for bucket public-read access
#
BucketPublicReadRemediation:
  Type: AWS::Config::RemediationConfiguration
  Condition: EnsureNoPublicRW
  Properties:
    Automatic: true
    MaximumAutomaticAttempts: 5
    RetryAttemptSeconds: 30
    ConfigRuleName: !Ref BucketPublicReadProhibitedRule
    TargetId: AWS-DisableS3BucketPublicReadWrite
    TargetType: SSM_DOCUMENT
    Parameters:
      AutomationAssumeRole:
        StaticValue:
          Values:
            - !GetAtt [ RemediationRole, Arn ]
      S3BucketName:
        ResourceValue:
          Value: RESOURCE_ID

Public read remediation details

Rule #4: Bucket Public write access

This rule works exactly like the previous one. The reason for two separated rules is that AWS Config doesn’t provide an unified managed rule for both read/write permissions.

The only difference to the read rule described before is the managed rule used, which in this case is S3_BUCKET_PUBLIC_WRITE_PROHIBITED instead of S3_BUCKET_PUBLIC_READ_PROHIBITED.

#
# AWS Config rule for blocking public write
#
BucketPublicWriteProhibitedRule:
  Type: AWS::Config::ConfigRule
  Condition: EnsureNoPublicRW
  DependsOn: RecorderConfiguration
  Properties:
    ConfigRuleName: BucketPublicWriteProhibitedRule
    Scope:
      ComplianceResourceTypes:
        - AWS::S3::Bucket
    Source:
      Owner: AWS
      SourceIdentifier: S3_BUCKET_PUBLIC_WRITE_PROHIBITED

#
# AWS Config auto-remediation for bucket public-write access
#
BucketPublicWriteRemediation:
  Type: AWS::Config::RemediationConfiguration
  Condition: EnsureNoPublicRW
  Properties:
    Automatic: true
    MaximumAutomaticAttempts: 5
    RetryAttemptSeconds: 30
    ConfigRuleName: !Ref BucketPublicWriteProhibitedRule
    TargetId: AWS-DisableS3BucketPublicReadWrite
    TargetType: SSM_DOCUMENT
    Parameters:
      AutomationAssumeRole:
        StaticValue:
          Values:
            - !GetAtt [ RemediationRole, Arn ]
      S3BucketName:
        ResourceValue:
          Value: RESOURCE_ID

Public write rule created

Public write remediation details

Rule #5: Bucket SSE

Finally, our 5th and last rule is for bucket SSE (server-side encryption) setup. This rule checks if SSE is enabled in the bucket. If not, it’s considered non-compliant. For this, we will be using the managed rule s3-bucket-server-side-encryption-enabled.

#
# AWS Config rule for server-side encryption
#
BucketSSERule:
  Type: AWS::Config::ConfigRule
  Condition: EnsureSSE
  DependsOn: RecorderConfiguration
  Properties:
    ConfigRuleName: BucketSSERule
    Scope:
      ComplianceResourceTypes:
        - AWS::S3::Bucket
    Source:
      Owner: AWS
      SourceIdentifier: S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED

SSE (server-side encryption) rule created

For the automation part, we will be using the runbook AWS-EnableS3BucketEncryption.

As parameters for this runbook, we will provide:

  • AutomationAssumeRole: the role assumed by AWS SSM to perform the modifications in the non-compliant resource
  • BucketName: the name of the non-compliant bucket
  • SSEAlgorithm: the SSE algorithm to be set on the bucket. In our case, we will use AES256, which leverages an S3 managed key
#
# AWS Config auto-remediation for bucket server-side encryption
#
BucketSSERemediation:
  Type: AWS::Config::RemediationConfiguration
  Condition: EnsureSSE
  Properties:
    Automatic: true
    MaximumAutomaticAttempts: 5
    RetryAttemptSeconds: 30
    ConfigRuleName: !Ref BucketSSERule
    TargetId: AWS-EnableS3BucketEncryption
    TargetType: SSM_DOCUMENT
    Parameters:
      AutomationAssumeRole:
        StaticValue:
          Values:
            - !GetAtt [ RemediationRole, Arn ]
      BucketName:
        ResourceValue:
          Value: RESOURCE_ID
      SSEAlgorithm:
        StaticValue:
          Values:
            - AES256

SSE (server-side encryption) remediation details

Final template

Collecting all the snippets presented above we have the full Cloudformation implementation of the solution.

For the sake of convenience, here’s a gist with the full implementation. This Gist also includes toggles for all the rules, so you can enable/disable only those you’re interested in.

Validation

Now that we have our AWS Config setup done with automatic remediation, it’s time to see everything in practice.

For this, let’s create an S3 bucket violating all the rules covered by our AWS Config setup:

#
# Creates a bucket with:
#  - Versioning disabled (default)
#  - Logging disabled    (default)
#  - SSE disabled        (default)
#  - Public Read/Write access
#
$ aws s3api create-bucket \
    --bucket super-secret-data \
    --acl public-read-write \
    --create-bucket-configuration LocationConstraint="[YOUR AWS REGION HERE]"

Overly permissive bucket

Pretty bad, huh?!

Now, let’s see AWS Config taking care of fixing this mess.

In some minutes, AWS Config recorder will discover our new bucket and run the checks. Since at least one of our rules are non-compliant, the bucket is reported as non-compliant as well.

By checking the bucket details, we can see what rules are non-compliant for the resource (in our case, all of them):

Non-compliant bucket discovered

Since our rules are backed by automated remediation, AWS SSM will start dispatching the runbooks to remediate the bucket.

In the next recorder execution, this bucket will be checked again against our rules. This time, since SSM fixed the violations automatically, the recorder will realize that the violations were remediated, and the bucket is reported as compliant.

Rules are now reported as compliant

Going back to the S3 console and checking our bucket details we can now see that all violations were fixed, and our bucket is now secure.

Custom remediation

Sometimes the managed SSM runbooks aren’t enough for us, either because they have limitations or because our use case is very specific.

In such situations, you can create your own remediation logic using AWS Lambda. The idea is, instead of registering a managed SSM runbook on the rule auto-remediation, we will invoke a lambda that performs this remediation using the AWS SDK. In this case, we have full control over the lambda logic and can implement remediations tailored for our specific case.

Approach #1: using an SNS topic

AWS Config supports sending notifications to anSNS topic with configuration changes identified by the recorder (here’s a list of notifications AWS Config sends to an SNS topic). Among these changes, there are compliance change notifications, which is sent when a resource transitions between compliant or non-compliant states.

With a proper setup, we can invoke our remediation lambda whenever a compliance change lands the SNS topic.

To receive configuration changes in an SNS topic, we need to create a AWS Config Delivery Channel.

Topic:
  Type: AWS::SNS::Topic

DeliveryChannel: 
  Type: AWS::Config::DeliveryChannel
  Properties: 
    SnsTopicARN: !Ref Topic

You can have only one delivery channel per region

The snippet above will create a delivery channel for the existing Config recorder in the region. Now, we just need to connect our lambda to the SNS topic and execute the logic when a non-compliant resource is identified.

Approach #2: using EventBridge rule

The second approach is similar to the first one, except that, instead of using an SNS topic, we will be using a EventBridge Event Rule as the lambda trigger.

Using EventBridge, we can create a rule to invoke our lambda whenever a resource transitions to a non-compliant state.

{
  "source":[
    "aws.config"
  ],
  "detail":{
    "requestParameters":{
      "evaluations":{
        "complianceType":[
          "NON_COMPLIANT"
        ]
      }
    },
    "additionalEventData":{
      "managedRuleIdentifier":[
        "S3_BUCKET_LOGGING_ENABLED"
      ]
    }
  }
}

The rule above will be triggered on non-compliant S3_BUCKET_LOGGING_ENABLED rules. With the rule created, we just need to define our lambda function as a target.

EventBridge setup with a lambda function as trigger

Conclusion

In this article, you learned how to identify and fix insecure S3 buckets in an automated way. Having this automation is essential since you no longer need to keep track of newly created buckets and check for their compliance manually. AWS Config will do the heavy-lifting and guarantee that, even if a permissive bucket lands your account, measures to fix this will be applied automatically whenever necessary.

Resources

Liked this post?

Have questions? Suggestions?

Follow me on Twitter and let's engage in a conversation