aws-cloudtrail

AWS CloudTrail

Audit AWS account activity with CloudTrail for compliance, security investigation, and operational troubleshooting.

When to Use

• Enabling organization-wide audit logging across all AWS accounts
• Investigating security incidents or unauthorized API activity
• Meeting compliance requirements for SOC 2, HIPAA, PCI DSS, or FedRAMP
• Setting up automated alerting on sensitive AWS API calls
• Querying historical AWS activity for forensic analysis

Prerequisites and Change Scope

Verify the AWS identity, partition, region and organization management/delegated-administrator permissions before mutation. All names, account IDs and ARNs below are synthetic placeholders; choose a globally unique owned bucket. Have a working KMS key policy permitting the selected CloudTrail trail and authorized readers, an existing CloudWatch log group and scoped role/trust policy before passing their ARNs. The snippets do not provision those dependencies. Review paid data-event/Lake/Athena scope and retention costs first. Export existing policies/selectors/lifecycle settings and merge intended changes; put-* operations can replace existing configuration. Never apply expiration to audit evidence under a legal hold.

Create an Organization Trail

# Create the S3 bucket for log storage
aws s3api create-bucket \
  --bucket org-cloudtrail-audit-logs \
  --region us-east-1
# Apply bucket policy allowing CloudTrail to write
aws s3api put-bucket-policy \
  --bucket org-cloudtrail-audit-logs \
  --policy '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AWSCloudTrailAclCheck",
        "Effect": "Allow",
        "Principal": {"Service": "cloudtrail.amazonaws.com"},
        "Action": "s3:GetBucketAcl",
        "Resource": "arn:aws:s3:::org-cloudtrail-audit-logs",
        "Condition": {"StringEquals": {"aws:SourceArn": "arn:aws:cloudtrail:us-east-1:123456789012:trail/org-audit-trail"}}
      },
      {
        "Sid": "AWSCloudTrailWrite",
        "Effect": "Allow",
        "Principal": {"Service": "cloudtrail.amazonaws.com"},
        "Action": "s3:PutObject",
        "Resource": ["arn:aws:s3:::org-cloudtrail-audit-logs/AWSLogs/123456789012/*", "arn:aws:s3:::org-cloudtrail-audit-logs/AWSLogs/o-exampleorgid/*"],
        "Condition": {
          "StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control", "aws:SourceArn": "arn:aws:cloudtrail:us-east-1:123456789012:trail/org-audit-trail"}
        }
      }
    ]
  }'
# Block public access on the audit bucket
aws s3api put-public-access-block \
  --bucket org-cloudtrail-audit-logs \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
# Enable version history (not WORM/tamper prevention)
aws s3api put-bucket-versioning \
  --bucket org-cloudtrail-audit-logs \
  --versioning-configuration Status=Enabled
# Enable server-side encryption
aws s3api put-bucket-encryption \
  --bucket org-cloudtrail-audit-logs \
  --server-side-encryption-configuration '{
    "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "aws:kms", "KMSMasterKeyID": "alias/cloudtrail-key"}}]
  }'
# Set lifecycle policy for log retention
aws s3api put-bucket-lifecycle-configuration \
  --bucket org-cloudtrail-audit-logs \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "TransitionToGlacier",
        "Status": "Enabled",
        "Filter": {"Prefix": "AWSLogs/"},
        "Transitions": [
          {"Days": 90, "StorageClass": "GLACIER"}
        ]
      },
      {
        "ID": "ExpireOldLogs",
        "Status": "Enabled",
        "Filter": {"Prefix": "AWSLogs/"},
        "Expiration": {"Days": 2555}
      }
    ]
  }'
# Create the organization trail
aws cloudtrail create-trail \
  --name org-audit-trail \
  --s3-bucket-name org-cloudtrail-audit-logs \
  --is-organization-trail \
  --is-multi-region-trail \
  --enable-log-file-validation \
  --kms-key-id arn:aws:kms:us-east-1:123456789012:alias/cloudtrail-key \
  --cloud-watch-logs-log-group-arn arn:aws:logs:us-east-1:123456789012:log-group:CloudTrail:* \
  --cloud-watch-logs-role-arn arn:aws:iam::123456789012:role/CloudTrail-CWLogs-Role
# Start logging
aws cloudtrail start-logging --name org-audit-trail

The organization prefix must use the real organization ID and the trail ARN must use the management account ID. Log-file validation detects changes; it does not prevent deletion. Versioning alone is not immutable retention. Validate Object Lock/legal-hold design separately where required.

Event Selectors for Management and Data Events

# Configure advanced event selectors for granular control
aws cloudtrail put-event-selectors \
  --trail-name org-audit-trail \
  --advanced-event-selectors '[
    {
      "Name": "AllManagementEvents",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Management"]}
      ]
    },
    {
      "Name": "S3DataEventsForSensitiveBuckets",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
        {"Field": "resources.ARN", "StartsWith": [
          "arn:aws:s3:::sensitive-data-bucket/",
          "arn:aws:s3:::pii-bucket/",
          "arn:aws:s3:::financial-data/"
        ]}
      ]
    },
    {
      "Name": "LambdaInvocations",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::Lambda::Function"]}
      ]
    },
    {
      "Name": "DynamoDBDataEvents",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::DynamoDB::Table"]}
      ]
    }
  ]'

CloudWatch Alerts for Sensitive Activity

# Create metric filter for unauthorized API calls
aws logs put-metric-filter \
  --log-group-name CloudTrail \
  --filter-name UnauthorizedAPICalls \
  --filter-pattern '{ ($.errorCode = "*UnauthorizedAccess*") || ($.errorCode = "AccessDenied*") }' \
  --metric-transformations \
    metricName=UnauthorizedAPICalls,metricNamespace=CloudTrailMetrics,metricValue=1
# Create alarm for unauthorized calls
aws cloudwatch put-metric-alarm \
  --alarm-name UnauthorizedAPICallsAlarm \
  --metric-name UnauthorizedAPICalls \
  --namespace CloudTrailMetrics \
  --statistic Sum \
  --period 300 \
  --threshold 5 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:security-alerts
# Root account usage alarm
aws logs put-metric-filter \
  --log-group-name CloudTrail \
  --filter-name RootAccountUsage \
  --filter-pattern '{ ($.userIdentity.type = "Root") && ($.userIdentity.invokedBy NOT EXISTS) && ($.eventType != "AwsServiceEvent") }' \
  --metric-transformations \
    metricName=RootAccountUsage,metricNamespace=CloudTrailMetrics,metricValue=1
aws cloudwatch put-metric-alarm \
  --alarm-name RootAccountUsageAlarm \
  --metric-name RootAccountUsage \
  --namespace CloudTrailMetrics \
  --statistic Sum \
  --period 300 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:security-alerts
# Console login without MFA
aws logs put-metric-filter \
  --log-group-name CloudTrail \
  --filter-name ConsoleLoginWithoutMFA \
  --filter-pattern '{ ($.eventName = "ConsoleLogin") && ($.additionalEventData.MFAUsed != "Yes") && ($.userIdentity.type = "IAMUser") }' \
  --metric-transformations \
    metricName=ConsoleLoginWithoutMFA,metricNamespace=CloudTrailMetrics,metricValue=1
# IAM policy changes
aws logs put-metric-filter \
  --log-group-name CloudTrail \
  --filter-name IAMPolicyChanges \
  --filter-pattern '{ ($.eventName=CreatePolicy) || ($.eventName=DeletePolicy) || ($.eventName=AttachRolePolicy) || ($.eventName=DetachRolePolicy) || ($.eventName=AttachUserPolicy) || ($.eventName=PutUserPolicy) }' \
  --metric-transformations \
    metricName=IAMPolicyChanges,metricNamespace=CloudTrailMetrics,metricValue=1
# Security group changes
aws logs put-metric-filter \
  --log-group-name CloudTrail \
  --filter-name SecurityGroupChanges \
  --filter-pattern '{ ($.eventName=AuthorizeSecurityGroupIngress) || ($.eventName=RevokeSecurityGroupIngress) || ($.eventName=CreateSecurityGroup) || ($.eventName=DeleteSecurityGroup) }' \
  --metric-transformations \
    metricName=SecurityGroupChanges,metricNamespace=CloudTrailMetrics,metricValue=1

Athena Queries for CloudTrail Analysis

-- Create Athena table for CloudTrail logs
CREATE EXTERNAL TABLE IF NOT EXISTS cloudtrail_logs (
  eventVersion STRING,
  userIdentity STRUCT<
    type: STRING,
    principalId: STRING,
    arn: STRING,
    accountId: STRING,
    invokedBy: STRING,
    accessKeyId: STRING,
    userName: STRING,
    sessionContext: STRUCT<
      attributes: STRUCT<mfaAuthenticated: STRING, creationDate: STRING>,
      sessionIssuer: STRUCT<type: STRING, principalId: STRING, arn: STRING, accountId: STRING, userName: STRING>
    >
  >,
  eventTime STRING,
  eventSource STRING,
  eventName STRING,
  awsRegion STRING,
  sourceIPAddress STRING,
  userAgent STRING,
  errorCode STRING,
  errorMessage STRING,
  requestParameters STRING,
  responseElements STRING,
  additionalEventData STRING,
  requestId STRING,
  eventId STRING,
  readOnly STRING,
  resources ARRAY<STRUCT<arn: STRING, accountId: STRING, type: STRING>>,
  eventType STRING,
  recipientAccountId STRING
)
PARTITIONED BY (region STRING, year STRING, month STRING, day STRING)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://org-cloudtrail-audit-logs/AWSLogs/o-exampleorgid/123456789012/CloudTrail/';
-- Add a known delivered partition before querying (repeat for required account/dates).
ALTER TABLE cloudtrail_logs ADD IF NOT EXISTS
PARTITION (region='us-east-1', year='2026', month='10', day='22')
LOCATION 's3://org-cloudtrail-audit-logs/AWSLogs/o-exampleorgid/123456789012/CloudTrail/us-east-1/2026/10/22/';
-- Find all delete operations in the last 7 days
SELECT eventTime, userIdentity.arn, eventName, sourceIPAddress,
       requestParameters
FROM cloudtrail_logs
WHERE eventName LIKE '%Delete%'
  AND eventTime > date_format(date_add('day', -7, current_date), '%Y-%m-%dT%H:%i:%sZ')
ORDER BY eventTime DESC
LIMIT 100;
-- Identify console logins outside a synthetic exact-IP allowlist (not CIDR matching)
SELECT eventTime, userIdentity.userName, sourceIPAddress,
       additionalEventData
FROM cloudtrail_logs
WHERE eventName = 'ConsoleLogin'
  AND sourceIPAddress NOT IN ('198.51.100.10', '203.0.113.20')
  AND eventTime > date_format(date_add('day', -30, current_date), '%Y-%m-%dT%H:%i:%sZ')
ORDER BY eventTime DESC;
-- Access key usage patterns per principal
SELECT userIdentity.arn,
       count(*) AS api_call_count,
       count(DISTINCT eventName) AS unique_actions,
       count(DISTINCT sourceIPAddress) AS unique_ips,
       min(eventTime) AS first_seen,
       max(eventTime) AS last_seen
FROM cloudtrail_logs
WHERE eventTime > date_format(date_add('day', -30, current_date), '%Y-%m-%dT%H:%i:%sZ')
GROUP BY userIdentity.arn
ORDER BY api_call_count DESC
LIMIT 50;
-- Failed API calls indicating permission issues or reconnaissance
SELECT eventTime, userIdentity.arn, eventName, errorCode, errorMessage,
       sourceIPAddress
FROM cloudtrail_logs
WHERE errorCode IN ('AccessDenied', 'UnauthorizedAccess', 'Client.UnauthorizedAccess')
  AND eventTime > date_format(date_add('day', -7, current_date), '%Y-%m-%dT%H:%i:%sZ')
ORDER BY eventTime DESC
LIMIT 200;
-- Track KMS key usage
SELECT eventTime, userIdentity.arn, eventName, requestParameters,
       resources[1].arn AS key_arn
FROM cloudtrail_logs
WHERE eventSource = 'kms.amazonaws.com'
  AND eventName IN ('Decrypt', 'Encrypt', 'GenerateDataKey', 'DisableKey', 'ScheduleKeyDeletion')
  AND eventTime > date_format(date_add('day', -7, current_date), '%Y-%m-%dT%H:%i:%sZ')
ORDER BY eventTime DESC;

For production query volumes, register/project all intended partitions and constrain region/year/month/day to prune scan cost; eventTime alone does not prune these partitions. This DDL covers one organization/account prefix, not every member account. Values such as AWS service names or IPv6 addresses need an explicit policy when classifying unusual source IPs.

CloudTrail Lake (Existing Eligible Customers Only)

CloudTrail Lake closed to new customers on May 31, 2026. Verify existing eligibility and organization/account event-store coverage before using this optional example; new customers should choose a supported CloudWatch or S3/Athena design. Trails remain supported. See AWS availability change.

# Create an event data store for long-term queryable storage
aws cloudtrail create-event-data-store \
  --name org-audit-event-store \
  --multi-region-enabled \
  --organization-enabled \
  --retention-period 2555 \
  --advanced-event-selectors '[
    {
      "Name": "AllManagementEvents",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Management"]}
      ]
    }
  ]'

-- CloudTrail Lake SQL queries (run in console or via StartQuery API)
-- Investigate a specific user's activity
SELECT eventTime, eventName, eventSource, sourceIPAddress,
       errorCode, requestParameters
FROM EVENT_DATA_STORE_ID
WHERE userIdentity.arn = 'arn:aws:iam::123456789012:user/suspicious-user'
  AND eventTime > '2024-01-01 00:00:00'
ORDER BY eventTime DESC;
-- Cross-account activity summary
SELECT recipientAccountId, userIdentity.arn,
       count(*) AS event_count
FROM EVENT_DATA_STORE_ID
WHERE eventTime > '2024-01-01 00:00:00'
GROUP BY recipientAccountId, userIdentity.arn
ORDER BY event_count DESC;

Validate Trail Integrity

# Validate log file integrity for a date range
aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:us-east-1:123456789012:trail/org-audit-trail \
  --start-time "2024-01-01T00:00:00Z" \
  --end-time "2024-01-31T23:59:59Z"

# Check trail status
aws cloudtrail get-trail-status --name org-audit-trail

# Describe the trail configuration
aws cloudtrail describe-trails --trail-name-list org-audit-trail

Terraform Configuration

resource "aws_cloudtrail" "org_trail" {
  name                          = "org-audit-trail"
  s3_bucket_name                = aws_s3_bucket.cloudtrail.id
  is_organization_trail         = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true
  kms_key_id                    = aws_kms_key.cloudtrail.arn
  cloud_watch_logs_group_arn    = "${aws_cloudwatch_log_group.cloudtrail.arn}:*"
  cloud_watch_logs_role_arn     = aws_iam_role.cloudtrail_cw.arn
  include_global_service_events = true

  advanced_event_selector {
    name = "AllManagementEvents"
    field_selector {
      field  = "eventCategory"
      equals = ["Management"]
    }
  }

  advanced_event_selector {
    name = "SensitiveS3DataEvents"
    field_selector {
      field  = "eventCategory"
      equals = ["Data"]
    }
    field_selector {
      field  = "resources.type"
      equals = ["AWS::S3::Object"]
    }
    field_selector {
      field       = "resources.ARN"
      starts_with = ["arn:aws:s3:::sensitive-data-bucket/"]
    }
  }

  tags = {
    Environment = "production"
    Compliance  = "soc2,hipaa"
  }
}

The Terraform fragment references externally defined bucket/key/log-role resources and policies; it is not standalone. Plan and inspect changes before applying. For an organization-wide rollout, verify representative member-account/region delivery and CloudWatch access as well as the management account. A metric filter without an alarm does not notify anyone; the example only creates two alarms and the remaining checklist alarms must be implemented and tested separately.

Setup Checklist

cloudtrail_checklist:
  trail_configuration:
    - "Organization trail enabled across all accounts"
    - "Multi-region trail enabled"
    - "Log file validation enabled"
    - "KMS encryption configured with dedicated key"
    - "CloudWatch Logs integration active"
    - "S3 bucket policy restricts access to CloudTrail service only"

  s3_bucket_hardening:
    - "Public access blocked"
    - "Versioning enabled"
    - "Server-side encryption enabled"
    - "Lifecycle policy set for retention and archival"
    - "Access logging enabled on the bucket itself"
    - "Object Lock enabled for WORM compliance (if required)"

  monitoring_and_alerting:
    - "Metric filters for unauthorized API calls"
    - "Alarm on root account usage"
    - "Alarm on console login without MFA"
    - "Alarm on IAM policy changes"
    - "Alarm on security group and NACL changes"
    - "Alarm on CloudTrail configuration changes"
    - "Alarm on S3 bucket policy changes"

  analysis:
    - "Athena table created for ad-hoc queries"
    - "CloudTrail Lake event data store for long-term queries"
    - "Regular review of high-risk API patterns"
    - "Automated reports for compliance evidence"

  operational:
    - "Trail status health check automated"
    - "Log delivery latency monitored"
    - "Log file validation run periodically"
    - "SNS notification for trail configuration changes"

Verification

After the authorized change, inspect get-trail-status for delivery errors and delivered object/digest timestamps. Use a harmless authorized management event in a test account, find its event ID in S3 and the selected query system, and test alarm routing without disclosing sensitive log bodies. Retain the before/after configuration and unresolved regions/accounts. Compliance tags and a green trail status do not establish complete event coverage or certification.

Best Practices

• Enable organization-wide trails from the management account for full coverage
• Always enable log file validation to detect tampering
• Encrypt logs with a customer-managed KMS key and restrict key usage
• Use advanced event selectors to capture data events on sensitive resources without logging everything
• Integrate with CloudWatch Logs for real-time metric filters and alarms
• Set up Athena or CloudTrail Lake for efficient querying during investigations
• Apply S3 lifecycle policies to transition old logs to Glacier and enforce retention
• Monitor the trail itself (delivery errors, configuration changes) as a meta-control
• Validate log integrity periodically as part of compliance evidence collection
• Restrict access to the CloudTrail S3 bucket and KMS key with least-privilege IAM policies

Limitations

• Guidance and checklists only; not legal advice and not a substitute for a qualified auditor.
• Docs-only import: upstream templates and scripts not bundled.

Example

Map this skill's control checklist to our current evidence and list gaps.

Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.