Optimizing Cloud Costs with AWS S3 Glacier Lifecycle Policies

As your systems scale, logs and system backups accumulate rapidly. Storing gigabytes or terabytes of inactive logs in standard Amazon S3 buckets is a massive waste of budget. In this guide, we will design and deploy S3 Lifecycle Policies to transition assets automatically to S3 Glacier Deep Archive (costing only $0.00099 per GB/month).

The Retention Strategy

For system auditing and compliance, we want to establish the following automated pipeline:

1. 0 - 30 Days: Keep objects in S3 Standard (for instant access and daily analysis).
2. 31 - 90 Days: Transition logs to S3 Standard-Infrequent Access (IA) (saves 40% cost, millisecond retrieval).
3. 91+ Days: Move objects to S3 Glacier Deep Archive (saves 95% cost, retrieval takes 12 hours).
4. 365 Days: Permanently delete objects to avoid keeping endless junk.

Method 1: Using Terraform (Infrastructure as Code)

If you manage your cloud infrastructure using Terraform, you can declare these lifecycle rules natively in your S3 module.

Here is the declarative HCL configuration:

resource "aws_s3_bucket" "system_logs" {
  bucket = "company-system-logs-archive"
}

resource "aws_s3_bucket_lifecycle_configuration" "log_retention" {
  bucket = aws_s3_bucket.system_logs.id

  rule {
    id     = "archive-and-cleanup-logs"
    status = "Enabled"

    filter {
      prefix = "logs/"
    }

    # Step 1: Transition to Standard-IA after 30 days
    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    # Step 2: Transition to Glacier Deep Archive after 90 days
    transition {
      days          = 90
      storage_class = "DEEP_ARCHIVE"
    }

    # Step 3: Permanently expire objects after 365 days
    expiration {
      days = 365
    }
  }
}

Method 2: Deploying via AWS CLI

If you need to instantly apply a lifecycle configuration to an existing bucket via the command line, you can use the AWS CLI.

1. Create the Policy Schema File

Create a file named lifecycle.json on your local terminal:

{
  "Rules": [
    {
      "ID": "log-retention-policy",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "logs/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "DEEP_ARCHIVE"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    }
  ]
}

2. Apply the Lifecycle Configuration

Run the put-bucket-lifecycle-configuration command:

aws s3api put-bucket-lifecycle-configuration \
  --bucket company-system-logs-archive \
  --lifecycle-configuration file://lifecycle.json

Verification

To verify the policy was applied successfully to your bucket, execute the retrieval command:

aws s3api get-bucket-lifecycle-configuration \
  --bucket company-system-logs-archive