Skip to content

Logs Commands

Manage CloudWatch log groups and process log files once they are on disk.

Two distinct jobs live here. list-groups, download, set-retention, and delete-group talk to CloudWatch. combine and aggregate operate on local files - aggregate in particular takes a directory of delivered logs (CloudTrail, ALB, CloudFront) and compacts them into fewer, larger files.

Commands

logs aggregate

Aggregate AWS log files into larger files for efficient processing.

aws-cloud-utilities logs aggregate INPUT_DIRECTORY [OPTIONS]
Option Value Description
--output-dir TEXT Output directory for aggregated files (default: ./aggregated_logs)
--target-size INTEGER Target size for aggregated files in MB (default: 250)
--log-type cloudtrail | cloudfront | elb | alb | route53 | all Log type to process (auto-detect if not specified)
--prefix TEXT Prefix for output files (default: aggregated)
--keep-structure flag Keep original directory structure in output
--no-compression flag Disable compression of output files
--delete-source flag Delete source files after successful processing

logs combine

Combine multiple log files into a single sorted file.

aws-cloud-utilities logs combine LOG_FOLDER [OPTIONS]
Option Value Description
--output-file TEXT Output file for combined logs (default: combined_logs_.log)
--sort-lines flag Sort log lines chronologically (default: enabled)

logs delete-group

Delete a CloudWatch log group.

aws-cloud-utilities logs delete-group LOG_GROUP [OPTIONS]
Option Value Description
--region TEXT AWS region where the log group is located (default: current region)
--confirm flag Skip confirmation prompt

logs download

Download CloudWatch logs for a specific log group or all groups.

aws-cloud-utilities logs download LOG_GROUP [OPTIONS]
Option Value Description
--days INTEGER Number of days to look back for logs (default: 7)
--region TEXT AWS region where the log group is located (default: current region)
--output-dir TEXT Output directory for downloaded logs (default: ./logs_)
--all-groups flag Download logs from all log groups (use 'ALL' as log_group argument)

logs list-groups

List CloudWatch log groups with details.

aws-cloud-utilities logs list-groups [OPTIONS]
Option Value Description
--region TEXT AWS region to list log groups from (default: current region)
--all-regions flag List log groups from all available regions
--include-size flag Include storage size information for each log group
--output-file TEXT Output file for log groups list (supports .json, .yaml, .csv)

logs set-retention

Set retention policy for a CloudWatch log group.

aws-cloud-utilities logs set-retention LOG_GROUP [RETENTION] [OPTIONS]
Option Value Description
--region TEXT AWS region where the log group is located (default: current region)
--if-never flag Only set retention if current retention is 'Never'
--dry-run flag Show what would be changed without making changes

Examples

# Log groups, with storage size
aws-cloud-utilities logs list-groups --include-size
aws-cloud-utilities logs list-groups --all-regions --output-file log-groups.csv

# Download the last 7 days for one group
aws-cloud-utilities logs download /aws/lambda/my-function

# Last 30 days, to a specific directory
aws-cloud-utilities logs download /aws/lambda/my-function --days 30 --output-dir ./lambda-logs

# Download every group (note the literal ALL argument)
aws-cloud-utilities logs download ALL --all-groups --days 3

# Retention: see what would change, then apply
aws-cloud-utilities logs set-retention /aws/lambda/my-function 30 --dry-run
aws-cloud-utilities logs set-retention /aws/lambda/my-function 30

# Only set retention where it is currently 'Never'
aws-cloud-utilities logs set-retention /aws/lambda/my-function 90 --if-never

# Delete a group
aws-cloud-utilities logs delete-group /aws/lambda/old-function --confirm

# Local file processing
aws-cloud-utilities logs combine ./lambda-logs --output-file combined.log
aws-cloud-utilities logs aggregate ./cloudtrail-logs --log-type cloudtrail --target-size 500

Retention is a positional argument

set-retention takes the log group and the retention in days as positional arguments, not flags:

aws-cloud-utilities logs set-retention /aws/lambda/my-function 30

Retention must be one of the values CloudWatch accepts (1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653). Log groups with no retention policy keep data forever and bill for it, which is what --if-never is for.

aggregate works on directories, not log groups

aggregate reads files already on disk. Download first, then aggregate:

aws-cloud-utilities logs download /aws/lambda/my-function --days 30 --output-dir ./raw
aws-cloud-utilities logs aggregate ./raw --target-size 250 --prefix lambda

--delete-source removes the input files after a successful run. It is destructive; confirm the output first.