Video coming soon
Please continue with the lesson material below.
AWS CLI 명령어 완전 정복
Log in to save your progressLesson Material
Mastering AWS CLI Commands
A hands-on guide to operating AWS services from the terminal. From installation to troubleshooting, this is a comprehensive collection of commands you can use right away in practice.
1. CLI Basic Setup
Installation
AWS CLI v2 can be installed on all major operating systems. v1 is no longer recommended, so make sure to install v2.
macOS (pkg installation)
# Download and install the official pkg file
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /
# Verify installation
aws --version
aws-cli/2.15.30 Python/3.11.8 Darwin/23.4.0 exe/x86_64 prompt/off
Linux (x86_64)
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version
aws-cli/2.15.30 Python/3.11.8 Linux/5.15.0-1056-aws exe/x86_64.ubuntu.22 prompt/off
Windows (MSI installation)
# Download and run the official MSI
msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi
# Or download directly with PowerShell
Invoke-WebRequest -Uri "https://awscli.amazonaws.com/AWSCLIV2.msi" -OutFile "AWSCLIV2.msi"
Start-Process msiexec.exe -ArgumentList '/i AWSCLIV2.msi /quiet' -Wait
aws --version
aws-cli/2.15.30 Python/3.11.8 Windows/10 exe/AMD64 prompt/off
Practical Tip: If
aws --versiondoesn't work after installation, try opening a new terminal. In most cases, the PATH environment variable hasn't been refreshed yet.
aws configure -- Default Credential Setup
This is the first thing you need to do when using AWS CLI for the first time. You register the IAM user's Access Key and Secret Key.
aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: ap-northeast-2
Default output format [None]: json
Running this command creates two files.
# Credentials file
cat ~/.aws/credentials
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# Configuration file
cat ~/.aws/config
[default]
region = ap-northeast-2
output = json
Warning: The
~/.aws/credentialsfile stores secret keys in plaintext. Never commit this file to Git or share it with others. Make sure to add.aws/to your.gitignore.
Named Profiles -- Managing Multiple Accounts
In practice, you often work with multiple AWS accounts like dev, staging, and production simultaneously. Named Profiles let you manage credentials for each account.
# Create a dev environment profile
aws configure --profile dev
AWS Access Key ID [None]: AKIAI44QH8DHBEXAMPLE
AWS Secret Access Key [None]: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Default region name [None]: ap-northeast-2
Default output format [None]: json
# Create a production environment profile
aws configure --profile prod
AWS Access Key ID [None]: AKIAI77RR9CCPEXAMPLE
AWS Secret Access Key [None]: Qz8mBkTv0fE3xN7aPdR/w4KjLm9EXAMPLEKEY12
Default region name [None]: ap-northeast-2
Default output format [None]: table
After configuration, ~/.aws/credentials and ~/.aws/config look like this.
cat ~/.aws/credentials
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
[dev]
aws_access_key_id = AKIAI44QH8DHBEXAMPLE
aws_secret_access_key = je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
[prod]
aws_access_key_id = AKIAI77RR9CCPEXAMPLE
aws_secret_access_key = Qz8mBkTv0fE3xN7aPdR/w4KjLm9EXAMPLEKEY12
cat ~/.aws/config
[default]
region = ap-northeast-2
output = json
[profile dev]
region = ap-northeast-2
output = json
[profile prod]
region = ap-northeast-2
output = table
There are two ways to use a profile.
# Method 1: Use the --profile option
aws s3 ls --profile dev
# Method 2: Set via environment variable
export AWS_PROFILE=dev
aws s3 ls # Runs using the dev profile
Practical Tip: In the
configfile, profile section names have theprofileprefix like[profile dev], but in thecredentialsfile, they are written without the prefix like[dev]. If you mix these up, you'll get authentication errors.
Four Output Formats
AWS CLI supports four output formats. Choose the one that fits your situation.
1) JSON (default)
Best for programming or when used with jq.
aws iam get-user --output json
{
"User": {
"Path": "/",
"UserName": "devops-admin",
"UserId": "AIDAIOSFODNN7EXAMPLE",
"Arn": "arn:aws:iam::123456789012:user/devops-admin",
"CreateDate": "2024-03-15T09:30:00+00:00",
"PasswordLastUsed": "2026-03-18T02:15:00+00:00"
}
}
2) Table
A human-readable table format. Good for reports or quick verification.
aws iam list-users --output table
---------------------------------------------------------------------------------------------------------
| ListUsers |
+-------------------------------------------------------------------------------------------------------+
|| Users ||
|+------------+---------------------------+--------------------+----------+----------------------------+|
|| CreateDate | Arn | UserId | UserName | PasswordLastUsed ||
|+------------+---------------------------+--------------------+----------+----------------------------+|
|| 2024-03-15 | arn:aws:iam::123456789012 | AIDAIOSFODNN7EXAM | admin | 2026-03-18T02:15:00+00:00 ||
|| | :user/admin | PLE | | ||
|| 2024-06-20 | arn:aws:iam::123456789012 | AIDAI44QH8DHBEXAM | dev-user | 2026-03-17T08:30:00+00:00 ||
|| | :user/dev-user | PLE | | ||
|| 2025-01-10 | arn:aws:iam::123456789012 | AIDAI77RR9CCPEXAM | ci-bot | ||
|| | :user/ci-bot | PLE | | ||
|+------------+---------------------------+--------------------+----------+----------------------------+|
3) Text
Tab-delimited text. Convenient when used with shell tools like awk or cut.
aws iam list-users --output text
USERS 2024-03-15T09:30:00+00:00 arn:aws:iam::123456789012:user/admin AIDAIOSFODNN7EXAMPLE / admin 2026-03-18T02:15:00+00:00
USERS 2024-06-20T14:22:00+00:00 arn:aws:iam::123456789012:user/dev-user AIDAI44QH8DHBEXAMPLE / dev-user 2026-03-17T08:30:00+00:00
USERS 2025-01-10T11:00:00+00:00 arn:aws:iam::123456789012:user/ci-bot AIDAI77RR9CCPEXAMPLE / ci-bot None
4) YAML
Displays structured data in a human-readable form.
aws iam get-user --output yaml
User:
Arn: arn:aws:iam::123456789012:user/devops-admin
CreateDate: '2024-03-15T09:30:00+00:00'
PasswordLastUsed: '2026-03-18T02:15:00+00:00'
Path: /
UserId: AIDAIOSFODNN7EXAMPLE
UserName: devops-admin
Practical Tip: The most efficient approach is to use
json+--queryin scripts andtablefor visual inspection.textis useful for simple shell pipelines.
--query Option (JMESPath) -- 6 Practical Examples
--query is a powerful feature that uses JMESPath expressions to extract specific data from JSON responses. You don't need to install jq separately.
Example 1: Extract usernames only
aws iam list-users --query 'Users[*].UserName'
[
"admin",
"dev-user",
"ci-bot"
]
Example 2: Output specific fields as a table
aws iam list-users \
--query 'Users[*].[UserName, UserId, CreateDate]' \
--output table
------------------------------------------------------------
| ListUsers |
+-----------+------------------------+---------------------+
| admin | AIDAIOSFODNN7EXAMPLE | 2024-03-15T09:30.. |
| dev-user | AIDAI44QH8DHBEXAMPLE | 2024-06-20T14:22.. |
| ci-bot | AIDAI77RR9CCPEXAMPLE | 2025-01-10T11:00.. |
+-----------+------------------------+---------------------+
Example 3: Conditional filtering -- Users created after a specific date
aws iam list-users \
--query 'Users[?CreateDate>=`2025-01-01`].UserName'
[
"ci-bot"
]
Example 4: Get only the first element
aws iam list-users --query 'Users[0].UserName' --output text
admin
Example 5: Extract specific information from EC2 instances
aws ec2 describe-instances \
--query 'Reservations[*].Instances[*].[InstanceId, State.Name, PublicIpAddress, Tags[?Key==`Name`].Value | [0]]' \
--output table
--------------------------------------------------------------
| DescribeInstances |
+---------------------+---------+----------------+-----------+
| i-0a1b2c3d4e5f6789 | running | 13.125.248.137 | web-prod |
| i-0b2c3d4e5f6a7890 | running | 3.35.127.92 | api-prod |
| i-0c3d4e5f6a7b8901 | stopped | None | dev-test |
+---------------------+---------+----------------+-----------+
Example 6: Pipe and sort -- Sort S3 buckets by creation date
aws s3api list-buckets \
--query 'sort_by(Buckets, &CreationDate)[*].[Name, CreationDate]' \
--output table
-------------------------------------------------------------
| ListBuckets |
+----------------------------------+------------------------+
| my-app-logs-bucket | 2024-01-15T08:30:00 |
| my-static-assets-prod | 2024-06-22T14:15:00 |
| terraform-state-123456789012 | 2025-02-10T09:45:00 |
| my-data-lake-raw | 2025-11-03T16:20:00 |
+----------------------------------+------------------------+
Practical Tip: When comparing strings in
--query, wrap them with backticks (`). For example:`running`. Use[]for array flattening and|for pipes. You can practice on the official JMESPath website (jmespath.org).
Global Options
These are options that can be used with any AWS CLI command.
# Specify region (overrides default setting)
aws ec2 describe-instances --region us-east-1
# Specify output format
aws s3 ls --output table
# Specify profile
aws iam list-users --profile prod
# Debug mode -- outputs detailed API call logs
aws sts get-caller-identity --debug
# Dry run -- checks the request without actually executing it (only supported by some commands)
aws ec2 run-instances \
--image-id ami-0c9c942bd7bf113a2 \
--instance-type t3.micro \
--dry-run
An error occurred (DryRunOperation) when calling the RunInstances operation:
Request would have succeeded, but DryRun flag is set.
If you see this error message, it means the request would have succeeded if actually executed. If you see UnauthorizedOperation instead, it means you don't have the required permissions.
# Disable CLI auto-paging -- outputs everything at once instead of using less
aws ec2 describe-instances --no-cli-pager
# Or set it globally via environment variable
export AWS_PAGER=""
Practical Tip: The
--debugoption is extremely useful for diagnosing API call failures. You can see HTTP request/response headers, the signing process, and retry logic. However, since the output is very long, viewing just the tail with2>&1 | tail -50is also a good approach.
Environment Variables
AWS CLI can also be configured through environment variables. This is especially useful in CI/CD pipelines and Docker container environments.
# Credential-related
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export AWS_SESSION_TOKEN="FwoGZXIvYXdzEBYaDH..." # Required for STS temporary credentials
# Region and output
export AWS_DEFAULT_REGION="ap-northeast-2"
export AWS_DEFAULT_OUTPUT="json"
# Profile
export AWS_PROFILE="dev"
# Disable pager
export AWS_PAGER=""
# Change config file paths (default: ~/.aws/)
export AWS_CONFIG_FILE="/custom/path/config"
export AWS_SHARED_CREDENTIALS_FILE="/custom/path/credentials"
# Retry settings
export AWS_MAX_ATTEMPTS=5
export AWS_RETRY_MODE="adaptive"
To check the currently applied environment variables, do the following.
env | grep AWS_
AWS_DEFAULT_REGION=ap-northeast-2
AWS_DEFAULT_OUTPUT=json
AWS_PROFILE=dev
AWS_PAGER=
Caution: If you set both
AWS_PROFILEandAWS_ACCESS_KEY_IDat the same time, conflicts can occur. Since environment variable credentials take priority over profiles, always verify the intended account withaws sts get-caller-identity.
Credential Precedence
AWS CLI can pull credentials from multiple sources. Here they are listed from highest to lowest priority.
| Priority | Source | Description |
|---|---|---|
| 1 (highest) | Command-line options | Directly specified via --region, --profile, etc. |
| 2 | Environment variables | AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc. |
| 3 | CLI credentials file | ~/.aws/credentials |
| 4 | CLI config file | ~/.aws/config |
| 5 | Container credentials | ECS task role (169.254.170.2 endpoint) |
| 6 (lowest) | Instance profile | IAM role attached to EC2 instance (IMDS) |
# Check which credentials are currently in use
aws sts get-caller-identity
{
"UserId": "AIDAIOSFODNN7EXAMPLE",
"Account": "123456789012",
"Arn": "arn:aws:iam::123456789012:user/devops-admin"
}
Practical Tip: When using CLI on an EC2 instance, don't input Access Keys directly. Instead, attach an IAM role to the instance (instance profile). There's no key management needed, and credentials are rotated automatically. This is the AWS-recommended best practice.
Caution: Credentials set in environment variables persist until you
unsetthem. If you've switched to a different profile but an unexpected account is being used, check the environment variables first withenv | grep AWS_.
2. IAM Commands
IAM (Identity and Access Management) is the security backbone of AWS. Let's look at how to manage users, groups, roles, and policies via the CLI.
User CRUD
Create a User
aws iam create-user --user-name developer-kim
{
"User": {
"Path": "/",
"UserName": "developer-kim",
"UserId": "AIDAJQABLZS4A3QDU576Q",
"Arn": "arn:aws:iam::123456789012:user/developer-kim",
"CreateDate": "2026-03-18T06:30:00+00:00"
}
}
Set Console Login Password
aws iam create-login-profile \
--user-name developer-kim \
--password 'TempP@ssw0rd!2026' \
--password-reset-required
{
"LoginProfile": {
"UserName": "developer-kim",
"CreateDate": "2026-03-18T06:31:00+00:00",
"PasswordResetRequired": true
}
}
List Users
aws iam list-users --output table
-----------------------------------------------------------------------------------------------
| ListUsers |
+---------------------------------------------------------------------------------------------+
|| Users ||
|+----------------------------+----------------------------+---------+----------------------+|
|| CreateDate | Arn | UserName| UserId ||
|+----------------------------+----------------------------+---------+----------------------+|
|| 2024-03-15T09:30:00+00:00 | arn:aws:iam::123456789012 | admin | AIDAIOSFODNN7EXAMPLE ||
|| | :user/admin | | ||
|| 2025-06-20T14:22:00+00:00 | arn:aws:iam::123456789012 | dev-user| AIDAI44QH8DHBEXAMPLE ||
|| | :user/dev-user | | ||
|| 2026-03-18T06:30:00+00:00 | arn:aws:iam::123456789012 | develo | AIDAJQABLZS4A3QDU576 ||
|| | :user/developer-kim | per-kim | Q ||
|+----------------------------+----------------------------+---------+----------------------+|
Get a Specific User's Information
aws iam get-user --user-name developer-kim
{
"User": {
"Path": "/",
"UserName": "developer-kim",
"UserId": "AIDAJQABLZS4A3QDU576Q",
"Arn": "arn:aws:iam::123456789012:user/developer-kim",
"CreateDate": "2026-03-18T06:30:00+00:00"
}
}
Add Tags to a User
aws iam tag-user \
--user-name developer-kim \
--tags Key=Department,Value=Engineering Key=Team,Value=Backend
Tag addition produces no output on success (only returns HTTP 200).
Delete a User
To delete a user, you must first remove all associated resources.
# 1. Detach attached policies
aws iam list-attached-user-policies --user-name developer-kim
aws iam detach-user-policy \
--user-name developer-kim \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
# 2. Delete inline policies
aws iam list-user-policies --user-name developer-kim
aws iam delete-user-policy \
--user-name developer-kim \
--policy-name my-inline-policy
# 3. Delete access keys
aws iam list-access-keys --user-name developer-kim
aws iam delete-access-key \
--user-name developer-kim \
--access-key-id AKIAI44QH8DHBEXAMPLE
# 4. Delete login profile
aws iam delete-login-profile --user-name developer-kim
# 5. Deactivate and delete MFA devices
aws iam list-mfa-devices --user-name developer-kim
aws iam deactivate-mfa-device \
--user-name developer-kim \
--serial-number arn:aws:iam::123456789012:mfa/developer-kim
aws iam delete-virtual-mfa-device \
--serial-number arn:aws:iam::123456789012:mfa/developer-kim
# 6. Remove from groups
aws iam list-groups-for-user --user-name developer-kim
aws iam remove-user-from-group \
--user-name developer-kim \
--group-name developers
# 7. Final deletion
aws iam delete-user --user-name developer-kim
Warning: If you don't remove all policies, access keys, MFA devices, and group memberships associated with the user, you'll get a
DeleteConflicterror. Follow the steps above in order.
Access Key Creation / Deletion / Rotation
Create an Access Key
aws iam create-access-key --user-name developer-kim
{
"AccessKey": {
"UserName": "developer-kim",
"AccessKeyId": "AKIAZ3MHWWCPW7EXAMPLE",
"Status": "Active",
"SecretAccessKey": "7ZbQN8kPf2Ys/X4mR6vW+cE5jH3nKLpExample01",
"CreateDate": "2026-03-18T06:35:00+00:00"
}
}
Warning: The
SecretAccessKeyis only viewable at this moment. Once you close the screen, you can never see it again, so make sure to save it in a secure location.
List Access Keys
aws iam list-access-keys --user-name developer-kim
{
"AccessKeyMetadata": [
{
"UserName": "developer-kim",
"AccessKeyId": "AKIAZ3MHWWCPW7EXAMPLE",
"Status": "Active",
"CreateDate": "2026-03-18T06:35:00+00:00"
}
]
}
Deactivate an Access Key
During rotation, instead of immediately deleting the old key, you first deactivate it to check for any impact.
aws iam update-access-key \
--user-name developer-kim \
--access-key-id AKIAZ3MHWWCPW7EXAMPLE \
--status Inactive
Delete an Access Key
aws iam delete-access-key \
--user-name developer-kim \
--access-key-id AKIAZ3MHWWCPW7EXAMPLE
Practical Key Rotation Procedure
Key rotation is a security best practice. It is recommended to perform it every 90 days.
# Step 1: Create a new key (max 2 per user)
aws iam create-access-key --user-name developer-kim
# -> Apply the new key to your application/CI
# Step 2: Deactivate the old key (don't delete immediately!)
aws iam update-access-key \
--user-name developer-kim \
--access-key-id AKIAZ3MHWWCPW7EXAMPLE \
--status Inactive
# Step 3: Wait 1-2 days and verify no issues
# Step 4: If no issues, delete the old key
aws iam delete-access-key \
--user-name developer-kim \
--access-key-id AKIAZ3MHWWCPW7EXAMPLE
Practical Tip: If you delete the old key immediately during rotation, services that haven't yet switched to the new key will experience outages. Always follow the order: deactivate -> wait -> delete. AWS allows up to 2 access keys per user, so both keys can coexist during rotation.
Attaching / Detaching / Creating Custom Policies
Search AWS Managed Policies
# Search AWS managed policies containing "S3"
aws iam list-policies \
--scope AWS \
--query 'Policies[?contains(PolicyName, `S3`)].[PolicyName, Arn]' \
--output table
------------------------------------------------------------------------
| ListPolicies |
+-------------------------------+--------------------------------------+
| AmazonS3FullAccess | arn:aws:iam::aws:policy/AmazonS3F.. |
| AmazonS3ReadOnlyAccess | arn:aws:iam::aws:policy/AmazonS3R.. |
| AmazonS3ObjectLambdaExecutio | arn:aws:iam::aws:policy/AmazonS3O.. |
| nRolePolicy | |
| AmazonS3OutpostsFullAccess | arn:aws:iam::aws:policy/AmazonS3O.. |
| AmazonS3OutpostsReadOnlyAcce | arn:aws:iam::aws:policy/AmazonS3O.. |
| ss | |
+-------------------------------+--------------------------------------+
Attach a Managed Policy to a User
aws iam attach-user-policy \
--user-name developer-kim \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
Check Policies Attached to a User
aws iam list-attached-user-policies --user-name developer-kim
{
"AttachedPolicies": [
{
"PolicyName": "AmazonS3ReadOnlyAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
]
}
Detach a Policy
aws iam detach-user-policy \
--user-name developer-kim \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
Create a Custom Policy
In practice, it's common to create and use custom policies following the principle of least privilege.
# First, write the policy document as a JSON file
cat > /tmp/s3-limited-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowListBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::my-app-data-bucket"
},
{
"Sid": "AllowObjectOperations",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::my-app-data-bucket/*"
}
]
}
EOF
# Create the policy
aws iam create-policy \
--policy-name S3AppDataBucketAccess \
--policy-document file:///tmp/s3-limited-policy.json \
--description "Read/write access to my-app-data-bucket"
{
"Policy": {
"PolicyName": "S3AppDataBucketAccess",
"PolicyId": "ANPAJQABLZS4A3QDU576Q",
"Arn": "arn:aws:iam::123456789012:policy/S3AppDataBucketAccess",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 0,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"Description": "Read/write access to my-app-data-bucket",
"CreateDate": "2026-03-18T07:00:00+00:00",
"UpdateDate": "2026-03-18T07:00:00+00:00"
}
}
# Attach the custom policy to the user
aws iam attach-user-policy \
--user-name developer-kim \
--policy-arn arn:aws:iam::123456789012:policy/S3AppDataBucketAccess
Practical Tip: When passing a policy document via the
file://protocol, on Linux/macOS usefile:///tmp/policy.json(three slashes), and on Windows usefile://C:\path\policy.json. The three slashes come from combiningfile://with the leading slash of the absolute path.
Role Creation / Viewing / Policy Attachment
IAM roles are used when AWS services or other accounts need to temporarily obtain permissions.
Create a Role for EC2 Instances
# Write the Trust Policy
cat > /tmp/ec2-trust-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
EOF
# Create the role
aws iam create-role \
--role-name EC2-S3-Access-Role \
--assume-role-policy-document file:///tmp/ec2-trust-policy.json \
--description "Role for EC2 instances to access S3"
{
"Role": {
"Path": "/",
"RoleName": "EC2-S3-Access-Role",
"RoleId": "AROAJQABLZS4A3QDU576Q",
"Arn": "arn:aws:iam::123456789012:role/EC2-S3-Access-Role",
"CreateDate": "2026-03-18T07:10:00+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Description": "Role for EC2 instances to access S3"
}
}
Attach a Policy to a Role
aws iam attach-role-policy \
--role-name EC2-S3-Access-Role \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
List Roles
aws iam list-roles \
--query 'Roles[?starts_with(RoleName, `EC2`)].[RoleName, Arn, CreateDate]' \
--output table
--------------------------------------------------------------------------------------------
| ListRoles |
+---------------------+--------------------------------------------------+-----------------+
| EC2-S3-Access-Role | arn:aws:iam::123456789012:role/EC2-S3-Access-Ro | 2026-03-18T07 |
| | le | :10:00+00:00 |
+---------------------+--------------------------------------------------+-----------------+
Get Role Details
aws iam get-role --role-name EC2-S3-Access-Role
{
"Role": {
"Path": "/",
"RoleName": "EC2-S3-Access-Role",
"RoleId": "AROAJQABLZS4A3QDU576Q",
"Arn": "arn:aws:iam::123456789012:role/EC2-S3-Access-Role",
"CreateDate": "2026-03-18T07:10:00+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Description": "Role for EC2 instances to access S3",
"MaxSessionDuration": 3600,
"RoleLastUsed": {}
}
}
Check Policies Attached to a Role
aws iam list-attached-role-policies --role-name EC2-S3-Access-Role
{
"AttachedPolicies": [
{
"PolicyName": "AmazonS3ReadOnlyAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
]
}
Create an Instance Profile and Link the Role
An instance profile is required to attach an IAM role to an EC2 instance.
# Create instance profile
aws iam create-instance-profile \
--instance-profile-name EC2-S3-Access-Profile
# Add the role to the instance profile
aws iam add-role-to-instance-profile \
--instance-profile-name EC2-S3-Access-Profile \
--role-name EC2-S3-Access-Role
{
"InstanceProfile": {
"Path": "/",
"InstanceProfileName": "EC2-S3-Access-Profile",
"InstanceProfileId": "AIPAJQABLZS4A3QDU576Q",
"Arn": "arn:aws:iam::123456789012:instance-profile/EC2-S3-Access-Profile",
"CreateDate": "2026-03-18T07:15:00+00:00",
"Roles": []
}
}
Practical Tip: When you create an EC2 role through the AWS Console, an instance profile is automatically created with the same name. However, via the CLI, you must create the role and instance profile separately and link them. If you don't know this difference, you'll struggle with the CLI.
Account Summary
You can get a bird's-eye view of the IAM resource status across the entire account.
aws iam get-account-summary
{
"SummaryMap": {
"Users": 8,
"UsersQuota": 5000,
"Groups": 3,
"GroupsQuota": 300,
"Roles": 25,
"RolesQuota": 1000,
"Policies": 12,
"PoliciesQuota": 1500,
"AccountMFAEnabled": 1,
"MFADevices": 5,
"MFADevicesInUse": 5,
"AccountAccessKeysPresent": 0,
"AccessKeysPerUserQuota": 2,
"ServerCertificates": 0,
"ServerCertificatesQuota": 20,
"AccountSigningCertificatesPresent": 0,
"AttachedPoliciesPerUserQuota": 10,
"AttachedPoliciesPerGroupQuota": 10,
"AttachedPoliciesPerRoleQuota": 10,
"PolicySizeQuota": 6144,
"PolicyVersionsInUseQuota": 5,
"GroupPolicySizeQuota": 5120
}
}
Here's what the key fields mean.
Users: 8-- Currently 8 IAM usersAccountMFAEnabled: 1-- Root account MFA is enabled (0 means danger!)AccountAccessKeysPresent: 0-- No access keys on the Root account (this is good for security)AccessKeysPerUserQuota: 2-- Maximum 2 access keys per user
Practical Tip: If
AccountMFAEnabledis 0, it means MFA is not configured on the Root account. This is the first item flagged in security audits, so make sure to enable it.
Account Password Policy -- View and Configure
# Check the current password policy
aws iam get-account-password-policy
{
"PasswordPolicy": {
"MinimumPasswordLength": 12,
"RequireSymbols": true,
"RequireNumbers": true,
"RequireUppercaseCharacters": true,
"RequireLowercaseCharacters": true,
"AllowUsersToChangePassword": true,
"ExpirePasswords": true,
"MaxPasswordAge": 90,
"PasswordReusePrevention": 12,
"HardExpiry": false
}
}
# Strengthen the password policy
aws iam update-account-password-policy \
--minimum-password-length 14 \
--require-symbols \
--require-numbers \
--require-uppercase-characters \
--require-lowercase-characters \
--allow-users-to-change-password \
--max-password-age 90 \
--password-reuse-prevention 24 \
--no-hard-expiry
IAM Troubleshooting
A collection of frequently used commands for debugging IAM-related issues.
Query Entities (Users/Groups/Roles) Attached to a Policy
Useful when you need to answer "Who is using this policy?"
aws iam list-entities-for-policy \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
{
"PolicyGroups": [
{
"GroupName": "developers",
"GroupId": "AGPAJQABLZS4A3QDU576Q"
}
],
"PolicyUsers": [
{
"UserName": "developer-kim",
"UserId": "AIDAJQABLZS4A3QDU576Q"
}
],
"PolicyRoles": [
{
"RoleName": "EC2-S3-Access-Role",
"RoleId": "AROAJQABLZS4A3QDU576Q"
}
]
}
Check When an Access Key Was Last Used
Keys that haven't been used for a long time are a security risk. Check them regularly.
aws iam get-access-key-last-used \
--access-key-id AKIAZ3MHWWCPW7EXAMPLE
{
"UserName": "developer-kim",
"AccessKeyLastUsed": {
"LastUsedDate": "2026-03-17T14:30:00+00:00",
"ServiceName": "s3",
"Region": "ap-northeast-2"
}
}
If the last used date is more than 90 days ago, or ServiceName is N/A, you should consider deactivating that key.
# Script to check all users' access key status at once
for user in $(aws iam list-users --query 'Users[*].UserName' --output text); do
echo "=== $user ==="
aws iam list-access-keys --user-name "$user" \
--query 'AccessKeyMetadata[*].[AccessKeyId, Status, CreateDate]' \
--output table
done
=== admin ===
-----------------------------------------------------
| ListAccessKeys |
+------------------------+--------+-----------------+
| AKIAIOSFODNN7EXAMPLE | Active | 2024-03-15T09.. |
+------------------------+--------+-----------------+
=== developer-kim ===
-----------------------------------------------------
| ListAccessKeys |
+------------------------+--------+-----------------+
| AKIAZ3MHWWCPW7EXAMPLE | Active | 2026-03-18T06.. |
+------------------------+--------+-----------------+
=== ci-bot ===
-----------------------------------------------------
| ListAccessKeys |
+------------------------+----------+---------------+
| AKIAI99XX2ZZQREXAMPLE | Inactive | 2025-01-10T.. |
| AKIAI88YY3WWPREXAMPLE | Active | 2025-12-01T.. |
+------------------------+----------+---------------+
Check Policy Contents
To see what permissions a specific policy grants, you need to query the policy version's document.
# Check the policy's default version
aws iam get-policy \
--policy-arn arn:aws:iam::123456789012:policy/S3AppDataBucketAccess \
--query 'Policy.DefaultVersionId' \
--output text
v1
# View the policy document contents
aws iam get-policy-version \
--policy-arn arn:aws:iam::123456789012:policy/S3AppDataBucketAccess \
--version-id v1
{
"PolicyVersion": {
"Document": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowListBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::my-app-data-bucket"
},
{
"Sid": "AllowObjectOperations",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::my-app-data-bucket/*"
}
]
},
"VersionId": "v1",
"IsDefaultVersion": true,
"CreateDate": "2026-03-18T07:00:00+00:00"
}
}
Practical Tip:
get-policyonly returns the policy's metadata. To see the actual permission contents, you must useget-policy-version. Many people get stuck at first wondering why the policy contents aren't showing up.
Credential Report
You can download the credential status of all IAM users in the account as a CSV. This is essential for security audits.
# Request report generation
aws iam generate-credential-report
{
"State": "STARTED",
"Description": "No report exists. Starting a new report generation."
}
# Download the report after a few seconds
aws iam get-credential-report --output text --query Content | base64 --decode
user,arn,user_creation_time,password_enabled,password_last_used,password_last_changed,password_next_rotation,mfa_active,access_key_1_active,access_key_1_last_rotated,access_key_1_last_used_date,access_key_1_last_used_region,access_key_1_last_used_service,access_key_2_active,access_key_2_last_rotated,access_key_2_last_used_date,access_key_2_last_used_region,access_key_2_last_used_service
<root_account>,arn:aws:iam::123456789012:root,2024-01-10T08:00:00+00:00,not_supported,2026-03-18T02:15:00+00:00,not_supported,not_supported,true,false,N/A,N/A,N/A,N/A,false,N/A,N/A,N/A,N/A
admin,arn:aws:iam::123456789012:user/admin,2024-03-15T09:30:00+00:00,true,2026-03-18T02:15:00+00:00,2025-12-01T10:00:00+00:00,2026-03-01T10:00:00+00:00,true,true,2025-12-01T10:00:00+00:00,2026-03-17T14:30:00+00:00,ap-northeast-2,s3,false,N/A,N/A,N/A,N/A
developer-kim,arn:aws:iam::123456789012:user/developer-kim,2026-03-18T06:30:00+00:00,true,N/A,2026-03-18T06:31:00+00:00,2026-06-16T06:31:00+00:00,false,true,2026-03-18T06:35:00+00:00,N/A,N/A,N/A,false,N/A,N/A,N/A,N/A
ci-bot,arn:aws:iam::123456789012:user/ci-bot,2025-01-10T11:00:00+00:00,false,N/A,N/A,N/A,false,false,2025-01-10T11:00:00+00:00,N/A,N/A,N/A,true,2025-12-01T09:00:00+00:00,2026-03-18T01:00:00+00:00,ap-northeast-2,sts
Here are the key items to check in this report.
| Check Item | Safe Value | Risky Value |
|---|---|---|
<root_account> mfa_active | true | false |
<root_account> access_key_1_active | false | true |
| Regular user mfa_active | true | false |
| access_key_last_rotated | Within 90 days | Over 90 days |
| access_key_last_used_date | Recent | Over 90 days or N/A (unused) |
# Save the report to a file for analysis
aws iam get-credential-report \
--output text --query Content | base64 --decode > /tmp/credential-report.csv
Practical Tip: If you run
get-credential-reportimmediately aftergenerate-credential-report, you may get an error saying it's not ready yet. It usually completes in a few seconds, but for accounts with many users, it can take several seconds to tens of seconds. Download it whenStateisCOMPLETE.
IAM Policy Simulator (CLI Version)
You can test in advance whether a specific user can perform a specific action.
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::123456789012:user/developer-kim \
--action-names s3:GetObject s3:PutObject s3:DeleteBucket \
--resource-arns "arn:aws:s3:::my-app-data-bucket/*" \
--query 'EvaluationResults[*].[EvalActionName, EvalDecision]' \
--output table
--------------------------------------------
| SimulatePrincipalPolicy |
+------------------+-----------------------+
| s3:GetObject | allowed |
| s3:PutObject | allowed |
| s3:DeleteBucket | implicitDeny |
+------------------+-----------------------+
implicitDeny means there is no explicit Deny, but there's also no Allow permission, so the request is denied. explicitDeny means it was blocked by an explicit Deny policy.
Practical Tip: Before changing policies in a production environment, test them first with
simulate-principal-policy. It's safe because it verifies permissions without actually making API calls.
3. STS Commands
STS (Security Token Service) is a service that issues temporary security credentials. It's used for role switching, cross-account access, federation, and more.
get-caller-identity -- Check Current Credentials
This is the most basic yet most frequently used STS command. It lets you check "who am I right now."
aws sts get-caller-identity
{
"UserId": "AIDAIOSFODNN7EXAMPLE",
"Account": "123456789012",
"Arn": "arn:aws:iam::123456789012:user/devops-admin"
}
You can also check per profile.
aws sts get-caller-identity --profile dev
{
"UserId": "AIDAI44QH8DHBEXAMPLE",
"Account": "111122223333",
"Arn": "arn:aws:iam::111122223333:user/dev-deployer"
}
aws sts get-caller-identity --profile prod
{
"UserId": "AIDAI77RR9CCPEXAMPLE",
"Account": "444455556666",
"Arn": "arn:aws:iam::444455556666:user/prod-admin"
}
Practical Tip: Whenever an AWS CLI command fails, start by running
aws sts get-caller-identityto verify you're using the intended account/user. More than half of IAM permission issues are caused by "accessing with the wrong account/user."
Caution:
get-caller-identityis the only AWS API that can be called even without any IAM permissions. Even if all other commands fail due to permission issues, this command will still work.
assume-role -- Role Switching
Used when you need to temporarily switch to a role in another account or a role with elevated permissions.
Basic assume-role
aws sts assume-role \
--role-arn arn:aws:iam::444455556666:role/ProductionDeployRole \
--role-session-name deploy-session-kim \
--duration-seconds 3600
{
"Credentials": {
"AccessKeyId": "ASIAZ3MHWWCPEXAMPLE1",
"SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/ExAmPlEkEy1234567",
"SessionToken": "FwoGZXIvYXdzEBYaDCj6X8K9n7pT4lBmHSLKAR7sFvmGJV2+oYdTRiBk2EYdJ4qDLmhPKEAkPMnzOQ7ZzJ4W6fQpS5CjwVIm3sNU0xJ7vFGnE1D+jGKJ3fPWIkBdR5l5qZkCIH5Kpxy3rO2LzR4N+q6n7nBXM9EXAMPLESESSIONTOKEN...",
"Expiration": "2026-03-18T08:30:00+00:00"
},
"AssumedRoleUser": {
"AssumedRoleId": "AROAZ3MHWWCPEXAMPLE:deploy-session-kim",
"Arn": "arn:aws:sts::444455556666:assumed-role/ProductionDeployRole/deploy-session-kim"
}
}
Apply Temporary Credentials via Environment Variables
If you set the credentials received from assume-role as environment variables, subsequent commands will run under that role.
# Store the assume-role result in a variable
CREDS=$(aws sts assume-role \
--role-arn arn:aws:iam::444455556666:role/ProductionDeployRole \
--role-session-name deploy-session-kim \
--duration-seconds 3600 \
--output json)
# Set environment variables
export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $CREDS | jq -r '.Credentials.SessionToken')
# Verify -- now operating as ProductionDeployRole
aws sts get-caller-identity
{
"UserId": "AROAZ3MHWWCPEXAMPLE:deploy-session-kim",
"Account": "444455556666",
"Arn": "arn:aws:sts::444455556666:assumed-role/ProductionDeployRole/deploy-session-kim"
}
# To return to the original credentials, remove the environment variables
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
# Verify
aws sts get-caller-identity
{
"UserId": "AIDAIOSFODNN7EXAMPLE",
"Account": "123456789012",
"Arn": "arn:aws:iam::123456789012:user/devops-admin"
}
Warning: If you forget
AWS_SESSION_TOKEN, you'll get anInvalidAccessKeyIderror. Temporary credentials from STS require all three values (Access Key, Secret Key, Session Token) to be set.
Practical Tip: Temporary credentials expire after the time specified in
--duration-seconds. The default is 3600 seconds (1 hour), and the maximum is 43200 seconds (12 hours). However, it may be limited by the role'sMaxSessionDurationsetting.
Cross-Account Access via Profiles
Running assume-role + setting environment variables every time is tedious. By configuring profiles in ~/.aws/config, role switching happens automatically.
cat ~/.aws/config
[default]
region = ap-northeast-2
output = json
[profile dev]
region = ap-northeast-2
output = json
[profile prod-deploy]
role_arn = arn:aws:iam::444455556666:role/ProductionDeployRole
source_profile = default
region = ap-northeast-2
output = json
duration_seconds = 3600
[profile staging-readonly]
role_arn = arn:aws:iam::777788889999:role/StagingReadOnlyRole
source_profile = default
region = ap-northeast-2
output = json
mfa_serial = arn:aws:iam::123456789012:mfa/devops-admin
# Now cross-account access is possible with just the --profile option
aws s3 ls --profile prod-deploy
2025-06-15 14:22:00 prod-app-assets
2025-09-01 10:30:00 prod-backup-data
2026-01-20 16:45:00 prod-terraform-state
# Profiles that require MFA will prompt for the code at execution time
aws ec2 describe-instances --profile staging-readonly
Enter MFA code for arn:aws:iam::123456789012:mfa/devops-admin: 123456
{
"Reservations": [
{
"Instances": [
{
"InstanceId": "i-0staging1234abcd",
"InstanceType": "t3.medium",
"State": {
"Name": "running"
}
}
]
}
]
}
Practical Tip:
source_profilespecifies the source credentials to use when assuming the role. Addingmfa_serialforces MFA authentication during role switching. Requiring MFA for production access is a security best practice.
decode-authorization-message -- Decoding Permission Error Messages
When a permission error occurs in AWS, an encoded error message is returned. Decoding it reveals exactly which permission is missing.
Error Scenario Example
aws ec2 run-instances \
--image-id ami-0c9c942bd7bf113a2 \
--instance-type t3.micro
An error occurred (UnauthorizedOperation) when calling the RunInstances operation:
You are not authorized to perform this operation.
Encoded authorization failure message:
VTJ3cGVYSnBiMlJ6TFhSbGMzUXRNakF5TmkweE1pMHhOR1F6WXpjd05EWTRPVEEwTURrNU56QXhNREF3TURBd01E...
Decode the Message
aws sts decode-authorization-message \
--encoded-message "VTJ3cGVYSnBiMlJ6TFhSbGMzUXRNakF5TmkweE1pMHhOR1F6WXpjd05EWTRPVEEwTURrNU56QXhNREF3TURBd01E..." \
--query DecodedMessage --output text | jq .
{
"allowed": false,
"explicitDeny": false,
"matchedStatements": {
"items": []
},
"failures": {
"items": []
},
"context": {
"principal": {
"id": "AIDAJQABLZS4A3QDU576Q",
"name": "developer-kim",
"arn": "arn:aws:iam::123456789012:user/developer-kim"
},
"action": "ec2:RunInstances",
"resource": "arn:aws:ec2:ap-northeast-2:123456789012:instance/*",
"conditions": {
"items": [
{
"key": "ec2:InstanceType",
"values": {
"items": [
{
"value": "t3.micro"
}
]
}
}
]
}
}
}
Analyzing the decoded message tells us the following.
allowed: false-- The request was deniedexplicitDeny: false-- There was no explicit Deny (= denied because there's no Allow policy)action: ec2:RunInstances-- The denied actionresource-- The target resourceprincipal-- The entity that made the request
Based on this information, you need to add the ec2:RunInstances permission to the developer-kim user.
Warning: To run
decode-authorization-message, you need thests:DecodeAuthorizationMessagepermission. Without this permission, the decoding itself will fail. You'll need to ask an administrator, or the administrator will need to decode it directly.
STS Troubleshooting Checklist
A checklist to follow when permission-related issues occur with the AWS CLI.
Step 1: Verify Current Credentials
# Check who I am
aws sts get-caller-identity
{
"UserId": "AIDAIOSFODNN7EXAMPLE",
"Account": "123456789012",
"Arn": "arn:aws:iam::123456789012:user/devops-admin"
}
Verify that the account and user match your expectations. If they don't, check your environment variables or profile.
Step 2: Check Environment Variables
env | grep AWS_
AWS_PROFILE=dev
AWS_DEFAULT_REGION=ap-northeast-2
If you see unintended AWS_ACCESS_KEY_ID or AWS_SESSION_TOKEN values set, remove them with unset.
# Clean up unnecessary environment variables
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN
Step 3: Check Policies Attached to the User/Role
# For a user
aws iam list-attached-user-policies --user-name devops-admin
aws iam list-user-policies --user-name devops-admin # Inline policies
aws iam list-groups-for-user --user-name devops-admin # Check group memberships
{
"AttachedPolicies": [
{
"PolicyName": "AdministratorAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AdministratorAccess"
}
]
}
# Also check policies attached to the group
aws iam list-attached-group-policies --group-name developers
Step 4: Policy Simulation
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::123456789012:user/developer-kim \
--action-names ec2:RunInstances \
--query 'EvaluationResults[*].[EvalActionName, EvalDecision]' \
--output table
-----------------------------------------
| SimulatePrincipalPolicy |
+-------------------+-------------------+
| ec2:RunInstances | implicitDeny |
+-------------------+-------------------+
If you see implicitDeny, you need to add an Allow policy. If you see explicitDeny, you need to find and modify the Deny policy.
Step 5: Check STS Temporary Credential Expiration
If you're using temporary credentials from assume-role, check the expiration time.
# Expiration time of temporary credentials is in the assume-role response
# Indirectly check whether the current session has expired
aws sts get-caller-identity 2>&1
If you see an error like this, the credentials have expired.
An error occurred (ExpiredTokenException) when calling the GetCallerIdentity operation:
The security token included in the request is expired
# Fix: Remove environment variables and re-run assume-role
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
# Re-run assume-role...
Step 6: Check Detailed Logs with Debug Mode
If the above steps don't resolve the issue, use the --debug option to inspect request details.
aws ec2 describe-instances --debug 2>&1 | tail -30
Key information to look for in the debug output.
- Request signature: Which credentials were used for signing
- Request region: Whether the request went to the correct region
- HTTP response code: 403 (no permission), 400 (bad request), 404 (resource not found), etc.
- Error message: The specific reason for failure
Step 7: Check API Call History with CloudTrail
# Query recently failed API calls
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances \
--max-results 5 \
--query 'Events[*].[EventTime, EventName, Username, Resources[0].ResourceName]' \
--output table
------------------------------------------------------------------------
| LookupEvents |
+----------------------------+----------------+----------+-------------+
| 2026-03-18T07:45:00+00:00 | RunInstances | dev-kim | i-0a1b2c.. |
| 2026-03-18T07:30:00+00:00 | RunInstances | dev-kim | None |
+----------------------------+----------------+----------+-------------+
If Resources is None, that call failed.
Practical Tip: Memorize this troubleshooting sequence: 1. Who am I? -> 2. Are environment variables clean? -> 3. Do policies exist? -> 4. Simulate -> 5. Token expired? -> 6. Debug -> 7. CloudTrail. Following this order will resolve most IAM/STS issues.
Additional STS Commands
get-session-token -- Issue MFA-Authenticated Temporary Token
When using policies that require MFA, you must first obtain a session token using the MFA code.
aws sts get-session-token \
--serial-number arn:aws:iam::123456789012:mfa/devops-admin \
--token-code 123456 \
--duration-seconds 43200
{
"Credentials": {
"AccessKeyId": "ASIAZ3MHWWCPMFAEXAMP",
"SecretAccessKey": "mfaSecretKeyExample1234567890ABCDEFGHIJK",
"SessionToken": "FwoGZXIvYXdzEBYaDMfaSessionTokenExampleLongString...",
"Expiration": "2026-03-18T18:30:00+00:00"
}
}
# Apply to environment variables
export AWS_ACCESS_KEY_ID="ASIAZ3MHWWCPMFAEXAMP"
export AWS_SECRET_ACCESS_KEY="mfaSecretKeyExample1234567890ABCDEFGHIJK"
export AWS_SESSION_TOKEN="FwoGZXIvYXdzEBYaDMfaSessionTokenExampleLongString..."
# Now you can access MFA-protected resources
aws s3 ls s3://mfa-protected-bucket/
get-access-key-info -- Check the Account ID of an Access Key
You can find out which account an access key belongs to.
aws sts get-access-key-info --access-key-id AKIAIOSFODNN7EXAMPLE
{
"Account": "123456789012"
}
Practical Tip: When managing keys across multiple accounts, use this command to quickly identify which account a key belongs to. It's also useful for quickly determining the account of a leaked key during security incident response.
Summary
Here is a summary of the key commands covered in this part.
| Category | Key Commands | Purpose |
|---|---|---|
| CLI Configuration | aws configure | Set default credentials |
aws configure --profile | Multi-account profiles | |
--query (JMESPath) | Filter JSON responses | |
--output json/table/text/yaml | Specify output format | |
| IAM Users | create-user, delete-user | User CRUD |
create-access-key, delete-access-key | Access key management | |
attach-user-policy, detach-user-policy | Attach/detach policies | |
create-policy | Create custom policies | |
| IAM Roles | create-role | Create roles |
attach-role-policy | Attach policies to roles | |
create-instance-profile | Instance profiles for EC2 | |
| IAM Auditing | get-account-summary | Account IAM overview |
generate-credential-report | Credential reports | |
simulate-principal-policy | Permission simulation | |
| STS | get-caller-identity | Verify current credentials |
assume-role | Switch roles | |
decode-authorization-message | Decode permission errors | |
get-session-token | Issue MFA session tokens |
Practical Tip: Bookmark this document and refer back to it whenever you need to. It is more important to be able to quickly look up CLI commands when needed than to memorize them.
aws <service> helpandaws <service> <command> helpare also great offline references.
The next part covers CLI commands for core infrastructure services like EC2, VPC, S3, and ELB.
AWS CLI Command Reference - Part 2: EC2 & VPC
In Part 1 we covered IAM and S3 commands. This time, we will go over EC2 and VPC commands, which are the backbone of your infrastructure. We will focus on the commands most commonly used in practice, along with actual output examples and troubleshooting guidance.
4. EC2 Commands
EC2 is the most fundamental compute service in AWS. Managing instances via the CLI is much faster than using the console.
4.1 Querying Instances
List All Instances
aws ec2 describe-instances \
--region ap-northeast-2
This command outputs all instance information as JSON, but the volume of data is so large that it is rarely used as-is in practice. You need to combine --query and --filters to extract just the information you need.
Query Only Running Instances
aws ec2 describe-instances \
--region ap-northeast-2 \
--filters "Name=instance-state-name,Values=running" \
--query "Reservations[].Instances[].{
ID:InstanceId,
Type:InstanceType,
State:State.Name,
PrivateIP:PrivateIpAddress,
PublicIP:PublicIpAddress,
Name:Tags[?Key=='Name']|[0].Value
}" \
--output table
------------------------------------------------------------------------------------------
| DescribeInstances |
+----------------------+-----------+-----------+----------------+---------------+----------+
| ID | Name | PrivateIP | PublicIP | State | Type |
+----------------------+-----------+-----------+----------------+---------------+----------+
| i-0a1b2c3d4e5f60001 | web-01 | 10.0.1.10 | 3.35.120.45 | running | t3.micro |
| i-0a1b2c3d4e5f60002 | web-02 | 10.0.1.11 | 3.35.120.87 | running | t3.micro |
| i-0a1b2c3d4e5f60003 | api-01 | 10.0.2.20 | None | running | t3.small |
| i-0a1b2c3d4e5f60004 | bastion | 10.0.0.5 | 52.79.231.12 | running | t3.nano |
+----------------------+-----------+-----------+----------------+---------------+----------+
--query uses JMESPath syntax. Tags[?Key=='Name']|[0].Value means "from the tags, get the Value of the one where Key equals Name."
Filter Instances by Specific Tag
# Query only instances where Environment=production
aws ec2 describe-instances \
--region ap-northeast-2 \
--filters \
"Name=tag:Environment,Values=production" \
"Name=instance-state-name,Values=running" \
--query "Reservations[].Instances[].{
ID:InstanceId,
Name:Tags[?Key=='Name']|[0].Value,
Type:InstanceType,
AZ:Placement.AvailabilityZone,
LaunchTime:LaunchTime
}" \
--output table
--------------------------------------------------------------------------------------------
| DescribeInstances |
+----------------------+-----------+-----------+------------------+-------------------------+
| ID | Name | Type | AZ | LaunchTime |
+----------------------+-----------+-----------+------------------+-------------------------+
| i-0a1b2c3d4e5f60001 | web-01 | t3.micro | ap-northeast-2a | 2026-01-15T09:30:00+00:00|
| i-0a1b2c3d4e5f60002 | web-02 | t3.micro | ap-northeast-2c | 2026-01-15T09:31:00+00:00|
| i-0a1b2c3d4e5f60003 | api-01 | t3.small | ap-northeast-2a | 2026-02-01T14:20:00+00:00|
+----------------------+-----------+-----------+------------------+-------------------------+
Query Detailed Information by Instance ID
aws ec2 describe-instances \
--region ap-northeast-2 \
--instance-ids i-0a1b2c3d4e5f60001 \
--query "Reservations[].Instances[].{
ID:InstanceId,
Type:InstanceType,
State:State.Name,
VpcId:VpcId,
SubnetId:SubnetId,
SecurityGroups:SecurityGroups[].GroupId,
IAMRole:IamInstanceProfile.Arn,
PrivateIP:PrivateIpAddress,
PublicIP:PublicIpAddress,
KeyName:KeyName
}" \
--output json
[
{
"ID": "i-0a1b2c3d4e5f60001",
"Type": "t3.micro",
"State": "running",
"VpcId": "vpc-0abc1234def567890",
"SubnetId": "subnet-0aaa1111bbb22222",
"SecurityGroups": [
"sg-0123456789abcdef0"
],
"IAMRole": "arn:aws:iam::123456789012:instance-profile/web-server-role",
"PrivateIP": "10.0.1.10",
"PublicIP": "3.35.120.45",
"KeyName": "my-keypair"
}
]
💡 Practical Tip:
--output tableis easy for humans to read, while--output jsonis great for parsing withjqin scripts. Use--output textwhen processing withawk/cut.
Count Running Instances by Type
aws ec2 describe-instances \
--region ap-northeast-2 \
--filters "Name=instance-state-name,Values=running" \
--query "Reservations[].Instances[].InstanceType" \
--output text | tr '\t' '\n' | sort | uniq -c | sort -rn
5 t3.micro
3 t3.small
2 t3.medium
1 m5.large
This gives you an at-a-glance view of how many running instances you have of each type.
4.2 Starting/Stopping/Terminating Instances
Stop an Instance
aws ec2 stop-instances \
--region ap-northeast-2 \
--instance-ids i-0a1b2c3d4e5f60001
{
"StoppingInstances": [
{
"CurrentState": {
"Code": 64,
"Name": "stopping"
},
"InstanceId": "i-0a1b2c3d4e5f60001",
"PreviousState": {
"Code": 16,
"Name": "running"
}
}
]
}
When you stop an instance, you are not charged for compute time, but EBS volume charges continue. Also, when you restart a stopped instance, the Public IP will change (unless you are using an Elastic IP).
Start an Instance
aws ec2 start-instances \
--region ap-northeast-2 \
--instance-ids i-0a1b2c3d4e5f60001
{
"StartingInstances": [
{
"CurrentState": {
"Code": 0,
"Name": "pending"
},
"InstanceId": "i-0a1b2c3d4e5f60001",
"PreviousState": {
"Code": 80,
"Name": "stopped"
}
}
]
}
Stop Multiple Instances at Once
aws ec2 stop-instances \
--region ap-northeast-2 \
--instance-ids i-0a1b2c3d4e5f60001 i-0a1b2c3d4e5f60002 i-0a1b2c3d4e5f60003
You can stop multiple instances at once by separating their IDs with spaces. This is useful for bulk-stopping a development environment at the end of the workday.
Terminate an Instance
aws ec2 terminate-instances \
--region ap-northeast-2 \
--instance-ids i-0a1b2c3d4e5f60001
{
"TerminatingInstances": [
{
"CurrentState": {
"Code": 32,
"Name": "shutting-down"
},
"InstanceId": "i-0a1b2c3d4e5f60001",
"PreviousState": {
"Code": 16,
"Name": "running"
}
}
]
}
⚠️ Caution:
terminatepermanently deletes the instance. This cannot be undone! IfDeleteOnTermination=true(the default), the EBS volume is also deleted. Always enable termination protection on production instances.
Enable/Disable Termination Protection
# Enable termination protection
aws ec2 modify-instance-attribute \
--region ap-northeast-2 \
--instance-id i-0a1b2c3d4e5f60001 \
--disable-api-termination
# Disable termination protection (when you need to delete)
aws ec2 modify-instance-attribute \
--region ap-northeast-2 \
--instance-id i-0a1b2c3d4e5f60001 \
--no-disable-api-termination
Wait Until an Instance Changes State
# Wait until the instance reaches running state
aws ec2 wait instance-running \
--region ap-northeast-2 \
--instance-ids i-0a1b2c3d4e5f60001
echo "The instance is now running!"
The wait command polls until the state changes. This is essential in scripts when you need to perform follow-up actions after starting an instance.
# Available wait conditions
aws ec2 wait instance-running # Wait until running
aws ec2 wait instance-stopped # Wait until stopped
aws ec2 wait instance-terminated # Wait until terminated
aws ec2 wait instance-status-ok # Wait until status check is OK
💡 Practical Tip:
instance-runningandinstance-status-okare different.runningmeans the instance has started, whilestatus-okmeans the OS has fully booted. SSH access becomes possible only afterstatus-ok.
4.3 Creating Instances (run-instances)
Basic Instance Creation
aws ec2 run-instances \
--region ap-northeast-2 \
--image-id ami-0c9c942bd7bf113a2 \
--instance-type t3.micro \
--key-name my-keypair \
--security-group-ids sg-0123456789abcdef0 \
--subnet-id subnet-0aaa1111bbb22222 \
--tag-specifications \
'ResourceType=instance,Tags=[{Key=Name,Value=web-03},{Key=Environment,Value=development}]' \
--count 1
{
"Groups": [],
"Instances": [
{
"InstanceId": "i-0a1b2c3d4e5f60005",
"InstanceType": "t3.micro",
"KeyName": "my-keypair",
"LaunchTime": "2026-03-18T06:45:12+00:00",
"PrivateIpAddress": "10.0.1.25",
"State": {
"Code": 0,
"Name": "pending"
},
"SubnetId": "subnet-0aaa1111bbb22222",
"VpcId": "vpc-0abc1234def567890",
"SecurityGroups": [
{
"GroupId": "sg-0123456789abcdef0",
"GroupName": "web-sg"
}
],
"Tags": [
{
"Key": "Name",
"Value": "web-03"
},
{
"Key": "Environment",
"Value": "development"
}
]
}
],
"OwnerId": "123456789012",
"ReservationId": "r-0a1b2c3d4e5f67890"
}
Here is what each option does:
--image-id: The AMI ID. Specify the AMI for your desired OS such as Amazon Linux 2023 or Ubuntu--instance-type: The instance size. Free Tier coverst2.microort3.micro--key-name: The name of the key pair to use for SSH access--security-group-ids: The security group ID to apply--subnet-id: The subnet where the instance will be placed--tag-specifications: Tag assignments. The Name tag is displayed as the instance name in the console--count: Number of instances to create
Instance Creation with Advanced Options
aws ec2 run-instances \
--region ap-northeast-2 \
--image-id ami-0c9c942bd7bf113a2 \
--instance-type t3.small \
--key-name my-keypair \
--security-group-ids sg-0123456789abcdef0 \
--subnet-id subnet-0aaa1111bbb22222 \
--iam-instance-profile Name=web-server-role \
--user-data file://setup.sh \
--block-device-mappings '[{
"DeviceName": "/dev/xvda",
"Ebs": {
"VolumeSize": 30,
"VolumeType": "gp3",
"Iops": 3000,
"Throughput": 125,
"DeleteOnTermination": true,
"Encrypted": true
}
}]' \
--metadata-options '{
"HttpTokens": "required",
"HttpEndpoint": "enabled"
}' \
--tag-specifications \
'ResourceType=instance,Tags=[{Key=Name,Value=api-02},{Key=Environment,Value=production}]' \
'ResourceType=volume,Tags=[{Key=Name,Value=api-02-root},{Key=Environment,Value=production}]' \
--disable-api-termination \
--count 1
Explanation of the additional options:
--iam-instance-profile: Assigns an IAM Role to the instance. Required when accessing S3, DynamoDB, etc.--user-data: A script to run when the instance starts. Used for package installation, config file deployment, etc.--block-device-mappings: EBS volume configuration. gp3 offers better cost-performance than gp2--metadata-options: Enforces IMDSv2. SettingHttpTokens=requiredis recommended for security--disable-api-termination: Prevents accidental deletion
⚠️ Caution: When you set
HttpTokens=requiredin--metadata-options, it allows only IMDSv2. Some older SDKs and tools may only support IMDSv1, so check for compatibility. However, IMDSv2 is strongly recommended for security (to prevent SSRF attacks).
Automatically Look Up the Latest Amazon Linux 2023 AMI
aws ec2 describe-images \
--region ap-northeast-2 \
--owners amazon \
--filters \
"Name=name,Values=al2023-ami-2023*-x86_64" \
"Name=state,Values=available" \
--query "sort_by(Images, &CreationDate)[-1].{
ImageId:ImageId,
Name:Name,
Created:CreationDate
}" \
--output table
-------------------------------------------------------------------
| DescribeImages |
+-----------+-------------------------+---------------------------+
| Created | ImageId | Name |
+-----------+-------------------------+---------------------------+
| 2026-03-10| ami-0c9c942bd7bf113a2 | al2023-ami-2023.6-x86_64 |
+-----------+-------------------------+---------------------------+
💡 Practical Tip: AMI IDs differ by region. If you use an AMI ID from
ap-northeast-2inus-east-1, you will get an error. In scripts, it is best to automatically look up the latest AMI using the command above.
4.4 Security Groups
List Security Groups
aws ec2 describe-security-groups \
--region ap-northeast-2 \
--query "SecurityGroups[].{
ID:GroupId,
Name:GroupName,
VPC:VpcId,
Description:Description
}" \
--output table
------------------------------------------------------------------------------------
| DescribeSecurityGroups |
+-------------------------+------------------+-------------------------+------------+
| Description | ID | Name | VPC |
+-------------------------+------------------+-------------------------+------------+
| Default SG | sg-0def000000001 | default | vpc-0abc.. |
| Web server access | sg-0123456789abc | web-sg | vpc-0abc.. |
| API server access | sg-0123456789abd | api-sg | vpc-0abc.. |
| Database access | sg-0123456789abe | db-sg | vpc-0abc.. |
| Bastion host access | sg-0123456789abf | bastion-sg | vpc-0abc.. |
+-------------------------+------------------+-------------------------+------------+
View Detailed Rules for a Specific Security Group
aws ec2 describe-security-groups \
--region ap-northeast-2 \
--group-ids sg-0123456789abc \
--query "SecurityGroups[].{
GroupName:GroupName,
InboundRules:IpPermissions[].{
Protocol:IpProtocol,
FromPort:FromPort,
ToPort:ToPort,
Source:join(', ', IpRanges[].CidrIp),
SourceSG:join(', ', UserIdGroupPairs[].GroupId)
},
OutboundRules:IpPermissionsEgress[].{
Protocol:IpProtocol,
FromPort:FromPort,
ToPort:ToPort,
Destination:join(', ', IpRanges[].CidrIp)
}
}" \
--output json
[
{
"GroupName": "web-sg",
"InboundRules": [
{
"Protocol": "tcp",
"FromPort": 80,
"ToPort": 80,
"Source": "0.0.0.0/0",
"SourceSG": ""
},
{
"Protocol": "tcp",
"FromPort": 443,
"ToPort": 443,
"Source": "0.0.0.0/0",
"SourceSG": ""
},
{
"Protocol": "tcp",
"FromPort": 22,
"ToPort": 22,
"Source": "",
"SourceSG": "sg-0123456789abf"
}
],
"OutboundRules": [
{
"Protocol": "-1",
"FromPort": null,
"ToPort": null,
"Destination": "0.0.0.0/0"
}
]
}
]
This is a typical web server security group configuration that allows SSH (port 22) only from the bastion security group (sg-0123456789abf) while allowing HTTP/HTTPS from everywhere.
Add an Inbound Rule to a Security Group
# Allow SSH access (from a specific IP only)
aws ec2 authorize-security-group-ingress \
--region ap-northeast-2 \
--group-id sg-0123456789abc \
--protocol tcp \
--port 22 \
--cidr 203.0.113.50/32
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-0a1b2c3d4e5f60001",
"GroupId": "sg-0123456789abc",
"IpProtocol": "tcp",
"FromPort": 22,
"ToPort": 22,
"CidrIpv4": "203.0.113.50/32",
"IsEgress": false
}
]
}
# Allow access from another security group (e.g., API -> DB)
aws ec2 authorize-security-group-ingress \
--region ap-northeast-2 \
--group-id sg-0123456789abe \
--protocol tcp \
--port 3306 \
--source-group sg-0123456789abd
⚠️ Warning: If you open SSH (port 22) with
--cidr 0.0.0.0/0, it becomes accessible from the entire internet. Always restrict it to your company VPN IP or the bastion's security group. AWS Security Hub will flag this kind of configuration with a warning.
Remove a Rule from a Security Group
# Remove a specific inbound rule
aws ec2 revoke-security-group-ingress \
--region ap-northeast-2 \
--group-id sg-0123456789abc \
--protocol tcp \
--port 22 \
--cidr 203.0.113.50/32
{
"Return": true
}
# Remove by Security Group Rule ID (more precise method)
aws ec2 revoke-security-group-ingress \
--region ap-northeast-2 \
--group-id sg-0123456789abc \
--security-group-rule-ids sgr-0a1b2c3d4e5f60001
💡 Practical Tip: When adding security group rules, include a description. This helps prevent the "why was this rule opened?" situation later on.
aws ec2 authorize-security-group-ingress \
--region ap-northeast-2 \
--group-id sg-0123456789abc \
--ip-permissions '[{
"IpProtocol": "tcp",
"FromPort": 8080,
"ToPort": 8080,
"IpRanges": [{
"CidrIp": "10.0.0.0/16",
"Description": "Allow API server access from within VPC - 2026-03-18 John Doe"
}]
}]'
4.5 Key Pairs
List Key Pairs
aws ec2 describe-key-pairs \
--region ap-northeast-2 \
--query "KeyPairs[].{
Name:KeyName,
ID:KeyPairId,
Type:KeyType,
Fingerprint:KeyFingerprint
}" \
--output table
------------------------------------------------------------------------
| DescribeKeyPairs |
+---------------------------+--------------------+-------+--------------+
| Fingerprint | ID | Name | Type |
+---------------------------+--------------------+-------+--------------+
| ab:cd:ef:12:34:56:78:... | key-0a1b2c3d4e5f6 | my-keypair | rsa |
| 12:34:56:78:ab:cd:ef:... | key-0b2c3d4e5f6a7 | dev-key | ed25519|
+---------------------------+--------------------+-------+--------------+
Create a Key Pair
# Create an RSA key pair (default)
aws ec2 create-key-pair \
--region ap-northeast-2 \
--key-name prod-keypair \
--key-type rsa \
--query "KeyMaterial" \
--output text > prod-keypair.pem
# Set file permissions (Linux/Mac)
chmod 400 prod-keypair.pem
# Create an ED25519 key pair (more secure and faster)
aws ec2 create-key-pair \
--region ap-northeast-2 \
--key-name prod-keypair-ed25519 \
--key-type ed25519 \
--query "KeyMaterial" \
--output text > prod-keypair-ed25519.pem
chmod 400 prod-keypair-ed25519.pem
⚠️ Caution: The private key (.pem file) of a key pair can only be downloaded once at creation time. If you lose it, it cannot be reissued. Make sure to store it in a safe place. When sharing within a team, use a secrets management tool like AWS Secrets Manager or 1Password.
Delete a Key Pair
aws ec2 delete-key-pair \
--region ap-northeast-2 \
--key-name old-keypair
{
"Return": true,
"KeyPairId": "key-0a1b2c3d4e5f6"
}
Deleting a key pair does not affect instances that are already using it for access. This is because the public key is already registered in ~/.ssh/authorized_keys on the instance.
💡 Practical Tip: In practice, the trend is moving toward using AWS Systems Manager Session Manager to access instances without SSH instead of key pairs. It is much more secure because access is controlled through IAM policies, eliminating the burden of key management.
4.6 AMI (Amazon Machine Image)
Create an AMI
# Create an AMI from a running instance
aws ec2 create-image \
--region ap-northeast-2 \
--instance-id i-0a1b2c3d4e5f60001 \
--name "web-server-v1.2-$(date +%Y%m%d)" \
--description "Web server with nginx 1.24 and Node.js 20 configured" \
--no-reboot \
--tag-specifications \
'ResourceType=image,Tags=[{Key=Name,Value=web-server-v1.2},{Key=Environment,Value=production}]'
{
"ImageId": "ami-0fakeami12345abcd"
}
Specifying --no-reboot creates the AMI without rebooting the instance. This is useful when you need to take an AMI from a live server, but filesystem consistency may not be perfect.
⚠️ Caution: Creating an AMI without
--no-rebootwill briefly reboot the instance. On production servers, create AMIs during maintenance windows, or use--no-reboot.
List AMIs You Own
aws ec2 describe-images \
--region ap-northeast-2 \
--owners self \
--query "sort_by(Images, &CreationDate) | reverse(@)[].{
ImageId:ImageId,
Name:Name,
State:State,
Created:CreationDate,
Size:BlockDeviceMappings[0].Ebs.VolumeSize
}" \
--output table
-------------------------------------------------------------------------------------
| DescribeImages |
+-------------------------+---------------------------+----------+------------+------+
| Created | ImageId | Name | State | Size |
+-------------------------+---------------------------+----------+------------+------+
| 2026-03-18T07:15:00.000Z| ami-0fakeami12345abcd | web-v1.2 | available | 30 |
| 2026-03-01T14:30:00.000Z| ami-0fakeami12345abce | web-v1.1 | available | 30 |
| 2026-02-15T09:00:00.000Z| ami-0fakeami12345abcf | web-v1.0 | available | 20 |
| 2026-01-10T11:20:00.000Z| ami-0fakeami12345abd0 | api-v2.0 | available | 50 |
+-------------------------+---------------------------+----------+------------+------+
Clean Up Old AMIs
# Find AMIs older than 60 days
aws ec2 describe-images \
--region ap-northeast-2 \
--owners self \
--query "Images[?CreationDate<='2026-01-17'].{
ImageId:ImageId,
Name:Name,
Created:CreationDate
}" \
--output table
# Deregister (delete) an AMI
aws ec2 deregister-image \
--region ap-northeast-2 \
--image-id ami-0fakeami12345abd0
Even after deregistering an AMI, the associated EBS snapshots are not automatically deleted. You need to delete the snapshots separately to save costs.
💡 Practical Tip: When deleting an AMI, you should also delete the associated snapshots. Otherwise, snapshot charges continue to accrue. First check which snapshots are linked to the AMI as shown below.
# Check snapshots associated with an AMI
aws ec2 describe-images \
--region ap-northeast-2 \
--image-ids ami-0fakeami12345abd0 \
--query "Images[].BlockDeviceMappings[].Ebs.SnapshotId" \
--output text
snap-0a1b2c3d4e5f60001
4.7 EBS Volumes & Snapshots
List EBS Volumes
aws ec2 describe-volumes \
--region ap-northeast-2 \
--query "Volumes[].{
VolumeId:VolumeId,
Size:Size,
Type:VolumeType,
State:State,
AZ:AvailabilityZone,
Attached:Attachments[0].InstanceId,
Name:Tags[?Key=='Name']|[0].Value
}" \
--output table
--------------------------------------------------------------------------------------------------
| DescribeVolumes |
+------------------+----------------------+------+------+---------+------------------+------------+
| AZ | Attached | Name | Size | State | Type | VolumeId |
+------------------+----------------------+------+------+---------+------------------+------------+
| ap-northeast-2a | i-0a1b2c3d4e5f60001 | web-01-root| 30 | in-use | gp3 | vol-0a1... |
| ap-northeast-2a | i-0a1b2c3d4e5f60003 | api-01-root| 50 | in-use | gp3 | vol-0a2... |
| ap-northeast-2c | i-0a1b2c3d4e5f60002 | web-02-root| 30 | in-use | gp3 | vol-0a3... |
| ap-northeast-2a | None | old-data | 100| available| gp2 | vol-0a4... |
| ap-northeast-2c | None | test-vol | 20 | available| gp2 | vol-0a5... |
+------------------+----------------------+------+------+---------+------------------+------------+
Find Unattached Volumes - Key to Cost Savings!
aws ec2 describe-volumes \
--region ap-northeast-2 \
--filters "Name=status,Values=available" \
--query "Volumes[].{
VolumeId:VolumeId,
Size:Size,
Type:VolumeType,
AZ:AvailabilityZone,
Created:CreateTime,
Name:Tags[?Key=='Name']|[0].Value,
MonthlyCost:join('', ['$', to_string(Size * \`0.08\`)])
}" \
--output table
----------------------------------------------------------------------------------
| DescribeVolumes |
+------------------+------------+------+-----------+------------+---------+--------+
| AZ | Created | Name |MonthlyCost| Size | Type |VolumeId|
+------------------+------------+------+-----------+------------+---------+--------+
| ap-northeast-2a | 2025-11-20 |old-data| $8.00 | 100 | gp2 |vol-0a4 |
| ap-northeast-2c | 2026-01-05 |test-vol| $1.60 | 20 | gp2 |vol-0a5 |
+------------------+------------+------+-----------+------------+---------+--------+
Unattached volumes incur monthly charges even when not connected to any instance. You should check for them regularly and clean them up.
💡 Practical Tip: Unattached EBS volumes are the first target for cost optimization. Check with the command above once a month, and for unnecessary volumes, take a snapshot first and then delete. Snapshots are cheaper than volumes (gp3: $0.08/GB-month vs snapshot: $0.05/GB-month).
Create an EBS Snapshot
aws ec2 create-snapshot \
--region ap-northeast-2 \
--volume-id vol-0a1b2c3d4e5f60001 \
--description "web-01 root volume backup before v1.3 deploy - 2026-03-18" \
--tag-specifications \
'ResourceType=snapshot,Tags=[{Key=Name,Value=web-01-backup-20260318},{Key=Environment,Value=production}]'
{
"SnapshotId": "snap-0a1b2c3d4e5f60001",
"VolumeId": "vol-0a1b2c3d4e5f60001",
"State": "pending",
"StartTime": "2026-03-18T07:30:00+00:00",
"Progress": "",
"OwnerId": "123456789012",
"Description": "web-01 root volume backup before v1.3 deploy - 2026-03-18",
"VolumeSize": 30,
"Tags": [
{
"Key": "Name",
"Value": "web-01-backup-20260318"
}
]
}
List Snapshots
aws ec2 describe-snapshots \
--region ap-northeast-2 \
--owner-ids self \
--query "sort_by(Snapshots, &StartTime) | reverse(@)[:10].{
SnapshotId:SnapshotId,
VolumeId:VolumeId,
Size:VolumeSize,
State:State,
Started:StartTime,
Description:Description
}" \
--output table
-------------------------------------------------------------------------------------------------------
| DescribeSnapshots |
+-------------------+-----------+------+-----------+-------------------------+-------------------------+
| Description | Size |SnapshotId | Started | State | VolumeId |
+-------------------+-----------+------------------+-------------------------+-----------+-------------+
| web-01 backup... | 30 | snap-0a1b2c3d... | 2026-03-18T07:30:00Z | completed | vol-0a1... |
| api-01 weekly... | 50 | snap-0a2b3c4d... | 2026-03-15T03:00:00Z | completed | vol-0a2... |
| web-02 backup... | 30 | snap-0a3b4c5d... | 2026-03-10T07:15:00Z | completed | vol-0a3... |
+-------------------+-----------+------------------+-------------------------+-----------+-------------+
Delete a Snapshot
aws ec2 delete-snapshot \
--region ap-northeast-2 \
--snapshot-id snap-0a3b4c5d6e7f80001
If there is no response, it succeeded. If there is an error, an error message will be output in JSON.
4.8 EC2 Troubleshooting
Problem 1: Cannot SSH - Systematic 4-Step Verification
When SSH is not working, do not panic. Check the following 4 steps in order.
Step 1: Check Instance Status
aws ec2 describe-instance-status \
--region ap-northeast-2 \
--instance-ids i-0a1b2c3d4e5f60001 \
--query "InstanceStatuses[].{
InstanceId:InstanceId,
State:InstanceState.Name,
SystemStatus:SystemStatus.Status,
InstanceStatus:InstanceStatus.Status
}" \
--output table
--------------------------------------------------------------------
| DescribeInstanceStatus |
+----------------------+----------------+---------------+-----------+
| InstanceId | InstanceStatus | State |SystemStatus|
+----------------------+----------------+---------------+-----------+
| i-0a1b2c3d4e5f60001 | ok | running | ok |
+----------------------+----------------+---------------+-----------+
Verify that both SystemStatus and InstanceStatus are ok. If either shows impaired, it could be an AWS hardware issue.
SystemStatus: AWS physical host issues (network, power, etc.)InstanceStatus: Issues inside the instance (OS kernel panic, disk full, etc.)
Step 2: Verify SSH (Port 22) Is Open in the Security Group
aws ec2 describe-security-groups \
--region ap-northeast-2 \
--group-ids sg-0123456789abcdef0 \
--query "SecurityGroups[].IpPermissions[?FromPort==\`22\`].{
Protocol:IpProtocol,
Port:FromPort,
Source:IpRanges[].CidrIp,
SourceSG:UserIdGroupPairs[].GroupId
}" \
--output json
[
[
{
"Protocol": "tcp",
"Port": 22,
"Source": [
"203.0.113.50/32"
],
"SourceSG": []
}
]
]
Check whether your current IP is on the allowed list. Your IP may have changed if you are working from home, a cafe, etc.
# Check your current public IP
curl -s https://checkip.amazonaws.com
203.0.113.75
In the example above, the security group only allows 203.0.113.50/32, but your current IP is 203.0.113.75. That is the cause!
Step 3: Check the Subnet's NACL
# Check the NACL of the subnet the instance belongs to
aws ec2 describe-network-acls \
--region ap-northeast-2 \
--filters "Name=association.subnet-id,Values=subnet-0aaa1111bbb22222" \
--query "NetworkAcls[].{
NaclId:NetworkAclId,
InboundRules:Entries[?Egress==\`false\`].{
RuleNumber:RuleNumber,
Protocol:Protocol,
Action:RuleAction,
CidrBlock:CidrBlock,
PortRange:PortRange
},
OutboundRules:Entries[?Egress==\`true\`].{
RuleNumber:RuleNumber,
Protocol:Protocol,
Action:RuleAction,
CidrBlock:CidrBlock,
PortRange:PortRange
}
}" \
--output json
[
{
"NaclId": "acl-0a1b2c3d4e5f60001",
"InboundRules": [
{
"RuleNumber": 100,
"Protocol": "6",
"Action": "allow",
"CidrBlock": "0.0.0.0/0",
"PortRange": { "From": 22, "To": 22 }
},
{
"RuleNumber": 200,
"Protocol": "6",
"Action": "allow",
"CidrBlock": "0.0.0.0/0",
"PortRange": { "From": 80, "To": 80 }
},
{
"RuleNumber": 32767,
"Protocol": "-1",
"Action": "deny",
"CidrBlock": "0.0.0.0/0",
"PortRange": null
}
],
"OutboundRules": [
{
"RuleNumber": 100,
"Protocol": "-1",
"Action": "allow",
"CidrBlock": "0.0.0.0/0",
"PortRange": null
},
{
"RuleNumber": 32767,
"Protocol": "-1",
"Action": "deny",
"CidrBlock": "0.0.0.0/0",
"PortRange": null
}
]
}
]
Unlike security groups, NACLs are stateless. Even if you allow SSH on inbound, the response will not go back unless you also allow ephemeral ports (1024-65535) on outbound.
Step 4: Check the Route Table (When a Public IP Exists)
# Check the subnet's route table
aws ec2 describe-route-tables \
--region ap-northeast-2 \
--filters "Name=association.subnet-id,Values=subnet-0aaa1111bbb22222" \
--query "RouteTables[].Routes[].{
Destination:DestinationCidrBlock,
Target:GatewayId || NatGatewayId || InstanceId,
State:State
}" \
--output table
---------------------------------------------------------
| DescribeRouteTables |
+------------------+------------------+------------------+
| Destination | State | Target |
+------------------+------------------+------------------+
| 10.0.0.0/16 | active | local |
| 0.0.0.0/0 | active | igw-0a1b2c3d.. |
+------------------+------------------+------------------+
If 0.0.0.0/0 points to an Internet Gateway (igw-), it is a Public Subnet. A Public IP is required for SSH access from the outside.
💡 Practical Tip: SSH Troubleshooting Checklist:
- Instance state = running, status = ok
- Security group allows your IP -> port 22
- NACL allows port 22 inbound + ephemeral ports outbound
- Public Subnet + Public IP (or access via Bastion)
- Verify the key pair (.pem) file is correct
- Verify the SSH username is correct (Amazon Linux =
ec2-user, Ubuntu =ubuntu, RHEL =ec2-user)
Problem 2: Check Instance Console Output
If an instance hangs during boot or fails a status check, you need to check the console output.
aws ec2 get-console-output \
--region ap-northeast-2 \
--instance-id i-0a1b2c3d4e5f60001 \
--latest \
--query "Output" \
--output text
[ 0.000000] Linux version 6.1.75-99.163.amzn2023.x86_64 ...
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.1.75 root=UUID=...
...
[ 3.456789] systemd[1]: Started OpenSSH server daemon.
[ 3.567890] cloud-init[987]: Cloud-init v. 23.4 running 'modules:final'
[ 3.678901] cloud-init[987]: Cloud-init v. 23.4 finished at ...
Amazon Linux 2023
Kernel 6.1.75 on an x86_64
ip-10-0-1-10 login:
If you see the login: prompt, the OS has booted successfully. If you see a kernel panic or fsck error, there may be an issue with the EBS volume.
Problem 3: Handling InsufficientInstanceCapacity Errors
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation:
There is no Spot capacity available that matches your request.
This error means there is no available capacity for the requested instance type in the given AZ.
How to handle it:
# 1. Try a different AZ
aws ec2 run-instances \
--region ap-northeast-2 \
--subnet-id subnet-0bbb2222ccc33333 \
--instance-type t3.micro \
--image-id ami-0c9c942bd7bf113a2 \
--key-name my-keypair \
--count 1
# 2. Try a similar alternative instance type
# Instead of t3.micro, try t3a.micro (AMD processor, cheaper)
aws ec2 run-instances \
--region ap-northeast-2 \
--instance-type t3a.micro \
--image-id ami-0c9c942bd7bf113a2 \
--key-name my-keypair \
--subnet-id subnet-0aaa1111bbb22222 \
--count 1
💡 Practical Tip: In Auto Scaling Groups, you can specify multiple instance types using
MixedInstancesPolicy. Even if one type is out of capacity, it will automatically fall back to an alternative type.
Security Group vs NACL Comparison
This is one of the most commonly confused concepts in practice. Here is a comparison table.
| Aspect | Security Group | NACL (Network ACL) |
|---|---|---|
| Applied at | Instance (ENI) level | Subnet level |
| State | Stateful (responses automatically allowed) | Stateless (inbound/outbound configured separately) |
| Rule types | Allow only | Allow + Deny |
| Rule evaluation | All rules evaluated | Evaluated by rule number (first match applied) |
| Default behavior | Deny all inbound | Allow all traffic (default NACL) |
| Practical usage | Primarily used (90%+) | Used for subnet-level blocking |
Stateful vs Stateless is the key difference:
Security Group (Stateful):
-> If you allow inbound port 80, response traffic goes out automatically
-> No need to separately open outbound rules
NACL (Stateless):
-> Even if you allow inbound port 80, you must allow outbound ephemeral ports (1024-65535) for responses
-> Both directions must be explicitly configured
⚠️ Warning: If you accidentally block outbound ephemeral ports in the NACL, the instance will appear healthy but connections will not work. In such cases, if you only look at the security group, you will wonder "all the rules are correct, so why is it not working?" When SSH is not working, always check the NACL too.
5. VPC Commands
VPC (Virtual Private Cloud) is the service that lets you create your own virtual network in AWS. Nearly all services including EC2, RDS, and Lambda run on top of a VPC.
5.1 Query VPCs
aws ec2 describe-vpcs \
--region ap-northeast-2 \
--query "Vpcs[].{
VpcId:VpcId,
CidrBlock:CidrBlock,
State:State,
IsDefault:IsDefault,
Name:Tags[?Key=='Name']|[0].Value
}" \
--output table
----------------------------------------------------------------------
| DescribeVpcs |
+--------------+-----------+-----------+-------------------+----------+
| CidrBlock | IsDefault | Name | State | VpcId |
+--------------+-----------+-----------+-------------------+----------+
| 172.31.0.0/16| True | None | available | vpc-def..|
| 10.0.0.0/16 | False | prod-vpc | available | vpc-0abc.|
| 10.1.0.0/16 | False | dev-vpc | available | vpc-0def.|
+--------------+-----------+-----------+-------------------+----------+
IsDefault=True is the default VPC that AWS creates automatically. In production, you should not use it; instead, create a separate VPC.
5.2 Query Subnets
aws ec2 describe-subnets \
--region ap-northeast-2 \
--filters "Name=vpc-id,Values=vpc-0abc1234def567890" \
--query "Subnets[].{
SubnetId:SubnetId,
CidrBlock:CidrBlock,
AZ:AvailabilityZone,
AvailableIPs:AvailableIpAddressCount,
PublicIP:MapPublicIpOnLaunch,
Name:Tags[?Key=='Name']|[0].Value
}" \
--output table
------------------------------------------------------------------------------------------
| DescribeSubnets |
+--------------+------------------+-------------+-----------+-----------------------------+
| AvailableIPs | AZ | CidrBlock | PublicIP | Name |
+--------------+------------------+-------------+-----------+-----------------------------+
| 250 | ap-northeast-2a | 10.0.0.0/24 | True | prod-public-2a |
| 250 | ap-northeast-2c | 10.0.1.0/24 | True | prod-public-2c |
| 251 | ap-northeast-2a | 10.0.10.0/24| False | prod-private-app-2a |
| 251 | ap-northeast-2c | 10.0.11.0/24| False | prod-private-app-2c |
| 252 | ap-northeast-2a | 10.0.20.0/24| False | prod-private-db-2a |
| 252 | ap-northeast-2c | 10.0.21.0/24| False | prod-private-db-2c |
+--------------+------------------+-------------+-----------+-----------------------------+
Subnets with PublicIP=True are Public Subnets that automatically assign a Public IP when instances are created. Subnets with False are Private Subnets.
AvailableIPs is the number of IPs still available in that subnet. A /24 subnet has 256 total IPs, but AWS reserves 5, so you can use up to 251.
💡 Practical Tip: Subnet design pattern (2-AZ, 3-Tier):
- Public Subnet: ALB, Bastion, NAT Gateway (10.0.0.0/24, 10.0.1.0/24)
- Private App Subnet: EC2, ECS, Lambda (10.0.10.0/24, 10.0.11.0/24)
- Private DB Subnet: RDS, ElastiCache (10.0.20.0/24, 10.0.21.0/24)
5.3 Query Route Tables
aws ec2 describe-route-tables \
--region ap-northeast-2 \
--filters "Name=vpc-id,Values=vpc-0abc1234def567890" \
--query "RouteTables[].{
RouteTableId:RouteTableId,
Name:Tags[?Key=='Name']|[0].Value,
Routes:Routes[].{
Destination:DestinationCidrBlock,
Target:GatewayId || NatGatewayId || TransitGatewayId || VpcPeeringConnectionId,
State:State
},
Associations:Associations[].SubnetId
}" \
--output json
[
{
"RouteTableId": "rtb-0a1b2c3d4e5f60001",
"Name": "prod-public-rt",
"Routes": [
{
"Destination": "10.0.0.0/16",
"Target": "local",
"State": "active"
},
{
"Destination": "0.0.0.0/0",
"Target": "igw-0a1b2c3d4e5f60001",
"State": "active"
}
],
"Associations": [
"subnet-0aaa1111bbb22222",
"subnet-0aaa1111bbb33333"
]
},
{
"RouteTableId": "rtb-0a1b2c3d4e5f60002",
"Name": "prod-private-rt",
"Routes": [
{
"Destination": "10.0.0.0/16",
"Target": "local",
"State": "active"
},
{
"Destination": "0.0.0.0/0",
"Target": "nat-0a1b2c3d4e5f60001",
"State": "active"
}
],
"Associations": [
"subnet-0aaa1111bbb44444",
"subnet-0aaa1111bbb55555",
"subnet-0aaa1111bbb66666",
"subnet-0aaa1111bbb77777"
]
}
]
- Public RT:
0.0.0.0/0-> IGW (Internet Gateway) = direct internet connection - Private RT:
0.0.0.0/0-> NAT Gateway = outbound-only access through NAT
5.4 Query Internet Gateways
aws ec2 describe-internet-gateways \
--region ap-northeast-2 \
--filters "Name=attachment.vpc-id,Values=vpc-0abc1234def567890" \
--query "InternetGateways[].{
IGWId:InternetGatewayId,
VpcId:Attachments[0].VpcId,
State:Attachments[0].State
}" \
--output table
------------------------------------------------------
| DescribeInternetGateways |
+----------------------------+----------+-------------+
| IGWId | State | VpcId |
+----------------------------+----------+-------------+
| igw-0a1b2c3d4e5f60001 | attached | vpc-0abc.. |
+----------------------------+----------+-------------+
Only one Internet Gateway can be attached per VPC. If State is not attached, internet connectivity will not work.
5.5 Query NAT Gateways
aws ec2 describe-nat-gateways \
--region ap-northeast-2 \
--filter "Name=vpc-id,Values=vpc-0abc1234def567890" \
--query "NatGateways[].{
NatGatewayId:NatGatewayId,
State:State,
SubnetId:SubnetId,
PublicIP:NatGatewayAddresses[0].PublicIp,
PrivateIP:NatGatewayAddresses[0].PrivateIp,
Name:Tags[?Key=='Name']|[0].Value
}" \
--output table
----------------------------------------------------------------------------------------------
| DescribeNatGateways |
+----------------------------+----------------+---------+--------------+----------------------+
| NatGatewayId | Name | PrivateIP| PublicIP | State | SubnetId |
+----------------------------+----------------+---------+--------------+----------+-----------+
| nat-0a1b2c3d4e5f60001 | prod-nat-2a | 10.0.0.5| 3.35.88.100 | available| subnet-...|
| nat-0a1b2c3d4e5f60002 | prod-nat-2c |10.0.1.5 | 52.79.45.67 | available| subnet-...|
+----------------------------+----------------+---------+--------------+----------+-----------+
NAT Gateway is used when instances in Private Subnets need to access the internet (package updates, API calls, etc.). It must be placed in a Public Subnet.
⚠️ Caution: NAT Gateway costs approximately $0.059 per hour + data processing charges. In the Seoul region, that is roughly $43 per month. In development environments, consider using just one, or use a NAT Instance (EC2-based) to save costs.
5.6 VPC Troubleshooting
Problem 1: Private Instance Cannot Access the Internet
This is the case where yum update or apt update fails on an instance in a Private Subnet. You need to systematically check from routing onward.
Step 1: Check which subnet the instance is in
aws ec2 describe-instances \
--region ap-northeast-2 \
--instance-ids i-0a1b2c3d4e5f60003 \
--query "Reservations[].Instances[].{
SubnetId:SubnetId,
VpcId:VpcId,
PrivateIP:PrivateIpAddress,
PublicIP:PublicIpAddress
}" \
--output table
------------------------------------------------------------
| DescribeInstances |
+------------+---------------+------------------+-----------+
| PrivateIP | PublicIP | SubnetId | VpcId |
+------------+---------------+------------------+-----------+
| 10.0.10.20 | None | subnet-0aaa1111..| vpc-0abc..|
+------------+---------------+------------------+-----------+
If PublicIP is None, the instance is in a Private Subnet.
Step 2: Check whether the subnet's route table has a NAT Gateway route
aws ec2 describe-route-tables \
--region ap-northeast-2 \
--filters "Name=association.subnet-id,Values=subnet-0aaa1111bbb44444" \
--query "RouteTables[].Routes[?DestinationCidrBlock=='0.0.0.0/0'].{
Destination:DestinationCidrBlock,
Target:NatGatewayId,
State:State
}" \
--output table
-----------------------------------------------
| DescribeRouteTables |
+---------------+------------------+------------+
| Destination | State | Target |
+---------------+------------------+------------+
| 0.0.0.0/0 | active | nat-0a1b.. |
+---------------+------------------+------------+
If 0.0.0.0/0 points to a NAT Gateway, the routing is OK. If the state is blackhole or the route does not exist, that is the problem.
Step 3: Check the NAT Gateway status
aws ec2 describe-nat-gateways \
--region ap-northeast-2 \
--nat-gateway-ids nat-0a1b2c3d4e5f60001 \
--query "NatGateways[].{
NatGatewayId:NatGatewayId,
State:State,
SubnetId:SubnetId,
FailureCode:FailureCode,
FailureMessage:FailureMessage
}" \
--output json
[
{
"NatGatewayId": "nat-0a1b2c3d4e5f60001",
"State": "available",
"SubnetId": "subnet-0aaa1111bbb22222",
"FailureCode": null,
"FailureMessage": null
}
]
If the NAT Gateway state is available, it is healthy. If it is failed or deleted, you need to create a new one.
Possible State values:
pending: Being createdavailable: Ready for usedeleting: Being deleteddeleted: Already deletedfailed: Creation failed
💡 Practical Tip: Also check the routing of the subnet where the NAT Gateway itself is located. The NAT Gateway must be in a Public Subnet, and that subnet's route table must have an IGW connection. Accidentally creating a NAT Gateway in a Private Subnet is a surprisingly common mistake.
Problem 2: Diagnosing Network Issues with VPC Flow Logs
VPC Flow Logs record the traffic passing through network interfaces. They are essential for finding the cause of "connectivity is not working" issues.
Enable VPC Flow Logs
# Create a CloudWatch Logs log group
aws logs create-log-group \
--region ap-northeast-2 \
--log-group-name /vpc/flow-logs/prod-vpc
# Create VPC Flow Logs
aws ec2 create-flow-logs \
--region ap-northeast-2 \
--resource-type VPC \
--resource-ids vpc-0abc1234def567890 \
--traffic-type ALL \
--log-destination-type cloud-watch-logs \
--log-group-name /vpc/flow-logs/prod-vpc \
--deliver-logs-permission-arn arn:aws:iam::123456789012:role/VPCFlowLogsRole \
--max-aggregation-interval 60
{
"ClientToken": "abc123",
"FlowLogIds": [
"fl-0a1b2c3d4e5f60001"
],
"Unsuccessful": []
}
--max-aggregation-interval 60 collects logs at 1-minute intervals. The default is 10 minutes (600 seconds), but for troubleshooting, setting it to 1 minute is better.
Query VPC Flow Logs (CloudWatch Logs Insights)
aws logs start-query \
--region ap-northeast-2 \
--log-group-name /vpc/flow-logs/prod-vpc \
--start-time $(date -d '1 hour ago' +%s) \
--end-time $(date +%s) \
--query-string '
fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, protocol, action, bytes
| filter action = "REJECT"
| sort @timestamp desc
| limit 20
'
{
"queryId": "12345678-1234-1234-1234-123456789012"
}
# Retrieve query results (after a few seconds)
aws logs get-query-results \
--region ap-northeast-2 \
--query-id 12345678-1234-1234-1234-123456789012
{
"results": [
[
{"field": "@timestamp", "value": "2026-03-18 07:45:23.000"},
{"field": "srcAddr", "value": "203.0.113.100"},
{"field": "dstAddr", "value": "10.0.1.10"},
{"field": "srcPort", "value": "54321"},
{"field": "dstPort", "value": "22"},
{"field": "protocol", "value": "6"},
{"field": "action", "value": "REJECT"},
{"field": "bytes", "value": "0"}
],
[
{"field": "@timestamp", "value": "2026-03-18 07:44:58.000"},
{"field": "srcAddr", "value": "198.51.100.50"},
{"field": "dstAddr", "value": "10.0.1.10"},
{"field": "srcPort", "value": "12345"},
{"field": "dstPort", "value": "3389"},
{"field": "protocol", "value": "6"},
{"field": "action", "value": "REJECT"},
{"field": "bytes", "value": "0"}
]
],
"status": "Complete",
"statistics": {
"recordsMatched": 2,
"recordsScanned": 15420,
"bytesScanned": 2048576
}
}
Interpreting the results above:
- An SSH (port 22) connection attempt from
203.0.113.100was REJECTED -> blocked by the security group - An RDP (port 3389) connection attempt from
198.51.100.50was REJECTED -> a Windows remote access attempt with no matching rule
💡 Practical Tip: When you see
REJECTin Flow Logs, it means the traffic was blocked by a security group or NACL. To tell which one blocked it:
- Security group block: Since response traffic is auto-allowed, you only see inbound REJECT
- NACL block: You may see both inbound ACCEPT + outbound REJECT together
Problem 3: Detailed NACL Rule Verification
# Query all NACLs in the VPC
aws ec2 describe-network-acls \
--region ap-northeast-2 \
--filters "Name=vpc-id,Values=vpc-0abc1234def567890" \
--query "NetworkAcls[].{
NaclId:NetworkAclId,
IsDefault:IsDefault,
Name:Tags[?Key=='Name']|[0].Value,
Subnets:Associations[].SubnetId,
InboundRules:Entries[?Egress==\`false\`] | sort_by(@, &RuleNumber)[].{
Rule:RuleNumber,
Proto:Protocol,
Ports:PortRange && join('-', [to_string(PortRange.From), to_string(PortRange.To)]),
CIDR:CidrBlock,
Action:RuleAction
}
}" \
--output json
[
{
"NaclId": "acl-0a1b2c3d4e5f60001",
"IsDefault": false,
"Name": "prod-public-nacl",
"Subnets": [
"subnet-0aaa1111bbb22222",
"subnet-0aaa1111bbb33333"
],
"InboundRules": [
{
"Rule": 100,
"Proto": "6",
"Ports": "80-80",
"CIDR": "0.0.0.0/0",
"Action": "allow"
},
{
"Rule": 110,
"Proto": "6",
"Ports": "443-443",
"CIDR": "0.0.0.0/0",
"Action": "allow"
},
{
"Rule": 120,
"Proto": "6",
"Ports": "22-22",
"CIDR": "203.0.113.0/24",
"Action": "allow"
},
{
"Rule": 900,
"Proto": "6",
"Ports": "1024-65535",
"CIDR": "0.0.0.0/0",
"Action": "allow"
},
{
"Rule": 32767,
"Proto": "-1",
"Ports": null,
"CIDR": "0.0.0.0/0",
"Action": "deny"
}
]
}
]
NACL rule numbers are evaluated from the lowest number first, and the first matching rule is applied. Rule 32767 is the default deny rule.
Here is the interpretation of the NACL rules above:
- Rule 100: Allow HTTP (80) from all sources
- Rule 110: Allow HTTPS (443) from all sources
- Rule 120: Allow SSH (22) from
203.0.113.0/24only - Rule 900: Allow ephemeral ports (1024-65535) from all sources (for response traffic)
- Rule 32767: Deny everything else
⚠️ Caution: Leave gaps between NACL rule numbers (100, 110, 120...). This way you can insert new rules between existing ones later. If you number them 1, 2, 3, you will not be able to add rules in between.
Problem 4: VPC Peering Status and DNS Configuration
This covers cases where two VPCs are connected via peering but traffic is not flowing.
Check VPC Peering Status
aws ec2 describe-vpc-peering-connections \
--region ap-northeast-2 \
--filters "Name=status-code,Values=active,pending-acceptance" \
--query "VpcPeeringConnections[].{
PeeringId:VpcPeeringConnectionId,
Status:Status.Code,
RequesterVPC:RequesterVpcInfo.VpcId,
RequesterCIDR:RequesterVpcInfo.CidrBlock,
AccepterVPC:AccepterVpcInfo.VpcId,
AccepterCIDR:AccepterVpcInfo.CidrBlock,
Name:Tags[?Key=='Name']|[0].Value
}" \
--output table
---------------------------------------------------------------------------------------------------
| DescribeVpcPeeringConnections |
+-------------------+-----------------+----------------+--------------+--------------+-------------+
| AccepterCIDR | AccepterVPC | Name | PeeringId |RequesterCIDR |RequesterVPC |
+-------------------+-----------------+----------------+--------------+--------------+-------------+
| 10.1.0.0/16 | vpc-0def.. | prod-to-dev | pcx-0a1b.. | 10.0.0.0/16 | vpc-0abc.. |
+-------------------+-----------------+----------------+--------------+--------------+-------------+
If Status is not active, communication will not work. If it shows pending-acceptance, the peer VPC has not accepted the connection yet.
Check whether both VPC route tables have peering routes
# Check routing in the Requester VPC (prod-vpc)
aws ec2 describe-route-tables \
--region ap-northeast-2 \
--filters "Name=vpc-id,Values=vpc-0abc1234def567890" \
--query "RouteTables[].Routes[?VpcPeeringConnectionId!=null].{
Destination:DestinationCidrBlock,
PeeringId:VpcPeeringConnectionId,
State:State
}" \
--output table
----------------------------------------------------------
| DescribeRouteTables |
+------------------+------------------+-------------------+
| Destination | PeeringId | State |
+------------------+------------------+-------------------+
| 10.1.0.0/16 | pcx-0a1b.. | active |
+------------------+------------------+-------------------+
Both VPC route tables must have a peer CIDR -> Peering Connection route.
Check VPC Peering DNS Resolution Settings
aws ec2 describe-vpc-peering-connections \
--region ap-northeast-2 \
--vpc-peering-connection-ids pcx-0a1b2c3d4e5f60001 \
--query "VpcPeeringConnections[].{
PeeringId:VpcPeeringConnectionId,
RequesterDNS:RequesterVpcInfo.PeeringOptions.AllowDnsResolutionFromRemoteVpc,
AccepterDNS:AccepterVpcInfo.PeeringOptions.AllowDnsResolutionFromRemoteVpc
}" \
--output json
[
{
"PeeringId": "pcx-0a1b2c3d4e5f60001",
"RequesterDNS": true,
"AccepterDNS": false
}
]
In the result above, the Accepter VPC's DNS resolution is false. In this case, the Requester VPC cannot resolve the Accepter VPC's Private DNS names.
# Enable DNS resolution
aws ec2 modify-vpc-peering-connection-options \
--region ap-northeast-2 \
--vpc-peering-connection-id pcx-0a1b2c3d4e5f60001 \
--accepter-peering-connection-options '{
"AllowDnsResolutionFromRemoteVpc": true
}'
{
"AccepterPeeringConnectionOptions": {
"AllowDnsResolutionFromRemoteVpc": true
}
}
💡 Practical Tip: VPC Peering troubleshooting checklist:
- Verify that the peering status is
active- Verify that both VPC route tables have routes for the peer CIDR -> Peering connection
- Verify that both security groups allow the peer CIDR
- Verify that CIDRs do not overlap (peering cannot be established if they do)
- Verify that the DNS resolution option is
trueon both sides- Verify that the NACL is not blocking the peer CIDR
Problem 5: Full VPC Network Diagnostic Script
In practice, when a network issue occurs, run the following commands in order to get a complete picture of the situation.
#!/bin/bash
# vpc-diagnose.sh - VPC Network Diagnostic Script
REGION="ap-northeast-2"
VPC_ID="vpc-0abc1234def567890"
echo "============================================"
echo " VPC Network Diagnosis Starting"
echo " VPC: $VPC_ID"
echo " Region: $REGION"
echo " Time: $(date)"
echo "============================================"
echo ""
echo ">>> 1. VPC Information"
aws ec2 describe-vpcs \
--region $REGION \
--vpc-ids $VPC_ID \
--query "Vpcs[].{VpcId:VpcId, CidrBlock:CidrBlock, State:State}" \
--output table
echo ""
echo ">>> 2. Internet Gateway Connection Status"
aws ec2 describe-internet-gateways \
--region $REGION \
--filters "Name=attachment.vpc-id,Values=$VPC_ID" \
--query "InternetGateways[].{IGWId:InternetGatewayId, State:Attachments[0].State}" \
--output table
echo ""
echo ">>> 3. NAT Gateway Status"
aws ec2 describe-nat-gateways \
--region $REGION \
--filter "Name=vpc-id,Values=$VPC_ID" \
--query "NatGateways[?State!='deleted'].{
NatGWId:NatGatewayId,
State:State,
Subnet:SubnetId,
PublicIP:NatGatewayAddresses[0].PublicIp
}" \
--output table
echo ""
echo ">>> 4. Available IPs per Subnet"
aws ec2 describe-subnets \
--region $REGION \
--filters "Name=vpc-id,Values=$VPC_ID" \
--query "Subnets[].{
SubnetId:SubnetId,
AZ:AvailabilityZone,
CIDR:CidrBlock,
AvailableIPs:AvailableIpAddressCount,
Name:Tags[?Key=='Name']|[0].Value
}" \
--output table
echo ""
echo ">>> 5. Route Table Summary"
aws ec2 describe-route-tables \
--region $REGION \
--filters "Name=vpc-id,Values=$VPC_ID" \
--query "RouteTables[].{
RTId:RouteTableId,
Name:Tags[?Key=='Name']|[0].Value,
DefaultRoute:Routes[?DestinationCidrBlock=='0.0.0.0/0'].{
Target:GatewayId || NatGatewayId,
State:State
} | [0]
}" \
--output json
echo ""
echo ">>> 6. Security Group List"
aws ec2 describe-security-groups \
--region $REGION \
--filters "Name=vpc-id,Values=$VPC_ID" \
--query "SecurityGroups[].{
SGId:GroupId,
Name:GroupName,
InboundRuleCount:length(IpPermissions),
OutboundRuleCount:length(IpPermissionsEgress)
}" \
--output table
echo ""
echo ">>> 7. VPC Peering Connections"
aws ec2 describe-vpc-peering-connections \
--region $REGION \
--filters "Name=requester-vpc-info.vpc-id,Values=$VPC_ID" \
--query "VpcPeeringConnections[].{
PeeringId:VpcPeeringConnectionId,
Status:Status.Code,
PeerVPC:AccepterVpcInfo.VpcId,
PeerCIDR:AccepterVpcInfo.CidrBlock
}" \
--output table
echo ""
echo "============================================"
echo " Diagnosis Complete"
echo "============================================"
Running this script gives you a complete overview of the VPC's network status in one pass.
💡 Practical Tip: Save this script in your team's shared repository and run it immediately when a network issue occurs. If you paste the output into Slack or a Jira ticket, other team members can quickly understand the situation.
5.7 VPC Core Concepts Summary
Inbound traffic path from the internet:
User -> IGW -> Public Subnet (ALB) -> Private Subnet (EC2)
^
Security Group + NACL check
Outbound traffic path from a Private instance:
EC2 (Private) -> NAT Gateway (Public Subnet) -> IGW -> Internet
^ ^
Route Table Elastic IP required
(0.0.0.0/0 -> NAT)
💡 Practical Tip: Key things to remember when designing a VPC:
- Place only ALBs, Bastions, and NAT Gateways in Public Subnets
- Place EC2, RDS, and Lambda in Private Subnets
- Create one NAT Gateway per AZ for high availability
- Choose a large enough VPC CIDR (/16 recommended). It is hard to expand later
- If CIDRs overlap with other VPCs or on-premises networks, peering/VPN connections will not work, so plan IP ranges in advance
Appendix: Frequently Used EC2/VPC One-Liners
# 1. Check running instance count across all regions
for region in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do
count=$(aws ec2 describe-instances --region $region --filters "Name=instance-state-name,Values=running" --query "length(Reservations[].Instances[])" --output text)
if [ "$count" != "0" ] && [ "$count" != "None" ]; then
echo "$region: $count instances"
fi
done
# 2. Find instances stopped for more than 30 days
aws ec2 describe-instances \
--region ap-northeast-2 \
--filters "Name=instance-state-name,Values=stopped" \
--query "Reservations[].Instances[?StateTransitionReason!=null].{
ID:InstanceId,
Name:Tags[?Key=='Name']|[0].Value,
StoppedSince:StateTransitionReason
}" \
--output table
# 3. Find unused Elastic IPs (these incur charges!)
aws ec2 describe-addresses \
--region ap-northeast-2 \
--query "Addresses[?AssociationId==null].{
PublicIP:PublicIp,
AllocationId:AllocationId,
Name:Tags[?Key=='Name']|[0].Value
}" \
--output table
# 4. Find security groups with SSH (port 22) open to 0.0.0.0/0
aws ec2 describe-security-groups \
--region ap-northeast-2 \
--query "SecurityGroups[?IpPermissions[?FromPort==\`22\` && IpRanges[?CidrIp=='0.0.0.0/0']]].{
SGId:GroupId,
Name:GroupName,
VPC:VpcId
}" \
--output table
# 5. Find gp2 volumes (candidates for gp3 migration)
aws ec2 describe-volumes \
--region ap-northeast-2 \
--filters "Name=volume-type,Values=gp2" \
--query "Volumes[].{
VolumeId:VolumeId,
Size:Size,
State:State,
AZ:AvailabilityZone,
Attached:Attachments[0].InstanceId
}" \
--output table
💡 Practical Tip: Among the one-liners above, #3 (unused Elastic IPs) and #4 (SSH open to 0.0.0.0/0) are items that are frequently flagged in security audits. Run them once a month as a routine check. Unused EIPs cost $0.005 per hour.
The next Part 3 will cover CLI commands for the remaining AWS services such as S3, RDS, and CloudWatch.
AWS CLI Command Reference Part 3 - S3, RDS, DynamoDB
In Part 1, we covered IAM, STS, and EC2. In Part 2, we covered VPC, ELB, and Auto Scaling. In Part 3, we will cover CLI commands for S3, the most commonly used storage service in AWS, the relational database RDS, and the NoSQL database DynamoDB. S3 in particular is used in almost every project, so make sure you master it thoroughly.
6. S3 Commands
The S3 CLI is broadly divided into two categories:
aws s3(high-level): Intuitive commands likecp,mv,sync. Used for everyday file operations.aws s3api(low-level): Commands that call the S3 API directly. Used for detailed configurations like bucket policies, lifecycle rules, versioning, etc.
6.1 High-Level Commands (aws s3)
List Buckets
aws s3 ls
2025-03-15 09:12:33 my-app-logs-prod
2025-06-20 14:30:01 my-app-static-assets
2025-08-01 11:00:22 my-data-backup-2025
2025-11-10 16:45:10 my-terraform-state-prod
S3 bucket names are globally unique. Only one bucket with a given name can exist across the entire world.
List Objects in a Bucket
# List bucket root
aws s3 ls s3://my-app-static-assets/
# List a specific prefix (folder)
aws s3 ls s3://my-app-static-assets/images/
# Recursively list all objects (including subfolders)
aws s3 ls s3://my-app-static-assets/ --recursive
# Display sizes in human-readable format
aws s3 ls s3://my-app-static-assets/ --recursive --human-readable --summarize
PRE css/
PRE images/
PRE js/
2025-11-01 09:00:00 1.2 KiB index.html
2025-11-01 09:00:00 512 Bytes favicon.ico
Total Objects: 47
Total Size: 23.4 MiB
Adding the --summarize option gives you a quick overview of the total object count and size.
Create a Bucket
# Create a bucket in the Seoul region
aws s3 mb s3://my-new-bucket-20250318 --region ap-northeast-2
make_bucket: my-new-bucket-20250318
💡 Practical Tip: Including the date or environment (prod, staging) in bucket names makes management easier. For example:
mycompany-logs-prod-2025,mycompany-assets-staging
Delete a Bucket
# Only empty buckets can be deleted
aws s3 rb s3://my-new-bucket-20250318
# Delete all objects in the bucket and then delete the bucket (use with caution!)
aws s3 rb s3://my-new-bucket-20250318 --force
remove_bucket: my-new-bucket-20250318
⚠️ Caution: The
--forceoption deletes all objects in the bucket before removing the bucket itself. Buckets with versioning enabled cannot be deleted using--force. How to delete a versioned bucket is covered in the troubleshooting section later.
Copy Files (cp)
# Local -> S3 upload
aws s3 cp ./report.pdf s3://my-app-static-assets/documents/
# S3 -> Local download
aws s3 cp s3://my-app-static-assets/documents/report.pdf ./downloads/
# S3 -> S3 copy (cross-bucket copy is also possible)
aws s3 cp s3://my-app-static-assets/images/logo.png s3://my-data-backup-2025/images/
# Upload an entire folder (recursive)
aws s3 cp ./build/ s3://my-app-static-assets/ --recursive
# Upload only specific file extensions
aws s3 cp ./logs/ s3://my-app-logs-prod/ --recursive --exclude "*" --include "*.log"
# Upload with a specific storage class
aws s3 cp ./archive.tar.gz s3://my-data-backup-2025/ --storage-class GLACIER
upload: ./report.pdf to s3://my-app-static-assets/documents/report.pdf
💡 Practical Tip: The order of
--excludeand--includematters. Use the pattern of first excluding all files with--exclude "*", then including only the desired files with--include "*.log".
Move Files (mv)
# Move an object within S3 (also used for renaming)
aws s3 mv s3://my-app-static-assets/old-logo.png s3://my-app-static-assets/images/logo.png
# Move from local to S3 (uploads and then deletes the local file)
aws s3 mv ./temp-data.csv s3://my-data-backup-2025/
# Move an entire folder
aws s3 mv s3://my-app-static-assets/v1/ s3://my-app-static-assets/v2/ --recursive
move: s3://my-app-static-assets/old-logo.png to s3://my-app-static-assets/images/logo.png
S3 does not actually have a native "move" operation. Internally, it copies the object and then deletes the original.
Delete Files (rm)
# Delete a single object
aws s3 rm s3://my-app-logs-prod/old-log.txt
# Delete all objects under a specific prefix
aws s3 rm s3://my-app-logs-prod/2024/ --recursive
# Delete only specific file extensions
aws s3 rm s3://my-app-logs-prod/ --recursive --exclude "*" --include "*.tmp"
# Preview files to be deleted (does not actually delete)
aws s3 rm s3://my-app-logs-prod/2024/ --recursive --dryrun
delete: s3://my-app-logs-prod/old-log.txt
💡 Practical Tip: Always use the
--dryrunoption to verify what will be deleted before performing bulk deletions. This can prevent accidents like wiping out production data.
Sync (sync)
# Sync local -> S3 (only uploads changed files)
aws s3 sync ./dist/ s3://my-app-static-assets/
# Sync S3 -> local
aws s3 sync s3://my-app-static-assets/ ./local-backup/
# Sync S3 -> S3 (cross-bucket replication)
aws s3 sync s3://my-app-static-assets/ s3://my-data-backup-2025/assets-backup/
# Also delete files locally that were deleted from S3
aws s3 sync s3://my-app-static-assets/ ./local-backup/ --delete
# Sync while excluding specific files
aws s3 sync ./dist/ s3://my-app-static-assets/ --exclude "*.map" --exclude ".DS_Store"
upload: dist/js/app.bundle.js to s3://my-app-static-assets/js/app.bundle.js
upload: dist/css/style.css to s3://my-app-static-assets/css/style.css
upload: dist/index.html to s3://my-app-static-assets/index.html
sync compares file sizes and modification times, and only transfers files that have changed. It is frequently used for deploying static websites in CI/CD pipelines.
⚠️ Caution: The
--deleteoption removes files from the destination that are not present in the source. When using this on production buckets, always verify with--dryrunfirst.
6.2 Low-Level Commands (aws s3api)
List Buckets
aws s3api list-buckets --query 'Buckets[].{Name:Name, Created:CreationDate}' --output table
---------------------------------------------------------
| ListBuckets |
+------------------+------------------------------------+
| Created | Name |
+------------------+------------------------------------+
| 2025-03-15T... | my-app-logs-prod |
| 2025-06-20T... | my-app-static-assets |
| 2025-08-01T... | my-data-backup-2025 |
| 2025-11-10T... | my-terraform-state-prod |
+------------------+------------------------------------+
Check Object Metadata (head-object)
aws s3api head-object \
--bucket my-app-static-assets \
--key images/logo.png
{
"AcceptRanges": "bytes",
"LastModified": "2025-11-01T09:00:00+00:00",
"ContentLength": 45231,
"ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
"ContentType": "image/png",
"Metadata": {},
"StorageClass": "STANDARD",
"ServerSideEncryption": "AES256"
}
head-object is used to check metadata without downloading the file. You can quickly check things like whether the file exists, its size, last modification time, and encryption status.
Upload an Object (put-object)
# Basic upload
aws s3api put-object \
--bucket my-app-static-assets \
--key config/app-config.json \
--body ./app-config.json \
--content-type application/json
# Server-side encryption + metadata
aws s3api put-object \
--bucket my-app-static-assets \
--key sensitive/credentials.enc \
--body ./credentials.enc \
--server-side-encryption aws:kms \
--ssekms-key-id alias/my-s3-key \
--metadata '{"environment":"production","uploaded-by":"deploy-pipeline"}'
# Upload with tags
aws s3api put-object \
--bucket my-app-logs-prod \
--key reports/2025-q4-report.pdf \
--body ./report.pdf \
--tagging "Department=Engineering&Project=DataPlatform"
{
"ETag": "\"a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4\"",
"ServerSideEncryption": "aws:kms",
"SSEKMSKeyId": "arn:aws:kms:ap-northeast-2:123456789012:key/abcd-1234-efgh-5678",
"VersionId": "3HL4kqtJvjVBH40Nrjfkd"
}
💡 Practical Tip: What is the difference between
aws s3 cpandaws s3api put-object?s3 cpautomatically performs multipart uploads for large files, whiles3api put-objectuploads with a single PUT request. For files larger than 5GB, you must uses3 cp.
Download an Object (get-object)
aws s3api get-object \
--bucket my-app-static-assets \
--key config/app-config.json \
./downloaded-config.json
{
"AcceptRanges": "bytes",
"LastModified": "2025-11-15T14:30:00+00:00",
"ContentLength": 2048,
"ETag": "\"e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2\"",
"ContentType": "application/json",
"ServerSideEncryption": "AES256",
"Metadata": {}
}
List Objects (list-objects-v2)
# Basic listing
aws s3api list-objects-v2 \
--bucket my-app-static-assets \
--prefix images/ \
--query 'Contents[].{Key:Key, Size:Size, Modified:LastModified}' \
--output table
# Pagination when there are more than 1000 objects
aws s3api list-objects-v2 \
--bucket my-app-logs-prod \
--prefix 2025/12/ \
--max-items 100
# List only "folders" using a delimiter
aws s3api list-objects-v2 \
--bucket my-app-static-assets \
--delimiter "/" \
--query 'CommonPrefixes[].Prefix'
--------------------------------------------------------------
| ListObjectsV2 |
+-------------------------------+-------+--------------------+
| Key | Size | Modified |
+-------------------------------+-------+--------------------+
| images/banner.jpg | 89234 | 2025-11-01T09:00.. |
| images/hero.webp | 45678 | 2025-11-01T09:00.. |
| images/logo.png | 45231 | 2025-11-01T09:00.. |
| images/thumbnail-01.jpg | 12345 | 2025-11-15T14:30.. |
+-------------------------------+-------+--------------------+
💡 Practical Tip: Use
list-objects-v2instead oflist-objects(v1). v2 has better performance, and AWS also recommends using v2.
Generate a Pre-Signed URL (presign)
# Generate a URL valid for 1 hour by default
aws s3 presign s3://my-app-static-assets/documents/report.pdf
# Specify validity duration (in seconds, max 7 days = 604800 seconds)
aws s3 presign s3://my-app-static-assets/documents/report.pdf --expires-in 3600
https://my-app-static-assets.s3.ap-northeast-2.amazonaws.com/documents/report.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20251201%2Fap-northeast-2%2Fs3%2Faws4_request&X-Amz-Date=20251201T120000Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=abcdef1234567890abcdef1234567890abcdef1234567890
Anyone who receives this URL can download the file without AWS credentials. The URL automatically expires after the specified validity period.
💡 Practical Tip: Pre-signed URLs are used when you need to temporarily share files with customers or when the frontend needs to upload directly to S3. For upload pre-signed URLs, you need to generate a pre-signed URL for
s3api'sput-object.
6.3 Checking/Configuring Bucket Policies
View Bucket Policy
aws s3api get-bucket-policy \
--bucket my-app-static-assets \
--output text | jq .
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowCloudFrontAccess",
"Effect": "Allow",
"Principal": {
"Service": "cloudfront.amazonaws.com"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-app-static-assets/*",
"Condition": {
"StringEquals": {
"AWS:SourceArn": "arn:aws:cloudfront::123456789012:distribution/E1A2B3C4D5F6G7"
}
}
}
]
}
Set Bucket Policy
# Set a policy from a JSON file
aws s3api put-bucket-policy \
--bucket my-app-static-assets \
--policy file://bucket-policy.json
Example bucket-policy.json (allowing access only via CloudFront OAC):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowCloudFrontServicePrincipalReadOnly",
"Effect": "Allow",
"Principal": {
"Service": "cloudfront.amazonaws.com"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-app-static-assets/*",
"Condition": {
"StringEquals": {
"AWS:SourceArn": "arn:aws:cloudfront::123456789012:distribution/E1A2B3C4D5F6G7"
}
}
}
]
}
Delete Bucket Policy
aws s3api delete-bucket-policy --bucket my-app-static-assets
6.4 Checking/Configuring Public Access Block
Check Public Access Block Status
aws s3api get-public-access-block --bucket my-app-static-assets
{
"PublicAccessBlockConfiguration": {
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}
}
When all four settings are true, public access is completely blocked. This should be the default for security purposes.
Set Public Access Block
# Block all public access (recommended)
aws s3api put-public-access-block \
--bucket my-app-static-assets \
--public-access-block-configuration \
'BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true'
⚠️ Caution: Even when hosting a static website directly from S3, do not make the S3 bucket public. Use a CloudFront + OAC configuration instead. Public S3 buckets are always flagged in security audits.
Check Account-Level Public Access Block
# Check the public access block applied to the entire account
aws s3control get-public-access-block \
--account-id 123456789012
{
"PublicAccessBlockConfiguration": {
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}
}
When an account-level block is configured, any public policies set on individual buckets will be ignored. Security teams typically apply the block at the account level and grant exceptions only for specific buckets that need them.
6.5 Lifecycle Policies
View Lifecycle Policy
aws s3api get-bucket-lifecycle-configuration \
--bucket my-app-logs-prod
{
"Rules": [
{
"ID": "MoveLogsToGlacier",
"Filter": {
"Prefix": "logs/"
},
"Status": "Enabled",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 90,
"StorageClass": "GLACIER"
},
{
"Days": 365,
"StorageClass": "DEEP_ARCHIVE"
}
],
"Expiration": {
"Days": 730
}
},
{
"ID": "CleanupIncompleteUploads",
"Filter": {
"Prefix": ""
},
"Status": "Enabled",
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 7
}
}
]
}
This policy transitions data to Standard-IA after 30 days, to Glacier after 90 days, to Deep Archive after 365 days, and deletes it after 730 days (2 years). It also automatically cleans up multipart uploads that have been incomplete for more than 7 days.
Set Lifecycle Policy
aws s3api put-bucket-lifecycle-configuration \
--bucket my-app-logs-prod \
--lifecycle-configuration file://lifecycle-policy.json
Example lifecycle-policy.json:
{
"Rules": [
{
"ID": "ArchiveOldLogs",
"Filter": {
"Prefix": "logs/"
},
"Status": "Enabled",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 90,
"StorageClass": "GLACIER"
}
],
"Expiration": {
"Days": 365
}
},
{
"ID": "CleanupMultipartUploads",
"Filter": {
"Prefix": ""
},
"Status": "Enabled",
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 7
}
}
]
}
💡 Practical Tip: Always add the
CleanupMultipartUploadsrule to every bucket. Failed multipart uploads accumulate silently and generate hidden costs. This single rule alone can significantly reduce unnecessary S3 expenses.
6.6 Checking Versioning
aws s3api get-bucket-versioning --bucket my-terraform-state-prod
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
| Status Value | Meaning |
|---|---|
Enabled | Versioning is active. Every object gets a version ID |
Suspended | Versioning is paused. Existing versions are retained, but new versions are not created |
| (empty response) | Versioning has never been enabled |
# Enable versioning
aws s3api put-bucket-versioning \
--bucket my-terraform-state-prod \
--versioning-configuration Status=Enabled
# List all versions of a specific object
aws s3api list-object-versions \
--bucket my-terraform-state-prod \
--prefix terraform.tfstate \
--query 'Versions[].{Key:Key, VersionId:VersionId, Modified:LastModified, Size:Size}' \
--output table
------------------------------------------------------------------------
| ListObjectVersions |
+---------------------+----------+--------------------+----------------+
| Key | Size | Modified | VersionId |
+---------------------+----------+--------------------+----------------+
| terraform.tfstate | 45231 | 2025-12-01T10:00.. | v3HL4kqtJvj.. |
| terraform.tfstate | 44890 | 2025-11-28T15:30.. | kBq8fPmN2Xc.. |
| terraform.tfstate | 43102 | 2025-11-25T09:15.. | pQr7wYzA5Mn.. |
+---------------------+----------+--------------------+----------------+
💡 Practical Tip: Always enable versioning on buckets that store Terraform state files. This lets you recover to a previous version if the state file becomes corrupted.
6.7 S3 Troubleshooting
Access Denied Debugging: 5 Steps
When you encounter an Access Denied (403) error in S3, follow these 5 steps in order.
Step 1: Check IAM Policies
# Verify who the current user/role is
aws sts get-caller-identity
# Simulate that user's policies
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::123456789012:user/deploy-user \
--action-names s3:GetObject s3:PutObject \
--resource-arns arn:aws:s3:::my-app-static-assets/*
{
"EvaluationResults": [
{
"EvalActionName": "s3:GetObject",
"EvalResourceName": "arn:aws:s3:::my-app-static-assets/*",
"EvalDecision": "allowed"
},
{
"EvalActionName": "s3:PutObject",
"EvalResourceName": "arn:aws:s3:::my-app-static-assets/*",
"EvalDecision": "implicitDeny"
}
]
}
implicitDeny means there is no policy explicitly allowing this action. You need to add the s3:PutObject action to the IAM policy.
Step 2: Check Bucket Policy
aws s3api get-bucket-policy --bucket my-app-static-assets --output text | jq .
Check whether the bucket policy contains an explicit Deny. Even if IAM allows it, access will be blocked if the bucket policy denies it.
Step 3: Check Public Access Block
# Bucket level
aws s3api get-public-access-block --bucket my-app-static-assets
# Account level
aws s3control get-public-access-block --account-id 123456789012
If you have set a public policy but access is still denied, the account-level Public Access Block may be blocking it.
Step 4: Check KMS Key Permissions
# Check the bucket's default encryption settings
aws s3api get-bucket-encryption --bucket my-app-static-assets
{
"ServerSideEncryptionConfiguration": {
"Rules": [
{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "aws:kms",
"KMSMasterKeyID": "arn:aws:kms:ap-northeast-2:123456789012:key/abcd-1234-efgh-5678"
},
"BucketKeyEnabled": true
}
]
}
}
If KMS encryption is in use, the IAM user/role must have kms:Decrypt and kms:GenerateDataKey permissions to read and write objects.
Step 5: Check VPC Endpoint Policy
# List S3 VPC endpoints
aws ec2 describe-vpc-endpoints \
--filters Name=service-name,Values=com.amazonaws.ap-northeast-2.s3 \
--query 'VpcEndpoints[].{Id:VpcEndpointId, Type:VpcEndpointType, Policy:PolicyDocument}' \
--output json
If you are accessing S3 through a VPC endpoint, verify that the endpoint policy allows access to the relevant bucket.
💡 Practical Tip: The key to debugging Access Denied is finding "where the denial is happening." Looking for events with
errorCode: AccessDeniedin CloudTrail provides more detailed information about which policy caused the denial.
# Look up Access Denied events in CloudTrail
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=GetObject \
--start-time 2025-12-01T00:00:00Z \
--end-time 2025-12-01T23:59:59Z \
--query 'Events[?contains(CloudTrailEvent, `AccessDenied`)].{Time:EventTime, User:Username, Event:EventName}' \
--output table
Deleting a Versioned Bucket
A bucket with versioning enabled cannot be deleted with aws s3 rb --force. You need to remove all versions and delete markers first.
# 1. Delete all object versions
aws s3api list-object-versions \
--bucket my-old-versioned-bucket \
--query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' \
--output json > /tmp/versions.json
aws s3api delete-objects \
--bucket my-old-versioned-bucket \
--delete file:///tmp/versions.json
# 2. Delete all delete markers
aws s3api list-object-versions \
--bucket my-old-versioned-bucket \
--query '{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}' \
--output json > /tmp/delete-markers.json
aws s3api delete-objects \
--bucket my-old-versioned-bucket \
--delete file:///tmp/delete-markers.json
# 3. Now delete the empty bucket
aws s3 rb s3://my-old-versioned-bucket
💡 Practical Tip: If there are tens of thousands of objects, the approach above can be slow. In such cases, it is more efficient to set a lifecycle policy with
Expirationof 1 day and wait. However, keep in mind that lifecycle policies typically run once per day, so it may take some time.
Multipart Upload Failure
When a large file upload fails midway, incomplete multipart uploads remain and incur storage costs.
# List incomplete multipart uploads
aws s3api list-multipart-uploads --bucket my-data-backup-2025
{
"Uploads": [
{
"UploadId": "exampleUploadId1234567890",
"Key": "large-backup/database-dump.tar.gz",
"Initiated": "2025-11-28T10:00:00.000Z",
"StorageClass": "STANDARD",
"Initiator": {
"ID": "arn:aws:iam::123456789012:user/deploy-user"
}
}
]
}
# Abort (delete) a specific multipart upload
aws s3api abort-multipart-upload \
--bucket my-data-backup-2025 \
--key large-backup/database-dump.tar.gz \
--upload-id exampleUploadId1234567890
⚠️ Caution: Incomplete multipart uploads do not show up in
aws s3 ls, but they still incur storage costs. Make sure to configure theAbortIncompleteMultipartUploadlifecycle rule described earlier.
Cost Analysis: Checking Bucket Size
When S3 costs are higher than expected, here is how to find out which bucket is using how much storage.
# Check the total size and object count for a specific bucket
aws s3 ls s3://my-app-logs-prod/ --recursive --human-readable --summarize
2025-01-15 09:00:00 1.2 MiB logs/2025/01/app-2025-01-15.log
2025-01-16 09:00:00 1.5 MiB logs/2025/01/app-2025-01-16.log
...
2025-12-01 09:00:00 2.1 MiB logs/2025/12/app-2025-12-01.log
Total Objects: 12847
Total Size: 18.7 GiB
# Check bucket size via CloudWatch metrics (more accurate and faster)
aws cloudwatch get-metric-statistics \
--namespace AWS/S3 \
--metric-name BucketSizeBytes \
--dimensions Name=BucketName,Value=my-app-logs-prod Name=StorageType,Value=StandardStorage \
--start-time 2025-12-01T00:00:00Z \
--end-time 2025-12-02T00:00:00Z \
--period 86400 \
--statistics Average \
--output json
{
"Label": "BucketSizeBytes",
"Datapoints": [
{
"Timestamp": "2025-12-01T00:00:00+00:00",
"Average": 20079595520.0,
"Unit": "Bytes"
}
]
}
About 18.7GB in use. CloudWatch metrics are updated once a day, but they return results instantly even for large buckets, making them more practical.
# Check the number of objects
aws cloudwatch get-metric-statistics \
--namespace AWS/S3 \
--metric-name NumberOfObjects \
--dimensions Name=BucketName,Value=my-app-logs-prod Name=StorageType,Value=AllStorageTypes \
--start-time 2025-12-01T00:00:00Z \
--end-time 2025-12-02T00:00:00Z \
--period 86400 \
--statistics Average \
--output json
{
"Label": "NumberOfObjects",
"Datapoints": [
{
"Timestamp": "2025-12-01T00:00:00+00:00",
"Average": 12847.0,
"Unit": "Count"
}
]
}
💡 Practical Tip: S3 cost reduction checklist:
- Lifecycle policies: Automatically move old data to cheaper storage classes
- Clean up incomplete multipart uploads: The
AbortIncompleteMultipartUploadrule is essential- S3 Intelligent-Tiering: Consider this storage class when access patterns are unknown
- Clean up unnecessary versions: Set expiration policies for old versions in versioned buckets
- S3 Storage Lens: Analyze storage usage across the entire account with a dashboard
CORS Error Debugging
When CORS errors occur while the frontend accesses S3 directly, check the bucket's CORS configuration.
# Check the current CORS settings
aws s3api get-bucket-cors --bucket my-app-static-assets
{
"CORSRules": [
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET", "PUT", "POST"],
"AllowedOrigins": ["https://www.myapp.com", "https://staging.myapp.com"],
"ExposeHeaders": ["ETag", "x-amz-request-id"],
"MaxAgeSeconds": 3600
}
]
}
# Apply CORS settings
aws s3api put-bucket-cors \
--bucket my-app-static-assets \
--cors-configuration file://cors-config.json
Example cors-config.json:
{
"CORSRules": [
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
"AllowedOrigins": ["https://www.myapp.com", "https://staging.myapp.com"],
"ExposeHeaders": ["ETag", "x-amz-request-id"],
"MaxAgeSeconds": 3600
}
]
}
# Test CORS behavior (simulate a preflight request with curl)
curl -s -o /dev/null -w "%{http_code}" \
-H "Origin: https://www.myapp.com" \
-H "Access-Control-Request-Method: GET" \
-X OPTIONS \
https://my-app-static-assets.s3.ap-northeast-2.amazonaws.com/images/logo.png
200
If 200 is returned, CORS is working properly.
⚠️ Caution: Using
"*"inAllowedOriginsallows access from all domains, which is a security risk. Always specify only the domains you want to allow. Also, if you are using CloudFront, you need to configure the CloudFront distribution to forward CORS-related headers as Origin headers.
7. RDS Commands
RDS (Relational Database Service) is a managed relational database service. It supports MySQL, PostgreSQL, Oracle, SQL Server, MariaDB, and Aurora, AWS's proprietary database engine.
7.1 Querying Instances
# List all RDS instances
aws rds describe-db-instances \
--query 'DBInstances[].{ID:DBInstanceIdentifier, Engine:Engine, Class:DBInstanceClass, Status:DBInstanceStatus, AZ:AvailabilityZone, Storage:AllocatedStorage}' \
--output table
---------------------------------------------------------------------------------------
| DescribeDBInstances |
+--------+------------------+----------+---------------+-----------+------------------+
| AZ | Class | Engine | ID | Status | Storage |
+--------+------------------+----------+---------------+-----------+------------------+
| ap-..2a| db.r6g.large | postgres | myapp-prod | available | 100 |
| ap-..2c| db.t3.medium | mysql | myapp-staging| available | 50 |
| ap-..2a| db.t3.micro | postgres | myapp-dev | available | 20 |
+--------+------------------+----------+---------------+-----------+------------------+
# View detailed information for a specific instance
aws rds describe-db-instances \
--db-instance-identifier myapp-prod \
--query 'DBInstances[0].{
ID:DBInstanceIdentifier,
Engine:Engine,
EngineVersion:EngineVersion,
Class:DBInstanceClass,
Status:DBInstanceStatus,
MultiAZ:MultiAZ,
Storage:AllocatedStorage,
StorageType:StorageType,
Endpoint:Endpoint.Address,
Port:Endpoint.Port,
VPC:DBSubnetGroup.VpcId,
SecurityGroups:VpcSecurityGroups[].VpcSecurityGroupId,
BackupRetention:BackupRetentionPeriod,
EncryptionEnabled:StorageEncrypted
}'
{
"ID": "myapp-prod",
"Engine": "postgres",
"EngineVersion": "15.4",
"Class": "db.r6g.large",
"Status": "available",
"MultiAZ": true,
"Storage": 100,
"StorageType": "gp3",
"Endpoint": "myapp-prod.c9abcdef12gh.ap-northeast-2.rds.amazonaws.com",
"Port": 5432,
"VPC": "vpc-0a1b2c3d4e5f67890",
"SecurityGroups": [
"sg-0abcdef1234567890"
],
"BackupRetention": 7,
"EncryptionEnabled": true
}
💡 Practical Tip:
MultiAZ: truemeans there is a standby instance in a different Availability Zone, providing automatic failover in case of an outage. Always enable Multi-AZ for production databases.
7.2 Creating an Instance
aws rds create-db-instance \
--db-instance-identifier myapp-new-db \
--db-instance-class db.t3.medium \
--engine postgres \
--engine-version 15.4 \
--master-username admin \
--master-user-password 'MySecureP@ssw0rd!' \
--allocated-storage 50 \
--storage-type gp3 \
--vpc-security-group-ids sg-0abcdef1234567890 \
--db-subnet-group-name my-db-subnet-group \
--backup-retention-period 7 \
--multi-az \
--storage-encrypted \
--no-publicly-accessible \
--tags Key=Environment,Value=production Key=Team,Value=backend
{
"DBInstance": {
"DBInstanceIdentifier": "myapp-new-db",
"DBInstanceClass": "db.t3.medium",
"Engine": "postgres",
"DBInstanceStatus": "creating",
"MasterUsername": "admin",
"AllocatedStorage": 50,
"MultiAZ": true,
"StorageEncrypted": true,
"PubliclyAccessible": false
}
}
⚠️ Caution: Entering
--master-user-passworddirectly in the CLI leaves the password in your shell history. In production, use AWS Secrets Manager, or use the--manage-master-user-passwordoption to let AWS manage the password automatically.
7.3 Modifying an Instance
# Change instance class (applied immediately with --apply-immediately)
aws rds modify-db-instance \
--db-instance-identifier myapp-prod \
--db-instance-class db.r6g.xlarge \
--apply-immediately
# Expand storage (applied immediately without downtime)
aws rds modify-db-instance \
--db-instance-identifier myapp-prod \
--allocated-storage 200
# Change backup retention period
aws rds modify-db-instance \
--db-instance-identifier myapp-prod \
--backup-retention-period 14
{
"DBInstance": {
"DBInstanceIdentifier": "myapp-prod",
"DBInstanceStatus": "modifying",
"PendingModifiedValues": {
"DBInstanceClass": "db.r6g.xlarge"
}
}
}
⚠️ Caution: Using
--apply-immediatelyapplies changes right away, but operations like instance class changes cause downtime. In production, it is safer to omit this option so that changes are applied during the maintenance window.
7.4 Stopping/Starting an Instance
# Stop the instance (can be stopped for up to 7 days; it auto-starts after that)
aws rds stop-db-instance --db-instance-identifier myapp-dev
{
"DBInstance": {
"DBInstanceIdentifier": "myapp-dev",
"DBInstanceStatus": "stopping"
}
}
# Start the instance
aws rds start-db-instance --db-instance-identifier myapp-dev
{
"DBInstance": {
"DBInstanceIdentifier": "myapp-dev",
"DBInstanceStatus": "starting"
}
}
💡 Practical Tip: Stopping dev/staging databases outside of business hours can save costs. A common pattern is to set up an automatic stop/start schedule using Lambda + EventBridge. However, Multi-AZ instances cannot be stopped. You must first switch to Single-AZ before stopping.
7.5 Deleting an Instance
# Delete while creating a final snapshot
aws rds delete-db-instance \
--db-instance-identifier myapp-dev \
--final-db-snapshot-identifier myapp-dev-final-snapshot-20251201 \
--no-delete-automated-backups
# Delete immediately without a snapshot (use with caution!)
aws rds delete-db-instance \
--db-instance-identifier myapp-dev \
--skip-final-snapshot \
--delete-automated-backups
{
"DBInstance": {
"DBInstanceIdentifier": "myapp-dev",
"DBInstanceStatus": "deleting"
}
}
⚠️ Caution: Using
--skip-final-snapshotdeletes immediately without creating a snapshot. For production databases, enable deletion protection and always create a final snapshot when deleting.
# Enable deletion protection
aws rds modify-db-instance \
--db-instance-identifier myapp-prod \
--deletion-protection \
--apply-immediately
7.6 Querying Aurora Clusters
# List Aurora clusters
aws rds describe-db-clusters \
--query 'DBClusters[].{Cluster:DBClusterIdentifier, Engine:Engine, Status:Status, Writer:Endpoint, Reader:ReaderEndpoint, Members:DBClusterMembers[].DBInstanceIdentifier}' \
--output json
[
{
"Cluster": "myapp-aurora-prod",
"Engine": "aurora-postgresql",
"Status": "available",
"Writer": "myapp-aurora-prod.cluster-c9abcdef12gh.ap-northeast-2.rds.amazonaws.com",
"Reader": "myapp-aurora-prod.cluster-ro-c9abcdef12gh.ap-northeast-2.rds.amazonaws.com",
"Members": [
"myapp-aurora-prod-instance-1",
"myapp-aurora-prod-instance-2"
]
}
]
Aurora separates the Writer endpoint and Reader endpoint. Sending read queries to the Reader endpoint can distribute the read load.
7.7 Viewing/Creating/Restoring Snapshots
View Snapshots
# View manual snapshots
aws rds describe-db-snapshots \
--db-instance-identifier myapp-prod \
--snapshot-type manual \
--query 'DBSnapshots[].{ID:DBSnapshotIdentifier, Status:Status, Created:SnapshotCreateTime, Size:AllocatedStorage, Engine:Engine}' \
--output table
-----------------------------------------------------------------------------------
| DescribeDBSnapshots |
+------------------------------+--------+----------+--------------------+---------+
| ID | Engine | Size | Created | Status |
+------------------------------+--------+----------+--------------------+---------+
| myapp-prod-before-upgrade | postgres| 100 | 2025-11-15T02:00.. | available|
| myapp-prod-weekly-20251201 | postgres| 100 | 2025-12-01T03:00.. | available|
+------------------------------+--------+----------+--------------------+---------+
# View automated backup snapshots
aws rds describe-db-snapshots \
--db-instance-identifier myapp-prod \
--snapshot-type automated \
--query 'DBSnapshots[].{ID:DBSnapshotIdentifier, Created:SnapshotCreateTime, Status:Status}' \
--output table
Create a Manual Snapshot
aws rds create-db-snapshot \
--db-instance-identifier myapp-prod \
--db-snapshot-identifier myapp-prod-before-migration-20251201 \
--tags Key=Purpose,Value=pre-migration Key=CreatedBy,Value=ops-team
{
"DBSnapshot": {
"DBSnapshotIdentifier": "myapp-prod-before-migration-20251201",
"DBInstanceIdentifier": "myapp-prod",
"Status": "creating",
"Engine": "postgres",
"AllocatedStorage": 100,
"Encrypted": true
}
}
💡 Practical Tip: Always create a manual snapshot before major migrations or engine upgrades. Automated backup snapshots are deleted when the instance is deleted, but manual snapshots are retained until you explicitly delete them.
Restore from a Snapshot
aws rds restore-db-instance-from-db-snapshot \
--db-instance-identifier myapp-restored \
--db-snapshot-identifier myapp-prod-before-migration-20251201 \
--db-instance-class db.r6g.large \
--db-subnet-group-name my-db-subnet-group \
--vpc-security-group-ids sg-0abcdef1234567890 \
--no-publicly-accessible
{
"DBInstance": {
"DBInstanceIdentifier": "myapp-restored",
"DBInstanceStatus": "creating",
"Engine": "postgres",
"DBInstanceClass": "db.r6g.large"
}
}
⚠️ Caution: Restoring from a snapshot always creates a new DB instance. It does not overwrite the existing instance. After restoring, you must verify the security groups, parameter groups, and option groups again.
7.8 RDS Troubleshooting
Connection Failure Checklist
When you cannot connect to RDS, follow these steps in order.
# 1. Check the RDS instance status
aws rds describe-db-instances \
--db-instance-identifier myapp-prod \
--query 'DBInstances[0].{Status:DBInstanceStatus, Endpoint:Endpoint.Address, Port:Endpoint.Port}'
{
"Status": "available",
"Endpoint": "myapp-prod.c9abcdef12gh.ap-northeast-2.rds.amazonaws.com",
"Port": 5432
}
# 2. Check security group inbound rules
aws rds describe-db-instances \
--db-instance-identifier myapp-prod \
--query 'DBInstances[0].VpcSecurityGroups[].VpcSecurityGroupId' \
--output text
sg-0abcdef1234567890
# Check the inbound rules of that security group
aws ec2 describe-security-groups \
--group-ids sg-0abcdef1234567890 \
--query 'SecurityGroups[0].IpPermissions[].{Protocol:IpProtocol, Port:FromPort, Source:IpRanges[].CidrIp, SourceSG:UserIdGroupPairs[].GroupId}'
[
{
"Protocol": "tcp",
"Port": 5432,
"Source": [],
"SourceSG": ["sg-0111222333444555"]
}
]
In this case, access is only allowed from a specific security group (sg-0111222333444555). You need to verify that the client EC2 instance belongs to this security group.
# 3. Check DB subnet group (network connectivity)
aws rds describe-db-instances \
--db-instance-identifier myapp-prod \
--query 'DBInstances[0].DBSubnetGroup.{Name:DBSubnetGroupName, VPC:VpcId, Subnets:Subnets[].{AZ:SubnetAvailabilityZone.Name, SubnetId:SubnetIdentifier}}'
{
"Name": "my-db-subnet-group",
"VPC": "vpc-0a1b2c3d4e5f67890",
"Subnets": [
{
"AZ": "ap-northeast-2a",
"SubnetId": "subnet-0aaa1111bbb22222"
},
{
"AZ": "ap-northeast-2c",
"SubnetId": "subnet-0ccc3333ddd44444"
}
]
}
# 4. Check if the RDS instance is publicly accessible
aws rds describe-db-instances \
--db-instance-identifier myapp-prod \
--query 'DBInstances[0].PubliclyAccessible'
false
If PubliclyAccessible: false, the instance cannot be accessed from outside the VPC. To connect locally, you need to use a VPN or SSH tunnel.
# 5. Check route tables (verify the client -> RDS route exists)
aws ec2 describe-route-tables \
--filters Name=association.subnet-id,Values=subnet-0aaa1111bbb22222 \
--query 'RouteTables[0].Routes[].{Destination:DestinationCidrBlock, Target:GatewayId || NatGatewayId || TransitGatewayId}'
💡 Practical Tip: About 80% of RDS connection issues are due to security group misconfiguration. The first thing to check is whether the client's security group ID is listed in the RDS security group's inbound rules.
Checking Storage Capacity
# Check allocated storage and usage (CloudWatch metrics)
aws cloudwatch get-metric-statistics \
--namespace AWS/RDS \
--metric-name FreeStorageSpace \
--dimensions Name=DBInstanceIdentifier,Value=myapp-prod \
--start-time 2025-12-01T00:00:00Z \
--end-time 2025-12-02T00:00:00Z \
--period 3600 \
--statistics Average \
--output json
{
"Label": "FreeStorageSpace",
"Datapoints": [
{
"Timestamp": "2025-12-01T12:00:00+00:00",
"Average": 53687091200.0,
"Unit": "Bytes"
}
]
}
About 50GB remaining (50GB used out of 100GB). If the remaining capacity is less than 10% of the total, you need to urgently expand the storage.
# Check storage auto-scaling configuration
aws rds describe-db-instances \
--db-instance-identifier myapp-prod \
--query 'DBInstances[0].{MaxAllocatedStorage:MaxAllocatedStorage, AllocatedStorage:AllocatedStorage}'
{
"MaxAllocatedStorage": 500,
"AllocatedStorage": 100
}
If MaxAllocatedStorage is set, storage auto-scaling is enabled. When the remaining capacity drops below 10%, the storage automatically expands.
💡 Practical Tip: Setting storage auto-scaling (
MaxAllocatedStorage) can prevent DB outages caused by full disks. However, costs may increase unexpectedly, so make sure to set CloudWatch alarms as well.
Slow Query Logs
# Check slow query log settings in the parameter group (PostgreSQL)
aws rds describe-db-parameters \
--db-parameter-group-name my-postgres15-params \
--query 'Parameters[?ParameterName==`log_min_duration_statement`].{Name:ParameterName, Value:ParameterValue}' \
--output table
-----------------------------------------------------
| DescribeDBParameters |
+------------------------------+--------------------+
| Name | Value |
+------------------------------+--------------------+
| log_min_duration_statement | 1000 |
+------------------------------+--------------------+
log_min_duration_statement=1000 means that queries taking 1000ms (1 second) or longer are recorded in the logs.
# List RDS log files
aws rds describe-db-log-files \
--db-instance-identifier myapp-prod \
--query 'DescribeDBLogFiles[?contains(LogFileName, `postgresql`)].{FileName:LogFileName, Size:Size, Written:LastWritten}' \
--output table
--------------------------------------------------------------------
| DescribeDBLogFiles |
+-------------------------------------------+--------+------------+
| FileName | Size | Written |
+-------------------------------------------+--------+------------+
| error/postgresql.log.2025-12-01-00 | 45231 | 1701388800 |
| error/postgresql.log.2025-12-01-01 | 12890 | 1701392400 |
| error/postgresql.log.2025-12-01-02 | 8901 | 1701396000 |
+-------------------------------------------+--------+------------+
# Download a log file
aws rds download-db-log-file-portion \
--db-instance-identifier myapp-prod \
--log-file-name error/postgresql.log.2025-12-01-00 \
--output text > /tmp/rds-slow-query.log
💡 Practical Tip: If you configure slow query logs to be exported to CloudWatch Logs, you can analyze query patterns and build dashboards with CloudWatch Logs Insights. Enabling Performance Insights also lets you visually identify the queries generating the most load.
Parameter Group Check
# Check the parameter group attached to the instance
aws rds describe-db-instances \
--db-instance-identifier myapp-prod \
--query 'DBInstances[0].DBParameterGroups'
[
{
"DBParameterGroupName": "my-postgres15-params",
"ParameterApplyStatus": "in-sync"
}
]
# View modified parameters (only those changed from defaults)
aws rds describe-db-parameters \
--db-parameter-group-name my-postgres15-params \
--query 'Parameters[?Source==`user`].{Name:ParameterName, Value:ParameterValue, ApplyType:ApplyType}' \
--output table
----------------------------------------------------------------------
| DescribeDBParameters |
+---------+----------------------------------+-----------------------+
|ApplyType| Name | Value |
+---------+----------------------------------+-----------------------+
| dynamic | log_min_duration_statement | 1000 |
| dynamic | shared_preload_libraries | pg_stat_statements |
| static | max_connections | 200 |
| dynamic | work_mem | 65536 |
| dynamic | maintenance_work_mem | 524288 |
+---------+----------------------------------+-----------------------+
Parameters with ApplyType of static require a DB instance reboot to take effect. dynamic parameters are applied immediately.
⚠️ Caution: Changing
max_connectionsrequires a reboot. In production, schedule the reboot during the maintenance window. Also, settingmax_connectionstoo high can cause memory exhaustion and make the instance unstable.
Event Logs
# View RDS events from the last 24 hours
aws rds describe-events \
--source-identifier myapp-prod \
--source-type db-instance \
--duration 1440 \
--output table
------------------------------------------------------------------------------------------
| DescribeEvents |
+--------------------+-------------------+-----------------------------------------------+
| Date | SourceIdentifier | Message |
+--------------------+-------------------+-----------------------------------------------+
| 2025-12-01T03:00.. | myapp-prod | Backing up DB instance |
| 2025-12-01T03:15.. | myapp-prod | Finished DB Instance backup |
| 2025-12-01T06:30.. | myapp-prod | Multi-AZ instance failover started |
| 2025-12-01T06:31.. | myapp-prod | Multi-AZ instance failover completed |
+--------------------+-------------------+-----------------------------------------------+
# View only specific event categories (failover, maintenance, etc.)
aws rds describe-events \
--source-identifier myapp-prod \
--source-type db-instance \
--event-categories '["failover","maintenance"]' \
--duration 10080
💡 Practical Tip: When a Multi-AZ failover occurs, connections are interrupted for about 1-2 minutes. If this event happens frequently, the instance class may be too small to handle the load. By subscribing to RDS events via SNS, you can receive notifications through Slack or email.
8. DynamoDB Commands
DynamoDB is AWS's fully managed NoSQL database. It features millisecond response times and near-infinite scalability. Data is organized using a partition key (PK) and an optional sort key (SK).
8.1 Listing Tables / Viewing Details
# List tables
aws dynamodb list-tables
{
"TableNames": [
"Users",
"Orders",
"Sessions",
"AuditLogs"
]
}
# View detailed table information
aws dynamodb describe-table \
--table-name Orders \
--query 'Table.{Name:TableName, Status:TableStatus, ItemCount:ItemCount, SizeBytes:TableSizeBytes, PK:KeySchema, BillingMode:BillingModeSummary.BillingMode, GSI:GlobalSecondaryIndexes[].IndexName}'
{
"Name": "Orders",
"Status": "ACTIVE",
"ItemCount": 1547832,
"SizeBytes": 524288000,
"PK": [
{
"AttributeName": "userId",
"KeyType": "HASH"
},
{
"AttributeName": "orderId",
"KeyType": "RANGE"
}
],
"BillingMode": "PAY_PER_REQUEST",
"GSI": [
"OrderStatusIndex",
"CreatedAtIndex"
]
}
HASH refers to the partition key and RANGE refers to the sort key. This table partitions data by userId and sorts orders for the same user by orderId.
8.2 Writing an Item (put-item)
# Add an item
aws dynamodb put-item \
--table-name Orders \
--item '{
"userId": {"S": "user-12345"},
"orderId": {"S": "ord-20251201-001"},
"status": {"S": "PENDING"},
"totalAmount": {"N": "45000"},
"items": {"L": [
{"M": {"productId": {"S": "prod-001"}, "quantity": {"N": "2"}, "price": {"N": "15000"}}},
{"M": {"productId": {"S": "prod-002"}, "quantity": {"N": "1"}, "price": {"N": "15000"}}}
]},
"createdAt": {"S": "2025-12-01T10:30:00Z"},
"ttl": {"N": "1735689600"}
}'
DynamoDB data type notation:
S: StringN: Number (passed as a string)BOOL: BooleanL: List (array)M: Map (object)NULL: NullSS/NS: String Set / Number Set
💡 Practical Tip: When you specify the
ttlattribute as a Unix timestamp (in seconds), DynamoDB automatically deletes expired items. This is useful for session data, temporary tokens, etc. The TTL attribute name must be separately enabled in the table settings.
8.3 Reading an Item (get-item)
# Retrieve a single item (specify exactly with partition key + sort key)
aws dynamodb get-item \
--table-name Orders \
--key '{
"userId": {"S": "user-12345"},
"orderId": {"S": "ord-20251201-001"}
}'
{
"Item": {
"userId": {"S": "user-12345"},
"orderId": {"S": "ord-20251201-001"},
"status": {"S": "PENDING"},
"totalAmount": {"N": "45000"},
"items": {"L": [
{"M": {"productId": {"S": "prod-001"}, "quantity": {"N": "2"}, "price": {"N": "15000"}}},
{"M": {"productId": {"S": "prod-002"}, "quantity": {"N": "1"}, "price": {"N": "15000"}}}
]},
"createdAt": {"S": "2025-12-01T10:30:00Z"},
"ttl": {"N": "1735689600"}
}
}
# Retrieve only specific attributes (projection)
aws dynamodb get-item \
--table-name Orders \
--key '{"userId": {"S": "user-12345"}, "orderId": {"S": "ord-20251201-001"}}' \
--projection-expression "status, totalAmount, createdAt"
{
"Item": {
"status": {"S": "PENDING"},
"totalAmount": {"N": "45000"},
"createdAt": {"S": "2025-12-01T10:30:00Z"}
}
}
# Strongly consistent read (guarantees the latest data)
aws dynamodb get-item \
--table-name Orders \
--key '{"userId": {"S": "user-12345"}, "orderId": {"S": "ord-20251201-001"}}' \
--consistent-read
💡 Practical Tip: By default,
get-itemuses eventually consistent reads. If you need to immediately read data that was just written, use the--consistent-readoption. However, this doubles the RCU (Read Capacity Unit) consumption.
8.4 Query
# Query all orders for a specific user
aws dynamodb query \
--table-name Orders \
--key-condition-expression "userId = :uid" \
--expression-attribute-values '{":uid": {"S": "user-12345"}}' \
--output json
{
"Items": [
{
"userId": {"S": "user-12345"},
"orderId": {"S": "ord-20251201-001"},
"status": {"S": "PENDING"},
"totalAmount": {"N": "45000"},
"createdAt": {"S": "2025-12-01T10:30:00Z"}
},
{
"userId": {"S": "user-12345"},
"orderId": {"S": "ord-20251128-003"},
"status": {"S": "DELIVERED"},
"totalAmount": {"N": "32000"},
"createdAt": {"S": "2025-11-28T14:20:00Z"}
}
],
"Count": 2,
"ScannedCount": 2
}
# Add sort key range conditions (query orders within a specific period)
aws dynamodb query \
--table-name Orders \
--key-condition-expression "userId = :uid AND orderId BETWEEN :start AND :end" \
--expression-attribute-values '{
":uid": {"S": "user-12345"},
":start": {"S": "ord-20251101"},
":end": {"S": "ord-20251231"}
}'
# Add filter conditions (only orders with PENDING status)
aws dynamodb query \
--table-name Orders \
--key-condition-expression "userId = :uid" \
--filter-expression "#s = :status" \
--expression-attribute-names '{"#s": "status"}' \
--expression-attribute-values '{
":uid": {"S": "user-12345"},
":status": {"S": "PENDING"}
}'
⚠️ Caution:
--filter-expressionfilters data after DynamoDB reads it. This means RCU consumption is based on the total amount of data before filtering. For frequently used filter conditions, creating a GSI (Global Secondary Index) is far more efficient in terms of both cost and performance.
# Query using a GSI (query orders by status)
aws dynamodb query \
--table-name Orders \
--index-name OrderStatusIndex \
--key-condition-expression "#s = :status" \
--expression-attribute-names '{"#s": "status"}' \
--expression-attribute-values '{":status": {"S": "PENDING"}}' \
--limit 10
{
"Items": [
{
"userId": {"S": "user-12345"},
"orderId": {"S": "ord-20251201-001"},
"status": {"S": "PENDING"},
"totalAmount": {"N": "45000"}
},
{
"userId": {"S": "user-67890"},
"orderId": {"S": "ord-20251201-005"},
"status": {"S": "PENDING"},
"totalAmount": {"N": "28000"}
}
],
"Count": 2,
"ScannedCount": 2
}
status is a reserved word in DynamoDB, so you must use an alias like #s through --expression-attribute-names. Using a reserved word directly will cause an error.
8.5 Scan
# Scan the entire table (caution: very costly on large tables)
aws dynamodb scan \
--table-name Orders \
--select COUNT
{
"Count": 1547832,
"ScannedCount": 1547832
}
# Scan with filter conditions
aws dynamodb scan \
--table-name Orders \
--filter-expression "totalAmount > :amount" \
--expression-attribute-values '{":amount": {"N": "100000"}}' \
--limit 5
{
"Items": [
{
"userId": {"S": "user-55555"},
"orderId": {"S": "ord-20251130-012"},
"status": {"S": "DELIVERED"},
"totalAmount": {"N": "150000"}
},
{
"userId": {"S": "user-88888"},
"orderId": {"S": "ord-20251125-007"},
"status": {"S": "SHIPPED"},
"totalAmount": {"N": "230000"}
}
],
"Count": 2,
"ScannedCount": 5,
"LastEvaluatedKey": {
"userId": {"S": "user-88888"},
"orderId": {"S": "ord-20251125-007"}
}
}
ScannedCount: 5 means 5 items were read, and of those, only Count: 2 matched the filter condition. If LastEvaluatedKey is present, there is still more data to read.
⚠️ Caution:
scanreads every item in the table. Running a scan on a table with millions of items consumes an enormous amount of RCU and can affect other requests. In production environments, usequerywhenever possible. If you must usescan, use--limitand pagination.
8.6 Updating an Item (update-item)
# Change order status
aws dynamodb update-item \
--table-name Orders \
--key '{"userId": {"S": "user-12345"}, "orderId": {"S": "ord-20251201-001"}}' \
--update-expression "SET #s = :newStatus, updatedAt = :now" \
--expression-attribute-names '{"#s": "status"}' \
--expression-attribute-values '{
":newStatus": {"S": "SHIPPED"},
":now": {"S": "2025-12-02T09:00:00Z"}
}' \
--return-values ALL_NEW
{
"Attributes": {
"userId": {"S": "user-12345"},
"orderId": {"S": "ord-20251201-001"},
"status": {"S": "SHIPPED"},
"totalAmount": {"N": "45000"},
"createdAt": {"S": "2025-12-01T10:30:00Z"},
"updatedAt": {"S": "2025-12-02T09:00:00Z"}
}
}
# Increment/decrement numeric values (atomic counter)
aws dynamodb update-item \
--table-name Users \
--key '{"userId": {"S": "user-12345"}}' \
--update-expression "SET loginCount = loginCount + :inc, lastLoginAt = :now" \
--expression-attribute-values '{
":inc": {"N": "1"},
":now": {"S": "2025-12-02T09:15:00Z"}
}' \
--return-values UPDATED_NEW
{
"Attributes": {
"loginCount": {"N": "157"},
"lastLoginAt": {"S": "2025-12-02T09:15:00Z"}
}
}
# Conditional update (only update when a specific condition is met)
aws dynamodb update-item \
--table-name Orders \
--key '{"userId": {"S": "user-12345"}, "orderId": {"S": "ord-20251201-001"}}' \
--update-expression "SET #s = :newStatus" \
--condition-expression "#s = :currentStatus" \
--expression-attribute-names '{"#s": "status"}' \
--expression-attribute-values '{
":newStatus": {"S": "DELIVERED"},
":currentStatus": {"S": "SHIPPED"}
}' \
--return-values ALL_NEW
--condition-expression is a condition that only changes the status to DELIVERED when the current status is SHIPPED. If the condition is not met, a ConditionalCheckFailedException error is raised and the update does not occur. This is useful for concurrency control.
💡 Practical Tip: Commonly used operations in
update-expression:
SET: Add/modify attributesREMOVE: Delete attributesADD: Increment/decrement numbers, add elements to a setDELETE: Remove elements from a setYou can combine multiple operations in a single call:
SET #s = :val REMOVE oldField ADD viewCount :one
8.7 Deleting an Item (delete-item)
# Delete an item
aws dynamodb delete-item \
--table-name Orders \
--key '{"userId": {"S": "user-12345"}, "orderId": {"S": "ord-20251201-001"}}'
# Conditional delete (only delete if status is CANCELLED)
aws dynamodb delete-item \
--table-name Orders \
--key '{"userId": {"S": "user-12345"}, "orderId": {"S": "ord-20251201-001"}}' \
--condition-expression "#s = :cancelled" \
--expression-attribute-names '{"#s": "status"}' \
--expression-attribute-values '{":cancelled": {"S": "CANCELLED"}}' \
--return-values ALL_OLD
{
"Attributes": {
"userId": {"S": "user-12345"},
"orderId": {"S": "ord-20251201-001"},
"status": {"S": "CANCELLED"},
"totalAmount": {"N": "45000"},
"createdAt": {"S": "2025-12-01T10:30:00Z"}
}
}
Using --return-values ALL_OLD returns the content of the deleted item. This is useful for keeping deletion logs.
8.8 Checking Capacity Mode (Provisioned vs. On-Demand)
aws dynamodb describe-table \
--table-name Orders \
--query 'Table.{
TableName:TableName,
BillingMode:BillingModeSummary.BillingMode,
ReadCapacity:ProvisionedThroughput.ReadCapacityUnits,
WriteCapacity:ProvisionedThroughput.WriteCapacityUnits,
LastDecreaseTime:BillingModeSummary.LastUpdateToPayPerRequestDateTime
}'
{
"TableName": "Orders",
"BillingMode": "PAY_PER_REQUEST",
"ReadCapacity": 0,
"WriteCapacity": 0,
"LastDecreaseTime": "2025-09-15T10:00:00+00:00"
}
| Capacity Mode | BillingMode | Characteristics |
|---|---|---|
| On-Demand | PAY_PER_REQUEST | Pay for what you use. Suitable when traffic is hard to predict |
| Provisioned | PROVISIONED | Read/write capacity is specified in advance. Suitable for predictable traffic |
# Check capacity for a provisioned mode table
aws dynamodb describe-table \
--table-name Sessions \
--query 'Table.{
TableName:TableName,
BillingMode:BillingModeSummary.BillingMode,
ReadCapacity:ProvisionedThroughput.ReadCapacityUnits,
WriteCapacity:ProvisionedThroughput.WriteCapacityUnits
}'
{
"TableName": "Sessions",
"BillingMode": "PROVISIONED",
"ReadCapacity": 100,
"WriteCapacity": 50
}
💡 Practical Tip: Start with on-demand mode during the early stages when traffic patterns are unknown, then switch to provisioned + Auto Scaling once traffic stabilizes. This is the most cost-efficient approach. On-demand mode costs about 6.5 times more than provisioned.
8.9 DynamoDB Troubleshooting
Checking Throttling (CloudWatch)
When a ProvisionedThroughputExceededException error occurs in DynamoDB, it means the provisioned capacity has been exceeded.
# Check read throttling
aws cloudwatch get-metric-statistics \
--namespace AWS/DynamoDB \
--metric-name ReadThrottleEvents \
--dimensions Name=TableName,Value=Sessions \
--start-time 2025-12-01T00:00:00Z \
--end-time 2025-12-02T00:00:00Z \
--period 300 \
--statistics Sum \
--output json
{
"Label": "ReadThrottleEvents",
"Datapoints": [
{
"Timestamp": "2025-12-01T09:00:00+00:00",
"Sum": 0.0,
"Unit": "Count"
},
{
"Timestamp": "2025-12-01T09:05:00+00:00",
"Sum": 0.0,
"Unit": "Count"
},
{
"Timestamp": "2025-12-01T14:00:00+00:00",
"Sum": 347.0,
"Unit": "Count"
},
{
"Timestamp": "2025-12-01T14:05:00+00:00",
"Sum": 523.0,
"Unit": "Count"
},
{
"Timestamp": "2025-12-01T14:10:00+00:00",
"Sum": 189.0,
"Unit": "Count"
}
]
}
Read throttling was concentrated around 2 PM. This could indicate a traffic spike during that period or a hot partition issue.
# Check write throttling
aws cloudwatch get-metric-statistics \
--namespace AWS/DynamoDB \
--metric-name WriteThrottleEvents \
--dimensions Name=TableName,Value=Sessions \
--start-time 2025-12-01T00:00:00Z \
--end-time 2025-12-02T00:00:00Z \
--period 300 \
--statistics Sum \
--output json
# Check actual consumed read/write capacity
aws cloudwatch get-metric-statistics \
--namespace AWS/DynamoDB \
--metric-name ConsumedReadCapacityUnits \
--dimensions Name=TableName,Value=Sessions \
--start-time 2025-12-01T13:00:00Z \
--end-time 2025-12-01T15:00:00Z \
--period 300 \
--statistics Sum \
--output json
{
"Label": "ConsumedReadCapacityUnits",
"Datapoints": [
{
"Timestamp": "2025-12-01T14:00:00+00:00",
"Sum": 45230.0,
"Unit": "Count"
},
{
"Timestamp": "2025-12-01T14:05:00+00:00",
"Sum": 52100.0,
"Unit": "Count"
}
]
}
52,100 RCU were consumed in 5 minutes, which means roughly 174 RCU per second (52100 / 300). If the provisioned capacity is 100 RCU, this is clearly insufficient.
# Resolve throttling: Check Auto Scaling settings
aws application-autoscaling describe-scalable-targets \
--service-namespace dynamodb \
--resource-ids "table/Sessions" \
--output json
{
"ScalableTargets": [
{
"ResourceId": "table/Sessions",
"ScalableDimension": "dynamodb:table:ReadCapacityUnits",
"MinCapacity": 50,
"MaxCapacity": 500,
"SuspendedState": {
"DynamicScalingInSuspended": false,
"DynamicScalingOutSuspended": false,
"ScheduledScalingSuspended": false
}
}
]
}
💡 Practical Tip: Ways to resolve throttling:
- Enable Auto Scaling: Automatically adjusts within the Min/Max capacity range
- Switch to on-demand mode: When traffic is hard to predict
- DAX (DynamoDB Accelerator): Distribute read load through caching
- Fix hot partition issues: Improve partition key design (see below)
Hot Partition Analysis
A hot partition occurs when requests are concentrated on a specific partition key, causing throttling. For example, if status is used as the partition key, most orders are in PENDING or ACTIVE status, causing traffic to concentrate on specific partitions.
# Analyze hot partition keys with CloudWatch Contributor Insights
# (Contributor Insights must be enabled first)
aws dynamodb describe-contributor-insights \
--table-name Sessions
{
"TableName": "Sessions",
"ContributorInsightsRuleList": [
"DynamoDBContributorInsights-PKC-Sessions-1701388800000",
"DynamoDBContributorInsights-SKC-Sessions-1701388800000",
"DynamoDBContributorInsights-PKT-Sessions-1701388800000",
"DynamoDBContributorInsights-SKT-Sessions-1701388800000"
],
"ContributorInsightsStatus": "ENABLED"
}
# Enable Contributor Insights (if not already enabled)
aws dynamodb update-contributor-insights \
--table-name Sessions \
--contributor-insights-action ENABLE
# Check the most accessed partition keys in CloudWatch
aws cloudwatch get-metric-data \
--metric-data-queries '[
{
"Id": "throttled_keys",
"MetricStat": {
"Metric": {
"Namespace": "AWS/DynamoDB",
"MetricName": "ThrottledRequests",
"Dimensions": [
{"Name": "TableName", "Value": "Sessions"},
{"Name": "Operation", "Value": "GetItem"}
]
},
"Period": 300,
"Stat": "Sum"
}
}
]' \
--start-time 2025-12-01T13:00:00Z \
--end-time 2025-12-01T15:00:00Z
{
"MetricDataResults": [
{
"Id": "throttled_keys",
"Label": "ThrottledRequests",
"Timestamps": [
"2025-12-01T14:05:00+00:00",
"2025-12-01T14:00:00+00:00"
],
"Values": [
523.0,
347.0
],
"StatusCode": "Complete"
}
]
}
💡 Practical Tip: Partition key design principles to avoid hot partitions:
- High cardinality: Choose attributes with many unique values as the partition key (e.g.,
userId,orderId)- Even distribution: Choose attributes that are not skewed toward specific values
- Composite keys: If a single attribute is inadequate, combine multiple attributes (e.g.,
userId#2025-12-01)- Write sharding: Add a random suffix to the partition key (e.g.,
event#1,event#2, ...,event#10)
# When a hot partition is suspected: Check per-partition metrics
# (This command shows the overall throughput at the table level)
aws cloudwatch get-metric-statistics \
--namespace AWS/DynamoDB \
--metric-name SuccessfulRequestLatency \
--dimensions Name=TableName,Value=Sessions Name=Operation,Value=GetItem \
--start-time 2025-12-01T13:00:00Z \
--end-time 2025-12-01T15:00:00Z \
--period 300 \
--statistics Average p99 \
--output table
--------------------------------------------------------------
| GetMetricStatistics |
+--------------------+----------+---------------------------+
| Timestamp | Average | p99 |
+--------------------+----------+---------------------------+
| 2025-12-01T13:00.. | 4.2 | 12.5 |
| 2025-12-01T13:30.. | 3.8 | 11.2 |
| 2025-12-01T14:00.. | 18.7 | 89.3 |
| 2025-12-01T14:05.. | 25.1 | 145.6 |
| 2025-12-01T14:30.. | 5.1 | 14.8 |
+--------------------+----------+---------------------------+
You can see that the p99 latency spiked to 145ms around 2 PM. Under normal conditions it is around 12ms, so it was more than 10 times slower. This is likely due to retries caused by throttling.
⚠️ Warning: The per-partition throughput limit in DynamoDB is 3,000 RCU for reads and 1,000 WCU for writes. No matter how much you increase the overall table capacity, throttling will occur if a single partition exceeds its limit. This is precisely why partition key design is so important.
Wrap-Up
In Part 3, we covered S3, RDS, and DynamoDB -- the core services for storing and managing data in AWS infrastructure. Here is a summary:
| Service | Use Case | Key Commands |
|---|---|---|
| S3 | Object storage (files, logs, backups) | s3 sync, s3api put-bucket-lifecycle-configuration |
| RDS | Relational DB (MySQL, PostgreSQL, Aurora) | rds describe-db-instances, rds create-db-snapshot |
| DynamoDB | NoSQL DB (key-value, document) | dynamodb query, dynamodb update-item |
💡 Next Step: In Part 4, we will cover CLI commands related to serverless and monitoring services, including CloudWatch, Lambda, and API Gateway.
AWS CLI Command Practical Guide - Part 4: Containers, Serverless, Load Balancers
In Part 3, we covered core infrastructure services like S3, RDS, and CloudWatch. In Part 4, we cover containers (ECS/EKS), serverless (Lambda), and load balancer (ELB) commands with a focus on real-world troubleshooting. We have included the most commonly encountered failure scenarios and their solutions during operations.
9. ECS Commands
ECS (Elastic Container Service) is AWS's container orchestration service. It follows a Cluster > Service > Task structure, and using the CLI to check deployment status and troubleshoot issues is really important in practice.
9.1 List Clusters
aws ecs list-clusters
{
"clusterArns": [
"arn:aws:ecs:ap-northeast-2:111122223333:cluster/prod-api-cluster",
"arn:aws:ecs:ap-northeast-2:111122223333:cluster/staging-cluster",
"arn:aws:ecs:ap-northeast-2:111122223333:cluster/batch-processing-cluster"
]
}
This shows all ECS cluster ARNs in your account. If you want to quickly extract just the names, use --query.
# Extract cluster names only
aws ecs list-clusters \
--query 'clusterArns[*]' \
--output table
9.2 Describe Cluster in Detail
aws ecs describe-clusters \
--clusters prod-api-cluster \
--include STATISTICS ATTACHMENTS
{
"clusters": [
{
"clusterArn": "arn:aws:ecs:ap-northeast-2:111122223333:cluster/prod-api-cluster",
"clusterName": "prod-api-cluster",
"status": "ACTIVE",
"registeredContainerInstancesCount": 0,
"runningTasksCount": 6,
"pendingTasksCount": 0,
"activeServicesCount": 3,
"statistics": [
{
"name": "runningFargateTasksCount",
"value": "6"
},
{
"name": "runningEC2TasksCount",
"value": "0"
}
],
"capacityProviders": [
"FARGATE",
"FARGATE_SPOT"
],
"defaultCapacityProviderStrategy": [
{
"capacityProvider": "FARGATE",
"weight": 1,
"base": 2
},
{
"capacityProvider": "FARGATE_SPOT",
"weight": 3,
"base": 0
}
]
}
]
}
You need to include --include STATISTICS to get statistics like runningFargateTasksCount. If registeredContainerInstancesCount is 0, it means this is a Fargate-only cluster.
💡 Practical Tip: Compare
runningTasksCountwithpendingTasksCount. If pending tasks persist for a long time, you should suspect resource shortages or image pull failures.
9.3 List Services
aws ecs list-services \
--cluster prod-api-cluster
{
"serviceArns": [
"arn:aws:ecs:ap-northeast-2:111122223333:service/prod-api-cluster/api-gateway-svc",
"arn:aws:ecs:ap-northeast-2:111122223333:service/prod-api-cluster/user-service",
"arn:aws:ecs:ap-northeast-2:111122223333:service/prod-api-cluster/order-service"
]
}
9.4 Describe Service in Detail (Running vs Desired Comparison)
aws ecs describe-services \
--cluster prod-api-cluster \
--services user-service
{
"services": [
{
"serviceName": "user-service",
"clusterArn": "arn:aws:ecs:ap-northeast-2:111122223333:cluster/prod-api-cluster",
"status": "ACTIVE",
"desiredCount": 3,
"runningCount": 3,
"pendingCount": 0,
"launchType": "FARGATE",
"taskDefinition": "arn:aws:ecs:ap-northeast-2:111122223333:task-definition/user-service:42",
"deploymentConfiguration": {
"maximumPercent": 200,
"minimumHealthyPercent": 100,
"deploymentCircuitBreaker": {
"enable": true,
"rollback": true
}
},
"deployments": [
{
"id": "ecs-svc/1234567890123456789",
"status": "PRIMARY",
"taskDefinition": "arn:aws:ecs:ap-northeast-2:111122223333:task-definition/user-service:42",
"desiredCount": 3,
"runningCount": 3,
"pendingCount": 0,
"rolloutState": "COMPLETED",
"rolloutStateReason": "ECS deployment ecs-svc/1234567890123456789 completed."
}
],
"events": [
{
"id": "a1b2c3d4-5678-90ab-cdef-111122223333",
"createdAt": "2026-03-18T09:15:00+09:00",
"message": "(service user-service) has reached a steady state."
},
{
"id": "a1b2c3d4-5678-90ab-cdef-444455556666",
"createdAt": "2026-03-18T09:14:30+09:00",
"message": "(service user-service) registered 1 targets in target-group prod-user-tg"
}
],
"loadBalancers": [
{
"targetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:targetgroup/prod-user-tg/abcdef1234567890",
"containerName": "user-app",
"containerPort": 8080
}
],
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-0a1b2c3d4e5f00001",
"subnet-0a1b2c3d4e5f00002"
],
"securityGroups": [
"sg-0a1b2c3d4e5f00001"
],
"assignPublicIp": "DISABLED"
}
}
}
]
}
The key thing in this output is the comparison between desiredCount and runningCount.
| Status | Meaning |
|---|---|
desired: 3, running: 3, pending: 0 | Normal. All desired tasks are running |
desired: 3, running: 1, pending: 2 | 2 tasks are still starting up. Deployment in progress or waiting for resources |
desired: 3, running: 0, pending: 0 | All tasks are dead and none are restarting. Serious issue |
desired: 3, running: 2, pending: 1 | If 1 task stays pending, suspect resource shortage |
⚠️ Caution: The
eventsarray shows the most recent events at the top. If the service is behaving abnormally, you must check these events. Messages like "unable to pull image" or "essential container exited" will tell you the cause.
9.5 List Tasks
# List running tasks
aws ecs list-tasks \
--cluster prod-api-cluster \
--service-name user-service \
--desired-status RUNNING
{
"taskArns": [
"arn:aws:ecs:ap-northeast-2:111122223333:task/prod-api-cluster/a1b2c3d4e5f6789012345678",
"arn:aws:ecs:ap-northeast-2:111122223333:task/prod-api-cluster/b2c3d4e5f67890123456789a",
"arn:aws:ecs:ap-northeast-2:111122223333:task/prod-api-cluster/c3d4e5f6789012345678901b"
]
}
# List stopped tasks (for root cause analysis)
aws ecs list-tasks \
--cluster prod-api-cluster \
--service-name user-service \
--desired-status STOPPED
{
"taskArns": [
"arn:aws:ecs:ap-northeast-2:111122223333:task/prod-api-cluster/d4e5f678901234567890abcd",
"arn:aws:ecs:ap-northeast-2:111122223333:task/prod-api-cluster/e5f6789012345678901bcdef"
]
}
💡 Practical Tip: If tasks keep dying and restarting, the first step is to query
STOPPEDtasks to find out why they died.
9.6 Describe Task in Detail (Including Stopped Reason)
aws ecs describe-tasks \
--cluster prod-api-cluster \
--tasks d4e5f678901234567890abcd
{
"tasks": [
{
"taskArn": "arn:aws:ecs:ap-northeast-2:111122223333:task/prod-api-cluster/d4e5f678901234567890abcd",
"clusterArn": "arn:aws:ecs:ap-northeast-2:111122223333:cluster/prod-api-cluster",
"taskDefinitionArn": "arn:aws:ecs:ap-northeast-2:111122223333:task-definition/user-service:42",
"lastStatus": "STOPPED",
"desiredStatus": "STOPPED",
"stoppedReason": "Essential container in task exited",
"stopCode": "EssentialContainerExited",
"stoppedAt": "2026-03-18T08:45:12+09:00",
"startedAt": "2026-03-18T08:42:03+09:00",
"containers": [
{
"containerArn": "arn:aws:ecs:ap-northeast-2:111122223333:container/prod-api-cluster/d4e5f678901234567890abcd/a1b2c3d4",
"name": "user-app",
"lastStatus": "STOPPED",
"exitCode": 137,
"reason": "OutOfMemoryError: Container killed due to memory usage",
"networkInterfaces": [
{
"privateIpv4Address": "10.0.3.42"
}
],
"healthStatus": "UNKNOWN"
}
],
"cpu": "512",
"memory": "1024",
"platformVersion": "1.4.0",
"group": "service:user-service"
}
]
}
Here are the key fields you need to look at.
| Field | What to Check |
|---|---|
stoppedReason | The overall reason why the task stopped |
stopCode | EssentialContainerExited, TaskFailedToStart, UserInitiated, etc. |
containers[].exitCode | 0 = normal exit, 137 = OOM Kill, 1 = application error |
containers[].reason | Specific error message at the container level |
startedAt ~ stoppedAt | How long the task survived. If it only lived for a few seconds, it crashed immediately on start |
💡 Practical Tip: Exit code reference table
- 0: Normal exit (graceful shutdown)
- 1: Application error (code bug, configuration error)
- 137: OOM Kill (forcefully terminated due to memory shortage) - you need to increase the task memory
- 139: Segmentation Fault
- 143: Terminated after receiving SIGTERM (normal scale-down)
9.7 Describe Task Definition
aws ecs describe-task-definition \
--task-definition user-service:42
{
"taskDefinition": {
"taskDefinitionArn": "arn:aws:ecs:ap-northeast-2:111122223333:task-definition/user-service:42",
"family": "user-service",
"revision": 42,
"status": "ACTIVE",
"requiresCompatibilities": ["FARGATE"],
"cpu": "512",
"memory": "1024",
"networkMode": "awsvpc",
"executionRoleArn": "arn:aws:iam::111122223333:role/ecsTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::111122223333:role/user-service-task-role",
"containerDefinitions": [
{
"name": "user-app",
"image": "111122223333.dkr.ecr.ap-northeast-2.amazonaws.com/user-service:v2.1.3",
"cpu": 256,
"memory": 512,
"memoryReservation": 256,
"essential": true,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 8080,
"protocol": "tcp"
}
],
"environment": [
{
"name": "SPRING_PROFILES_ACTIVE",
"value": "prod"
},
{
"name": "SERVER_PORT",
"value": "8080"
}
],
"secrets": [
{
"name": "DB_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:ap-northeast-2:111122223333:secret:prod/db-password-AbCdEf"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/user-service",
"awslogs-region": "ap-northeast-2",
"awslogs-stream-prefix": "user-app"
}
},
"healthCheck": {
"command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
"interval": 30,
"timeout": 5,
"retries": 3,
"startPeriod": 60
}
}
]
}
}
A task definition is the "blueprint" for your container. You can verify everything here -- image version, environment variables, secrets, log settings, and health checks.
⚠️ Caution:
executionRoleArnandtaskRoleArnare different roles.
- executionRole: Permissions for the ECS agent to pull images from ECR and send logs to CloudWatch
- taskRole: Permissions for the application inside the container to access other AWS services (S3, DynamoDB, etc.)
9.8 Update Service (force-new-deployment)
# Force redeployment with the same task definition (useful when using the latest image tag)
aws ecs update-service \
--cluster prod-api-cluster \
--service user-service \
--force-new-deployment
{
"service": {
"serviceName": "user-service",
"desiredCount": 3,
"runningCount": 3,
"deployments": [
{
"id": "ecs-svc/9876543210987654321",
"status": "PRIMARY",
"taskDefinition": "arn:aws:ecs:ap-northeast-2:111122223333:task-definition/user-service:42",
"desiredCount": 3,
"runningCount": 0,
"pendingCount": 3,
"rolloutState": "IN_PROGRESS"
},
{
"id": "ecs-svc/1234567890123456789",
"status": "ACTIVE",
"taskDefinition": "arn:aws:ecs:ap-northeast-2:111122223333:task-definition/user-service:42",
"desiredCount": 0,
"runningCount": 3,
"pendingCount": 0,
"rolloutState": "IN_PROGRESS"
}
]
}
}
# Update to a new task definition revision
aws ecs update-service \
--cluster prod-api-cluster \
--service user-service \
--task-definition user-service:43
# Change desired count (manual scaling)
aws ecs update-service \
--cluster prod-api-cluster \
--service user-service \
--desired-count 5
When a deployment starts, you will see 2 entries in the deployments array. PRIMARY is the new deployment, and ACTIVE is the existing one. Once the new tasks are up and pass health checks, the old tasks are drained.
💡 Practical Tip:
--force-new-deploymentspins up new containers even with the same task definition. It is commonly used to pick up a new image after overwriting the:latesttag in ECR. However, in production, it is recommended to use explicit tags (v2.1.3) and increment the task definition revision.
9.9 ECS Troubleshooting
When Tasks Keep Dying (Similar to CrashLoopBackOff)
# Step 1: Check recently stopped tasks
aws ecs list-tasks \
--cluster prod-api-cluster \
--service-name user-service \
--desired-status STOPPED \
--query 'taskArns[0:3]' \
--output text
# Step 2: Check the stop reason
aws ecs describe-tasks \
--cluster prod-api-cluster \
--tasks d4e5f678901234567890abcd \
--query 'tasks[0].{stoppedReason:stoppedReason,stopCode:stopCode,exitCode:containers[0].exitCode,containerReason:containers[0].reason}'
{
"stoppedReason": "Essential container in task exited",
"stopCode": "EssentialContainerExited",
"exitCode": 1,
"containerReason": "CannotStartContainerError: Error response from daemon: failed to create shim task: OCI runtime create failed"
}
# Step 3: Check container logs (CloudWatch Logs)
aws logs get-log-events \
--log-group-name /ecs/user-service \
--log-stream-name "user-app/user-app/d4e5f678901234567890abcd" \
--limit 50 \
--query 'events[*].message' \
--output text
2026-03-18T08:42:05.123Z Starting application...
2026-03-18T08:42:06.456Z Loading configuration from /app/config/application.yml
2026-03-18T08:42:07.789Z ERROR: Failed to connect to database at prod-db.cluster-abcdef.ap-northeast-2.rds.amazonaws.com:3306
2026-03-18T08:42:07.790Z java.net.ConnectException: Connection refused (Connection refused)
2026-03-18T08:42:07.791Z Application failed to start. Shutting down.
In this case, the root cause is a DB connection failure. You should check the security groups and DB endpoint.
Check Service Events
aws ecs describe-services \
--cluster prod-api-cluster \
--services user-service \
--query 'services[0].events[0:10].[createdAt,message]' \
--output table
---------------------------------------------------------------------------------------------------------------------
| DescribeServices |
+---------------------------+-----------------------------------------------------------------------------------+
| 2026-03-18T09:15:00+09:00| (service user-service) has reached a steady state. |
| 2026-03-18T09:14:55+09:00| (service user-service) registered 1 targets in target-group prod-user-tg |
| 2026-03-18T09:14:30+09:00| (service user-service) has started 1 tasks: task d4e5f678901234567890abcd. |
| 2026-03-18T09:13:00+09:00| (service user-service) stopped 1 tasks: task c3d4e5f6789012345678901b. |
| 2026-03-18T09:12:45+09:00| (service user-service, taskSet ecs-svc/1234567890) unable to pull secrets or |
| | registry auth: unable to retrieve secret from arn:aws:secretsmanager:... |
+---------------------------+-----------------------------------------------------------------------------------+
💡 Practical Tip: Here are commonly seen service event messages and their solutions.
| Event Message | Cause | Solution |
|---|---|---|
unable to pull image | ECR image not found or insufficient permissions | Verify the image tag, add ECR permissions to executionRole |
unable to pull secrets | No access to Secrets Manager | Add secretsmanager:GetSecretValue permission to executionRole |
essential container exited | Application crash | Check CloudWatch logs for errors |
has been unable to place a task | Insufficient resources (CPU/memory) | For Fargate, check subnet IP availability; for EC2, add more instances |
service ... is unhealthy in target-group | Health check failure | Check health check path, port, and grace period |
circuit breaker: rolling back | Deployment failure auto-rollback | Check for configuration errors in the new task definition |
Debugging Deployment Issues
# Monitor deployment status (check the deployments array)
aws ecs describe-services \
--cluster prod-api-cluster \
--services user-service \
--query 'services[0].deployments[*].{id:id,status:status,desired:desiredCount,running:runningCount,pending:pendingCount,rollout:rolloutState}'
[
{
"id": "ecs-svc/9876543210987654321",
"status": "PRIMARY",
"desired": 3,
"running": 1,
"pending": 2,
"rollout": "IN_PROGRESS"
},
{
"id": "ecs-svc/1234567890123456789",
"status": "ACTIVE",
"desired": 0,
"running": 2,
"pending": 0,
"rollout": "IN_PROGRESS"
}
]
If a deployment has been running for a while but the PRIMARY running count is not going up, it means the new tasks are failing health checks. If the Circuit Breaker is enabled, it will automatically roll back.
# Wait for deployment to complete (useful in scripts)
aws ecs wait services-stable \
--cluster prod-api-cluster \
--services user-service
This command waits until the service reaches a steady state. It is commonly used as a post-deployment verification step in CI/CD pipelines.
⚠️ Caution:
wait services-stablepolls 40 times by default (at 15-second intervals = up to 10 minutes). If the service does not stabilize within 10 minutes, it returns an error. Java applications with long startup times may hit this timeout.
10. EKS Commands
EKS (Elastic Kubernetes Service) is AWS's managed Kubernetes service. The control plane is managed by AWS, and we only manage the worker nodes. You will often need to use EKS CLI commands together with kubectl.
10.1 List Clusters
aws eks list-clusters
{
"clusters": [
"prod-eks-cluster",
"staging-eks-cluster",
"dev-eks-cluster"
]
}
Unlike ECS, this returns cluster names directly instead of ARNs.
10.2 Describe Cluster in Detail
aws eks describe-cluster \
--name prod-eks-cluster
{
"cluster": {
"name": "prod-eks-cluster",
"arn": "arn:aws:eks:ap-northeast-2:111122223333:cluster/prod-eks-cluster",
"createdAt": "2025-06-15T10:30:00+09:00",
"version": "1.29",
"status": "ACTIVE",
"endpoint": "https://ABCDEF1234567890.gr7.ap-northeast-2.eks.amazonaws.com",
"roleArn": "arn:aws:iam::111122223333:role/eks-cluster-role",
"resourcesVpcConfig": {
"subnetIds": [
"subnet-0a1b2c3d4e5f00001",
"subnet-0a1b2c3d4e5f00002",
"subnet-0a1b2c3d4e5f00003",
"subnet-0a1b2c3d4e5f00004"
],
"securityGroupIds": [
"sg-0a1b2c3d4e5f00010"
],
"clusterSecurityGroupId": "sg-0a1b2c3d4e5f00011",
"vpcId": "vpc-0a1b2c3d4e5f00001",
"endpointPublicAccess": true,
"endpointPrivateAccess": true,
"publicAccessCidrs": [
"203.0.113.0/24"
]
},
"kubernetesNetworkConfig": {
"serviceIpv4Cidr": "172.20.0.0/16"
},
"logging": {
"clusterLogging": [
{
"types": ["api", "audit", "authenticator", "controllerManager", "scheduler"],
"enabled": true
}
]
},
"platformVersion": "eks.8",
"tags": {
"Environment": "production",
"Team": "platform"
}
}
}
Here are the important things to check.
| Field | What to Check |
|---|---|
version | Kubernetes version. AWS typically supports 4 minor versions |
status | Anything other than ACTIVE indicates a problem |
endpointPublicAccess / endpointPrivateAccess | API server access method |
publicAccessCidrs | IP ranges allowed to access the public endpoint |
logging.enabled | Whether control plane logging is enabled |
💡 Practical Tip: In production, it is recommended to set
endpointPublicAccess: falseandendpointPrivateAccess: true, allowing access only through a VPN or Bastion host. Restricting access to specific IPs viapublicAccessCidrsis also a good approach.
10.3 Update kubeconfig
aws eks update-kubeconfig \
--name prod-eks-cluster \
--region ap-northeast-2
Added new context arn:aws:eks:ap-northeast-2:111122223333:cluster/prod-eks-cluster to /home/devops/.kube/config
This command adds EKS cluster connection information to your ~/.kube/config file. After this, you can access the cluster using kubectl commands.
# Configure with a specific profile and alias
aws eks update-kubeconfig \
--name prod-eks-cluster \
--region ap-northeast-2 \
--alias prod-cluster \
--profile production
Added new context prod-cluster to /home/devops/.kube/config
# Verify the configuration
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* arn:aws:eks:ap-northeast-2:111122223333:cluster/prod-eks-cluster arn:aws:eks:ap-northeast-2:111122223333:cluster/prod-eks-cluster arn:aws:eks:ap-northeast-2:111122223333:cluster/prod-eks-cluster
prod-cluster arn:aws:eks:ap-northeast-2:111122223333:cluster/prod-eks-cluster arn:aws:eks:ap-northeast-2:111122223333:cluster/prod-eks-cluster
⚠️ Caution:
update-kubeconfiguses IAM credentials from the AWS CLI. If you switch to a different AWS profile, kubectl will also operate with different permissions. If you get a "You must be logged in to the server (Unauthorized)" error, check your AWS profile.
10.4 Describe Node Groups
aws eks list-nodegroups \
--cluster-name prod-eks-cluster
{
"nodegroups": [
"prod-general-ng",
"prod-compute-ng",
"prod-spot-ng"
]
}
aws eks describe-nodegroup \
--cluster-name prod-eks-cluster \
--nodegroup-name prod-general-ng
{
"nodegroup": {
"nodegroupName": "prod-general-ng",
"nodegroupArn": "arn:aws:eks:ap-northeast-2:111122223333:nodegroup/prod-eks-cluster/prod-general-ng/a1b2c3d4",
"clusterName": "prod-eks-cluster",
"version": "1.29",
"status": "ACTIVE",
"capacityType": "ON_DEMAND",
"scalingConfig": {
"minSize": 2,
"maxSize": 10,
"desiredSize": 4
},
"instanceTypes": [
"m5.xlarge"
],
"amiType": "AL2_x86_64",
"nodeRole": "arn:aws:iam::111122223333:role/eks-node-role",
"subnets": [
"subnet-0a1b2c3d4e5f00003",
"subnet-0a1b2c3d4e5f00004"
],
"labels": {
"workload-type": "general"
},
"taints": [],
"health": {
"issues": []
},
"updateConfig": {
"maxUnavailable": 1
},
"launchTemplate": {
"name": "prod-eks-node-lt",
"version": "3",
"id": "lt-0a1b2c3d4e5f00001"
}
}
}
If health.issues is empty, everything is healthy. If there are problems, error messages will appear here.
10.5 Node Group Scaling
aws eks update-nodegroup-config \
--cluster-name prod-eks-cluster \
--nodegroup-name prod-general-ng \
--scaling-config minSize=2,maxSize=15,desiredSize=6
{
"update": {
"id": "a1b2c3d4-5678-90ab-cdef-111122223333",
"status": "InProgress",
"type": "ConfigUpdate",
"params": [
{
"type": "MinSize",
"value": "2"
},
{
"type": "MaxSize",
"value": "15"
},
{
"type": "DesiredSize",
"value": "6"
}
],
"createdAt": "2026-03-18T10:30:00+09:00"
}
}
# Check update status
aws eks describe-update \
--name prod-eks-cluster \
--nodegroup-name prod-general-ng \
--update-id a1b2c3d4-5678-90ab-cdef-111122223333
{
"update": {
"id": "a1b2c3d4-5678-90ab-cdef-111122223333",
"status": "Successful",
"type": "ConfigUpdate",
"createdAt": "2026-03-18T10:30:00+09:00"
}
}
💡 Practical Tip: Node group scaling works through Auto Scaling Groups. When scaling up, it takes 2-5 minutes for new nodes to reach Ready status. For urgent traffic spikes, it is best to set up Cluster Autoscaler or Karpenter in advance.
10.6 EKS Troubleshooting
kubeconfig Issues
# Symptom: Error when running kubectl commands
kubectl get nodes
error: You must be logged in to the server (Unauthorized)
# Fix 1: Reconfigure kubeconfig
aws eks update-kubeconfig \
--name prod-eks-cluster \
--region ap-northeast-2
# Fix 2: Verify current IAM credentials
aws sts get-caller-identity
{
"UserId": "AIDACKCEVSQ6C2EXAMPLE",
"Account": "111122223333",
"Arn": "arn:aws:iam::111122223333:user/devops-admin"
}
# Fix 3: Check if the current IAM user/role is mapped in the aws-auth ConfigMap
kubectl describe configmap aws-auth -n kube-system
Name: aws-auth
Namespace: kube-system
Data:
====
mapRoles:
----
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::111122223333:role/eks-node-role
username: system:node:{{EC2PrivateDNSName}}
- groups:
- system:masters
rolearn: arn:aws:iam::111122223333:role/devops-eks-admin
username: devops-admin
mapUsers:
----
- groups:
- system:masters
userarn: arn:aws:iam::111122223333:user/devops-admin
username: devops-admin
⚠️ Caution: The IAM user/role that created the EKS cluster automatically has
system:masterspermissions, but it does not appear in theaws-authConfigMap. To grant access to other users, you must explicitly add them toaws-auth.
Node NotReady Status
# Check node status with kubectl
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-3-42.ap-northeast-2.compute.internal Ready <none> 30d v1.29.3-eks-a1b2c3d
ip-10-0-4-15.ap-northeast-2.compute.internal NotReady <none> 30d v1.29.3-eks-a1b2c3d
ip-10-0-3-78.ap-northeast-2.compute.internal Ready <none> 15d v1.29.3-eks-a1b2c3d
ip-10-0-4-91.ap-northeast-2.compute.internal Ready <none> 15d v1.29.3-eks-a1b2c3d
# Check the EC2 instance status of the NotReady node
aws ec2 describe-instance-status \
--filters "Name=private-dns-name,Values=ip-10-0-4-15.ap-northeast-2.compute.internal" \
--query 'InstanceStatuses[0].{State:InstanceState.Name,System:SystemStatus.Status,Instance:InstanceStatus.Status}'
{
"State": "running",
"System": "ok",
"Instance": "ok"
}
If the EC2 instance itself is running but the node is NotReady, there is a high chance it is a kubelet or networking issue.
# Check detailed node events
kubectl describe node ip-10-0-4-15.ap-northeast-2.compute.internal | tail -20
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning NodeNotReady 5m kubelet Node condition Ready is now: False
Warning ContainerGCFailed 5m kubelet rpc error: code = ResourceExhausted desc = disk full
💡 Practical Tip: Here are the main causes of node NotReady status.
- Disk full: When container images or logs fill up the disk. If kubelet cannot perform image GC, the node goes NotReady
- kubelet process died: Connect via SSM Session Manager and check
systemctl status kubelet- ENI/IP exhaustion: Check available IPs in the subnet
- Security group changes: Verify that communication ports (443, 10250) between nodes and the control plane are not blocked
Private Endpoint Access
# Check cluster endpoint settings
aws eks describe-cluster \
--name prod-eks-cluster \
--query 'cluster.resourcesVpcConfig.{public:endpointPublicAccess,private:endpointPrivateAccess,cidrs:publicAccessCidrs}'
{
"public": false,
"private": true,
"cidrs": []
}
If public access is disabled, you can only access the API server from within the VPC.
# When you need access without a VPN or Bastion -> Enable public access (specific IPs only)
aws eks update-cluster-config \
--name prod-eks-cluster \
--resources-vpc-config endpointPublicAccess=true,publicAccessCidrs="203.0.113.50/32"
{
"update": {
"id": "b2c3d4e5-6789-01ab-cdef-222233334444",
"status": "InProgress",
"type": "EndpointAccessUpdate"
}
}
⚠️ Caution: Changing endpoint settings can take 10-20 minutes. During the change, API server access may become temporarily unstable, so it is recommended to do this during a maintenance window for production clusters.
11. Lambda Commands
Lambda is AWS's serverless computing service. You just upload your code without managing servers, and it runs automatically when events come in. You can use the CLI for function management, invocation, and debugging.
11.1 List Functions
aws lambda list-functions \
--query 'Functions[*].{Name:FunctionName,Runtime:Runtime,Memory:MemorySize,Timeout:Timeout,LastModified:LastModified}' \
--output table
---------------------------------------------------------------------------------------------------------
| ListFunctions |
+----------------------------+-----------------+--------+-----------+------------------------------------+
| Name | Runtime | Memory | Timeout | LastModified |
+----------------------------+-----------------+--------+-----------+------------------------------------+
| order-processor | python3.12 | 512 | 30 | 2026-03-17T14:30:00.000+0000 |
| image-resizer | nodejs20.x | 1024 | 60 | 2026-03-15T09:20:00.000+0000 |
| payment-webhook | python3.12 | 256 | 10 | 2026-03-10T11:45:00.000+0000 |
| data-sync-batch | java21 | 2048 | 900 | 2026-03-12T16:00:00.000+0000 |
| auth-token-validator | python3.12 | 128 | 5 | 2026-02-28T08:15:00.000+0000 |
+----------------------------+-----------------+--------+-----------+------------------------------------+
11.2 Describe Function in Detail
aws lambda get-function \
--function-name order-processor
{
"Configuration": {
"FunctionName": "order-processor",
"FunctionArn": "arn:aws:lambda:ap-northeast-2:111122223333:function:order-processor",
"Runtime": "python3.12",
"Role": "arn:aws:iam::111122223333:role/lambda-order-processor-role",
"Handler": "app.handler",
"CodeSize": 2456789,
"Description": "Process incoming orders from SQS",
"Timeout": 30,
"MemorySize": 512,
"LastModified": "2026-03-17T14:30:00.000+0000",
"Version": "$LATEST",
"Environment": {
"Variables": {
"ORDER_TABLE": "prod-orders",
"NOTIFICATION_TOPIC_ARN": "arn:aws:sns:ap-northeast-2:111122223333:order-notifications",
"LOG_LEVEL": "INFO"
}
},
"TracingConfig": {
"Mode": "Active"
},
"State": "Active",
"LastUpdateStatus": "Successful",
"Architectures": ["arm64"],
"EphemeralStorage": {
"Size": 512
},
"SnapStart": {
"ApplyOn": "None",
"OptimizationStatus": "Off"
},
"LoggingConfig": {
"LogFormat": "JSON",
"LogGroup": "/aws/lambda/order-processor"
},
"VpcConfig": {
"SubnetIds": [
"subnet-0a1b2c3d4e5f00003",
"subnet-0a1b2c3d4e5f00004"
],
"SecurityGroupIds": [
"sg-0a1b2c3d4e5f00020"
],
"VpcId": "vpc-0a1b2c3d4e5f00001"
}
},
"Code": {
"RepositoryType": "S3",
"Location": "https://awslambda-ap-ne-2-tasks.s3.ap-northeast-2.amazonaws.com/..."
}
}
11.3 View Function Configuration Only
aws lambda get-function-configuration \
--function-name order-processor \
--query '{Runtime:Runtime,Memory:MemorySize,Timeout:Timeout,State:State,LastUpdate:LastUpdateStatus,VPC:VpcConfig.VpcId,Env:Environment.Variables}'
{
"Runtime": "python3.12",
"Memory": 512,
"Timeout": 30,
"State": "Active",
"LastUpdate": "Successful",
"VPC": "vpc-0a1b2c3d4e5f00001",
"Env": {
"ORDER_TABLE": "prod-orders",
"NOTIFICATION_TOPIC_ARN": "arn:aws:sns:ap-northeast-2:111122223333:order-notifications",
"LOG_LEVEL": "INFO"
}
}
💡 Practical Tip: Always check
StateandLastUpdateStatus.
State: Active+LastUpdateStatus: Successful= NormalState: Pending= Code update or VPC ENI setup in progressState: Failed= Deployment failed. The function cannot be invoked in this state
11.4 Invoke Function
# Synchronous invocation (get the result immediately)
aws lambda invoke \
--function-name order-processor \
--payload '{"orderId": "ORD-20260318-001", "userId": "user-123", "amount": 45000}' \
--cli-binary-format raw-in-base64-out \
response.json
{
"StatusCode": 200,
"FunctionError": null,
"ExecutedVersion": "$LATEST",
"LogResult": null
}
# Check the response body
cat response.json
{"statusCode": 200, "body": "{\"message\": \"Order ORD-20260318-001 processed successfully\", \"processedAt\": \"2026-03-18T10:45:30Z\"}"}
# View logs along with the result (--log-type Tail)
aws lambda invoke \
--function-name order-processor \
--payload '{"orderId": "ORD-20260318-002", "userId": "user-456", "amount": 89000}' \
--cli-binary-format raw-in-base64-out \
--log-type Tail \
response.json \
--query 'LogResult' \
--output text | base64 --decode
START RequestId: a1b2c3d4-5678-90ab-cdef-111122223333 Version: $LATEST
[INFO] 2026-03-18T01:45:30.123Z a1b2c3d4-5678-90ab-cdef-111122223333 Processing order: ORD-20260318-002
[INFO] 2026-03-18T01:45:30.456Z a1b2c3d4-5678-90ab-cdef-111122223333 Writing to DynamoDB table: prod-orders
[INFO] 2026-03-18T01:45:30.789Z a1b2c3d4-5678-90ab-cdef-111122223333 Publishing notification to SNS
END RequestId: a1b2c3d4-5678-90ab-cdef-111122223333
REPORT RequestId: a1b2c3d4-5678-90ab-cdef-111122223333 Duration: 234.56 ms Billed Duration: 235 ms Memory Size: 512 MB Max Memory Used: 89 MB Init Duration: 345.67 ms
The REPORT line is important.
| Field | Meaning |
|---|---|
Duration | Actual execution time |
Billed Duration | Billable time (rounded up to 1ms) |
Memory Size | Allocated memory |
Max Memory Used | Maximum memory actually used |
Init Duration | Initialization time during Cold Start (not shown for Warm Starts) |
# Asynchronous invocation (Fire and Forget)
aws lambda invoke \
--function-name data-sync-batch \
--payload '{"source": "legacy-db", "target": "new-db", "date": "2026-03-18"}' \
--cli-binary-format raw-in-base64-out \
--invocation-type Event \
response.json
{
"StatusCode": 202,
"FunctionError": null
}
Asynchronous invocation returns StatusCode 202 and finishes immediately. Lambda runs the function in the background and retries up to 2 times on failure.
💡 Practical Tip: If you do not include
--cli-binary-format raw-in-base64-out, you will need to base64-encode the payload. Make sure to include this option with AWS CLI v2. You can setcli_binary_format = raw-in-base64-outin~/.aws/configso you do not have to include it every time.
11.5 Update Code (zip file)
# Step 1: Package your code as a zip file
cd /path/to/lambda-code
zip -r function.zip . -x "*.pyc" "__pycache__/*" ".git/*"
# Step 2: Update the code
aws lambda update-function-code \
--function-name order-processor \
--zip-file fileb://function.zip
{
"FunctionName": "order-processor",
"FunctionArn": "arn:aws:lambda:ap-northeast-2:111122223333:function:order-processor",
"Runtime": "python3.12",
"CodeSize": 2567890,
"LastModified": "2026-03-18T11:00:00.000+0000",
"LastUpdateStatus": "InProgress",
"State": "Active",
"CodeSha256": "YFgDgE1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9="
}
# Wait for the update to complete
aws lambda wait function-updated \
--function-name order-processor
# Verify completion
aws lambda get-function-configuration \
--function-name order-processor \
--query '{LastUpdate:LastUpdateStatus,CodeSha:CodeSha256}'
{
"LastUpdate": "Successful",
"CodeSha": "YFgDgE1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9="
}
⚠️ Caution: There are zip file size limits.
- Direct upload: 50MB (zip)
- Upload via S3: 250MB (after decompression)
- Container image: 10GB
For large packages, consider using Lambda Layers or the container image approach.
11.6 Event Source Mapping
# List event source mappings
aws lambda list-event-source-mappings \
--function-name order-processor
{
"EventSourceMappings": [
{
"UUID": "a1b2c3d4-5678-90ab-cdef-aaaaaaaaaaaa",
"EventSourceArn": "arn:aws:sqs:ap-northeast-2:111122223333:order-queue",
"FunctionArn": "arn:aws:lambda:ap-northeast-2:111122223333:function:order-processor",
"State": "Enabled",
"BatchSize": 10,
"MaximumBatchingWindowInSeconds": 5,
"LastProcessingResult": "OK",
"FunctionResponseTypes": [
"ReportBatchItemFailures"
],
"ScalingConfig": {
"MaximumConcurrency": 50
}
}
]
}
If LastProcessingResult is OK, everything is normal. No records processed means the queue is empty, and PROBLEM: ... means there is an error.
# Disable event source mapping (stop message intake during an incident)
aws lambda update-event-source-mapping \
--uuid a1b2c3d4-5678-90ab-cdef-aaaaaaaaaaaa \
--no-enabled
{
"UUID": "a1b2c3d4-5678-90ab-cdef-aaaaaaaaaaaa",
"State": "Disabling",
"LastProcessingResult": "OK"
}
💡 Practical Tip: If errors keep occurring on an SQS trigger, messages can be reprocessed endlessly, causing Lambda costs to skyrocket. Disable the event source mapping, analyze the messages accumulated in the DLQ (Dead Letter Queue), and then fix the problem.
11.7 Concurrency Settings
# Set reserved concurrency (guarantee dedicated slots for this function)
aws lambda put-function-concurrency \
--function-name order-processor \
--reserved-concurrent-executions 100
{
"ReservedConcurrentExecutions": 100
}
# Set provisioned concurrency (eliminate Cold Starts)
aws lambda put-provisioned-concurrency-config \
--function-name order-processor \
--qualifier prod \
--provisioned-concurrent-executions 20
{
"RequestedProvisionedConcurrentExecutions": 20,
"AvailableProvisionedConcurrentExecutions": 0,
"AllocatedProvisionedConcurrentExecutions": 0,
"Status": "IN_PROGRESS"
}
# Verify provisioning is complete
aws lambda get-provisioned-concurrency-config \
--function-name order-processor \
--qualifier prod
{
"RequestedProvisionedConcurrentExecutions": 20,
"AvailableProvisionedConcurrentExecutions": 20,
"AllocatedProvisionedConcurrentExecutions": 20,
"Status": "READY",
"LastModified": "2026-03-18T11:30:00.000+0000"
}
| Concurrency Type | Use Case | Cost |
|---|---|---|
| Reserved Concurrency | Ensures this function's concurrency cannot be taken by other functions | Free (pay only for what you use) |
| Provisioned Concurrency | Completely eliminates Cold Starts | Charged even while idle |
⚠️ Caution: The default concurrency limit per account is 1,000 per region. If you distribute reserved concurrency across multiple functions, the remaining functions will have less concurrency available. Setting it to
0completely blocks function invocations, which can be used as an emergency kill switch.
11.8 Lambda Troubleshooting
Checking Error Logs
# Query recent error logs
aws logs filter-log-events \
--log-group-name /aws/lambda/order-processor \
--filter-pattern "ERROR" \
--start-time $(date -d '1 hour ago' +%s)000 \
--limit 20 \
--query 'events[*].{time:timestamp,msg:message}' \
--output table
-------------------------------------------------------------------------------------------------------------------
| FilterLogEvents |
+-----------------------+------------------------------------------------------------------------------------------+
| time | msg |
+-----------------------+------------------------------------------------------------------------------------------+
| 1710726330123 | [ERROR] 2026-03-18T02:45:30.123Z KeyError: 'orderId' |
| 1710726330124 | Traceback (most recent call last): |
| 1710726330125 | File "/var/task/app.py", line 15, in handler |
| 1710726330126 | order_id = event['orderId'] |
| 1710726390456 | [ERROR] 2026-03-18T02:46:30.456Z botocore.exceptions.ClientError: An error occurred |
| | (ConditionalCheckFailedException) when calling the PutItem operation |
+-----------------------+------------------------------------------------------------------------------------------+
# Trace full logs by a specific RequestId
aws logs filter-log-events \
--log-group-name /aws/lambda/order-processor \
--filter-pattern "a1b2c3d4-5678-90ab-cdef-111122223333" \
--query 'events[*].message' \
--output text
START RequestId: a1b2c3d4-5678-90ab-cdef-111122223333 Version: $LATEST
[INFO] 2026-03-18T02:45:30.100Z a1b2c3d4-5678-90ab-cdef-111122223333 Processing order...
[ERROR] 2026-03-18T02:45:30.123Z a1b2c3d4-5678-90ab-cdef-111122223333 KeyError: 'orderId'
Traceback (most recent call last):
File "/var/task/app.py", line 15, in handler
order_id = event['orderId']
KeyError: 'orderId'
END RequestId: a1b2c3d4-5678-90ab-cdef-111122223333
REPORT RequestId: a1b2c3d4-5678-90ab-cdef-111122223333 Duration: 23.45 ms Billed Duration: 24 ms Memory Size: 512 MB Max Memory Used: 78 MB
Timeout / Memory Issues
# Find timeout occurrence logs
aws logs filter-log-events \
--log-group-name /aws/lambda/data-sync-batch \
--filter-pattern "Task timed out" \
--start-time $(date -d '24 hours ago' +%s)000 \
--query 'events[*].message' \
--output text
2026-03-17T15:30:00.000Z a1b2c3d4-5678-90ab-cdef-bbbbbbbbbbbb Task timed out after 900.10 seconds
# Analyze memory usage patterns (extract from REPORT lines)
aws logs filter-log-events \
--log-group-name /aws/lambda/order-processor \
--filter-pattern "REPORT" \
--start-time $(date -d '1 hour ago' +%s)000 \
--limit 10 \
--query 'events[*].message' \
--output text
REPORT RequestId: ... Duration: 234.56 ms Billed Duration: 235 ms Memory Size: 512 MB Max Memory Used: 89 MB Init Duration: 345.67 ms
REPORT RequestId: ... Duration: 45.12 ms Billed Duration: 46 ms Memory Size: 512 MB Max Memory Used: 91 MB
REPORT RequestId: ... Duration: 189.34 ms Billed Duration: 190 ms Memory Size: 512 MB Max Memory Used: 95 MB
REPORT RequestId: ... Duration: 56.78 ms Billed Duration: 57 ms Memory Size: 512 MB Max Memory Used: 88 MB
REPORT RequestId: ... Duration: 2345.67 ms Billed Duration: 2346 ms Memory Size: 512 MB Max Memory Used: 478 MB
The last request shows Max Memory Used: 478 MB which is close to Memory Size: 512 MB. In cases like this, an OOM (Out of Memory) error may occur soon.
💡 Practical Tip: If memory usage exceeds 80% of the allocated amount, it is recommended to increase the memory. In Lambda, increasing memory also proportionally increases CPU, so doubling the memory can cut execution time in half, resulting in similar costs. Use the AWS Lambda Power Tuning tool to find the optimal memory setting.
Cold Start Analysis
# Filter invocations with Init Duration (= Cold Start occurred)
aws logs filter-log-events \
--log-group-name /aws/lambda/order-processor \
--filter-pattern "Init Duration" \
--start-time $(date -d '6 hours ago' +%s)000 \
--query 'events[*].message' \
--output text
REPORT RequestId: ... Duration: 234.56 ms Billed Duration: 580 ms Memory Size: 512 MB Max Memory Used: 89 MB Init Duration: 345.67 ms
REPORT RequestId: ... Duration: 189.34 ms Billed Duration: 523 ms Memory Size: 512 MB Max Memory Used: 95 MB Init Duration: 334.12 ms
REPORT RequestId: ... Duration: 212.45 ms Billed Duration: 556 ms Memory Size: 512 MB Max Memory Used: 92 MB Init Duration: 343.89 ms
If Init Duration is 300ms or higher, Cold Starts are impacting performance.
| Cold Start Solution | Description | Cost |
|---|---|---|
| Provisioned Concurrency | Pre-warm instances in advance | Incurs standby costs |
| SnapStart (Java) | Reduces initialization time by 90% using JVM snapshots | Free |
| Reduce package size | Remove unnecessary dependencies, use Lambda Layers | Free |
| arm64 architecture | 20% cheaper with Graviton processors + faster initialization | 20% savings |
| Remove VPC | Being inside a VPC adds Cold Start time for ENI creation. Remove VPC if not needed | Free |
Resolving Task timed out
# Check the current timeout setting
aws lambda get-function-configuration \
--function-name data-sync-batch \
--query '{Timeout:Timeout,Memory:MemorySize}'
{
"Timeout": 900,
"Memory": 2048
}
# Increase the timeout (maximum is 900 seconds = 15 minutes)
aws lambda update-function-configuration \
--function-name data-sync-batch \
--timeout 900
# If already at the maximum (900 seconds) -> architecture change is needed
💡 Practical Tip: If 15 minutes is still not enough for Lambda, consider these alternatives.
- Step Functions: Split the job into multiple Lambda functions and chain them together
- ECS Fargate: Container tasks with no time limit
- SQS + Lambda: Split large jobs into smaller message units for parallel processing
- Lambda internal optimization: Reduce batch size, introduce parallel processing, reuse DB connection pools
12. ELB Commands
ELB (Elastic Load Balancing) is a service that distributes incoming traffic across multiple targets (EC2, ECS, Lambda). In the AWS CLI, you use elbv2 commands to manage ALB (Application Load Balancer) and NLB (Network Load Balancer). This is one of the most commonly used commands for troubleshooting.
12.1 List ALB/NLB
aws elbv2 describe-load-balancers \
--query 'LoadBalancers[*].{Name:LoadBalancerName,Type:Type,Scheme:Scheme,State:State.Code,DNSName:DNSName}' \
--output table
----------------------------------------------------------------------------------------------------------------------
| DescribeLoadBalancers |
+---------------------+------------------------------------------------------------------+-----------+--------+--------+
| Name | DNSName | Scheme | State | Type |
+---------------------+------------------------------------------------------------------+-----------+--------+--------+
| prod-api-alb | prod-api-alb-1234567890.ap-northeast-2.elb.amazonaws.com | internet-facing | active | application |
| prod-internal-alb | prod-internal-alb-0987654321.ap-northeast-2.elb.amazonaws.com | internal | active | application |
| prod-grpc-nlb | prod-grpc-nlb-abcdef1234.elb.ap-northeast-2.amazonaws.com | internal | active | network |
+---------------------+------------------------------------------------------------------+-----------+--------+--------+
| Field | Meaning |
|---|---|
Type: application | ALB (HTTP/HTTPS, path-based routing) |
Type: network | NLB (TCP/UDP, ultra-low latency) |
Scheme: internet-facing | Accessible from the public internet |
Scheme: internal | Accessible only from within the VPC |
12.2 Describe ALB in Detail
aws elbv2 describe-load-balancers \
--names prod-api-alb
{
"LoadBalancers": [
{
"LoadBalancerArn": "arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:loadbalancer/app/prod-api-alb/abcdef1234567890",
"DNSName": "prod-api-alb-1234567890.ap-northeast-2.elb.amazonaws.com",
"CanonicalHostedZoneId": "ZWKZPGTI48KDX",
"CreatedTime": "2025-09-01T10:00:00+09:00",
"LoadBalancerName": "prod-api-alb",
"Scheme": "internet-facing",
"VpcId": "vpc-0a1b2c3d4e5f00001",
"State": {
"Code": "active"
},
"Type": "application",
"AvailabilityZones": [
{
"ZoneName": "ap-northeast-2a",
"SubnetId": "subnet-0a1b2c3d4e5f00001"
},
{
"ZoneName": "ap-northeast-2c",
"SubnetId": "subnet-0a1b2c3d4e5f00002"
}
],
"SecurityGroups": [
"sg-0a1b2c3d4e5f00030"
],
"IpAddressType": "ipv4"
}
]
}
12.3 Describe Target Groups
# List target groups associated with a specific ALB
aws elbv2 describe-target-groups \
--load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:loadbalancer/app/prod-api-alb/abcdef1234567890 \
--query 'TargetGroups[*].{Name:TargetGroupName,Port:Port,Protocol:Protocol,TargetType:TargetType,HealthPath:HealthCheckPath}' \
--output table
---------------------------------------------------------------------------------------------
| DescribeTargetGroups |
+---------------------+---------------+----------+------------+-----------------------------+
| HealthPath | Name | Port | Protocol | TargetType |
+---------------------+---------------+----------+------------+-----------------------------+
| /health | prod-user-tg | 8080 | HTTP | ip |
| /api/health | prod-order-tg| 8080 | HTTP | ip |
| /actuator/health | prod-auth-tg | 8080 | HTTP | ip |
+---------------------+---------------+----------+------------+-----------------------------+
TargetType: ip is the type used for Fargate tasks (awsvpc mode). For EC2 it would show instance, and for Lambda it would show lambda.
12.4 Check Target Health
aws elbv2 describe-target-health \
--target-group-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:targetgroup/prod-user-tg/abcdef1234567890
{
"TargetHealthDescriptions": [
{
"Target": {
"Id": "10.0.3.42",
"Port": 8080,
"AvailabilityZone": "ap-northeast-2a"
},
"HealthCheckPort": "8080",
"TargetHealth": {
"State": "healthy"
}
},
{
"Target": {
"Id": "10.0.4.15",
"Port": 8080,
"AvailabilityZone": "ap-northeast-2c"
},
"HealthCheckPort": "8080",
"TargetHealth": {
"State": "healthy"
}
},
{
"Target": {
"Id": "10.0.3.78",
"Port": 8080,
"AvailabilityZone": "ap-northeast-2a"
},
"HealthCheckPort": "8080",
"TargetHealth": {
"State": "unhealthy",
"Reason": "Target.ResponseCodeMismatch",
"Description": "Health checks failed with these codes: [503]"
}
}
]
}
Here are the target health status codes and their meanings.
| State | Meaning |
|---|---|
healthy | Health check passed successfully |
unhealthy | Health check failed |
initial | Target was just registered and is waiting for the first health check |
draining | Being deregistered (existing connections are being drained) |
unavailable | Target availability cannot be determined |
unused | Target group is not associated with any listener/rule |
💡 Practical Tip: Make sure to check the
Reasonfield forunhealthytargets. Here are commonly seen causes.
Reason Cause Solution Target.ResponseCodeMismatchHealth check response is not the expected code (200) Investigate why the app's health endpoint is returning 503 Target.TimeoutNo response to the health check request Check security groups, app port, and timeout settings Target.FailedHealthChecksExceeded consecutive failure count Check application logs for errors Elb.InternalErrorALB internal error Check AWS service status
12.5 Describe Listeners
aws elbv2 describe-listeners \
--load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:loadbalancer/app/prod-api-alb/abcdef1234567890 \
--query 'Listeners[*].{Port:Port,Protocol:Protocol,DefaultAction:DefaultActions[0].Type,CertificateArn:Certificates[0].CertificateArn}'
[
{
"Port": 80,
"Protocol": "HTTP",
"DefaultAction": "redirect",
"CertificateArn": null
},
{
"Port": 443,
"Protocol": "HTTPS",
"DefaultAction": "forward",
"CertificateArn": "arn:aws:acm:ap-northeast-2:111122223333:certificate/a1b2c3d4-5678-90ab-cdef-111122223333"
}
]
Port 80 redirects to HTTPS, and port 443 handles the actual traffic -- a typical pattern.
12.6 Describe Listener Rules
# List rules for the HTTPS listener
aws elbv2 describe-rules \
--listener-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:listener/app/prod-api-alb/abcdef1234567890/1234567890abcdef \
--query 'Rules[*].{Priority:Priority,Conditions:Conditions[0].Values[0],Action:Actions[0].Type,Target:Actions[0].TargetGroupArn}' \
--output table
------------------------------------------------------------------------------------------------------------------------------------------------------------
| DescribeRules |
+----------+------------------------------------------+----------+-----------------------------------------------------------------------------------------+
| Priority | Conditions | Action | Target |
+----------+------------------------------------------+----------+-----------------------------------------------------------------------------------------+
| 1 | /api/users/* | forward | arn:aws:elasticloadbalancing:...:targetgroup/prod-user-tg/abcdef1234567890 |
| 2 | /api/orders/* | forward | arn:aws:elasticloadbalancing:...:targetgroup/prod-order-tg/fedcba0987654321 |
| 3 | /api/auth/* | forward | arn:aws:elasticloadbalancing:...:targetgroup/prod-auth-tg/123456abcdef7890 |
| default | None | fixed-response | None |
+----------+------------------------------------------+----------+-----------------------------------------------------------------------------------------+
This shows routing to different target groups (microservices) based on path. The default rule is the fallback that activates when no other rule matches.
# View rule details (full conditions and actions)
aws elbv2 describe-rules \
--listener-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:listener/app/prod-api-alb/abcdef1234567890/1234567890abcdef \
--query 'Rules[?Priority==`1`]'
[
{
"RuleArn": "arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:listener-rule/app/prod-api-alb/abcdef1234567890/1234567890abcdef/aabb1122",
"Priority": "1",
"Conditions": [
{
"Field": "path-pattern",
"Values": ["/api/users/*"],
"PathPatternConfig": {
"Values": ["/api/users/*"]
}
}
],
"Actions": [
{
"Type": "forward",
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:targetgroup/prod-user-tg/abcdef1234567890",
"Order": 1,
"ForwardConfig": {
"TargetGroups": [
{
"TargetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:targetgroup/prod-user-tg/abcdef1234567890",
"Weight": 1
}
],
"TargetGroupStickinessConfig": {
"Enabled": false
}
}
}
],
"IsDefault": false
}
]
12.7 ELB Troubleshooting
Analyzing Unhealthy Target Causes
# Step 1: Identify unhealthy targets
aws elbv2 describe-target-health \
--target-group-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:targetgroup/prod-user-tg/abcdef1234567890 \
--query 'TargetHealthDescriptions[?TargetHealth.State!=`healthy`].{IP:Target.Id,Port:Target.Port,State:TargetHealth.State,Reason:TargetHealth.Reason,Detail:TargetHealth.Description}'
[
{
"IP": "10.0.3.78",
"Port": 8080,
"State": "unhealthy",
"Reason": "Target.ResponseCodeMismatch",
"Detail": "Health checks failed with these codes: [503]"
}
]
# Step 2: Check the target group's health check settings
aws elbv2 describe-target-groups \
--target-group-arns arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:targetgroup/prod-user-tg/abcdef1234567890 \
--query 'TargetGroups[0].{Path:HealthCheckPath,Port:HealthCheckPort,Protocol:HealthCheckProtocol,Interval:HealthCheckIntervalSeconds,Timeout:HealthCheckTimeoutSeconds,Healthy:HealthyThresholdCount,Unhealthy:UnhealthyThresholdCount,Matcher:Matcher.HttpCode}'
{
"Path": "/health",
"Port": "8080",
"Protocol": "HTTP",
"Interval": 30,
"Timeout": 5,
"Healthy": 3,
"Unhealthy": 3,
"Matcher": "200"
}
# Step 3: Check security groups (whether the port from ALB to target is allowed)
aws ec2 describe-security-groups \
--group-ids sg-0a1b2c3d4e5f00001 \
--query 'SecurityGroups[0].IpPermissions[*].{Proto:IpProtocol,FromPort:FromPort,ToPort:ToPort,Source:UserIdGroupPairs[0].GroupId}' \
--output table
------------------------------------------------------------
| DescribeSecurityGroups |
+--------+----------+--------+-----------------------------+
| Proto | FromPort | ToPort | Source |
+--------+----------+--------+-----------------------------+
| tcp | 8080 | 8080 | sg-0a1b2c3d4e5f00030 |
+--------+----------+--------+-----------------------------+
Traffic from the ALB's security group (sg-0a1b2c3d4e5f00030) to the target's port 8080 must be allowed.
💡 Practical Tip: Here is an unhealthy target checklist.
- Verify the target application is actually running (ECS task status, EC2 instance status)
- Confirm the health check path (
/health) actually responds- Verify the health check response code matches the Matcher setting (200)
- Check that security groups allow traffic from ALB to the target port
- Confirm the target is in the correct subnet within the VPC
Tracking 502 Bad Gateway Errors
A 502 occurs when the ALB sent a request to the target but did not receive a valid response.
# Check ALB access logs in S3 (if enabled)
# First, verify whether logging is enabled
aws elbv2 describe-load-balancer-attributes \
--load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:loadbalancer/app/prod-api-alb/abcdef1234567890 \
--query 'Attributes[?Key==`access_logs.s3.enabled` || Key==`access_logs.s3.bucket` || Key==`access_logs.s3.prefix`]'
[
{
"Key": "access_logs.s3.enabled",
"Value": "true"
},
{
"Key": "access_logs.s3.bucket",
"Value": "prod-alb-access-logs"
},
{
"Key": "access_logs.s3.prefix",
"Value": "prod-api-alb"
}
]
# Check 502/504 occurrence trends via CloudWatch metrics
aws cloudwatch get-metric-statistics \
--namespace AWS/ApplicationELB \
--metric-name HTTPCode_ELB_502_Count \
--dimensions Name=LoadBalancer,Value=app/prod-api-alb/abcdef1234567890 \
--start-time 2026-03-18T00:00:00Z \
--end-time 2026-03-18T12:00:00Z \
--period 3600 \
--statistics Sum \
--query 'Datapoints[*].{Time:Timestamp,Count:Sum}' \
--output table
-------------------------------------------------
| GetMetricStatistics |
+---------------------------+--------------------+
| Time | Count |
+---------------------------+--------------------+
| 2026-03-18T01:00:00Z | 0.0 |
| 2026-03-18T02:00:00Z | 0.0 |
| 2026-03-18T03:00:00Z | 12.0 |
| 2026-03-18T04:00:00Z | 45.0 |
| 2026-03-18T05:00:00Z | 3.0 |
| 2026-03-18T06:00:00Z | 0.0 |
+---------------------------+--------------------+
502 errors were concentrated between 3-5 AM. You need to check whether there were deployments or scale-down events during this time.
💡 Practical Tip: Know the difference between 502 and 504.
- 502 Bad Gateway: The target sent a response but it was invalid, or the target connection was dropped. Typically occurs when the target app crashes or when old containers are terminated during deployment
- 504 Gateway Timeout: The target did not respond within the time limit. Occurs when the ALB idle timeout (default 60 seconds) is exceeded. If your target app has long processing times, you need to increase the ALB timeout
Resolving 504 Gateway Timeout
# Check the ALB idle timeout
aws elbv2 describe-load-balancer-attributes \
--load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:loadbalancer/app/prod-api-alb/abcdef1234567890 \
--query 'Attributes[?Key==`idle_timeout.timeout_seconds`]'
[
{
"Key": "idle_timeout.timeout_seconds",
"Value": "60"
}
]
# Increase the idle timeout to 120 seconds
aws elbv2 modify-load-balancer-attributes \
--load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:loadbalancer/app/prod-api-alb/abcdef1234567890 \
--attributes Key=idle_timeout.timeout_seconds,Value=120
{
"Attributes": [
{
"Key": "idle_timeout.timeout_seconds",
"Value": "120"
}
]
}
⚠️ Warning: Before increasing the ALB timeout, first check the request processing time of your target application. The priority should be fixing the root cause of slow responses (DB query optimization, external API timeouts, etc.). Simply increasing the ALB timeout blindly can cause slow connections to be held open for too long, degrading overall performance.
Debugging Listener Rules
When requests are not routing to the intended target group, you need to check the listener rule priorities and conditions.
# View all rule priorities and conditions at a glance
aws elbv2 describe-rules \
--listener-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:listener/app/prod-api-alb/abcdef1234567890/1234567890abcdef \
--query 'Rules[*].{Priority:Priority,Conditions:Conditions[*].{Field:Field,Values:Values[0]},ActionType:Actions[0].Type,TargetGroup:Actions[0].TargetGroupArn}' \
--output json
[
{
"Priority": "1",
"Conditions": [{"Field": "path-pattern", "Values": "/api/users/*"}],
"ActionType": "forward",
"TargetGroup": "arn:aws:elasticloadbalancing:...:targetgroup/prod-user-tg/abcdef1234567890"
},
{
"Priority": "2",
"Conditions": [{"Field": "path-pattern", "Values": "/api/orders/*"}],
"ActionType": "forward",
"TargetGroup": "arn:aws:elasticloadbalancing:...:targetgroup/prod-order-tg/fedcba0987654321"
},
{
"Priority": "3",
"Conditions": [{"Field": "host-header", "Values": "admin.example.com"}],
"ActionType": "forward",
"TargetGroup": "arn:aws:elasticloadbalancing:...:targetgroup/prod-admin-tg/112233aabbccddee"
},
{
"Priority": "default",
"Conditions": [],
"ActionType": "fixed-response",
"TargetGroup": null
}
]
Rules are matched in order of lowest Priority number first. If a request matches a different rule before the intended one, it will be routed to the wrong target.
💡 Practical Tip: Here is a listener rule debugging checklist.
- Verify rule priorities are correct (more specific paths should have higher priority)
- Check that path-pattern wildcards (
*) are correctly placed- Verify that the host-header condition domain matches the actual request
- Confirm that multiple conditions are combined with AND (all conditions in the same rule must be met for a match)
- Check that the default rule behavior is appropriate (return 404 vs forward)
Enabling Access Logs
# Enable access logs (saved to an S3 bucket)
aws elbv2 modify-load-balancer-attributes \
--load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-2:111122223333:loadbalancer/app/prod-api-alb/abcdef1234567890 \
--attributes \
Key=access_logs.s3.enabled,Value=true \
Key=access_logs.s3.bucket,Value=prod-alb-access-logs \
Key=access_logs.s3.prefix,Value=prod-api-alb
{
"Attributes": [
{
"Key": "access_logs.s3.enabled",
"Value": "true"
},
{
"Key": "access_logs.s3.bucket",
"Value": "prod-alb-access-logs"
},
{
"Key": "access_logs.s3.prefix",
"Value": "prod-api-alb"
}
]
}
⚠️ Warning: The S3 bucket that stores access logs must have write permissions for the ELB service principal. The ELB account ID for the Seoul region (ap-northeast-2) is
600734575887.
# S3 bucket policy example (this should already be configured)
aws s3api get-bucket-policy \
--bucket prod-alb-access-logs \
--query 'Policy' \
--output text | python -m json.tool
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::600734575887:root"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::prod-alb-access-logs/prod-api-alb/AWSLogs/111122223333/*"
}
]
}
# Check access logs (saved to S3 every 5 minutes)
aws s3 ls s3://prod-alb-access-logs/prod-api-alb/AWSLogs/111122223333/elasticloadbalancing/ap-northeast-2/2026/03/18/
2026-03-18 10:05:30 125432 111122223333_elasticloadbalancing_ap-northeast-2_app.prod-api-alb.abcdef1234567890_20260318T0100Z_52.78.xxx.xxx_abcdef12.log.gz
2026-03-18 10:10:30 134567 111122223333_elasticloadbalancing_ap-northeast-2_app.prod-api-alb.abcdef1234567890_20260318T0105Z_52.78.xxx.xxx_fedcba98.log.gz
💡 Practical Tip: Access logs are your last resort for troubleshooting. They let you identify the exact cause of 502/504 errors (which target sent what response), processing times for slow requests, and abnormal traffic patterns. However, since log volume can be large, the recommended approach is to monitor with CloudWatch metrics day-to-day and only analyze access logs when issues arise. Using Athena allows you to query S3-stored access logs with SQL, which is very convenient.
Summary: Part 4 Key Command Reference
ECS Command Summary
# Status checks
aws ecs list-clusters
aws ecs describe-clusters --clusters <name> --include STATISTICS
aws ecs list-services --cluster <name>
aws ecs describe-services --cluster <name> --services <svc>
aws ecs list-tasks --cluster <name> --service-name <svc> --desired-status STOPPED
aws ecs describe-tasks --cluster <name> --tasks <task-id>
aws ecs describe-task-definition --task-definition <family:revision>
# Operations
aws ecs update-service --cluster <name> --service <svc> --force-new-deployment
aws ecs update-service --cluster <name> --service <svc> --desired-count <n>
aws ecs wait services-stable --cluster <name> --services <svc>
EKS Command Summary
# Status checks
aws eks list-clusters
aws eks describe-cluster --name <name>
aws eks list-nodegroups --cluster-name <name>
aws eks describe-nodegroup --cluster-name <name> --nodegroup-name <ng>
# Operations
aws eks update-kubeconfig --name <name> --region ap-northeast-2
aws eks update-nodegroup-config --cluster-name <name> --nodegroup-name <ng> --scaling-config minSize=X,maxSize=Y,desiredSize=Z
Lambda Command Summary
# Status checks
aws lambda list-functions
aws lambda get-function --function-name <name>
aws lambda get-function-configuration --function-name <name>
aws lambda list-event-source-mappings --function-name <name>
# Operations
aws lambda invoke --function-name <name> --payload '<json>' --cli-binary-format raw-in-base64-out response.json
aws lambda update-function-code --function-name <name> --zip-file fileb://function.zip
aws lambda put-function-concurrency --function-name <name> --reserved-concurrent-executions <n>
# Log analysis
aws logs filter-log-events --log-group-name /aws/lambda/<name> --filter-pattern "ERROR"
aws logs filter-log-events --log-group-name /aws/lambda/<name> --filter-pattern "Task timed out"
aws logs filter-log-events --log-group-name /aws/lambda/<name> --filter-pattern "Init Duration"
ELB Command Summary
# Status checks
aws elbv2 describe-load-balancers
aws elbv2 describe-target-groups --load-balancer-arn <arn>
aws elbv2 describe-target-health --target-group-arn <arn>
aws elbv2 describe-listeners --load-balancer-arn <arn>
aws elbv2 describe-rules --listener-arn <arn>
# Operations
aws elbv2 describe-load-balancer-attributes --load-balancer-arn <arn>
aws elbv2 modify-load-balancer-attributes --load-balancer-arn <arn> --attributes Key=idle_timeout.timeout_seconds,Value=120
aws elbv2 modify-load-balancer-attributes --load-balancer-arn <arn> --attributes Key=access_logs.s3.enabled,Value=true Key=access_logs.s3.bucket,Value=<bucket>
# Troubleshooting metrics
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_ELB_502_Count --dimensions Name=LoadBalancer,Value=app/<name>/<id> --start-time <start> --end-time <end> --period 3600 --statistics Sum
Part 5 will cover CLI commands for the remaining services including Route 53, CloudFront, CloudFormation, and more.
AWS CLI Commands Part 5 - Monitoring, DNS, CDN, Messaging
Once you have built your infrastructure, the next steps are monitoring it, connecting domains, delivering content quickly, and exchanging messages between services. This part covers the essential CLI commands for CloudWatch, Route 53, CloudFront, SQS, and SNS.
13. CloudWatch Commands
CloudWatch is a monitoring service responsible for metric collection, alarm configuration, and log management for AWS resources. Think of it as an "alert system" that notifies you before your server goes down.
Listing Metrics (list-metrics)
Query the list of available metrics within a specific namespace.
# List EC2-related metrics
aws cloudwatch list-metrics \
--namespace "AWS/EC2" \
--region ap-northeast-2
{
"Metrics": [
{
"Namespace": "AWS/EC2",
"MetricName": "CPUUtilization",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-0a1b2c3d4e5f67890"
}
]
},
{
"Namespace": "AWS/EC2",
"MetricName": "NetworkIn",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-0a1b2c3d4e5f67890"
}
]
},
{
"Namespace": "AWS/EC2",
"MetricName": "StatusCheckFailed",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-0a1b2c3d4e5f67890"
}
]
},
{
"Namespace": "AWS/EC2",
"MetricName": "DiskReadOps",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-0a1b2c3d4e5f67890"
}
]
}
]
}
When you specify a service using --namespace, it filters and shows only that service's metrics. Besides EC2, there are many other namespaces like AWS/RDS, AWS/ELB, AWS/Lambda, and more.
You can also filter by a specific metric name.
# Query only the CPUUtilization metric
aws cloudwatch list-metrics \
--namespace "AWS/EC2" \
--metric-name "CPUUtilization" \
--region ap-northeast-2
{
"Metrics": [
{
"Namespace": "AWS/EC2",
"MetricName": "CPUUtilization",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-0a1b2c3d4e5f67890"
}
]
},
{
"Namespace": "AWS/EC2",
"MetricName": "CPUUtilization",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-0b9a8c7d6e5f43210"
}
]
}
]
}
You can see that separate metrics exist for each instance. The InstanceId in Dimensions tells you which instance a metric belongs to.
Querying Metric Statistics (get-metric-statistics)
Query the actual numeric data for a specific metric. Let's look at EC2 CPU utilization over the last hour at 5-minute intervals.
# Query CPU utilization statistics over the last hour (5-minute intervals)
aws cloudwatch get-metric-statistics \
--namespace "AWS/EC2" \
--metric-name "CPUUtilization" \
--dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f67890 \
--start-time 2026-03-18T08:00:00Z \
--end-time 2026-03-18T09:00:00Z \
--period 300 \
--statistics Average Maximum \
--region ap-northeast-2
{
"Label": "CPUUtilization",
"Datapoints": [
{
"Timestamp": "2026-03-18T08:00:00+00:00",
"Average": 12.34,
"Maximum": 23.5,
"Unit": "Percent"
},
{
"Timestamp": "2026-03-18T08:05:00+00:00",
"Average": 15.67,
"Maximum": 31.2,
"Unit": "Percent"
},
{
"Timestamp": "2026-03-18T08:10:00+00:00",
"Average": 78.92,
"Maximum": 95.1,
"Unit": "Percent"
},
{
"Timestamp": "2026-03-18T08:15:00+00:00",
"Average": 45.23,
"Maximum": 67.8,
"Unit": "Percent"
},
{
"Timestamp": "2026-03-18T08:20:00+00:00",
"Average": 8.45,
"Maximum": 12.3,
"Unit": "Percent"
}
]
}
--period 300 means data is aggregated at 5-minute (300-second) intervals. For --statistics, you can specify Average, Maximum, Minimum, Sum, or SampleCount. Looking at the results, CPU spiked to 78% at 08:10 and then came back down.
💡 Practical Tip:
--periodmust be set to a multiple of 60. Basic monitoring collects data at 5-minute (300-second) intervals. If you enable Detailed Monitoring, you can receive data at 1-minute (60-second) intervals. Note that Detailed Monitoring incurs additional costs.
Querying ALB Metrics
You can also check ALB request counts and response times through CloudWatch.
# Query ALB 5xx error count
aws cloudwatch get-metric-statistics \
--namespace "AWS/ApplicationELB" \
--metric-name "HTTPCode_ELB_5XX_Count" \
--dimensions Name=LoadBalancer,Value=app/my-web-alb/50dc6c495c0c9188 \
--start-time 2026-03-18T06:00:00Z \
--end-time 2026-03-18T09:00:00Z \
--period 3600 \
--statistics Sum \
--region ap-northeast-2
{
"Label": "HTTPCode_ELB_5XX_Count",
"Datapoints": [
{
"Timestamp": "2026-03-18T06:00:00+00:00",
"Sum": 0.0,
"Unit": "Count"
},
{
"Timestamp": "2026-03-18T07:00:00+00:00",
"Sum": 3.0,
"Unit": "Count"
},
{
"Timestamp": "2026-03-18T08:00:00+00:00",
"Sum": 127.0,
"Unit": "Count"
}
]
}
At 08:00, there were 127 5xx errors. This signals that something was wrong with the backend servers. If you connect this to an alarm, you can quickly detect incidents.
Creating Alarms (put-metric-alarm)
Let's set up an alarm that triggers when CPU utilization exceeds 80%.
# Create an alarm that sends SNS notification when CPU exceeds 80%
aws cloudwatch put-metric-alarm \
--alarm-name "high-cpu-alarm" \
--alarm-description "EC2 CPU utilization exceeds 80% alarm" \
--metric-name "CPUUtilization" \
--namespace "AWS/EC2" \
--statistic Average \
--period 300 \
--threshold 80 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 2 \
--alarm-actions "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts" \
--ok-actions "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts" \
--dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f67890 \
--treat-missing-data notBreaching \
--region ap-northeast-2
If it succeeds with no output, the alarm has been created. Let's go through the key parameters one by one.
| Parameter | Meaning |
|---|---|
--period 300 | Check metrics every 5 minutes |
--threshold 80 | 80% threshold |
--comparison-operator GreaterThanThreshold | Trigger when threshold is exceeded |
--evaluation-periods 2 | Trigger alarm after 2 consecutive breaches (10 minutes) |
--alarm-actions | Send notification to SNS topic when entering ALARM state |
--ok-actions | Also send notification when returning to OK state (recovery notification) |
--treat-missing-data notBreaching | Treat missing data as normal |
💡 Practical Tip: Setting
--evaluation-periods 2prevents the alarm from triggering on temporary spikes. It ignores cases where CPU briefly spikes and immediately drops back down, and only triggers when there is genuinely sustained high load.
⚠️ Caution: The SNS topic specified in
--alarm-actionsmust be created in advance. If you specify a non-existent ARN, the alarm will be created but no notifications will be sent.
Creating an ALB 5xx Error Alarm
# Alarm when ALB 5xx errors exceed 10 in 5 minutes
aws cloudwatch put-metric-alarm \
--alarm-name "alb-5xx-alarm" \
--alarm-description "ALB 5xx errors exceed 10" \
--metric-name "HTTPCode_ELB_5XX_Count" \
--namespace "AWS/ApplicationELB" \
--statistic Sum \
--period 300 \
--threshold 10 \
--comparison-operator GreaterThanOrEqualToThreshold \
--evaluation-periods 1 \
--alarm-actions "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts" \
--dimensions Name=LoadBalancer,Value=app/my-web-alb/50dc6c495c0c9188 \
--treat-missing-data notBreaching \
--region ap-northeast-2
Since 5xx errors indicate server-side issues, immediate awareness is important, so we set --evaluation-periods 1 to trigger the alarm on the very first breach.
Checking Alarm Status (describe-alarms)
Let's check the current status of our alarms.
# Check all alarm statuses
aws cloudwatch describe-alarms \
--region ap-northeast-2
{
"MetricAlarms": [
{
"AlarmName": "high-cpu-alarm",
"AlarmDescription": "EC2 CPU utilization exceeds 80% alarm",
"StateValue": "OK",
"StateReason": "Threshold Crossed: 1 out of the last 2 datapoints [12.34 (18/03/26 08:25:00)] was not greater than the threshold (80.0) (minimum 2 datapoints for ALARM -> OK transition).",
"StateUpdatedTimestamp": "2026-03-18T08:30:00+00:00",
"MetricName": "CPUUtilization",
"Namespace": "AWS/EC2",
"Statistic": "Average",
"Period": 300,
"EvaluationPeriods": 2,
"Threshold": 80.0,
"ComparisonOperator": "GreaterThanThreshold",
"AlarmActions": [
"arn:aws:sns:ap-northeast-2:123456789012:ops-alerts"
],
"TreatMissingData": "notBreaching"
},
{
"AlarmName": "alb-5xx-alarm",
"AlarmDescription": "ALB 5xx errors exceed 10",
"StateValue": "ALARM",
"StateReason": "Threshold Crossed: 1 datapoint [127.0 (18/03/26 08:00:00)] was greater than or equal to the threshold (10.0).",
"StateUpdatedTimestamp": "2026-03-18T08:05:00+00:00",
"MetricName": "HTTPCode_ELB_5XX_Count",
"Namespace": "AWS/ApplicationELB",
"Statistic": "Sum",
"Period": 300,
"EvaluationPeriods": 1,
"Threshold": 10.0,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"TreatMissingData": "notBreaching"
}
]
}
Looking at StateValue, the CPU alarm is in OK state and the ALB 5xx alarm is in ALARM state. StateReason provides the detailed reason why each alarm is in its current state.
You can also filter to see only alarms in the ALARM state.
# Query only alarms in ALARM state
aws cloudwatch describe-alarms \
--state-value ALARM \
--region ap-northeast-2
{
"MetricAlarms": [
{
"AlarmName": "alb-5xx-alarm",
"StateValue": "ALARM",
"StateReason": "Threshold Crossed: 1 datapoint [127.0 (18/03/26 08:00:00)] was greater than or equal to the threshold (10.0).",
"StateUpdatedTimestamp": "2026-03-18T08:05:00+00:00"
}
]
}
To query a specific alarm by name, use --alarm-names.
# Query a specific alarm
aws cloudwatch describe-alarms \
--alarm-names "high-cpu-alarm" \
--region ap-northeast-2
Listing Log Groups
Check the list of log groups in CloudWatch Logs.
# List log groups
aws logs describe-log-groups \
--region ap-northeast-2
{
"logGroups": [
{
"logGroupName": "/aws/lambda/my-api-handler",
"creationTime": 1710700800000,
"metricFilterCount": 0,
"arn": "arn:aws:logs:ap-northeast-2:123456789012:log-group:/aws/lambda/my-api-handler:*",
"storedBytes": 52428800,
"retentionInDays": 14
},
{
"logGroupName": "/aws/ecs/my-web-service",
"creationTime": 1710614400000,
"metricFilterCount": 1,
"arn": "arn:aws:logs:ap-northeast-2:123456789012:log-group:/aws/ecs/my-web-service:*",
"storedBytes": 1073741824,
"retentionInDays": 30
},
{
"logGroupName": "/aws/rds/cluster/my-aurora/error",
"creationTime": 1710528000000,
"metricFilterCount": 0,
"arn": "arn:aws:logs:ap-northeast-2:123456789012:log-group:/aws/rds/cluster/my-aurora/error:*",
"storedBytes": 10485760
}
]
}
Log groups without retentionInDays set are stored indefinitely. This means storage costs keep accumulating, so you should always set a retention period.
⚠️ Caution: Check the
storedBytes. If logs have accumulated to over 1GB, you could be incurring significant monthly storage costs. It is best to clean up unnecessary log groups and set retention periods.
To set a log retention period, use the following command.
# Set log retention period to 30 days
aws logs put-retention-policy \
--log-group-name "/aws/ecs/my-web-service" \
--retention-in-days 30 \
--region ap-northeast-2
Listing Log Streams
Check the list of streams within a specific log group. A stream represents an individual source (instance, container, etc.) that sends logs.
# List recent log streams (sorted by last event time)
aws logs describe-log-streams \
--log-group-name "/aws/ecs/my-web-service" \
--order-by LastEventTime \
--descending \
--limit 5 \
--region ap-northeast-2
{
"logStreams": [
{
"logStreamName": "ecs/my-web-service/a1b2c3d4e5f6",
"creationTime": 1710756000000,
"firstEventTimestamp": 1710756001000,
"lastEventTimestamp": 1710759600000,
"lastIngestionTime": 1710759605000,
"uploadSequenceToken": "49642675913...",
"arn": "arn:aws:logs:ap-northeast-2:123456789012:log-group:/aws/ecs/my-web-service:log-stream:ecs/my-web-service/a1b2c3d4e5f6",
"storedBytes": 2097152
},
{
"logStreamName": "ecs/my-web-service/f6e5d4c3b2a1",
"creationTime": 1710752400000,
"firstEventTimestamp": 1710752401000,
"lastEventTimestamp": 1710759300000,
"lastIngestionTime": 1710759305000,
"storedBytes": 1572864
}
]
}
You can also read recent log events directly.
# Query recent log events from a specific log stream
aws logs get-log-events \
--log-group-name "/aws/ecs/my-web-service" \
--log-stream-name "ecs/my-web-service/a1b2c3d4e5f6" \
--limit 5 \
--region ap-northeast-2
{
"events": [
{
"timestamp": 1710759590000,
"message": "2026-03-18T09:59:50.000Z INFO [main] Starting health check...",
"ingestionTime": 1710759595000
},
{
"timestamp": 1710759591000,
"message": "2026-03-18T09:59:51.000Z INFO [main] Health check passed. Status: 200",
"ingestionTime": 1710759596000
},
{
"timestamp": 1710759595000,
"message": "2026-03-18T09:59:55.000Z ERROR [worker-3] Connection to RDS timed out after 30000ms",
"ingestionTime": 1710759600000
},
{
"timestamp": 1710759596000,
"message": "2026-03-18T09:59:56.000Z WARN [worker-3] Retrying database connection (attempt 2/3)...",
"ingestionTime": 1710759601000
},
{
"timestamp": 1710759598000,
"message": "2026-03-18T09:59:58.000Z INFO [worker-3] Database connection re-established",
"ingestionTime": 1710759603000
}
],
"nextForwardToken": "f/37419234...",
"nextBackwardToken": "b/37419230..."
}
You can see actual application logs here. There are traces of an RDS connection timeout that was recovered after a retry.
Running CloudWatch Logs Insights Queries
Logs Insights is a powerful analytics tool that lets you query logs like SQL. From the CLI, it runs in two steps: start the query, then retrieve the results.
# Start an Insights query to search for ERROR logs from the last hour
aws logs start-query \
--log-group-name "/aws/ecs/my-web-service" \
--start-time $(date -d '1 hour ago' +%s) \
--end-time $(date +%s) \
--query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20' \
--region ap-northeast-2
{
"queryId": "12345678-abcd-efgh-ijkl-567890abcdef"
}
Once a queryId is returned, use it to retrieve the results. Since queries run asynchronously, they need a little time to complete.
# Retrieve query results
aws logs get-query-results \
--query-id "12345678-abcd-efgh-ijkl-567890abcdef" \
--region ap-northeast-2
{
"results": [
[
{
"field": "@timestamp",
"value": "2026-03-18 09:59:55.000"
},
{
"field": "@message",
"value": "2026-03-18T09:59:55.000Z ERROR [worker-3] Connection to RDS timed out after 30000ms"
}
],
[
{
"field": "@timestamp",
"value": "2026-03-18 09:45:12.000"
},
{
"field": "@message",
"value": "2026-03-18T09:45:12.000Z ERROR [worker-1] OutOfMemoryError: Java heap space"
}
],
[
{
"field": "@timestamp",
"value": "2026-03-18 09:30:03.000"
},
{
"field": "@message",
"value": "2026-03-18T09:30:03.000Z ERROR [api-handler] Request timeout: POST /api/orders exceeded 30s limit"
}
]
],
"statistics": {
"recordsMatched": 15.0,
"recordsScanned": 48523.0,
"bytesScanned": 52428800.0
},
"status": "Complete"
}
If status is Running, the query is still in progress and you need to call it again after a short wait. Complete means the results are ready. Looking at statistics, it scanned 48,523 log records and found 15 ERROR logs.
💡 Practical Tip: Here is a collection of useful Logs Insights query examples.
# Aggregate error count by time interval
--query-string 'stats count(*) as errorCount by bin(5m) | filter @message like /ERROR/'
# Top 10 slowest response times
--query-string 'fields @timestamp, @message | parse @message "duration=*ms" as duration | sort duration desc | limit 10'
# Trace request logs from a specific IP
--query-string 'fields @timestamp, @message | filter @message like /192.168.1.100/ | sort @timestamp desc'
CloudWatch Troubleshooting: When an Alarm Shows INSUFFICIENT_DATA
Sometimes after creating an alarm, the status shows as INSUFFICIENT_DATA.
# Check alarms in INSUFFICIENT_DATA state
aws cloudwatch describe-alarms \
--state-value INSUFFICIENT_DATA \
--region ap-northeast-2
{
"MetricAlarms": [
{
"AlarmName": "rds-cpu-alarm",
"StateValue": "INSUFFICIENT_DATA",
"StateReason": "Unchecked: Initial alarm creation",
"MetricName": "CPUUtilization",
"Namespace": "AWS/RDS",
"Dimensions": [
{
"Name": "DBInstanceIdentifier",
"Value": "my-database"
}
],
"TreatMissingData": "missing"
}
]
}
Solutions by cause:
1) Alarm was just created: Not enough data has been collected yet. Once enough data accumulates for the number of evaluation-periods, the state will transition. Wait about 5-10 minutes.
2) Instance is stopped: Stopped EC2 instances or deleted RDS instances do not generate metrics. In this case, the treat-missing-data setting is key.
# treat-missing-data option behaviors:
# missing -> Maintain INSUFFICIENT_DATA state (default)
# notBreaching -> Treat missing data as OK
# breaching -> Treat missing data as ALARM
# ignore -> Maintain current state
# Change alarm's treat-missing-data to notBreaching
aws cloudwatch put-metric-alarm \
--alarm-name "rds-cpu-alarm" \
--metric-name "CPUUtilization" \
--namespace "AWS/RDS" \
--statistic Average \
--period 300 \
--threshold 80 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 2 \
--dimensions Name=DBInstanceIdentifier,Value=my-database \
--treat-missing-data notBreaching \
--alarm-actions "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts" \
--region ap-northeast-2
3) Namespace or Dimensions are incorrect: Use list-metrics to verify that the metric name and Dimensions values match the actual resource.
💡 Practical Tip: In most production environments,
--treat-missing-data notBreachingis recommended. It prevents unnecessary alarms from firing when a server is properly shut down. However, for alarms like StatusCheckFailed, you should set it tobreachingso you can detect when an instance is down.
14. Route 53 Commands
Route 53 is AWS's managed DNS service. It connects domain names to IP addresses or AWS resources.
Listing Hosted Zones
Query the list of registered Hosted Zones.
# List hosted zones
aws route53 list-hosted-zones
{
"HostedZones": [
{
"Id": "/hostedzone/Z0123456789ABCDEFGHIJ",
"Name": "myapp.com.",
"CallerReference": "2026-01-15T10:30:00Z",
"Config": {
"Comment": "Production domain",
"PrivateZone": false
},
"ResourceRecordSetCount": 8
},
{
"Id": "/hostedzone/Z9876543210KLMNOPQRST",
"Name": "internal.myapp.com.",
"CallerReference": "2026-02-01T14:00:00Z",
"Config": {
"Comment": "Internal DNS for VPC",
"PrivateZone": true
},
"ResourceRecordSetCount": 15
}
]
}
PrivateZone: false means it is a public DNS zone accessible from the internet, while true is a private DNS zone used only within a VPC. The trailing dot (.) at the end of domain names is standard DNS notation.
💡 Practical Tip: Route 53 is a global service, so the
--regionoption is not needed. You get the same results regardless of which region you call from.
Listing DNS Records
Query all DNS records in a specific hosted zone.
# List records in a hosted zone
aws route53 list-resource-record-sets \
--hosted-zone-id Z0123456789ABCDEFGHIJ
{
"ResourceRecordSets": [
{
"Name": "myapp.com.",
"Type": "NS",
"TTL": 172800,
"ResourceRecords": [
{ "Value": "ns-1234.awsdns-56.org." },
{ "Value": "ns-789.awsdns-01.net." },
{ "Value": "ns-456.awsdns-23.co.uk." },
{ "Value": "ns-012.awsdns-78.com." }
]
},
{
"Name": "myapp.com.",
"Type": "SOA",
"TTL": 900,
"ResourceRecords": [
{ "Value": "ns-1234.awsdns-56.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400" }
]
},
{
"Name": "myapp.com.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "Z3AADJGX6KTTL2",
"DNSName": "dualstack.my-web-alb-1234567890.ap-northeast-2.elb.amazonaws.com.",
"EvaluateTargetHealth": true
}
},
{
"Name": "www.myapp.com.",
"Type": "CNAME",
"TTL": 300,
"ResourceRecords": [
{ "Value": "myapp.com" }
]
},
{
"Name": "api.myapp.com.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "Z3AADJGX6KTTL2",
"DNSName": "dualstack.my-api-alb-0987654321.ap-northeast-2.elb.amazonaws.com.",
"EvaluateTargetHealth": true
}
}
]
}
NS and SOA records are automatically created when the hosted zone is created. myapp.com is connected to an ALB via an Alias, and www.myapp.com points to the root domain via a CNAME.
Creating/Updating Records (UPSERT)
When creating or updating records in Route 53, you use a JSON file. UPSERT is a convenient action that creates a record if it doesn't exist, or updates it if it does.
First, prepare the JSON file.
# dns-change.json file contents
cat << 'EOF' > dns-change.json
{
"Comment": "A record for staging environment",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "staging.myapp.com",
"Type": "A",
"TTL": 300,
"ResourceRecords": [
{
"Value": "3.36.100.50"
}
]
}
}
]
}
EOF
# Apply DNS record change
aws route53 change-resource-record-sets \
--hosted-zone-id Z0123456789ABCDEFGHIJ \
--change-batch file://dns-change.json
{
"ChangeInfo": {
"Id": "/change/C0123456789ABCDEF",
"Status": "PENDING",
"SubmittedAt": "2026-03-18T10:00:00.000Z",
"Comment": "A record for staging environment"
}
}
Status: PENDING means the change has been submitted. DNS changes take time to propagate (usually within 60 seconds, up to 48 hours).
To check the change status, use the following command.
# Check DNS change status
aws route53 get-change \
--id /change/C0123456789ABCDEF
{
"ChangeInfo": {
"Id": "/change/C0123456789ABCDEF",
"Status": "INSYNC",
"SubmittedAt": "2026-03-18T10:00:00.000Z",
"Comment": "A record for staging environment"
}
}
When it changes to INSYNC, the change has been fully applied to all Route 53 name servers.
💡 Practical Tip: There are three values you can use for
Action:
CREATE- Create a record (errors if it already exists)DELETE- Delete a recordUPSERT- Update if exists, create if not (the safest and most commonly used)
Creating Alias Records (ALB Connection)
When connecting a domain to an ALB, you use an Alias record. Unlike regular A records, Alias records incur no additional DNS query costs.
# alias-record.json file contents
cat << 'EOF' > alias-record.json
{
"Comment": "Alias record pointing to ALB",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "app.myapp.com",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "ZWKZPGTI48KDX",
"DNSName": "dualstack.my-web-alb-1234567890.ap-northeast-2.elb.amazonaws.com",
"EvaluateTargetHealth": true
}
}
}
]
}
EOF
# Apply Alias record
aws route53 change-resource-record-sets \
--hosted-zone-id Z0123456789ABCDEFGHIJ \
--change-batch file://alias-record.json
{
"ChangeInfo": {
"Id": "/change/C9876543210FEDCBA",
"Status": "PENDING",
"SubmittedAt": "2026-03-18T10:05:00.000Z",
"Comment": "Alias record pointing to ALB"
}
}
⚠️ Caution: The
HostedZoneIdis a fixed value per region where the ALB is located. You must specify it manually. The ALB hosted zone ID for the ap-northeast-2 (Seoul) region isZWKZPGTI48KDX. You can find this value in theCanonicalHostedZoneIdfield fromdescribe-load-balancers.
# Check the ALB's hosted zone ID
aws elbv2 describe-load-balancers \
--names my-web-alb \
--query "LoadBalancers[0].CanonicalHostedZoneId" \
--output text \
--region ap-northeast-2
ZWKZPGTI48KDX
DNS Response Testing (test-dns-answer)
You can test how Route 53 responds to DNS queries. This is useful for verifying that your configuration is correct before actual DNS propagation.
# DNS response test
aws route53 test-dns-answer \
--hosted-zone-id Z0123456789ABCDEFGHIJ \
--record-name "app.myapp.com" \
--record-type A
{
"Nameserver": "ns-1234.awsdns-56.org",
"RecordName": "app.myapp.com.",
"RecordType": "A",
"RecordData": [
"3.36.100.50",
"3.36.100.51"
],
"ResponseCode": "NOERROR",
"Protocol": "UDP"
}
ResponseCode: NOERROR means it is responding normally. RecordData shows the ALB's IP addresses (the result of the Alias record resolving to the ALB's actual IPs).
# CNAME record test
aws route53 test-dns-answer \
--hosted-zone-id Z0123456789ABCDEFGHIJ \
--record-name "www.myapp.com" \
--record-type CNAME
{
"Nameserver": "ns-1234.awsdns-56.org",
"RecordName": "www.myapp.com.",
"RecordType": "CNAME",
"RecordData": [
"myapp.com"
],
"ResponseCode": "NOERROR",
"Protocol": "UDP"
}
Route 53 Troubleshooting: Verifying DNS Propagation
If you changed a DNS record but the old IP is still responding, you need to check the propagation status.
# 1. Check Route 53 internal status (verify it's INSYNC)
aws route53 get-change --id /change/C0123456789ABCDEF
{
"ChangeInfo": {
"Id": "/change/C0123456789ABCDEF",
"Status": "INSYNC",
"SubmittedAt": "2026-03-18T10:00:00.000Z"
}
}
If it shows INSYNC but still isn't working, it could be a local DNS cache issue.
# 2. Query Route 53 name server directly with dig (bypass local cache)
dig @ns-1234.awsdns-56.org staging.myapp.com A
# 3. Verify with other public DNS servers
dig @8.8.8.8 staging.myapp.com A
dig @1.1.1.1 staging.myapp.com A
💡 Practical Tip: If the TTL is set to a long value, the old record stays in cache for a long time. A safe pattern is to lower the TTL to 60 seconds before making changes, and then restore it to 300-3600 seconds after the change is complete.
Route 53 Troubleshooting: Checking Health Check Status
# List Health Checks
aws route53 list-health-checks
{
"HealthChecks": [
{
"Id": "abcd1234-5678-efgh-ijkl-mnop9012qrst",
"CallerReference": "health-check-api-2026",
"HealthCheckConfig": {
"IPAddress": "3.36.100.50",
"Port": 443,
"Type": "HTTPS",
"ResourcePath": "/health",
"FullyQualifiedDomainName": "api.myapp.com",
"RequestInterval": 30,
"FailureThreshold": 3
},
"HealthCheckVersion": 1
}
]
}
# Check Health Check status
aws route53 get-health-check-status \
--health-check-id abcd1234-5678-efgh-ijkl-mnop9012qrst
{
"HealthCheckObservations": [
{
"Region": "us-east-1",
"IPAddress": "15.177.2.100",
"StatusReport": {
"Status": "Success: HTTP Status Code 200, OK",
"CheckedTime": "2026-03-18T09:58:30.000Z"
}
},
{
"Region": "eu-west-1",
"IPAddress": "15.177.10.50",
"StatusReport": {
"Status": "Failure: Connection timed out",
"CheckedTime": "2026-03-18T09:58:25.000Z"
}
},
{
"Region": "ap-southeast-1",
"IPAddress": "15.177.18.200",
"StatusReport": {
"Status": "Success: HTTP Status Code 200, OK",
"CheckedTime": "2026-03-18T09:58:28.000Z"
}
}
]
}
Health Checks are performed from multiple regions, and some regions are failing. If eu-west-1 is timing out, it could be that the security group or network configuration is not allowing the Health Check IP range from that region.
⚠️ Warning: Route 53 Health Checks send requests from multiple regions worldwide. You need to allow the Route 53 Health Check IP ranges in your security groups. You can check the IPs for the
ROUTE53_HEALTHCHECKSservice in the AWS IP Ranges JSON.
15. CloudFront Commands
CloudFront is AWS's CDN (Content Delivery Network) service. It caches content at Edge Locations around the world to deliver it quickly to users.
Listing Distributions
# List CloudFront distributions
aws cloudfront list-distributions
{
"DistributionList": {
"Items": [
{
"Id": "E1A2B3C4D5E6F7",
"ARN": "arn:aws:cloudfront::123456789012:distribution/E1A2B3C4D5E6F7",
"Status": "Deployed",
"LastModifiedTime": "2026-03-15T08:00:00.000Z",
"DomainName": "d111111abcdef8.cloudfront.net",
"Aliases": {
"Quantity": 2,
"Items": [
"cdn.myapp.com",
"static.myapp.com"
]
},
"Origins": {
"Quantity": 2,
"Items": [
{
"Id": "S3-my-static-assets",
"DomainName": "my-static-assets.s3.ap-northeast-2.amazonaws.com",
"S3OriginConfig": {
"OriginAccessIdentity": ""
}
},
{
"Id": "ALB-my-web-alb",
"DomainName": "my-web-alb-1234567890.ap-northeast-2.elb.amazonaws.com",
"CustomOriginConfig": {
"HTTPPort": 80,
"HTTPSPort": 443,
"OriginProtocolPolicy": "https-only"
}
}
]
},
"DefaultCacheBehavior": {
"ViewerProtocolPolicy": "redirect-to-https",
"CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6",
"Compress": true
},
"ViewerCertificate": {
"ACMCertificateArn": "arn:aws:acm:us-east-1:123456789012:certificate/abcd1234-5678-efgh-ijkl-mnop9012qrst",
"SSLSupportMethod": "sni-only",
"MinimumProtocolVersion": "TLSv1.2_2021"
},
"Enabled": true
}
],
"Quantity": 1
}
}
There is a distribution with two Origins: S3 and ALB. Two custom domains are connected (cdn.myapp.com and static.myapp.com), HTTPS is enforced, and only TLS 1.2 or higher is allowed.
💡 Practical Tip: CloudFront ACM certificates must be created in the us-east-1 (N. Virginia) region. Certificates from other regions cannot be used with CloudFront. This is a fixed AWS constraint.
Getting Distribution Details
# Get detailed information for a specific distribution
aws cloudfront get-distribution \
--id E1A2B3C4D5E6F7
{
"Distribution": {
"Id": "E1A2B3C4D5E6F7",
"Status": "Deployed",
"DomainName": "d111111abcdef8.cloudfront.net",
"DistributionConfig": {
"Origins": {
"Quantity": 2,
"Items": [
{
"Id": "S3-my-static-assets",
"DomainName": "my-static-assets.s3.ap-northeast-2.amazonaws.com"
},
{
"Id": "ALB-my-web-alb",
"DomainName": "my-web-alb-1234567890.ap-northeast-2.elb.amazonaws.com"
}
]
},
"CacheBehaviors": {
"Quantity": 1,
"Items": [
{
"PathPattern": "/api/*",
"TargetOriginId": "ALB-my-web-alb",
"ViewerProtocolPolicy": "https-only",
"CachePolicyId": "4135ea2d-6df8-44a3-9df3-4b5a84be39ad",
"AllowedMethods": {
"Quantity": 7,
"Items": ["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"]
}
}
]
},
"DefaultCacheBehavior": {
"TargetOriginId": "S3-my-static-assets",
"ViewerProtocolPolicy": "redirect-to-https",
"Compress": true,
"DefaultTTL": 86400,
"MaxTTL": 31536000,
"MinTTL": 0
},
"Enabled": true
},
"ETag": "E2QWRUHAP1234"
}
}
The /api/* path is forwarded to the ALB, while everything else serves static files from S3. Remember the ETag value -- you will need it when updating the distribution configuration.
Creating Cache Invalidations
When you update deployed static files, you may need to forcefully delete the cache from Edge Locations.
# Invalidate cache for specific paths
aws cloudfront create-invalidation \
--distribution-id E1A2B3C4D5E6F7 \
--paths "/index.html" "/css/*" "/js/*"
{
"Location": "https://cloudfront.amazonaws.com/2020-05-31/distribution/E1A2B3C4D5E6F7/invalidation/I1A2B3C4D5E6F7",
"Invalidation": {
"Id": "I1A2B3C4D5E6F7",
"Status": "InProgress",
"CreateTime": "2026-03-18T10:15:00.000Z",
"InvalidationBatch": {
"Paths": {
"Quantity": 3,
"Items": [
"/index.html",
"/css/*",
"/js/*"
]
},
"CallerReference": "cli-1710756900-abcdef"
}
}
}
To invalidate all files, use "/*".
# Full cache invalidation
aws cloudfront create-invalidation \
--distribution-id E1A2B3C4D5E6F7 \
--paths "/*"
⚠️ Caution: The first 1,000 invalidation paths per month are free. After that, there is a per-path cost.
/*counts as 1 path, so it can be cheaper than invalidating 10 individual files separately.
Checking Invalidation Status
# Check invalidation status
aws cloudfront get-invalidation \
--distribution-id E1A2B3C4D5E6F7 \
--id I1A2B3C4D5E6F7
{
"Invalidation": {
"Id": "I1A2B3C4D5E6F7",
"Status": "Completed",
"CreateTime": "2026-03-18T10:15:00.000Z",
"InvalidationBatch": {
"Paths": {
"Quantity": 3,
"Items": [
"/index.html",
"/css/*",
"/js/*"
]
},
"CallerReference": "cli-1710756900-abcdef"
}
}
}
When Status: Completed, the cache has been cleared from all Edge Locations worldwide. This typically takes about 5-15 minutes.
You can also view the invalidation history at once.
# List invalidation history
aws cloudfront list-invalidations \
--distribution-id E1A2B3C4D5E6F7
{
"InvalidationList": {
"Items": [
{
"Id": "I1A2B3C4D5E6F7",
"CreateTime": "2026-03-18T10:15:00.000Z",
"Status": "Completed"
},
{
"Id": "I7F6E5D4C3B2A1",
"CreateTime": "2026-03-17T14:30:00.000Z",
"Status": "Completed"
}
],
"Quantity": 2
}
}
CloudFront Troubleshooting: Old Content Still Showing After Cache Invalidation
Sometimes old content continues to appear even after invalidation. Let's go through the checkpoints in order.
1) Verify that the invalidation status is Completed
aws cloudfront get-invalidation \
--distribution-id E1A2B3C4D5E6F7 \
--id I1A2B3C4D5E6F7 \
--query "Invalidation.Status" \
--output text
Completed
2) Check if it is a browser cache issue
The CloudFront cache and the browser cache are separate. Even if you invalidate CloudFront, the browser's cached version remains. Try accessing in incognito mode or verify directly with curl.
# Verify directly with curl (bypass browser cache)
curl -I https://cdn.myapp.com/index.html
HTTP/2 200
content-type: text/html
x-cache: Miss from cloudfront
x-amz-cf-pop: ICN54-C1
age: 0
x-cache: Miss from cloudfront means the content was freshly fetched from the Origin, not from cache. This indicates it is the latest content, not old content.
3) Verify the path pattern is correct
Invalidation paths are case-sensitive and must start with /. /images/logo.png and /Images/Logo.png are different paths.
4) If using query string-based caching
# You may also need to invalidate paths with query strings
aws cloudfront create-invalidation \
--distribution-id E1A2B3C4D5E6F7 \
--paths "/api/data?version=2" "/api/data?version=1"
💡 Practical Tip: If you apply version hashing to static files, cache invalidation becomes unnecessary altogether. For example, naming files like
main.abc123.jsso that the filename changes whenever the content changes. Build tools in frameworks like React/Vue/Next.js do this automatically.
CloudFront Troubleshooting: Checking Origin Errors
Here is how to check whether the Origin (S3 or ALB) has issues when 502/504 errors occur on CloudFront.
# Check CloudFront error rate metrics
aws cloudwatch get-metric-statistics \
--namespace "AWS/CloudFront" \
--metric-name "5xxErrorRate" \
--dimensions Name=DistributionId,Value=E1A2B3C4D5E6F7 Name=Region,Value=Global \
--start-time 2026-03-18T08:00:00Z \
--end-time 2026-03-18T10:00:00Z \
--period 300 \
--statistics Average \
--region us-east-1
{
"Label": "5xxErrorRate",
"Datapoints": [
{
"Timestamp": "2026-03-18T08:00:00+00:00",
"Average": 0.0,
"Unit": "Percent"
},
{
"Timestamp": "2026-03-18T08:30:00+00:00",
"Average": 12.5,
"Unit": "Percent"
},
{
"Timestamp": "2026-03-18T09:00:00+00:00",
"Average": 0.5,
"Unit": "Percent"
}
]
}
⚠️ Warning: CloudFront metrics can only be queried from the us-east-1 region. You must specify
--region us-east-1.
At 08:30, the 5xx error rate spiked to 12.5%. There was likely an issue with the Origin. If the Origin is an ALB, check the ALB's Target Group health status. If it is S3, check the bucket policy and whether the objects exist.
# When the Origin is ALB: Check Target Group health status
aws elbv2 describe-target-health \
--target-group-arn arn:aws:elasticloadbalancing:ap-northeast-2:123456789012:targetgroup/my-web-tg/50dc6c495c0c9188 \
--region ap-northeast-2
# When the Origin is S3: Check if the object exists
aws s3 ls s3://my-static-assets/index.html
16. SQS Commands
SQS (Simple Queue Service) is a message queue service. It enables asynchronous communication between services. Think of it like a "post office mailbox."
Listing Queues
# List SQS queues
aws sqs list-queues \
--region ap-northeast-2
{
"QueueUrls": [
"https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue",
"https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue-dlq",
"https://sqs.ap-northeast-2.amazonaws.com/123456789012/email-notification-queue",
"https://sqs.ap-northeast-2.amazonaws.com/123456789012/image-resize-queue.fifo"
]
}
SQS identifies queues by URL. Queues ending with -dlq are Dead Letter Queues (queues that store failed messages), and those ending with .fifo are FIFO queues that guarantee message ordering.
You can also filter by a specific name prefix.
# Query only queues with names containing "order"
aws sqs list-queues \
--queue-name-prefix "order" \
--region ap-northeast-2
{
"QueueUrls": [
"https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue",
"https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue-dlq"
]
}
Getting Queue Attributes
# Get detailed queue attributes
aws sqs get-queue-attributes \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue" \
--attribute-names All \
--region ap-northeast-2
{
"Attributes": {
"QueueArn": "arn:aws:sqs:ap-northeast-2:123456789012:order-processing-queue",
"ApproximateNumberOfMessages": "142",
"ApproximateNumberOfMessagesNotVisible": "3",
"ApproximateNumberOfMessagesDelayed": "0",
"CreatedTimestamp": "1704067200",
"LastModifiedTimestamp": "1710756000",
"VisibilityTimeout": "30",
"MaximumMessageSize": "262144",
"MessageRetentionPeriod": "345600",
"DelaySeconds": "0",
"ReceiveMessageWaitTimeSeconds": "20",
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-northeast-2:123456789012:order-processing-queue-dlq\",\"maxReceiveCount\":3}"
}
}
Let's go through the important attributes one by one.
| Attribute | Value | Meaning |
|---|---|---|
ApproximateNumberOfMessages | 142 | Number of messages waiting in the queue |
ApproximateNumberOfMessagesNotVisible | 3 | Number of messages currently being processed |
VisibilityTimeout | 30 seconds | Time a message is invisible to other consumers after being received |
MessageRetentionPeriod | 345600 seconds (4 days) | Message retention period |
ReceiveMessageWaitTimeSeconds | 20 seconds | Long Polling wait time |
RedrivePolicy | maxReceiveCount: 3 | Move to DLQ after 3 failures |
💡 Practical Tip: If
ReceiveMessageWaitTimeSecondsis 20, Long Polling is enabled. This reduces the number of API calls compared to Short Polling (default 0), which saves costs. It is recommended to use Long Polling in production environments.
Sending Messages (send-message)
# Send a message to the queue
aws sqs send-message \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue" \
--message-body '{"orderId": "ORD-20260318-001", "userId": "user-123", "amount": 59900, "items": ["item-A", "item-B"]}' \
--message-attributes '{"OrderType":{"DataType":"String","StringValue":"standard"},"Priority":{"DataType":"Number","StringValue":"1"}}' \
--region ap-northeast-2
{
"MD5OfMessageBody": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
"MD5OfMessageAttributes": "f6e5d4c3b2a1f0e9d8c7b6a5f4e3d2c1",
"MessageId": "12345678-abcd-efgh-ijkl-567890abcdef"
}
MessageId is the unique identifier for the message. MD5OfMessageBody is a hash value for verifying the integrity of the message body.
When sending messages to a FIFO queue, additional parameters are required.
# Send a message to a FIFO queue (MessageGroupId and MessageDeduplicationId are required)
aws sqs send-message \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/image-resize-queue.fifo" \
--message-body '{"imageId": "img-001", "size": "thumbnail"}' \
--message-group-id "user-123" \
--message-deduplication-id "img-001-thumbnail-20260318" \
--region ap-northeast-2
{
"MD5OfMessageBody": "b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6a7",
"MessageId": "23456789-bcde-fghi-jklm-678901bcdefg",
"SequenceNumber": "18872884153850340096"
}
⚠️ Caution: In FIFO queues, messages with the same
--message-group-idhave guaranteed ordering. There is no ordering guarantee between different groups.--message-deduplication-idprevents duplicate messages within a 5-minute window.
Receiving Messages (receive-message)
# Receive messages from the queue (up to 10)
aws sqs receive-message \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue" \
--max-number-of-messages 5 \
--wait-time-seconds 10 \
--message-attribute-names All \
--region ap-northeast-2
{
"Messages": [
{
"MessageId": "12345678-abcd-efgh-ijkl-567890abcdef",
"ReceiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a2bIwMFmXR+U4OPqvIGVuvj2V7o8IdCDxQlbyC/LV+GPJsD2nvYO9FjfDoYN7TguGY...",
"MD5OfBody": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
"Body": "{\"orderId\": \"ORD-20260318-001\", \"userId\": \"user-123\", \"amount\": 59900, \"items\": [\"item-A\", \"item-B\"]}",
"MessageAttributes": {
"OrderType": {
"StringValue": "standard",
"DataType": "String"
},
"Priority": {
"StringValue": "1",
"DataType": "Number"
}
}
}
]
}
The ReceiptHandle is very important. You need this value to delete the message. Once a message is received, it is invisible to other consumers for the duration of VisibilityTimeout. You must complete processing and delete it within this time.
Deleting Messages (delete-message)
Delete a message from the queue after processing is complete.
# Delete a message (using ReceiptHandle)
aws sqs delete-message \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue" \
--receipt-handle "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a2bIwMFmXR+U4OPqvIGVuvj2V7o8IdCDxQlbyC/LV+GPJsD2nvYO9FjfDoYN7TguGY..." \
--region ap-northeast-2
If it succeeds with no output, the message has been deleted.
⚠️ Caution: A
ReceiptHandleis newly issued each time a message is received. Attempting to delete with an oldReceiptHandlemay fail. The safe pattern is to process and delete immediately after receiving.
Purging a Queue (purge-queue)
Delete all messages in a queue at once.
# Delete all messages in the queue
aws sqs purge-queue \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue" \
--region ap-northeast-2
If it succeeds with no output, the purge has started. The actual deletion can take up to 60 seconds.
⚠️ Warning:
purge-queuecannot be undone. All messages are permanently deleted. Never run this carelessly in production environments. Also, callingpurge-queueagain within 60 seconds will result in an error.
SQS Troubleshooting: Checking the DLQ When Messages Are Piling Up
If messages keep accumulating and are not being processed, you should check whether failed messages are piling up in the DLQ (Dead Letter Queue).
# 1. Check the current message count in the main queue
aws sqs get-queue-attributes \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue" \
--attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible \
--region ap-northeast-2
{
"Attributes": {
"ApproximateNumberOfMessages": "1523",
"ApproximateNumberOfMessagesNotVisible": "0"
}
}
1,523 messages are waiting and 0 messages are being processed. This means the consumer is not running.
# 2. Check the message count in the DLQ
aws sqs get-queue-attributes \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue-dlq" \
--attribute-names ApproximateNumberOfMessages \
--region ap-northeast-2
{
"Attributes": {
"ApproximateNumberOfMessages": "87"
}
}
There are 87 messages in the DLQ. This means 87 messages attempted processing 3 times (maxReceiveCount) but all failed and were moved to the DLQ.
# 3. Check a sample failed message from the DLQ
aws sqs receive-message \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue-dlq" \
--max-number-of-messages 1 \
--region ap-northeast-2
{
"Messages": [
{
"MessageId": "34567890-cdef-ghij-klmn-789012cdefgh",
"ReceiptHandle": "AQEBzHnKxrGigUNZj7rYjgDgxmaT3TMy...",
"Body": "{\"orderId\": \"ORD-20260317-999\", \"userId\": \"user-456\", \"amount\": -500, \"items\": []}",
"Attributes": {
"ApproximateReceiveCount": "4",
"ApproximateFirstReceiveTimestamp": "1710669600000"
}
}
]
}
The amount is negative and items is empty. Processing failed due to invalid data. These messages should either be corrected and resent to the original queue, or the root cause should be identified and the code should be fixed.
SQS Troubleshooting: Monitoring Message Count (CloudWatch Integration)
You can monitor the message count in a queue using CloudWatch alarms.
# Alarm when 500 or more messages accumulate in the queue
aws cloudwatch put-metric-alarm \
--alarm-name "sqs-backlog-alarm" \
--alarm-description "SQS order-processing-queue message backlog exceeds 500" \
--metric-name "ApproximateNumberOfMessagesVisible" \
--namespace "AWS/SQS" \
--statistic Average \
--period 300 \
--threshold 500 \
--comparison-operator GreaterThanOrEqualToThreshold \
--evaluation-periods 2 \
--alarm-actions "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts" \
--dimensions Name=QueueName,Value=order-processing-queue \
--treat-missing-data notBreaching \
--region ap-northeast-2
# Alarm when even 1 message exists in the DLQ (messages in the DLQ indicate an abnormality)
aws cloudwatch put-metric-alarm \
--alarm-name "sqs-dlq-alarm" \
--alarm-description "Failed messages exist in DLQ" \
--metric-name "ApproximateNumberOfMessagesVisible" \
--namespace "AWS/SQS" \
--statistic Sum \
--period 300 \
--threshold 1 \
--comparison-operator GreaterThanOrEqualToThreshold \
--evaluation-periods 1 \
--alarm-actions "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts" \
--dimensions Name=QueueName,Value=order-processing-queue-dlq \
--treat-missing-data notBreaching \
--region ap-northeast-2
💡 Practical Tip: DLQ alarms should be set with
--evaluation-periods 1so they trigger immediately. Messages accumulating in the DLQ means there is either a bug in the processing logic or an issue with an external service, so it should be investigated as soon as possible.
17. SNS Commands
SNS (Simple Notification Service) is a Pub/Sub messaging service. It can deliver a single message to multiple subscribers simultaneously. Think of it like a "school bulletin board."
Listing Topics
# List SNS topics
aws sns list-topics \
--region ap-northeast-2
{
"Topics": [
{
"TopicArn": "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts"
},
{
"TopicArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events"
},
{
"TopicArn": "arn:aws:sns:ap-northeast-2:123456789012:deploy-notifications"
}
]
}
You can also check the detailed attributes of a topic.
# Get topic attributes
aws sns get-topic-attributes \
--topic-arn "arn:aws:sns:ap-northeast-2:123456789012:order-events" \
--region ap-northeast-2
{
"Attributes": {
"TopicArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events",
"Owner": "123456789012",
"DisplayName": "Order Events",
"SubscriptionsConfirmed": "3",
"SubscriptionsPending": "1",
"SubscriptionsDeleted": "0",
"EffectiveDeliveryPolicy": "{\"http\":{\"defaultHealthyRetryPolicy\":{\"minDelayTarget\":20,\"maxDelayTarget\":20,\"numRetries\":3}}}",
"Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"SNS:Publish\",\"Resource\":\"arn:aws:sns:ap-northeast-2:123456789012:order-events\",\"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"123456789012\"}}}]}"
}
}
SubscriptionsConfirmed: 3 means 3 subscriptions have been confirmed, and SubscriptionsPending: 1 means there is 1 subscriber who has not yet confirmed their subscription (for email subscriptions, they need to click the confirmation link to finalize).
Listing Subscriptions
# List subscriptions for a specific topic
aws sns list-subscriptions-by-topic \
--topic-arn "arn:aws:sns:ap-northeast-2:123456789012:order-events" \
--region ap-northeast-2
{
"Subscriptions": [
{
"SubscriptionArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events:a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"Owner": "123456789012",
"Protocol": "sqs",
"Endpoint": "arn:aws:sqs:ap-northeast-2:123456789012:order-processing-queue",
"TopicArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events"
},
{
"SubscriptionArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events:b2c3d4e5-f6a7-8901-bcde-f12345678901",
"Owner": "123456789012",
"Protocol": "lambda",
"Endpoint": "arn:aws:lambda:ap-northeast-2:123456789012:function:order-analytics",
"TopicArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events"
},
{
"SubscriptionArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events:c3d4e5f6-a7b8-9012-cdef-a12345678901",
"Owner": "123456789012",
"Protocol": "email",
"Endpoint": "ops-team@myapp.com",
"TopicArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events"
},
{
"SubscriptionArn": "PendingConfirmation",
"Owner": "123456789012",
"Protocol": "email",
"Endpoint": "manager@myapp.com",
"TopicArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events"
}
]
}
Subscriptions are set up for an SQS queue, Lambda function, and email respectively. manager@myapp.com is in PendingConfirmation status because the email confirmation has not been completed yet.
Publishing Messages (publish)
When you publish a message to an SNS topic, it is delivered to all subscribers simultaneously.
# Publish a basic message
aws sns publish \
--topic-arn "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts" \
--subject "Server Incident Alert" \
--message "OOM (Out of Memory) occurred on Production server web-01. Automatic restart in progress." \
--region ap-northeast-2
{
"MessageId": "45678901-defg-hijk-lmno-901234defghi"
}
To send different messages per protocol, use --message-structure json. You can send a human-readable format to email and JSON to SQS.
# Publish different messages per protocol
aws sns publish \
--topic-arn "arn:aws:sns:ap-northeast-2:123456789012:order-events" \
--message-structure json \
--message '{
"default": "A new order has been placed. Order number: ORD-20260318-002",
"email": "Hello,\n\nA new order has been placed.\n\nOrder Number: ORD-20260318-002\nAmount: 89,000 KRW\nProduct: Wireless Keyboard x1\n\nThank you.",
"sqs": "{\"eventType\":\"ORDER_CREATED\",\"orderId\":\"ORD-20260318-002\",\"amount\":89000,\"timestamp\":\"2026-03-18T10:30:00Z\"}",
"lambda": "{\"eventType\":\"ORDER_CREATED\",\"orderId\":\"ORD-20260318-002\",\"amount\":89000,\"userId\":\"user-789\"}"
}' \
--region ap-northeast-2
{
"MessageId": "56789012-efgh-ijkl-mnop-012345efghij"
}
💡 Practical Tip: When using
--message-structure json, the"default"key is required. It serves as the fallback message for protocols that do not have a specific message defined.
Adding a Subscription (Email)
# Add an email subscription
aws sns subscribe \
--topic-arn "arn:aws:sns:ap-northeast-2:123456789012:ops-alerts" \
--protocol email \
--notification-endpoint "devops@myapp.com" \
--region ap-northeast-2
{
"SubscriptionArn": "pending confirmation"
}
Email subscriptions are only activated after clicking the confirmation email. Until confirmed, they remain in pending confirmation status.
Adding a Subscription (SQS)
# Subscribe an SQS queue to an SNS topic
aws sns subscribe \
--topic-arn "arn:aws:sns:ap-northeast-2:123456789012:order-events" \
--protocol sqs \
--notification-endpoint "arn:aws:sqs:ap-northeast-2:123456789012:order-processing-queue" \
--attributes '{"RawMessageDelivery":"true"}' \
--region ap-northeast-2
{
"SubscriptionArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events:d4e5f6a7-b8c9-0123-defa-b23456789012"
}
Setting RawMessageDelivery: true delivers only the raw message to SQS without the SNS metadata wrapper. This simplifies message parsing, so it is almost always enabled for SQS subscriptions.
⚠️ Caution: For an SQS queue to receive messages from an SNS topic, you need to add a permission to the SQS queue's access policy that allows SNS to send messages. Without it, the subscription will be created but messages will not be delivered.
# Add SNS send permission to the SQS queue (policy example)
aws sqs set-queue-attributes \
--queue-url "https://sqs.ap-northeast-2.amazonaws.com/123456789012/order-processing-queue" \
--attributes '{
"Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"sns.amazonaws.com\"},\"Action\":\"sqs:SendMessage\",\"Resource\":\"arn:aws:sqs:ap-northeast-2:123456789012:order-processing-queue\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"arn:aws:sns:ap-northeast-2:123456789012:order-events\"}}}]}"
}' \
--region ap-northeast-2
Adding a Subscription (Lambda)
# Subscribe a Lambda function to an SNS topic
aws sns subscribe \
--topic-arn "arn:aws:sns:ap-northeast-2:123456789012:order-events" \
--protocol lambda \
--notification-endpoint "arn:aws:lambda:ap-northeast-2:123456789012:function:order-analytics" \
--region ap-northeast-2
{
"SubscriptionArn": "arn:aws:sns:ap-northeast-2:123456789012:order-events:e5f6a7b8-c9d0-1234-efab-c34567890123"
}
Lambda subscriptions likewise require the Lambda function to have SNS invocation permissions.
# Add SNS invocation permission to the Lambda function
aws lambda add-permission \
--function-name order-analytics \
--statement-id sns-order-events \
--action lambda:InvokeFunction \
--principal sns.amazonaws.com \
--source-arn "arn:aws:sns:ap-northeast-2:123456789012:order-events" \
--region ap-northeast-2
{
"Statement": "{\"Sid\":\"sns-order-events\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"sns.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:ap-northeast-2:123456789012:function:order-analytics\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:sns:ap-northeast-2:123456789012:order-events\"}}}"
}
SNS + SQS Fan-Out Pattern Overview
The Fan-Out pattern combining SNS and SQS is very commonly used in practice. It is used when a single event needs to be independently processed by multiple services.
# Check current configuration: all subscribers of the order-events topic
aws sns list-subscriptions-by-topic \
--topic-arn "arn:aws:sns:ap-northeast-2:123456789012:order-events" \
--query "Subscriptions[].{Protocol:Protocol, Endpoint:Endpoint}" \
--output table \
--region ap-northeast-2
----------------------------------------------------------------------
| ListSubscriptionsByTopic |
+----------+---------------------------------------------------------+
| Protocol | Endpoint |
+----------+---------------------------------------------------------+
| sqs | arn:aws:sqs:...:order-processing-queue |
| lambda | arn:aws:lambda:...:function:order-analytics |
| email | ops-team@myapp.com |
+----------+---------------------------------------------------------+
💡 Practical Tip: The key advantage of the Fan-Out pattern is that each consumer is independent. Even if the order-analytics Lambda fails, the order-processing-queue still receives its messages normally. This is a core architectural pattern for preventing failure propagation between services.
Unsubscribing
You can remove subscriptions that are no longer needed.
# Unsubscribe
aws sns unsubscribe \
--subscription-arn "arn:aws:sns:ap-northeast-2:123456789012:order-events:c3d4e5f6-a7b8-9012-cdef-a12345678901" \
--region ap-northeast-2
If it succeeds with no output, the subscription has been removed. That subscriber will no longer receive messages.
⚠️ Caution: Email subscriptions in
PendingConfirmationstatus cannot be removed withunsubscribe. Unconfirmed subscriptions automatically expire after 3 days.
Summary: Key Commands Cheat Sheet by Service
CloudWatch
| Task | Command |
|---|---|
| List metrics | aws cloudwatch list-metrics --namespace "AWS/EC2" |
| Query metric values | aws cloudwatch get-metric-statistics --metric-name CPUUtilization ... |
| Create alarm | aws cloudwatch put-metric-alarm --alarm-name "..." ... |
| Check alarm status | aws cloudwatch describe-alarms --state-value ALARM |
| List log groups | aws logs describe-log-groups |
| Query log events | aws logs get-log-events --log-group-name "..." --log-stream-name "..." |
| Insights query | aws logs start-query ... -> aws logs get-query-results --query-id "..." |
Route 53
| Task | Command |
|---|---|
| List hosted zones | aws route53 list-hosted-zones |
| List DNS records | aws route53 list-resource-record-sets --hosted-zone-id "..." |
| Create/update record | aws route53 change-resource-record-sets --change-batch file://dns.json |
| DNS test | aws route53 test-dns-answer --record-name "..." --record-type A |
| Health Check status | aws route53 get-health-check-status --health-check-id "..." |
CloudFront
| Task | Command |
|---|---|
| List distributions | aws cloudfront list-distributions |
| Distribution details | aws cloudfront get-distribution --id "..." |
| Cache invalidation | aws cloudfront create-invalidation --distribution-id "..." --paths "/*" |
| Invalidation status | aws cloudfront get-invalidation --distribution-id "..." --id "..." |
SQS
| Task | Command |
|---|---|
| List queues | aws sqs list-queues |
| Queue attributes | aws sqs get-queue-attributes --queue-url "..." --attribute-names All |
| Send message | aws sqs send-message --queue-url "..." --message-body "..." |
| Receive message | aws sqs receive-message --queue-url "..." --max-number-of-messages 10 |
| Delete message | aws sqs delete-message --queue-url "..." --receipt-handle "..." |
| Purge queue | aws sqs purge-queue --queue-url "..." |
SNS
| Task | Command |
|---|---|
| List topics | aws sns list-topics |
| List subscriptions | aws sns list-subscriptions-by-topic --topic-arn "..." |
| Publish message | aws sns publish --topic-arn "..." --message "..." |
| Add subscription (email) | aws sns subscribe --protocol email --notification-endpoint "..." |
| Add subscription (SQS) | aws sns subscribe --protocol sqs --notification-endpoint "arn:..." |
| Add subscription (Lambda) | aws sns subscribe --protocol lambda --notification-endpoint "arn:..." |
| Unsubscribe | aws sns unsubscribe --subscription-arn "..." |
AWS CLI Commands Practical Guide - Part 6: IaC, Secrets, Operations Management, Cost
In this final part, we cover CloudFormation for managing infrastructure as code, Secrets Manager and Parameter Store for securely handling sensitive information, Systems Manager for remote server management, and commands for tracking and optimizing costs. We wrap up the series with a comprehensive troubleshooting cheat sheet you can use right away in production.
18. CloudFormation Commands
CloudFormation is an IaC (Infrastructure as Code) service that lets you define AWS infrastructure in YAML/JSON templates and deploy it automatically. You can manage the entire lifecycle of a stack from the CLI.
18.1 List Stacks
Query all CloudFormation stacks that exist in your current account.
# List all stacks
aws cloudformation list-stacks \
--stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE ROLLBACK_COMPLETE \
--query "StackSummaries[].{Name:StackName,Status:StackStatus,Created:CreationTime}" \
--output table
-----------------------------------------------------------------
| ListStacks |
+---------------------+-------------------+---------------------+
| Created | Name | Status |
+---------------------+-------------------+---------------------+
| 2025-12-01T09:15:00 | prod-vpc-stack | CREATE_COMPLETE |
| 2025-12-03T14:22:00 | prod-ecs-cluster | UPDATE_COMPLETE |
| 2025-11-28T11:00:00 | staging-api-stack | CREATE_COMPLETE |
| 2025-12-05T08:30:00 | dev-lambda-stack | ROLLBACK_COMPLETE |
+---------------------+-------------------+---------------------+
If you don't specify --stack-status-filter, stacks in DELETE_COMPLETE status will also show up and the list gets very long. It's better to filter for only the statuses you want to see.
18.2 Describe Stack Details
View detailed information about a specific stack.
# Describe a specific stack
aws cloudformation describe-stacks \
--stack-name prod-vpc-stack \
--query "Stacks[0].{
Name:StackName,
Status:StackStatus,
Created:CreationTime,
Updated:LastUpdatedTime,
Outputs:Outputs[].{Key:OutputKey,Value:OutputValue}
}"
{
"Name": "prod-vpc-stack",
"Status": "CREATE_COMPLETE",
"Created": "2025-12-01T09:15:00.000000+00:00",
"Updated": "2025-12-10T16:45:00.000000+00:00",
"Outputs": [
{
"Key": "VpcId",
"Value": "vpc-0a1b2c3d4e5f67890"
},
{
"Key": "PublicSubnetIds",
"Value": "subnet-0abc1234,subnet-0def5678"
},
{
"Key": "PrivateSubnetIds",
"Value": "subnet-0ghi9012,subnet-0jkl3456"
}
]
}
Outputs are used when referencing values from other stacks or when pulling resource IDs from scripts. For example, you might export the VPC ID as an Output from the VPC stack, and then import it in the ECS stack.
18.3 Check Stack Events (Tracking Failure Causes)
You can view all events that occurred during stack creation/update/deletion in chronological order. This is the first place to look when tracking deployment failure causes.
# Query stack events (filter for recent failure events only)
aws cloudformation describe-stack-events \
--stack-name dev-lambda-stack \
--query "StackEvents[?ResourceStatus=='CREATE_FAILED'].{
Time:Timestamp,
Resource:LogicalResourceId,
Type:ResourceType,
Status:ResourceStatus,
Reason:ResourceStatusReason
}" \
--output table
--------------------------------------------------------------------------------------------------------------
| DescribeStackEvents |
+---------------------------+-------------------+-------------------------------+---------------+------------+
| Reason | Resource | Type | Status | Time |
+---------------------------+-------------------+-------------------------------+---------------+------------+
| Resource handler returned | ApiFunction | AWS::Lambda::Function | CREATE_FAILED | 2025-12-05 |
| message: "The runtime | | | | T08:32:15 |
| parameter of python3.8 | | | | |
| is no longer supported" | | | | |
+---------------------------+-------------------+-------------------------------+---------------+------------+
| Resource creation cancelled| ApiGateway | AWS::ApiGateway::RestApi | CREATE_FAILED | 2025-12-05 |
| : the resource depends on | | | | T08:32:16 |
| resource 'ApiFunction' | | | | |
| which failed to create | | | | |
+---------------------------+-------------------+-------------------------------+---------------+------------+
Looking at the events, the root cause of the failure is that ApiFunction failed because the python3.8 runtime is no longer supported. ApiGateway was cancelled as a cascading failure due to its dependency. The key is always to find the very first failure event.
💡 Practical Tip: Events are listed in reverse chronological order, so the root cause is usually at the bottom of the output (the oldest failure). Filtering for only
CREATE_FAILEDwith--queryhelps you find the cause quickly.
18.4 List Stack Resources
View the Physical IDs of all AWS resources created by a stack.
# List stack resources
aws cloudformation list-stack-resources \
--stack-name prod-vpc-stack \
--query "StackResourceSummaries[].{
Logical:LogicalResourceId,
Physical:PhysicalResourceId,
Type:ResourceType,
Status:ResourceStatus
}" \
--output table
-------------------------------------------------------------------------------------------------
| ListStackResources |
+--------------------+----------------------------+----------------------------+----------------+
| Logical | Physical | Type | Status |
+--------------------+----------------------------+----------------------------+----------------+
| VPC | vpc-0a1b2c3d4e5f67890 | AWS::EC2::VPC | CREATE_COMPLETE|
| PublicSubnet1 | subnet-0abc1234def56789 | AWS::EC2::Subnet | CREATE_COMPLETE|
| PublicSubnet2 | subnet-0def5678abc01234 | AWS::EC2::Subnet | CREATE_COMPLETE|
| PrivateSubnet1 | subnet-0ghi9012def34567 | AWS::EC2::Subnet | CREATE_COMPLETE|
| PrivateSubnet2 | subnet-0jkl3456ghi78901 | AWS::EC2::Subnet | CREATE_COMPLETE|
| InternetGateway | igw-0abc1234def56789 | AWS::EC2::InternetGateway | CREATE_COMPLETE|
| NatGateway | nat-0def5678abc01234 | AWS::EC2::NatGateway | CREATE_COMPLETE|
| PublicRouteTable | rtb-0abc1234567890def | AWS::EC2::RouteTable | CREATE_COMPLETE|
+--------------------+----------------------------+----------------------------+----------------+
The Logical ID is the name defined in the template, and the Physical ID is the actual resource ID created in AWS. Use the Physical ID when looking up resources in the console.
18.5 Creating/Viewing/Executing Change Sets
A Change Set is a feature that lets you preview what changes will occur before updating a stack. For production stacks, you should always create a Change Set first, review it, and then execute.
# Create a Change Set
aws cloudformation create-change-set \
--stack-name prod-vpc-stack \
--change-set-name add-vpc-endpoint-20251215 \
--template-body file://vpc-template-v2.yaml \
--parameters ParameterKey=Environment,ParameterValue=prod
{
"Id": "arn:aws:cloudformation:ap-northeast-2:123456789012:changeSet/add-vpc-endpoint-20251215/a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"StackId": "arn:aws:cloudformation:ap-northeast-2:123456789012:stack/prod-vpc-stack/f1e2d3c4-b5a6-7890-1234-567890abcdef"
}
# Describe the Change Set (preview what changes will occur)
aws cloudformation describe-change-set \
--stack-name prod-vpc-stack \
--change-set-name add-vpc-endpoint-20251215 \
--query "Changes[].{
Action:ResourceChange.Action,
Resource:ResourceChange.LogicalResourceId,
Type:ResourceChange.ResourceType,
Replacement:ResourceChange.Replacement
}" \
--output table
---------------------------------------------------------------------------
| DescribeChangeSet |
+--------+--------------------+----------------------------+--------------+
| Action | Resource | Type | Replacement |
+--------+--------------------+----------------------------+--------------+
| Add | S3VpcEndpoint | AWS::EC2::VPCEndpoint | None |
| Modify | PrivateRouteTable | AWS::EC2::RouteTable | False |
+--------+--------------------+----------------------------+--------------+
The Replacement column is important. If it shows True, it means the resource will be deleted and recreated. If a stateful resource like RDS or EC2 shows Replacement: True, there is a risk of data loss!
# Execute the Change Set (apply after review)
aws cloudformation execute-change-set \
--stack-name prod-vpc-stack \
--change-set-name add-vpc-endpoint-20251215
⚠️ Caution: If any resource shows
Replacement: True, make sure to back it up first. If an RDS instance gets replaced, the existing DB data will be lost. A Change Set is just a "preview" -- execution cannot be undone.
18.6 Drift Detection
Drift occurs when someone manually modifies a resource created by CloudFormation via the console or CLI. Drift detection identifies differences between the template and the actual state.
# Start drift detection
aws cloudformation detect-stack-drift \
--stack-name prod-vpc-stack
{
"StackDriftDetectionId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}
# Check drift detection status (it's asynchronous, so check until complete)
aws cloudformation describe-stack-drift-detection-status \
--stack-drift-detection-id a1b2c3d4-e5f6-7890-abcd-ef1234567890
{
"StackId": "arn:aws:cloudformation:ap-northeast-2:123456789012:stack/prod-vpc-stack/f1e2d3c4-b5a6",
"StackDriftDetectionId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"StackDriftStatus": "DRIFTED",
"DriftedStackResourceCount": 1,
"DetectionStatus": "DETECTION_COMPLETE",
"Timestamp": "2025-12-15T10:30:45.000000+00:00"
}
# View details of drifted resources
aws cloudformation describe-stack-resource-drifts \
--stack-name prod-vpc-stack \
--stack-resource-drift-status-filters MODIFIED \
--query "StackResourceDrifts[].{
Resource:LogicalResourceId,
Type:ResourceType,
Status:StackResourceDriftStatus,
Diffs:PropertyDifferences[].{
Path:PropertyPath,
Expected:ExpectedValue,
Actual:ActualValue
}
}"
[
{
"Resource": "PublicSecurityGroup",
"Type": "AWS::EC2::SecurityGroup",
"Status": "MODIFIED",
"Diffs": [
{
"Path": "/SecurityGroupIngress/0/CidrIp",
"Expected": "10.0.0.0/16",
"Actual": "0.0.0.0/0"
}
]
}
]
In this example, someone manually changed the security group's inbound rule from 10.0.0.0/16 (internal only) to 0.0.0.0/0 (open to all). This is a serious drift that could lead to a security incident.
💡 Practical Tip: It's a good idea to set up a Lambda that periodically runs drift detection on an EventBridge schedule, and sends an SNS notification when drift is detected. The core principle of IaC is maintaining "code = actual infrastructure."
18.7 CloudFormation Troubleshooting
Tracking Events on Deployment Failure
# Track all events for a failed stack in chronological order
aws cloudformation describe-stack-events \
--stack-name dev-lambda-stack \
--query "StackEvents[?contains(ResourceStatus,'FAILED')].{
Time:Timestamp,
Resource:LogicalResourceId,
Reason:ResourceStatusReason
}" \
--output table
--------------------------------------------------------------------------------------------------------------
| DescribeStackEvents |
+----------------------------+----------------------+--------------------------------------------------------+
| Time | Resource | Reason |
+----------------------------+----------------------+--------------------------------------------------------+
| 2025-12-05T08:32:15+00:00 | ApiFunction | The runtime parameter of python3.8 is no longer |
| | | supported for creating or updating AWS Lambda functions.|
| | | Use the UpdateFunctionConfiguration API to update to a |
| | | supported runtime. |
+----------------------------+----------------------+--------------------------------------------------------+
| 2025-12-05T08:32:16+00:00 | ApiGateway | Resource creation cancelled |
+----------------------------+----------------------+--------------------------------------------------------+
| 2025-12-05T08:32:30+00:00 | dev-lambda-stack | The following resource(s) failed to create: |
| | | [ApiFunction, ApiGateway]. Rollback requested by user. |
+----------------------------+----------------------+--------------------------------------------------------+
Handling ROLLBACK_COMPLETE Status
A stack in ROLLBACK_COMPLETE status cannot be updated, and if you leave it as-is, you can't reuse the name either. The only solution is to delete it and recreate it.
# Find stacks in ROLLBACK_COMPLETE status
aws cloudformation list-stacks \
--stack-status-filter ROLLBACK_COMPLETE \
--query "StackSummaries[].{Name:StackName,Status:StackStatus,Reason:StackStatusReason}" \
--output table
---------------------------------------------------------------------------
| ListStacks |
+-----------------------+--------------------+----------------------------+
| Name | Status | Reason |
+-----------------------+--------------------+----------------------------+
| dev-lambda-stack | ROLLBACK_COMPLETE | The following resource(s) |
| | | failed to create: |
| | | [ApiFunction] |
+-----------------------+--------------------+----------------------------+
# Delete a ROLLBACK_COMPLETE stack
aws cloudformation delete-stack \
--stack-name dev-lambda-stack
# Wait for deletion to complete
aws cloudformation wait stack-delete-complete \
--stack-name dev-lambda-stack
echo "Stack deletion complete"
Handling Delete Failures
When stack deletion fails, you can skip certain resources and proceed with deletion.
# Check delete failure events
aws cloudformation describe-stack-events \
--stack-name stuck-stack \
--query "StackEvents[?ResourceStatus=='DELETE_FAILED'].{
Resource:LogicalResourceId,
Reason:ResourceStatusReason
}"
[
{
"Resource": "S3Bucket",
"Reason": "The bucket you tried to delete is not empty (Service: Amazon S3)"
}
]
# Delete the stack while skipping the problematic resource (the resource needs manual cleanup)
aws cloudformation delete-stack \
--stack-name stuck-stack \
--retain-resources S3Bucket
If an S3 bucket is not empty, CloudFormation cannot delete it. You can either empty the bucket first, or use --retain-resources to retain that resource, delete just the stack, and then manually clean up the bucket afterward.
💡 Practical Tip: When managing S3 buckets with CloudFormation, it's a good idea to either set
DeletionPolicy: Retainin your template or add a custom resource Lambda that empties the bucket. This prevents issues when deleting the stack.
19. Secrets Manager / SSM Parameter Store
These are two services for securely storing and managing sensitive information (DB passwords, API keys, etc.). The choice depends on your use case.
| Comparison | Secrets Manager | Parameter Store |
|---|---|---|
| Primary Use | DB passwords, API keys | Configuration values, environment variables |
| Auto Rotation | Supported (Lambda-based) | Not supported |
| Cost | $0.40/month per secret | Standard free, Advanced $0.05/month |
| Encryption | KMS encryption by default | KMS only for SecureString |
| Size Limit | Max 64KB | Standard 4KB, Advanced 8KB |
19.1 Secrets Manager: Querying Secrets
# List secrets
aws secretsmanager list-secrets \
--query "SecretList[].{Name:Name,Description:Description,LastAccessed:LastAccessedDate}" \
--output table
---------------------------------------------------------------------------
| ListSecrets |
+---------------------------+-------------------------+-------------------+
| Name | Description | LastAccessed |
+---------------------------+-------------------------+-------------------+
| prod/rds/master-password | Production RDS master | 2025-12-15 |
| prod/api/stripe-key | Stripe API secret key | 2025-12-14 |
| staging/rds/password | Staging DB password | 2025-12-10 |
+---------------------------+-------------------------+-------------------+
# Retrieve the secret value (view the actual password)
aws secretsmanager get-secret-value \
--secret-id prod/rds/master-password \
--query "SecretString" \
--output text
{"username":"admin","password":"xK9#mP2$vL7nQ4wR","engine":"mysql","host":"prod-db.cluster-abc123.ap-northeast-2.rds.amazonaws.com","port":3306,"dbname":"myapp"}
Secrets are typically stored in JSON format with multiple values in a single secret. You can combine --query with jq to extract specific fields.
# Extract only the password
aws secretsmanager get-secret-value \
--secret-id prod/rds/master-password \
--query "SecretString" \
--output text | jq -r '.password'
xK9#mP2$vL7nQ4wR
19.2 Secrets Manager: Creating/Updating Secrets
# Create a new secret
aws secretsmanager create-secret \
--name prod/api/openai-key \
--description "OpenAI API Key for production" \
--secret-string '{"api_key":"sk-proj-abc123def456ghi789"}'
{
"ARN": "arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:prod/api/openai-key-AbCdEf",
"Name": "prod/api/openai-key",
"VersionId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}
# Update a secret value
aws secretsmanager update-secret \
--secret-id prod/api/openai-key \
--secret-string '{"api_key":"sk-proj-new-key-xyz789"}'
{
"ARN": "arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:prod/api/openai-key-AbCdEf",
"Name": "prod/api/openai-key",
"VersionId": "b2c3d4e5-f6a7-8901-bcde-f23456789012"
}
⚠️ Caution: When you run
update-secret, the previous version of the secret is moved to theAWSPREVIOUSstaging label. You can recover the previous version if you need to roll back, but it may be deleted over time, so make sure to back up important keys separately.
19.3 Secrets Manager: Automatic Rotation
# Set up automatic rotation (every 30 days)
aws secretsmanager rotate-secret \
--secret-id prod/rds/master-password \
--rotation-lambda-arn arn:aws:lambda:ap-northeast-2:123456789012:function:SecretsRotationFunction \
--rotation-rules "{\"AutomaticallyAfterDays\": 30}"
{
"ARN": "arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:prod/rds/master-password-XyZ123",
"Name": "prod/rds/master-password",
"VersionId": "c3d4e5f6-a7b8-9012-cdef-345678901234"
}
# Trigger immediate rotation
aws secretsmanager rotate-secret \
--secret-id prod/rds/master-password
# Check rotation status
aws secretsmanager describe-secret \
--secret-id prod/rds/master-password \
--query "{
Name:Name,
RotationEnabled:RotationEnabled,
RotationDays:RotationRules.AutomaticallyAfterDays,
LastRotated:LastRotatedDate,
NextRotation:NextRotationDate
}"
{
"Name": "prod/rds/master-password",
"RotationEnabled": true,
"RotationDays": 30,
"LastRotated": "2025-12-01T03:00:00+00:00",
"NextRotation": "2025-12-31T03:00:00+00:00"
}
💡 Practical Tip: For RDS password rotation, using the default Lambda template provided by AWS is convenient. However, if your application is caching the secret, make sure to use the SDK's secret caching library so that it automatically fetches the new password after rotation.
19.4 SSM Parameter Store: Querying Parameters
# Query a specific parameter
aws ssm get-parameter \
--name "/prod/app/database-url" \
--with-decryption
{
"Parameter": {
"Name": "/prod/app/database-url",
"Type": "SecureString",
"Value": "mysql://admin:xK9mP2vL7n@prod-db.cluster-abc123.ap-northeast-2.rds.amazonaws.com:3306/myapp",
"Version": 3,
"LastModifiedDate": "2025-12-10T14:30:00+00:00",
"ARN": "arn:aws:ssm:ap-northeast-2:123456789012:parameter/prod/app/database-url",
"DataType": "text"
}
}
The --with-decryption flag decrypts and displays SecureString type parameters. Without this flag, the value appears in its encrypted form.
19.5 SSM Parameter Store: Creating/Updating Parameters
# Create a plain parameter (String)
aws ssm put-parameter \
--name "/prod/app/feature-flag/new-ui" \
--type String \
--value "true" \
--description "New UI feature flag for production"
{
"Version": 1,
"Tier": "Standard"
}
# Create an encrypted parameter (SecureString)
aws ssm put-parameter \
--name "/prod/app/redis-auth-token" \
--type SecureString \
--value "aR3dIs$ecureT0ken!" \
--description "Redis AUTH token" \
--key-id alias/prod-app-key
{
"Version": 1,
"Tier": "Standard"
}
# Update an existing parameter (--overwrite is required)
aws ssm put-parameter \
--name "/prod/app/feature-flag/new-ui" \
--type String \
--value "false" \
--overwrite
{
"Version": 2,
"Tier": "Standard"
}
If you forget --overwrite, you'll get a ParameterAlreadyExists error for an existing parameter.
19.6 SSM Parameter Store: Path-Based Queries
The most powerful feature of Parameter Store is path (hierarchy) based queries. You can fetch all parameters under /prod/app/ at once.
# Query all parameters under a path
aws ssm get-parameters-by-path \
--path "/prod/app/" \
--recursive \
--with-decryption \
--query "Parameters[].{Name:Name,Type:Type,Value:Value}" \
--output table
---------------------------------------------------------------------------
| GetParametersByPath |
+----------------------------------+--------------+-----------------------+
| Name | Type | Value |
+----------------------------------+--------------+-----------------------+
| /prod/app/database-url | SecureString | mysql://admin:xK9... |
| /prod/app/redis-auth-token | SecureString | aR3dIs$ecureT0ken! |
| /prod/app/feature-flag/new-ui | String | false |
| /prod/app/api-endpoint | String | https://api.myapp.com|
| /prod/app/log-level | String | INFO |
+----------------------------------+--------------+-----------------------+
# Search parameters (filtering)
aws ssm describe-parameters \
--parameter-filters "Key=Name,Values=/prod/" \
--query "Parameters[].{Name:Name,Type:Type,LastModified:LastModifiedDate,Description:Description}" \
--output table
-----------------------------------------------------------------------------------------------
| DescribeParameters |
+-------------------------------+----------------------------+--------------+------------------+
| Description | LastModified | Name | Type |
+-------------------------------+----------------------------+--------------+------------------+
| Production DB URL | 2025-12-10T14:30:00+00:00 | /prod/app/.. | SecureString |
| Redis AUTH token | 2025-12-12T09:15:00+00:00 | /prod/app/.. | SecureString |
| New UI feature flag | 2025-12-15T11:00:00+00:00 | /prod/app/.. | String |
+-------------------------------+----------------------------+--------------+------------------+
💡 Practical Tip: If you design parameter names in the
/{environment}/{service}/{key}format, you can control access at the path level in IAM policies, likearn:aws:ssm:*:*:parameter/prod/*. This way, the development team can access only/dev/*, while the operations team can access up to/prod/*.
19.7 Troubleshooting: KMS Key Permission Issues
The most common error with SecureString parameters and Secrets Manager is KMS key permission issues.
# Error scenario: Insufficient KMS key permissions
aws ssm get-parameter \
--name "/prod/app/database-url" \
--with-decryption
An error occurred (AccessDeniedException) when calling the GetParameter operation:
The ciphertext refers to a customer master key that does not exist,
does not exist in this region, or you are not allowed to access.
# Step 1: Check which KMS key was used to encrypt the parameter
aws ssm describe-parameters \
--parameter-filters "Key=Name,Values=/prod/app/database-url" \
--query "Parameters[0].{Name:Name,KeyId:KeyId,Type:Type}"
{
"Name": "/prod/app/database-url",
"KeyId": "alias/prod-app-key",
"Type": "SecureString"
}
# Step 2: Check the KMS key details
aws kms describe-key \
--key-id alias/prod-app-key \
--query "KeyMetadata.{KeyId:KeyId,State:KeyState,Manager:KeyManager}"
{
"KeyId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"State": "Enabled",
"Manager": "CUSTOMER"
}
# Step 3: Check if the current user/role has permission to use the KMS key
aws kms list-grants \
--key-id alias/prod-app-key \
--query "Grants[?GranteePrincipal=='arn:aws:iam::123456789012:role/my-role'].{
Operations:Operations,
Grantee:GranteePrincipal
}"
To resolve KMS permission issues, you need to either add kms:Decrypt permission to the IAM policy, or allow the IAM role in the KMS key policy.
⚠️ Warning: If the KMS key is in
PendingDeletionstatus, decryption is impossible. If you don't cancel within the key deletion waiting period (7-30 days), all data encrypted with that key becomes permanently inaccessible. You can cancel withaws kms cancel-key-deletion.
20. Systems Manager (SSM) Management
Systems Manager is a service for securely managing EC2 instances without SSH. There's no need to open port 22 in security groups, and Session Manager automatically logs all access history.
20.1 Querying Managed Instances
Only instances with the SSM Agent installed and the correct IAM role attached will appear.
# List SSM managed instances
aws ssm describe-instance-information \
--query "InstanceInformationList[].{
Id:InstanceId,
Name:ComputerName,
Platform:PlatformType,
PlatformName:PlatformName,
AgentVersion:AgentVersion,
PingStatus:PingStatus,
LastPing:LastPingDateTime
}" \
--output table
-----------------------------------------------------------------------------------------------------------------
| DescribeInstanceInformation |
+---------------+--------------+-------------------+----------+-----------+--------------+----------------------+
| AgentVersion | Id | LastPing | Name | PingStatus| Platform | PlatformName |
+---------------+--------------+-------------------+----------+-----------+--------------+----------------------+
| 3.3.987.0 | i-0abc1234de | 2025-12-15T10:30 | ip-10-0 | Online | Linux | Amazon Linux 2023 |
| 3.3.987.0 | i-0def5678ab | 2025-12-15T10:29 | ip-10-0 | Online | Linux | Ubuntu 22.04 |
| 3.3.987.0 | i-0ghi9012cd | 2025-12-15T10:31 | ip-10-0 | Online | Windows | Windows Server 2022 |
| 3.3.880.0 | i-0jkl3456ef | 2025-12-14T08:15 | ip-10-0 | ConnectionLost | Linux | Amazon Linux 2 |
+---------------+--------------+-------------------+----------+-----------+--------------+----------------------+
If PingStatus is Online, the SSM Agent is operating normally. ConnectionLost means the instance is either powered off or there is an issue with the SSM Agent.
20.2 Connecting with Session Manager
Connect to an instance shell via the browser or CLI without SSH.
# Connect to an instance with Session Manager
aws ssm start-session \
--target i-0abc1234def56789
Starting session with SessionId: user@example.com-0abc1234def56789
sh-5.2$
sh-5.2$ whoami
ssm-user
sh-5.2$ hostname
ip-10-0-1-42.ap-northeast-2.compute.internal
sh-5.2$ exit
Exiting session with sessionId: user@example.com-0abc1234def56789
Session logs can be automatically saved to S3 or CloudWatch Logs, enabling audit trails of who ran which commands and when.
💡 Practical Tip: To use Session Manager, you need to have the Session Manager plugin for AWS CLI installed locally. If you get a "SessionManagerPlugin is not found" error when running
aws ssm start-session, install the plugin.
20.3 Port Forwarding (e.g., RDS Access)
When you want to connect directly from your local machine to an RDS in a private subnet, use Session Manager port forwarding. No bastion host is needed.
# RDS port forwarding (local 3306 -> remote RDS 3306)
aws ssm start-session \
--target i-0abc1234def56789 \
--document-name AWS-StartPortForwardingSessionToRemoteHost \
--parameters "{
\"host\":[\"prod-db.cluster-abc123.ap-northeast-2.rds.amazonaws.com\"],
\"portNumber\":[\"3306\"],
\"localPortNumber\":[\"13306\"]
}"
Starting session with SessionId: user@example.com-0abc1234def56789
Port 13306 opened for sessionId user@example.com-0abc1234def56789.
Waiting for connections...
# Connect to RDS via the local port from another terminal
mysql -h 127.0.0.1 -P 13306 -u admin -p myapp
i-0abc1234def56789 is a "relay" EC2 instance in the private subnet. The connection tunnels through this instance to reach the RDS.
# Standard port forwarding (to the instance's own port)
aws ssm start-session \
--target i-0abc1234def56789 \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["8080"],"localPortNumber":["18080"]}'
Starting session with SessionId: user@example.com-0abc1234def56789
Port 18080 opened for sessionId user@example.com-0abc1234def56789.
Waiting for connections...
💡 Practical Tip: Port forwarding is useful not only for RDS but also for accessing private services like ElastiCache Redis and OpenSearch from your local machine. It enables DB access during development without a VPN.
20.4 Executing Remote Commands with Run Command
You can execute commands on multiple instances simultaneously. This is useful for applying patches, collecting logs, restarting agents, and more.
# Execute commands on multiple instances
aws ssm send-command \
--document-name "AWS-RunShellScript" \
--targets "Key=tag:Environment,Values=production" \
--parameters '{"commands":["df -h","free -m","uptime"]}' \
--comment "Check disk and memory usage on prod servers"
{
"Command": {
"CommandId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"DocumentName": "AWS-RunShellScript",
"Comment": "Check disk and memory usage on prod servers",
"Status": "Pending",
"TargetCount": 3,
"CompletedCount": 0,
"DeliveryTimedOutCount": 0,
"RequestedDateTime": "2025-12-15T11:00:00.000000+00:00"
}
}
# Check command execution results
aws ssm list-command-invocations \
--command-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
--details \
--query "CommandInvocations[].{
Instance:InstanceId,
Status:Status,
Output:CommandPlugins[0].Output
}"
[
{
"Instance": "i-0abc1234def56789",
"Status": "Success",
"Output": "Filesystem Size Used Avail Use% Mounted on\n/dev/xvda1 50G 32G 18G 64% /\n\n total used free\nMem: 7964 5123 2841\n\n 11:00:15 up 45 days, 3:22, 0 users, load average: 0.12, 0.08, 0.05\n"
},
{
"Instance": "i-0def5678abc01234",
"Status": "Success",
"Output": "Filesystem Size Used Avail Use% Mounted on\n/dev/xvda1 50G 45G 5G 90% /\n\n total used free\nMem: 7964 7102 862\n\n 11:00:16 up 120 days, 8:45, 0 users, load average: 2.45, 1.89, 1.56\n"
},
{
"Instance": "i-0ghi9012cde34567",
"Status": "Success",
"Output": "Filesystem Size Used Avail Use% Mounted on\n/dev/xvda1 50G 12G 38G 24% /\n\n total used free\nMem: 15928 4012 11916\n\n 11:00:16 up 10 days, 12:33, 0 users, load average: 0.03, 0.05, 0.02\n"
}
]
Looking at the results, i-0def5678abc01234 has 90% disk usage, nearly exhausted memory, and high load average -- it needs immediate attention!
⚠️ Warning: Run Command is a powerful feature, but sending dangerous commands like
rm -rforrebootto multiple servers at once via--targetsin production can cause a disaster. Always verify the scope of--targets, and whenever possible, test on a single instance first.
20.5 SSM Troubleshooting: 3 Steps When an Instance Doesn't Appear
When an instance doesn't show up in describe-instance-information, check these three things in order.
# Step 1: Check if an SSM IAM role is attached to the EC2 instance
aws ec2 describe-instances \
--instance-ids i-0abc1234def56789 \
--query "Reservations[0].Instances[0].{
InstanceId:InstanceId,
State:State.Name,
IamProfile:IamInstanceProfile.Arn
}"
{
"InstanceId": "i-0abc1234def56789",
"State": "running",
"IamProfile": null
}
The IAM Instance Profile is null! You need to attach an IAM role that includes the AmazonSSMManagedInstanceCore policy.
# Step 2: Check if SSM Agent is installed/running (if you can already access the instance)
# Verify via EC2 User Data or another method
aws ec2 describe-instances \
--instance-ids i-0abc1234def56789 \
--query "Reservations[0].Instances[0].{
Platform:Platform,
LaunchTime:LaunchTime,
ImageId:ImageId
}"
{
"Platform": null,
"LaunchTime": "2025-11-01T09:00:00+00:00",
"ImageId": "ami-0abc1234def56789"
}
Modern AMIs like Amazon Linux 2023, Amazon Linux 2, and Ubuntu 20.04+ come with SSM Agent pre-installed. Older AMIs or custom images may require manual installation.
# Step 3: Check VPC endpoints or internet connectivity
# SSM Agent needs to communicate with the SSM service endpoint over HTTPS
aws ec2 describe-vpc-endpoints \
--filters "Name=service-name,Values=com.amazonaws.ap-northeast-2.ssm" \
--query "VpcEndpoints[].{
VpcId:VpcId,
ServiceName:ServiceName,
State:State
}"
[]
There are no VPC endpoints, and if it's in a private subnet, the SSM Agent cannot connect to the service. Solution: Either add a NAT Gateway or create the following three VPC endpoints:
com.amazonaws.ap-northeast-2.ssmcom.amazonaws.ap-northeast-2.ssmmessagescom.amazonaws.ap-northeast-2.ec2messages
💡 Practical Tip: 90% of the time when an SSM instance doesn't appear, the cause is one of these three: (1) no IAM role, (2) SSM Agent not running, (3) no network connectivity. Checking them in order will resolve most cases.
20.6 Debugging Run Command Failures
# View details of a failed command
aws ssm get-command-invocation \
--command-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
--instance-id i-0abc1234def56789 \
--query "{
Status:Status,
StatusDetails:StatusDetails,
StandardOutput:StandardOutputContent,
StandardError:StandardErrorContent,
ResponseCode:ResponseCode
}"
{
"Status": "Failed",
"StatusDetails": "Failed",
"StandardOutput": "",
"StandardError": "/bin/sh: yum: command not found\n",
"ResponseCode": 127,
"ExecutionElapsedTime": "PT0.5S"
}
The command failed because it tried to run yum on an Ubuntu instance. Ubuntu uses apt. When sending commands to instances with different operating systems via --targets, you need to separate the commands by OS.
21. Cost Management Commands
Here's how to query and track AWS costs using the CLI. This uses the AWS Cost Explorer API.
21.1 Monthly/Per-Service Cost Query
# Query this month's costs by service
aws ce get-cost-and-usage \
--time-period Start=2025-12-01,End=2025-12-31 \
--granularity MONTHLY \
--metrics "UnblendedCost" \
--group-by Type=DIMENSION,Key=SERVICE \
--query "ResultsByTime[0].Groups[?Metrics.UnblendedCost.Amount > '10'].{
Service:Keys[0],
Cost:Metrics.UnblendedCost.Amount,
Unit:Metrics.UnblendedCost.Unit
}" \
--output table
---------------------------------------------------------------------------
| GetCostAndUsage |
+----------------------------------------------+------------+------------+
| Service | Cost | Unit |
+----------------------------------------------+------------+------------+
| Amazon Elastic Compute Cloud - Compute | 2847.32 | USD |
| Amazon Relational Database Service | 1523.67 | USD |
| Amazon Simple Storage Service | 342.15 | USD |
| Amazon CloudFront | 187.43 | USD |
| AWS Lambda | 45.89 | USD |
| Amazon ElastiCache | 312.50 | USD |
| Amazon Elastic Container Service | 856.21 | USD |
| AWS Key Management Service | 23.40 | USD |
| Amazon Route 53 | 12.50 | USD |
+----------------------------------------------+------------+------------+
The
> '10'filter in--queryshows only services costing more than $10. If you include every small-cost service, the list gets too long.
# Monthly total cost trend over the last 3 months
aws ce get-cost-and-usage \
--time-period Start=2025-10-01,End=2025-12-31 \
--granularity MONTHLY \
--metrics "UnblendedCost" \
--query "ResultsByTime[].{
Month:TimePeriod.Start,
Cost:Total.UnblendedCost.Amount,
Unit:Total.UnblendedCost.Unit
}" \
--output table
------------------------------------------
| GetCostAndUsage |
+--------------+------------+------------+
| Month | Cost | Unit |
+--------------+------------+------------+
| 2025-10-01 | 5432.18 | USD |
| 2025-11-01 | 5891.45 | USD |
| 2025-12-01 | 6151.07 | USD |
+--------------+------------+------------+
Costs are increasing every month. You should compare by service to see which ones are driving the increase.
21.2 Daily Cost Query (Tracking Cost Spikes)
# Daily cost trend for the last 7 days
aws ce get-cost-and-usage \
--time-period Start=2025-12-09,End=2025-12-16 \
--granularity DAILY \
--metrics "UnblendedCost" \
--query "ResultsByTime[].{
Date:TimePeriod.Start,
Cost:Total.UnblendedCost.Amount
}" \
--output table
---------------------------------
| GetCostAndUsage |
+--------------+----------------+
| Date | Cost |
+--------------+----------------+
| 2025-12-09 | 198.42 |
| 2025-12-10 | 201.35 |
| 2025-12-11 | 195.89 |
| 2025-12-12 | 487.23 |
| 2025-12-13 | 523.67 |
| 2025-12-14 | 498.12 |
| 2025-12-15 | 512.45 |
+--------------+----------------+
Starting December 12, costs suddenly increased by 2.5x! You need to investigate what happened on that date.
# Analyze costs by service for the date of the spike
aws ce get-cost-and-usage \
--time-period Start=2025-12-12,End=2025-12-13 \
--granularity DAILY \
--metrics "UnblendedCost" \
--group-by Type=DIMENSION,Key=SERVICE \
--query "ResultsByTime[0].Groups | sort_by(@, &to_number(Metrics.UnblendedCost.Amount)) | reverse(@) | [:5].{
Service:Keys[0],
Cost:Metrics.UnblendedCost.Amount
}" \
--output table
---------------------------------------------------------------------------
| GetCostAndUsage |
+----------------------------------------------+------------------------+
| Service | Cost |
+----------------------------------------------+------------------------+
| Amazon Elastic Compute Cloud - Compute | 289.45 |
| Amazon Relational Database Service | 87.32 |
| Amazon Elastic Container Service | 52.18 |
| Amazon Simple Storage Service | 31.24 |
| Amazon CloudFront | 15.67 |
+----------------------------------------------+------------------------+
EC2 costs increased significantly. Someone may have launched large instances, or Auto Scaling may have scaled up to many instances.
21.3 Cost Forecast
# Forecasted total cost for this month
aws ce get-cost-forecast \
--time-period Start=2025-12-16,End=2025-12-31 \
--granularity MONTHLY \
--metric "UNBLENDED_COST" \
--query "{
ForecastedCost:Total.Amount,
Unit:Total.Unit,
Confidence:Total.Unit
}"
{
"ForecastedCost": "7823.45",
"Unit": "USD"
}
Including the remaining period of the month, the projected total is about $7,823. That's a 33% increase compared to last month's $5,891.
21.4 Budget Review
Check the budget status configured with AWS Budgets.
# List all budgets
aws budgets describe-budgets \
--account-id 123456789012 \
--query "Budgets[].{
Name:BudgetName,
Type:BudgetType,
Limit:BudgetLimit.Amount,
Unit:BudgetLimit.Unit,
Actual:CalculatedSpend.ActualSpend.Amount,
Forecast:CalculatedSpend.ForecastedSpend.Amount
}" \
--output table
---------------------------------------------------------------------------
| DescribeBudgets |
+-----------+------------+----------+---------+--------+-----------------+
| Actual | Forecast | Limit | Name | Type | Unit |
+-----------+------------+----------+---------+--------+-----------------+
| 6151.07 | 7823.45 | 6000.00 | Monthly | COST | USD |
| | | | -Total | | |
+-----------+------------+----------+---------+--------+-----------------+
| 2847.32 | 3612.18 | 3000.00 | EC2 | COST | USD |
| | | | -Budget | | |
+-----------+------------+----------+---------+--------+-----------------+
The Monthly-Total budget of $6,000 has already been exceeded, and the EC2 budget of $3,000 is almost maxed out. Immediate action is needed.
💡 Practical Tip: If you set up alerts at 80%, 100%, and 120% of your budget in AWS Budgets, you can detect cost overruns early. You can also create budgets and alerts from the CLI using
aws budgets create-budget.
21.5 Cost Troubleshooting: Tracking Resources During Cost Spikes
Here's how to use CloudTrail to track who created what when costs spike.
# Find EC2 instances created on the date of the cost spike
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances \
--start-time 2025-12-12T00:00:00Z \
--end-time 2025-12-13T00:00:00Z \
--query "Events[].{
Time:EventTime,
User:Username,
Event:EventName
}" \
--output table
---------------------------------------------------------------------------
| LookupEvents |
+---------------------------+------------------+-------------------------+
| Time | User | Event |
+---------------------------+------------------+-------------------------+
| 2025-12-12T09:15:00+00:00 | dev-user-kim | RunInstances |
| 2025-12-12T09:15:01+00:00 | dev-user-kim | RunInstances |
| 2025-12-12T09:15:02+00:00 | dev-user-kim | RunInstances |
| 2025-12-12T09:15:03+00:00 | dev-user-kim | RunInstances |
| 2025-12-12T09:15:04+00:00 | dev-user-kim | RunInstances |
+---------------------------+------------------+-------------------------+
dev-user-kim created 5 EC2 instances at once at 9 AM on December 12. This was the cause of the cost spike!
# View event details (check which instance type)
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances \
--start-time 2025-12-12T09:00:00Z \
--end-time 2025-12-12T10:00:00Z \
--max-results 1 \
--query "Events[0].CloudTrailEvent" \
--output text | jq '.requestParameters.instanceType'
"p3.2xlarge"
They launched 5 GPU instances (p3.2xlarge, ~$3.06/hour). Running them for just one day costs $367! If they were for testing, they should be terminated immediately after use. You might also consider using Spot Instances.
⚠️ Caution: The Cost Explorer API itself incurs charges. Each
get-cost-and-usagecall costs $0.01. If your automation scripts call it every minute, the costs can add up unexpectedly, so call at reasonable intervals (e.g., once per day).
22. Comprehensive Troubleshooting Cheat Sheet
This is a collection of the most common problem scenarios you'll encounter in practice, along with commands you can run immediately to investigate.
22.1 Commands by Symptom
| Symptom | Command to Run | Description |
|---|---|---|
| Access Denied | aws sts get-caller-identity | Check current authentication info |
aws iam simulate-principal-policy --policy-source-arn <ARN> --action-names <ACTION> --resource-arns <RESOURCE> | Simulate permissions for a specific action | |
aws iam list-attached-user-policies --user-name <USER> | List attached policies | |
| EC2 Connection Failure | aws ec2 describe-instance-status --instance-ids <ID> | Check instance status/health checks |
aws ec2 describe-security-groups --group-ids <SG_ID> | Security group inbound rules | |
aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<SUBNET> | Check NACL rules | |
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<SUBNET> | Check route table | |
| Service Not Responding (5xx) | aws elbv2 describe-target-health --target-group-arn <ARN> | Target group health check |
aws logs filter-log-events --log-group-name <GROUP> --filter-pattern "ERROR" | Application error logs | |
aws cloudwatch get-metric-statistics --namespace AWS/ECS --metric-name CPUUtilization ... | Check CPU/memory overload | |
| Deployment Failure | aws cloudformation describe-stack-events --stack-name <STACK> | CloudFormation events |
aws ecs describe-services --cluster <CLUSTER> --services <SVC> | ECS service events | |
aws deploy get-deployment --deployment-id <ID> | CodeDeploy deployment status | |
| Cost Spike | aws ce get-cost-and-usage --granularity DAILY ... | Daily cost trend |
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances | Track resource creation | |
| DNS Issues | aws route53 test-dns-answer --hosted-zone-id <ID> --record-name <NAME> --record-type A | Test DNS response |
aws route53 list-resource-record-sets --hosted-zone-id <ID> | List DNS records | |
| SSL Certificate Issues | aws acm describe-certificate --certificate-arn <ARN> | Certificate status/expiration |
aws acm list-certificates --certificate-statuses EXPIRED INACTIVE | Expired/inactive certificates | |
| Lambda Errors | aws lambda get-function --function-name <NAME> | Check function configuration |
aws logs filter-log-events --log-group-name /aws/lambda/<NAME> --filter-pattern "ERROR" | Lambda error logs | |
aws lambda list-event-source-mappings --function-name <NAME> | Event source mappings | |
| RDS Connection Failure | aws rds describe-db-instances --db-instance-identifier <ID> | DB status/endpoint |
aws ec2 describe-security-groups --group-ids <RDS_SG> | RDS security group | |
aws rds describe-db-subnet-groups --db-subnet-group-name <NAME> | Subnet group |
22.2 One-Liner Collection
Here are useful one-liners you can copy and use right away in practice.
Find EC2 Instances Across All Regions
# Find running EC2 instances across all regions
for region in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do
echo "=== $region ==="
aws ec2 describe-instances \
--region $region \
--filters "Name=instance-state-name,Values=running" \
--query "Reservations[].Instances[].{Id:InstanceId,Type:InstanceType,Name:Tags[?Key=='Name']|[0].Value}" \
--output table 2>/dev/null
done
=== ap-northeast-2 ===
-----------------------------------------------
| DescribeInstances |
+-----------+---------------+-----------------+
| Id | Name | Type |
+-----------+---------------+-----------------+
| i-0abc... | prod-web-01 | t3.large |
| i-0def... | prod-web-02 | t3.large |
| i-0ghi... | prod-api-01 | c5.xlarge |
+-----------+---------------+-----------------+
=== us-east-1 ===
-----------------------------------------------
| DescribeInstances |
+-----------+---------------+-----------------+
| Id | Name | Type |
+-----------+---------------+-----------------+
| i-0jkl... | test-ml-gpu | p3.2xlarge |
+-----------+---------------+-----------------+
=== eu-west-1 ===
(No instances found)
It turns out there was a GPU instance running in us-east-1! Scanning all regions like this helps you find resources that have been forgotten and left running.
Find Access Keys Unused for 30+ Days
# Find Access Keys unused for more than 30 days
aws iam generate-credential-report > /dev/null 2>&1
sleep 3
aws iam get-credential-report \
--query "Content" \
--output text | base64 -d | \
awk -F',' 'NR>1 && $9!="N/A" && $11!="N/A" {
split($11,a,"T");
cmd="date -d \""a[1]"\" +%s";
cmd | getline last_used;
close(cmd);
cmd2="date +%s";
cmd2 | getline now;
close(cmd2);
diff=(now-last_used)/86400;
if(diff>30) print $1, "| Last used:", a[1], "| Days ago:", int(diff)
}'
dev-user-park | Last used: 2025-10-15 | Days ago: 61
svc-legacy-app | Last used: 2025-09-20 | Days ago: 86
intern-2025 | Last used: 2025-08-01 | Days ago: 136
These Access Keys should be deactivated or deleted to reduce security risks.
Find Public S3 Buckets
# Find S3 buckets with public access enabled
for bucket in $(aws s3api list-buckets --query "Buckets[].Name" --output text); do
status=$(aws s3api get-public-access-block --bucket $bucket 2>/dev/null | \
jq -r 'if .PublicAccessBlockConfiguration.BlockPublicAcls == true and
.PublicAccessBlockConfiguration.BlockPublicPolicy == true
then "BLOCKED" else "POTENTIALLY_PUBLIC" end' 2>/dev/null)
if [ "$status" = "POTENTIALLY_PUBLIC" ] || [ -z "$status" ]; then
echo "⚠️ $bucket - Public Access Block not configured or partially allowed"
fi
done
⚠️ legacy-static-assets - Public Access Block not configured or partially allowed
⚠️ dev-test-uploads - Public Access Block not configured or partially allowed
Find Untagged Resources
# Find running EC2 instances with no tags at all
aws ec2 describe-instances \
--filters "Name=instance-state-name,Values=running" \
--query "Reservations[].Instances[?!not_null(Tags)].{
Id:InstanceId,
Type:InstanceType,
LaunchTime:LaunchTime
}[]" \
--output table
---------------------------------------------------
| DescribeInstances |
+------------------+----------+-------------------+
| Id | Type | LaunchTime |
+------------------+----------+-------------------+
| i-0xyz9876abc543 | m5.large | 2025-11-20T14:00 |
+------------------+----------+-------------------+
# Find running EC2 instances without a Name tag
aws ec2 describe-instances \
--filters "Name=instance-state-name,Values=running" \
--query "Reservations[].Instances[?!contains(Tags[].Key, 'Name')].{
Id:InstanceId,
Type:InstanceType,
AZ:Placement.AvailabilityZone
}[]" \
--output table
Resources without tags can't be tracked for cost allocation and have no identifiable owner. It's best to enforce a mandatory tagging policy.
Top 5 Services by Cost
# Top 5 services by cost this month
aws ce get-cost-and-usage \
--time-period Start=2025-12-01,End=2025-12-31 \
--granularity MONTHLY \
--metrics "UnblendedCost" \
--group-by Type=DIMENSION,Key=SERVICE \
--query "ResultsByTime[0].Groups | sort_by(@, &to_number(Metrics.UnblendedCost.Amount)) | reverse(@) | [:5].{
Service:Keys[0],
Cost:Metrics.UnblendedCost.Amount
}" \
--output table
---------------------------------------------------------------------------
| GetCostAndUsage |
+----------------------------------------------+------------------------+
| Service | Cost |
+----------------------------------------------+------------------------+
| Amazon Elastic Compute Cloud - Compute | 2847.32 |
| Amazon Relational Database Service | 1523.67 |
| Amazon Elastic Container Service | 856.21 |
| Amazon Simple Storage Service | 342.15 |
| Amazon ElastiCache | 312.50 |
+----------------------------------------------+------------------------+
Find Regions in Use
# Find regions with actual resources (based on EC2)
for region in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do
count=$(aws ec2 describe-instances \
--region $region \
--filters "Name=instance-state-name,Values=running" \
--query "length(Reservations[].Instances[])" \
--output text 2>/dev/null)
if [ "$count" -gt 0 ] 2>/dev/null; then
echo "$region: $count instances"
fi
done
ap-northeast-2: 12 instances
us-east-1: 3 instances
ap-southeast-1: 1 instances
# More comprehensive method: Find regions with recent API calls via CloudTrail
for region in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do
events=$(aws cloudtrail lookup-events \
--region $region \
--max-results 1 \
--query "length(Events)" \
--output text 2>/dev/null)
if [ "$events" -gt 0 ] 2>/dev/null; then
echo "$region: active"
fi
done
ap-northeast-2: active
us-east-1: active
ap-southeast-1: active
eu-west-1: active
Find SSL Certificates Expiring Soon
# Find ACM certificates expiring within 30 days
aws acm list-certificates \
--query "CertificateSummaryList[].CertificateArn" \
--output text | tr '\t' '\n' | while read arn; do
expiry=$(aws acm describe-certificate \
--certificate-arn "$arn" \
--query "Certificate.{Domain:DomainName,NotAfter:NotAfter,Status:Status}" \
--output json 2>/dev/null)
echo "$expiry"
done | jq -s '[.[] | select(.NotAfter != null)]
| sort_by(.NotAfter)
| .[] | select(
(.NotAfter | fromdateiso8601) < (now + 2592000)
) | {Domain, NotAfter, Status}'
{
"Domain": "staging.myapp.com",
"NotAfter": "2026-01-05T00:00:00+00:00",
"Status": "ISSUED"
}
The certificate for staging.myapp.com expires in 21 days. If it's a DNS-validated certificate, it will auto-renew, but email-validated certificates require manual renewal.
Find Security Groups Allowing 0.0.0.0/0
# Find security group rules allowing access from all IPs (0.0.0.0/0)
aws ec2 describe-security-groups \
--query "SecurityGroups[].{
GroupId:GroupId,
GroupName:GroupName,
OpenPorts:IpPermissions[?IpRanges[?CidrIp=='0.0.0.0/0']].{
Port:FromPort,
Protocol:IpProtocol
}
}" \
--output json | jq '[.[] | select(.OpenPorts | length > 0)]'
[
{
"GroupId": "sg-0abc1234def56789",
"GroupName": "web-server-sg",
"OpenPorts": [
{ "Port": 80, "Protocol": "tcp" },
{ "Port": 443, "Protocol": "tcp" }
]
},
{
"GroupId": "sg-0def5678abc01234",
"GroupName": "dev-temp-sg",
"OpenPorts": [
{ "Port": 22, "Protocol": "tcp" },
{ "Port": 3306, "Protocol": "tcp" }
]
}
]
The 80/443 ports being open on web-server-sg is normal, but dev-temp-sg has SSH (22) and MySQL (3306) open to the entire internet. This is a security issue that needs to be fixed immediately!
💡 Practical Tip: Collecting these security audit one-liners and running them as a weekly/monthly cron job that sends results to Slack is a great way to prevent security incidents. You can also set up automatic detection and remediation with AWS Config Rules.
Find Unattached EBS Volumes (Prevent Cost Waste)
# Find EBS volumes not attached to any instance (wasting money!)
aws ec2 describe-volumes \
--filters "Name=status,Values=available" \
--query "Volumes[].{
VolumeId:VolumeId,
Size:Size,
Type:VolumeType,
Created:CreateTime,
MonthlyCost:Size
}" \
--output table
---------------------------------------------------------------------------
| DescribeVolumes |
+------------+-------------------+-------+--------+-----------------------+
| Created | MonthlyCost(GB) | Size | Type | VolumeId |
+------------+-------------------+-------+--------+-----------------------+
| 2025-10-01 | 500 | 500 | gp3 | vol-0abc1234def56789 |
| 2025-11-15 | 100 | 100 | gp3 | vol-0def5678abc01234 |
| 2025-09-20 | 1000 | 1000 | io2 | vol-0ghi9012cde34567 |
+------------+-------------------+-------+--------+-----------------------+
There are 1,600GB of unattached EBS volumes sitting idle. At $0.08/month per GB for gp3 and $0.125/month per GB for io2, cleaning this up alone would save about $173 per month. Create snapshots first, then delete the volumes.
22.3 Troubleshooting Order Summary
No matter what the problem is, following this order will help you find the cause quickly.
Problem occurred!
|
+-- 1. Check current state
| +-- aws sts get-caller-identity (who am I?)
| +-- aws <service> describe-* (current resource state)
|
+-- 2. Track recent changes
| +-- aws cloudtrail lookup-events (who changed what?)
| +-- aws cloudformation describe-stack-events (deployment events)
|
+-- 3. Check logs
| +-- aws logs filter-log-events (application logs)
| +-- aws logs get-log-events (specific log stream)
|
+-- 4. Check networking
| +-- aws ec2 describe-security-groups (SG rules)
| +-- aws ec2 describe-network-acls (NACL rules)
| +-- aws ec2 describe-route-tables (routing)
|
+-- 5. Service-specific detailed checks
+-- (refer to the symptom-based command table above)
💡 Practical Tip: Bookmark this document and when a problem occurs, find the command from the symptom-based table and copy-paste it right away. Most AWS operational issues can be quickly diagnosed with these commands.
Wrapping Up the Series
We've covered a comprehensive summary of AWS CLI commands from Part 1 through Part 6. Here's an overview of everything covered in this series:
| Part | Scope | Key Services |
|---|---|---|
| Part 1 | Basic setup, IAM, EC2, VPC | Authentication, compute, networking basics |
| Part 2 | S3, EBS, ELB, Auto Scaling | Storage, load balancing, scaling |
| Part 3 | RDS, DynamoDB, ElastiCache | Databases |
| Part 4 | Lambda, API Gateway, ECS, ECR | Serverless, containers |
| Part 5 | Route 53, CloudFront, CloudWatch, SNS/SQS | DNS, CDN, monitoring, messaging |
| Part 6 | CloudFormation, Secrets, SSM, Cost, Troubleshooting | IaC, security, operations, cost management |
You don't need to memorize every command. What matters is knowing that these things are possible, and then referring to this document to find the right command when you need it. Building a habit of using the CLI instead of clicking through the console will make a real difference in automating repetitive tasks and speeding up troubleshooting.
Q&A0
Only students taking this course can post questions.
No questions yet. Be the first to ask!