Monitoring with CloudWatch and CloudTrail
6 pages · ~5 min✓ Reviewed
Builds on Containers on ECS and EKS. Next up: CloudFormation, CDK and Cost Basics.
CloudWatch Metrics, Alarms and Dashboards
1. Metrics
- Metric = data points by namespace + name + dimensions
- Each unique dimension combo is a separate metric
- EC2 basic = 5 min; detailed = 1 min (paid)
- EC2 does not report memory or disk-used
- Install the CloudWatch agent to collect them
2. Alarms
| Setting | Meaning |
|---|---|
| Statistic | Average, Sum, Max |
| Period | seconds per window |
| Threshold | value that counts as breach |
| Evaluation periods (N) | latest windows checked |
| Datapoints to alarm (M) | breaches needed, e.g. 3 of 5 |
- OK
- ALARMM of N breach
- INSUFFICIENT_DATAtoo few points
| Missing data | Effect |
|---|---|
missing (default) | uses existing points; sparse metric sticks in INSUFFICIENT_DATA |
notBreaching | gaps count as OK |
breaching | gaps breach: stopped instance falsely fires |
ignore | keeps current state |
- Actions: SNS, EC2 stop/terminate/reboot/recover, Auto Scaling
- Test:
aws cloudwatch set-alarm-state --alarm-name X --state-value ALARM --state-reason test
3. Dashboards
- Mix metrics from multiple regions
- Mix metrics from multiple accounts
- Metric math:
errors / requests * 100= error rate %
CloudWatch Logs, Logs Insights and Custom Metrics
1. Hierarchy and ingest
| Level | Holds | Example |
|---|---|---|
| Log group | retention, permissions | /aws/lambda/orders |
| Log stream | one instance or container | i-0abc |
| Log event | timestamped line | one log line |
- Retention defaults to Never expire
- Set retention to control cost
- Ingest: CloudWatch agent, awslogs driver, SDK
- Role needs
logs:CreateLogStream,logs:PutLogEvents
2. Insights and metric filters
- Insights queries logs with a command pipeline
- Metric filter turns a log pattern into a metric
- An alarm can watch that metric
- Counts only events ingested after creation
text
fields @timestamp, @message | filter @message like /ERROR/ | stats count() as errors by bin(5m) as t | sort t desc | limit 20
3. Custom metrics and cost
CloudTrail: Management vs Data Events and Trails to S3
1. Event Types
- CloudTrail records AWS API calls from console, CLI and SDK
- Event history: last 90 days, management events, free, per region
- History cannot be extended
| Management | Data | |
|---|---|---|
| Plane | Control | Data |
| Examples | RunInstances, CreateBucket, AttachRolePolicy | S3 GetObject/PutObject, Lambda Invoke, DynamoDB item ops |
| Default | ON | OFF |
| Volume | Low | High, billed |
2. Trail to S3
- A trail delivers events to S3 (optionally CloudWatch Logs)
- Use it for retention beyond 90 days
bash
aws cloudtrail create-trail --name org-trail \ --s3-bucket-name my-logs --is-multi-region-trail aws cloudtrail start-logging --name org-trail
Who-Did-What Investigations and AWS Config
1. Reading a CloudTrail event
| Field | Answers |
|---|---|
eventTime | when |
eventName / eventSource | which call, which service |
userIdentity | who: type, arn, principalId, sessionContext |
sourceIPAddress / userAgent | from where, which tool |
requestParameters | inputs sent |
errorCode | why it failed |
2. Finding the actor
bash
aws cloudtrail lookup-events \
--lookup-attributes \
AttributeKey=EventName,AttributeValue=TerminateInstancesrecent management events
- Older or data events: query the S3 trail with
Athena - An assumed role shows the role session, not the person
- 1Role sessionuserIdentity
- 2sessionContextsourceIdentity
- 3AssumeRole eventoriginal call
- 4Real userhuman or federated
Observability and Audit Cheat Sheet
1. Metrics and Alarms
| Mode | Resolution |
|---|---|
| Basic | 5 min |
| Detailed | 1 min (paid) |
| High resolution | 1 sec |
- Metric = namespace + name + dimensions
- Memory and disk need the agent
- States: OK / ALARM / INSUFFICIENT_DATA
- M of N datapoints to alarm
- Set treat missing data deliberately
2. Logs
- Group > stream > event
- Set retention
- Metric filter counts new events only
- Insights bills per GB scanned
3. CloudTrail and Config
- Management events on, data events off
- History 90 days; trail to S3 for longer
- Enable log file validation
Check yourself
You create a metric filter for ERROR today, but yesterday's 40 ERROR lines never show in the metric. Why, and what now?
- Answer: a metric filter only counts events ingested after it was created.
- Fix: run Logs Insights over yesterday:
filter @message like /ERROR/ | stats count() by bin(5m).
An alarm needs 3 of 5 datapoints breaching. The last 5 are: breach, OK, breach, OK, breach. State?
- Answer: ALARM, 3 of the last 5 breach.
- The breaches need not be consecutive; any M of the last N count.
Someone deleted an S3 object. lookup-events for DeleteObject returns nothing. Why?
- Answer: object operations are data events, which are off by default.
- Fix: enable data events on a trail for that bucket; past deletes are not recoverable from history.