With usage-billed logging services like CloudWatch, cost optimization starts with your code. It's not just that applications generate the logs; applications spend. Every console.log, every stack trace, every log statement contributes to your bill.
Taking an application- and code-centric approach to cost reduction means understanding what your code logs before and after each deployment. Here's a practical walk-through of what to look for and how to optimize CloudWatch Logs costs at the code level.
| Cost Trap | Efficiency Pattern |
|---|---|
| Logging at DEBUG level in production | Set log level to INFO; silence AWS SDKs |
| Unfiltered health check logging | Sample health checks (1 in 100) |
| Logging every Lambda invocation | Sample routine invocations (1%); always log errors |
| Logging every database query | Disable ORM query logging in production |
| High-frequency counting via logs | Convert to CloudWatch Metrics |
| Logging every request | Filter at source—only log errors/significant events |
| Multiple log messages for single operation | Consolidate sequential messages into one log |
| Logging full request/response payloads | Log only key identifiers and metadata |
| Redundant timestamp and service fields | Trim redundant fields—CloudWatch adds timestamps |
| Full stack traces for expected errors | Log only error message for business logic errors |
| Unstructured log messages | Use structured logging (JSON format) |
| Verbose repetitive field values | Encode repetitive fields (bitfield compression) |
| Indefinite log retention (CloudWatch default) | Set explicit retention periods (7-90 days) |
Attribute the Costs
Start with your bill. Break down costs by application, Lambda function, and log group to understand where spend concentrates. CloudWatch charges for log ingestion (~$0.50 per GB in US East) and storage (per GB-month for retained logs). Your bill reveals whether you're burning budget on excessive ingestion (high-volume application logs) or runaway storage (indefinite retention is the default trap).
Attribute costs down to specific log statements in your codebase. Which log messages generate millions of events per day? Which Lambda functions log on every invocation? This granular attribution reveals which of the 13 efficiency patterns below deliver the highest impact. The best way to decide if a log should be sent to CloudWatch isn't to ask "is this useful?" (the answer is always "yes")—ask instead: is this $1,000/month useful? Once you can attach a price tag to specific log statements, optimization priorities become clear.
Note: CloudWatch platform features like retention policies and S3 exports complement code-level optimizations. The patterns below focus on what your application code can control.
1. Set log level to INFO and silence AWS SDKs
Cost impact: Ingestion
Debug and verbose logging are valuable in development but wasteful in production. Logging at DEBUG level can generate 10-100x more volume than INFO level.
# Production: set log level to INFO
import logging
logging.basicConfig(level=logging.INFO)
# Silence noisy AWS SDKs
logging.getLogger('boto3').setLevel(logging.WARNING)
logging.getLogger('botocore').setLevel(logging.WARNING)
2. Sample health checks
Cost impact: Ingestion
Health check endpoints can generate thousands of logs per day with minimal signal. Sample them at the source to maintain visibility without paying for every ping:
import random
def health_check():
is_healthy = check_health()
# Log only 1 in 100 health checks
if random.randint(1, 100) == 1:
logger.info("Health check passed (sampled)")
return is_healthy
3. Sample Lambda invocations
Cost impact: Ingestion
For high-frequency Lambda functions, logging every invocation creates massive volume. Sample routine invocations while always logging errors:
// Bad: logs every Lambda invocation (millions/day at scale)
exports.handler = async (event) => {
console.log('Lambda invoked', JSON.stringify(event));
// ...
}
// Good: sample routine invocations, always log errors
exports.handler = async (event) => {
if (Math.random() < 0.01) { // 1% sample
console.log('Lambda invoked (sampled)');
}
try {
// ...
} catch (error) {
console.error('Lambda error', error); // Always log errors
}
}
4. Disable ORM query logging in production
Cost impact: Ingestion
ORM query logging is a major cost trap. Databases can execute thousands of queries per minute, and logging each one creates massive volume with minimal debugging value in production:
# Django: disable SQL query logging in production
LOGGING = {
'loggers': {
'django.db.backends': {
'level': 'INFO', # Not DEBUG
}
}
}
5. Convert to CloudWatch Metrics
Cost impact: Ingestion
For counting and aggregation, CloudWatch Metrics are dramatically cheaper than logs. If you're logging to count events, you're using the wrong tool:
import boto3
cloudwatch = boto3.client('cloudwatch')
# Expensive: one log per request
logger.info(f"Request processed: {endpoint}")
# Cheap: increment a metric
cloudwatch.put_metric_data(
Namespace='MyApp',
MetricData=[{
'MetricName': 'RequestCount',
'Value': 1,
'Unit': 'Count'
}]
)
6. Filter at source—only log errors and significant events
Cost impact: Ingestion
The most effective cost reduction happens before logs ever reach CloudWatch. Don't send low-value logs in the first place:
# Bad: logs every request
logger.info(f"Processing request {request_id}")
# Good: only log errors or significant events
if response.status_code >= 400:
logger.error(f"Request failed {request_id}: {response.status_code}")
7. Consolidate sequential log messages
Cost impact: Ingestion
Don't emit multiple logs for a single operation when one message would do. Each log message carries overhead from timestamps and metadata—four messages cost 4x what one message costs:
# Bad: 4 separate messages with redundant metadata
logger.info("Starting task")
logger.info("Processing item 1")
logger.info("Processing item 2")
logger.info("Task complete")
# Good: one consolidated message with all info
logger.info(json.dumps({
"event": "task_completed",
"items_processed": 2,
"duration_ms": 150
}))
8. Log only key identifiers and metadata
Cost impact: Ingestion
Logging entire HTTP request bodies or response payloads can increase log volume by orders of magnitude. Log only the identifiers and metadata needed for debugging:
# Bad: logs 100KB+ per request
logger.info(f"API response: {json.dumps(response.body)}")
# Good: log only key identifiers
logger.info(json.dumps({
"event": "api_response",
"request_id": request_id,
"status_code": response.status_code
}))
9. Trim redundant fields
Cost impact: Ingestion
CloudWatch automatically adds timestamps to every log event. Don't include redundant metadata that the platform already provides:
# Bad: redundant timestamp and service name
logger.info(f"{datetime.now()} [payment-service] Processing payment")
# Good: use log group naming, log just the event
logger.info("Processing payment")
10. Log only error message for expected errors
Cost impact: Ingestion
Stack traces are valuable for unexpected errors, but logging full traces for expected business logic errors wastes volume. Differentiate between programming errors (log stack trace) and expected validation failures (log error message only):
# Bad: logs entire stack trace for business logic errors
try:
process_order(order_id)
except OutOfStockError as e:
logger.error("Order failed", exc_info=True)
# Good: log only what's needed
try:
process_order(order_id)
except OutOfStockError as e:
logger.error(f"Order failed: {str(e)}", extra={"order_id": order_id})
11. Use structured logging (JSON format)
Cost impact: Ingestion (improves compression and querying)
Emit logs in JSON format with consistent field names. Structured logs compress better (3-10x) and enable efficient CloudWatch Insights queries:
# Use structured logging
logger.info(json.dumps({
"event": "checkout_completed",
"user_id": user_id,
"amount": amount
}))
12. Encode repetitive fields (bitfield compression)
Cost impact: Ingestion
If a single field with repetitive verbose data (OAuth scopes, permission lists, feature flags) accounts for a disproportionate percentage of your log volume, consider encoding it. Techniques like bitfield compression can reduce a 150+ character field to 22 characters—achieving 90-97% compression on that field alone:
# Bad: verbose permissions list logged on every request
logger.info(json.dumps({
"event": "request_authorized",
"permissions": "read:users write:users delete:users read:posts write:posts delete:posts"
}))
# Good: encode as bitfield
permission_bits = encode_permissions_as_bitfield(user.permissions)
logger.info(json.dumps({
"event": "request_authorized",
"permissions_encoded": permission_bits # "0x1F3A" instead of 150+ chars
}))
This can be a game-changer when one field dominates your bill. We've seen cases where a single field accounted for 25% of total logging costs.
13. Set explicit retention periods
Cost impact: Storage
CloudWatch's default retention is indefinite—this is a trap. Storage costs accumulate month after month. Always configure retention:
- 7 days for verbose application logs
- 30-90 days for operational logs
- 365+ days only for compliance logs
Export to S3 for long-term retention—S3 is 10-50x cheaper than CloudWatch storage for logs you need to keep but rarely query.
Closing Thoughts
Basic log hygiene is table stakes—set appropriate log levels, silence noisy SDKs, sample health checks and Lambda invocations. Set explicit retention periods (CloudWatch's indefinite default is a trap). This baseline discipline prevents the worst cost traps.
Getting to the next level of savings requires two steps. First, let observed costs guide what needs optimization. Attribute your bill down to specific log statements in your codebase—which messages generate millions of events per day, which Lambda functions log on every invocation. This granular attribution reveals which of the 13 efficiency patterns above matter most for your application. Second, apply the optimizations that address your cost concentrations. These patterns reduce volume without losing visibility: filtering at the source eliminates low-value logs, sampling maintains visibility on repetitive messages, converting to metrics handles high-frequency counting efficiently, and trimming payloads keeps messages focused on signal over noise.
You don't lose visibility. You gain precision. Cost-effective logging is an engineering discipline—treat each log statement as having a price tag and optimize based on measured impact.
Looking for help with cost optimizations like these? Sign up for Early Access to Frugal. Frugal attributes AWS CloudWatch Logs costs to your code, finds inefficient usage, provides Frugal Fixes that reduce your bill, and helps keep you out of cost traps for new and changing code.