The Frugal Approach to Datadog Log Management Costs

With usage-billed logging services like Datadog, cost optimization starts with your code. It's not just that applications generate the logs; applications spend. Every logger.info, every stack trace, every log statement contributes to your bill.

Taking an application- and code-centric approach to cost reduction means understanding what your code logs before and after each deployment. Here's a practical walk-through of what to look for and how to optimize Datadog Log Management costs at the code level.

Attribute the Costs

Start with your bill. Break down costs by application, service, and log source to understand where spend concentrates. Datadog charges for indexed logs (stored and searchable, typically ~$1.27 per million events for 7-day retention) and ingested logs (volume processed, ~$0.10 per GB). Your bill reveals whether you're burning budget on excessive indexing (too many low-value logs being stored) or excessive ingestion (high-volume, oversized log messages).

Attribute costs down to specific log statements in your codebase. Which log messages generate millions of events per day? Which ones carry massive payloads? This granular attribution reveals which of the 14 efficiency patterns below deliver the highest impact. The best way to decide if a log should be indexed isn't to ask "is this useful?" (the answer is always "yes")—ask instead: is this $1,000/month useful? Once you can attach a price tag to specific log statements, optimization priorities become clear.

Note: Datadog offers platform features like retention settings, Flex Logs, and exclusion filters that complement code-level optimizations. The patterns below focus on what your application code can control.

1. Set log level to INFO and silence noisy libraries

Cost impact: Both indexing and ingestion

Debug and verbose logging are valuable in development but wasteful in production. Logging at DEBUG level can generate 10-100x more volume than INFO level.

# Production: set log level to INFO
import logging
logging.basicConfig(level=logging.INFO)

# Silence noisy libraries
logging.getLogger('boto3').setLevel(logging.WARNING)

2. Sample health checks

Cost impact: Both indexing and ingestion

Health check endpoints can generate thousands of logs per day with minimal signal. Sample them at the source to maintain visibility without paying to index every ping:

import random

def health_check():
    is_healthy = check_health()

    # Log only 1 in 100 health checks
    if random.randint(1, 100) == 1:
        logger.info("Health check passed (sampled)")

    return is_healthy

3. Sample repetitive logs with counter logic

Cost impact: Both indexing and ingestion

If a log message appears hundreds of times per day, sample it. You maintain visibility without paying to index every instance:

request_counter = 0

def handle_request(request_id):
    global request_counter
    request_counter += 1

    # Log every 100th request
    if request_counter % 100 == 0:
        logger.info(f"Processed {request_counter} requests")

    # Always log errors
    if error:
        logger.error(f"Request failed: {request_id}")

4. Disable ORM query logging in production

Cost impact: Both indexing and ingestion

ORM query logging is a major cost trap. Databases can execute thousands of queries per minute, and logging each one creates massive volume with minimal debugging value in production:

# Django: disable SQL query logging in production
LOGGING = {
    'loggers': {
        'django.db.backends': {
            'level': 'INFO',  # Not DEBUG
        }
    }
}

5. Convert high-frequency counting to metrics

Cost impact: Both indexing and ingestion

For counting and aggregation, metrics are 10-100x cheaper than logs. If you're logging to count events, you're using the wrong tool:

# Expensive: one log per request
logger.info(f"Request processed: {endpoint}")

# Cheap: increment a counter metric
statsd.increment('requests.processed', tags=[f'endpoint:{endpoint}'])

6. Filter at source—only log errors and significant events

Cost impact: Both indexing and ingestion

The most effective cost reduction happens before logs ever reach Datadog. Configure applications to filter out low-value logs at the logging library level:

# Bad: logs every request
logger.info(f"Processing request {request_id}")

# Good: only log errors or significant events
if response.status_code >= 400:
    logger.error(f"Request failed {request_id}: {response.status_code}")

7. Feature flag detailed logs

Cost impact: Both indexing and ingestion

Keep the ability to turn on high-verbosity logs per region, service, or customer without paying to index them all the time. Feature flags let you enable detailed logging on demand for debugging specific issues:

# Use feature flags to control log verbosity
if feature_flags.is_enabled('detailed_logging', service='payment-service'):
    logger.debug(f"Processing payment: {payment_details}")
else:
    logger.info("Payment processed", extra={"payment_id": payment_id})

8. Consolidate sequential log messages

Cost impact: Both indexing and ingestion

Don't emit multiple logs for a single operation when one message would do. Each log message carries overhead from timestamps and metadata—four messages cost 4x what one message costs:

# Bad: 4 separate messages with redundant metadata
logger.info("Starting task")
logger.info("Processing item 1")
logger.info("Processing item 2")
logger.info("Task complete")

# Good: one consolidated message with all info
logger.info("task_completed", extra={
    "items_processed": 2,
    "duration_ms": 150
})

9. Log only key identifiers and metadata

Cost impact: Ingestion

Logging entire HTTP request bodies or response payloads can increase log volume by orders of magnitude. Log only the identifiers and metadata needed for debugging:

# Bad: logs 100KB+ per request
logger.info(f"Request: {json.dumps(request.body)}")

# Good: log only key identifiers
logger.info("Request received", extra={
    "request_id": request.headers.get('x-request-id'),
    "endpoint": request.path,
    "method": request.method
})

10. Trim redundant fields

Cost impact: Ingestion

JSON context is powerful, but if every log carries redundant metadata that the platform already provides, you're paying for duplication. Datadog automatically adds timestamps, service names, and environment tags—your application doesn't need to include them:

# Bad: redundant timestamp and service name
logger.info(f"{datetime.now()} [payment-service] Processing payment")

# Good: Datadog adds timestamp, use tags for service name
logger.info("Processing payment")

11. Log only error message for expected errors

Cost impact: Ingestion

Stack traces are valuable for unexpected errors, but logging full traces for expected business logic errors wastes volume. Differentiate between programming errors (log stack trace) and expected validation failures (log error message only):

# Bad: logs entire stack trace for business logic errors
try:
    process_payment(amount)
except InsufficientFundsError as e:
    logger.error("Payment failed", exc_info=True)  # Full stack trace

# Good: log only what's needed
try:
    process_payment(amount)
except InsufficientFundsError as e:
    logger.error(f"Payment failed: {str(e)}")

12. Use structured logging (JSON format)

Cost impact: Ingestion (improves compression and filtering)

Emit logs in JSON format with consistent field names. Structured logs compress better (3-10x), enable precise Datadog filtering, and facilitate metrics extraction:

# Use structured logging libraries
logger.info("checkout_completed", extra={
    "user_id": user_id,
    "amount": amount,
    "cart_items": item_count
})

13. Encode repetitive fields (bitfield compression)

Cost impact: Ingestion

If a single field with repetitive verbose data (OAuth scopes, permission lists, feature flags) accounts for a disproportionate percentage of your log volume, consider encoding it. Techniques like bitfield compression can reduce a 150+ character field to 22 characters—achieving 90-97% compression on that field alone:

# Bad: verbose permissions list logged on every request
logger.info("Request authorized", extra={
    "permissions": "read:users write:users delete:users read:posts write:posts delete:posts read:comments write:comments"
})

# Good: encode as bitfield
permission_bits = encode_permissions_as_bitfield(user.permissions)
logger.info("Request authorized", extra={
    "permissions_encoded": permission_bits  # "0x1F3A" instead of 150+ chars
})

This can be a game-changer when one field dominates your bill. We've seen cases where a single field accounted for 25% of total logging costs.

14. Enable compression in Datadog agent

Cost impact: Ingestion

Enable gzip compression in your Datadog agent configuration. Text-heavy logs can compress significantly, reducing the volume sent over the network:

# datadog.yaml
logs_config:
  use_compression: true
  compression_level: 6  # Balance between speed and compression ratio

Compression happens before ingestion metering, reducing your ingested log volume costs directly.

Closing Thoughts

Basic log hygiene is table stakes—set appropriate log levels, silence noisy libraries, sample health checks. Configure Datadog features like retention settings, Flex Logs, and exclusion filters to match your operational needs. This baseline discipline prevents the worst cost traps.

Getting to the next level of savings requires two steps. First, let observed costs guide what needs optimization. Attribute your bill down to specific log statements in your codebase—which messages generate millions of events per day, which ones carry oversized payloads. This granular attribution reveals which of the 14 efficiency patterns above matter most for your application. Second, apply the optimizations that address your cost concentrations. These patterns reduce volume without losing debuggability: filtering at the source eliminates low-value logs, sampling maintains visibility on repetitive messages, converting to metrics handles high-frequency counting efficiently, and trimming payloads keeps messages focused on signal over noise.

You don't lose debuggability. You gain precision. Cost-effective logging is an engineering discipline—treat each log statement as having a price tag and optimize based on measured impact.

Looking for help with cost optimizations like these? Sign up for Early Access to Frugal. Frugal attributes Datadog Log Management costs to your code, finds inefficient usage, provides Frugal Fixes that reduce your bill, and helps keep you out of cost traps for new and changing code.