Cost Engineering
GCP
The Frugal Approach to GCP Cloud Logging Costs
Craig Conboy

With usage-billed logging services like Google Cloud Logging, cost optimization starts with your code. It's not just that applications generate the logs; applications spend. Every console.log, every stack trace, every log statement contributes to your bill.

Taking an application- and code-centric approach to cost reduction means understanding what your code logs before and after each deployment. Here's a practical walk-through of what to look for and how to optimize Cloud Logging costs at the code level.

Cost Trap Efficiency Pattern
Logging at DEBUG level in production Set log level to INFO in production
Noisy library logging Silence verbose GCP/urllib3 loggers
Unfiltered health check logging Sample health checks (1 in 100)
Logging every Cloud Function/Run invocation Sample routine invocations (1%); always log errors
Logging repeated identical messages Sample repetitive logs with counter logic
Unstructured log messages Use structured logging (JSON format)
Redundant timestamp fields Don't include timestamps—GCP adds them
Redundant service name/environment Use GCP resource labels instead
Logging full request/response payloads Log only key identifiers and metadata
Full stack traces for expected errors Log only error message for business logic errors
Logging every database query Disable verbose ORM logging; log only slow queries
High-frequency counting via logs Convert to Cloud Monitoring Metrics
Multiple log messages for single operation Consolidate sequential messages into one log
Verbose repetitive field values Encode repetitive fields (bitfield compression)

Attribute the Costs

Start with your bill. Break down costs by application, service, and log type to understand where spend concentrates. GCP Cloud Logging primarily charges for ingestion (~$0.50 per GB, with first 50 GB per project per month free). Ingestion includes 30-day retention—GCP's most developer-friendly aspect. But that free 50 GB per project can disappear quickly at scale.

Attribute costs down to specific log statements in your codebase. Which log messages generate millions of events per day? Which Cloud Functions log on every invocation? This granular attribution reveals which of the 14 efficiency patterns below deliver the highest impact. The best way to decide if a log should be sent to Cloud Logging isn't to ask "is this useful?" (the answer is always "yes")—ask instead: is this $1,000/month useful? Once you can attach a price tag to specific log statements, optimization priorities become clear.

Note: GCP platform features like exclusion filters and log routing complement code-level optimizations. The patterns below focus on what your application code can control, all targeting ingestion volume reduction.

1. Set log level to INFO in production

Debug and info logs are valuable in development but wasteful in production. Logging at DEBUG level can generate 10-100x more volume than INFO level.

# Development
import logging
logging.basicConfig(level=logging.DEBUG)

# Production
logging.basicConfig(level=logging.INFO)
// Go: set log level via environment variable
logLevel := os.Getenv("LOG_LEVEL")
if logLevel == "" {
    logLevel = "INFO"
}

2. Silence verbose GCP/urllib3 loggers

GCP client libraries and dependencies can generate excessive debug output. Silence them to reduce noise:

# GCP libraries can be verbose
logging.getLogger('google').setLevel(logging.WARNING)
logging.getLogger('google.cloud').setLevel(logging.WARNING)
logging.getLogger('urllib3').setLevel(logging.WARNING)

3. Sample health checks

Health check endpoints can generate thousands of logs per day with minimal signal. Sample them to maintain visibility without paying for every occurrence:

import random

# Sample 1 in 100 health checks
def health_check():
    is_healthy = check_health()

    if random.randint(1, 100) == 1:
        logger.info("Health check passed (sampled)")

    return is_healthy

4. Sample Cloud Function/Run invocations

For high-frequency Cloud Functions or Cloud Run services, logging every invocation creates massive volume. Sample routine invocations while always logging errors:

// Bad: logs every invocation (millions/day at scale)
exports.handler = (req, res) => {
    console.log('Function invoked', req.body);
    // ...
};

// Good: sample routine invocations, always log errors
exports.handler = (req, res) => {
    if (Math.random() < 0.01) {  // 1% sample
        console.log('Function invoked (sampled)');
    }

    try {
        // ...
    } catch (error) {
        console.error('Function error', error);  // Always log errors
    }
};

5. Sample repetitive logs with counter logic

For repeated identical messages or high-frequency operations, use counter-based sampling to maintain visibility without paying for every occurrence:

var requestCounter int64

func handleRequest(r *http.Request) {
    counter := atomic.AddInt64(&requestCounter, 1)

    // Log every 100th request
    if counter%100 == 0 {
        log.Printf("Processed %d requests", counter)
    }

    // Always log errors
    if err != nil {
        log.Printf("Request error: %v", err)
    }
}

This can reduce log volume by 90-99% for high-frequency operations.

6. Use structured logging (JSON format)

Structured logs (JSON format) are more compressible, enable better GCP exclusion filters, and facilitate metrics extraction:

import json

# Instead of this:
logger.info(f"User {user_id} completed checkout for ${amount}")

# Do this:
logger.info(json.dumps({
    "event": "checkout_completed",
    "user_id": user_id,
    "amount": amount
}))
// Go: use structured logging
log.Printf(`{"event":"checkout_completed","user_id":"%s","amount":%d}`,
    userId, amount)

Structured logs enable precise GCP exclusion filters (which prevent ingestion entirely), compress better, and facilitate metrics extraction.

7. Don't include timestamps—GCP adds them

GCP automatically adds timestamps to every log event. Don't include redundant timestamp fields in your log messages:

# Bad: includes redundant timestamp
logger.info(f"{datetime.now().isoformat()} - User logged in")

# Good: just the message
logger.info("User logged in")

8. Use GCP resource labels instead of logging service name

Service name, environment, and region are automatically added by GCP via resource labels. Don't repeat this static context in every log message:

# Bad: repeats service name in every log
logger.info(f"[payment-service] Processing payment")

# Good: GCP resource labeling handles this
logger.info("Processing payment")

9. Log only key identifiers and metadata

Logging entire HTTP request bodies or response payloads can increase log volume by orders of magnitude. Log only the identifiers and metadata needed for debugging:

# Bad: logs 100KB+ per request
logger.info(f"API response: {json.dumps(response.body)}")

# Good: log only key identifiers
logger.info(json.dumps({
    "event": "api_response",
    "request_id": request_id,
    "status_code": response.status_code,
    "size_bytes": len(response.body)
}))

10. Log only error message for expected errors

Stack traces are valuable for unexpected errors, but logging full traces for expected business logic errors wastes volume. Differentiate between programming errors (log stack trace) and expected validation failures (log error message only):

# Bad: logs entire stack trace for business logic errors
try:
    process_order(order_id)
except OutOfStockError as e:
    logger.error("Order failed", exc_info=True)  # Full stack trace

# Good: log only what's needed
try:
    process_order(order_id)
except OutOfStockError as e:
    logger.error(f"Order failed: {str(e)}", extra={"order_id": order_id})

This can reduce per-log size by 20-50%, directly reducing ingestion costs.

11. Disable verbose ORM logging; log only slow queries

ORM query logging is a major cost trap in high-throughput applications. Databases can execute thousands of queries per minute, and logging each one creates massive volume with minimal debugging value in production.

Disable verbose database logging:

# Django: disable SQL query logging in production
LOGGING = {
    'loggers': {
        'django.db.backends': {
            'level': 'INFO',  # Not DEBUG
        }
    }
}

Log only slow queries:

import time

def execute_query(query):
    start = time.time()
    result = db.execute(query)
    duration = time.time() - start

    # Only log slow queries
    if duration > 1.0:
        logger.warning(json.dumps({
            "event": "slow_query",
            "duration_seconds": duration,
            "query": query[:100]  # Truncate
        }))

    return result

12. Convert to Cloud Monitoring Metrics

Cloud Monitoring Metrics are dramatically cheaper than equivalent log volume. For counting and aggregation, use metrics instead of logs.

Instead of logging each request:

from google.cloud import monitoring_v3

client = monitoring_v3.MetricServiceClient()
project_name = f"projects/{project_id}"

# Expensive: one log per request
logger.info(f"Request processed: {endpoint}")

# Cheap: increment a metric
series = monitoring_v3.TimeSeries()
series.metric.type = "custom.googleapis.com/request_count"
series.metric.labels["endpoint"] = endpoint
# ... (complete metric configuration)
client.create_time_series(name=project_name, time_series=[series])

Or use log-based metrics (still cheaper than full log volume):

# Instead of logging every occurrence
logger.info(f"Request processed: {endpoint}")

# Log once with a counter field, then use GCP log-based metrics
logger.info(json.dumps({
    "event": "request_processed",
    "endpoint": endpoint,
    "count": 1  # Create log-based metric from this
}))

Reserve logs for context-rich events:

# Use metrics for counting
# (metric code)

# Use logs for errors with context
if latency_ms > 5000:
    logger.error(json.dumps({
        "event": "slow_request",
        "request_id": request_id,
        "latency_ms": latency_ms,
        "query_count": query_count
    }))

Metrics are orders of magnitude cheaper than equivalent log volume.

13. Consolidate sequential messages into one log

Don't emit multiple logs for a single operation when one message would do. Each log message carries overhead from timestamps and metadata—four messages cost 4x what one message costs, even if the actual data is the same:

# Bad: 4 separate messages with redundant metadata
logger.info("Starting task")
logger.info("Processing item 1")
logger.info("Processing item 2")
logger.info("Task complete")

# Good: one consolidated message with all info
logger.info(json.dumps({
    "event": "task_completed",
    "items_processed": 2,
    "duration_ms": 150
}))

14. Encode repetitive fields (bitfield compression)

If a single field with repetitive verbose data (OAuth scopes, permission lists, feature flags) accounts for a disproportionate percentage of your log volume, consider encoding it. Techniques like bitfield compression can reduce a 150+ character field to 22 characters—achieving 90-97% compression on that field alone.

This can be a game-changer when one field dominates your bill. We've seen cases where a single field accounted for 25% of total logging costs.

# Bad: verbose permissions list logged on every request
logger.info(json.dumps({
    "event": "request_authorized",
    "permissions": "read:users write:users delete:users read:posts write:posts delete:posts"
}))

# Good: encode as bitfield
permission_bits = encode_permissions_as_bitfield(user.permissions)
logger.info(json.dumps({
    "event": "request_authorized",
    "permissions_encoded": permission_bits  # "0x1F3A" instead of 150+ chars
}))

Closing Thoughts

Basic log hygiene is table stakes—set appropriate log levels, silence noisy GCP/urllib3 loggers, sample health checks and Cloud Function/Run invocations. Use GCP exclusion filters to prevent low-value log ingestion. This baseline discipline prevents the worst cost traps.

Getting to the next level of savings requires two steps. First, let observed costs guide what needs optimization. Attribute your bill down to specific log statements in your codebase—which messages generate millions of events per day, which Cloud Functions log on every invocation. This granular attribution reveals which of the 14 efficiency patterns above matter most for your application. Second, apply the optimizations that address your cost concentrations. These patterns reduce volume without losing visibility: filtering at the source eliminates low-value logs, sampling maintains visibility on repetitive messages, converting to metrics handles high-frequency counting efficiently, and trimming payloads keeps messages focused on signal over noise.

You don't lose visibility. You gain precision. Cost-effective logging is an engineering discipline—treat each log statement as having a price tag and optimize based on measured impact.

Looking for help with cost optimizations like these? Sign up for Early Access to Frugal. Frugal attributes GCP Cloud Logging costs to your code, finds inefficient usage, provides Frugal Fixes that reduce your bill, and helps keep you out of cost traps for new and changing code.

Back to Top