How to Build a Logging System

Many indie developers start logging with a few console.log lines.

That works during local debugging, but once the product is live, it breaks down quickly. A user says “it failed just now,” but you do not know which request it was. A payment webhook fails, but you do not know which step failed. A background job disappears, and all you can do is search server files. A production issue happens occasionally, and you cannot reproduce it.

The goal of logging is not “print more text.” The goal is to answer three questions when something goes wrong:

What happened?
Who did it happen to?
Why did it happen?

This article references official materials:

• OpenTelemetry Logs
• OpenTelemetry Logs Specification
• OWASP Logging Cheat Sheet
• Grafana Loki Labels
• Grafana Loki Label Best Practices

What Logs Are For

Logs are event records produced while your system is running.

They are different from metrics, which answer whether the system is healthy overall. They are also different from error monitoring, which focuses on exceptions and stack traces. Logs are better for detailed investigation.

For example:

What did this user just do?
Which services handled this request?
What status did the third-party API return?
Which step did the background job reach?
Why did this order not become paid?

Logs are the most common “scene record” during production debugging. Without logs, many issues become guesswork.

Do Not Only Write String Logs

Early logs often look like this:

payment failed
user login success
send email error

These logs are readable, but they are hard to query, aggregate, and correlate.

A better approach is structured logging. Each log entry should contain consistent fields so it can be searched and filtered later.

A basic log entry can include:

timestamp
level
service
environment
request_id
user_id
team_id
route
action
status_code
duration_ms
error_code
message
metadata

Do not put everything inside message. Fields you need to query should be structured fields.

Standardize Log Levels

Log levels are not decoration. They determine how you filter noise.

Common levels can be understood this way:

debug: local or temporary investigation detail
info: normal business event
warn: unexpected, but the system can continue
error: request or job failed and needs attention
fatal: process-level serious failure that needs immediate action

An early product often makes one of two mistakes: everything is info, or every exception is error.

If levels are unclear, production debugging becomes noisy.

A simple rule:

Normal user action succeeds: info
Third-party API temporarily times out but retry succeeds: warn
Request ultimately fails: error
Process is about to exit: fatal
Local debugging detail: debug

Request ID Is the Backbone

If your system has HTTP requests, frontend calls, background jobs, webhooks, or payment callbacks, add request_id or trace_id early.

Its purpose is to connect scattered logs.

One payment flow may include:

Frontend starts checkout
Backend creates checkout session
Payment platform sends webhook
Database updates order
Email system sends receipt
Background job syncs subscription state

If each step has the same request_id, trace_id, or business identifier such as order_id, you can reconstruct the entire path.

OpenTelemetry's logs documentation also emphasizes that logs can be correlated with traces and spans, so logs are no longer isolated text but part of an observability flow.

Where You Must Log

Do not add logs to every line of code. Cover important boundaries first.

The most important places:

User login, signup, and logout
Permission changes
Payment creation, success, failure, and refund
Webhook receive, verification, and processing result
Email sending result
File upload and deletion
Background job start, completion, and failure
Third-party API call result
Important configuration change
Security-related event

These events affect money, permissions, user assets, or system stability.

Normal page views and ordinary button clicks do not always belong in server logs. Behavioral analytics can be handled by an analytics system. Logs should focus on debugging, auditing, and traceability.

Separate Log Types

A product usually has several types of logs.

Application logs: runtime records from business code
Access logs: HTTP path, status code, duration
Error logs: exceptions, stack traces, failure context
Audit logs: who changed what and when
Security logs: login failure, permission denial, suspicious access
Job logs: background jobs, queues, scheduled tasks

They can go into the same logging platform, but their fields and retention policies may differ.

Audit logs and security logs usually need to be more complete and retained longer. Debug logs can be short-lived. Access logs can be high-volume, so cost matters more.

Do Not Log Sensitive Data

Once logs enter a system, they may be searched, forwarded, backed up, aggregated, and retained for a long time.

So do not write sensitive data into logs.

Especially avoid:

Passwords
Verification codes
Access tokens
Refresh tokens
Card numbers
Identity document images
Full phone numbers or emails
Private user content
Third-party platform secrets
Cookie and Authorization headers

The OWASP Logging Cheat Sheet also emphasizes security, privacy, and data protection concerns in logging.

If you need to identify a user, log an internal user_id, masked email, order ID, or team ID instead of raw private data.

Centralize Log Collection

If your logs only live in local server files, debugging becomes painful.

Once you have multiple machines, services, serverless functions, background jobs, or containers, you need one place to search logs.

Early on, platform logs may be enough:

Vercel Logs
Cloudflare Logs
Render Logs
Railway Logs
Fly.io Logs
Cloud provider console logs

When volume and query needs grow, consider a dedicated platform:

ELK / OpenSearch
Grafana Loki
Datadog
Better Stack
New Relic
Sentry Logs

Do not start with an oversized logging stack. But also do not wait until an incident to discover that logs are scattered across many places.

Be Careful With Indexed Fields

Structured logs do not mean every field should become an index.

In systems like Loki, labels affect query and storage behavior. Grafana Loki documentation warns that high-cardinality labels can create performance problems.

High-cardinality fields often include:

user_id
request_id
order_id
email
ip
uuid
session_id

These fields can live inside the log body or structured metadata, but they may not be good top-level labels or indexed fields.

Early on, use low-cardinality fields as primary filters:

service
environment
level
region
route
job_name

Use high-cardinality fields for precise search, but do not let them overload your logging system.

Design Retention and Cost

Logs are usually not expensive because one line is expensive. They become expensive because volume is high, retention is long, and queries are frequent.

Define retention rules early:

debug logs: local or short-term
info logs: 7 to 30 days
error logs: 30 to 90 days
audit logs: retained longer based on business and compliance needs
security logs: retained longer based on security policy
access logs: designed separately based on traffic and cost

There is no universal retention period, but you need rules.

A logging system without retention rules can quietly become a cost sink.

Alerts Are Not the Same as Logs

Logs can produce alerts, but every error log should not trigger a notification.

Better alerts aggregate patterns:

Login failures spike within 5 minutes
Payment webhooks fail repeatedly
The same job fails after multiple retries
Error rate exceeds a threshold
A third-party API timeout rate increases
Security-related events appear

Logs record facts. Alerts tell you “this needs action.” If alerts are too noisy, the team will start ignoring them.

A Minimum Viable Setup

If you are an indie developer, start here:

1. All services output JSON structured logs
2. Each request gets a request_id
3. Logs include service, env, level, user_id, route, duration_ms
4. Payment, login, permission, webhook, and job flows must be logged
5. Passwords, tokens, and raw private data are forbidden in logs
6. Logs first go into your deployment platform's built-in log system
7. error logs and key business failures connect to alerts
8. Retention differs by log type

This setup is not heavy, but it is enough to debug most early product issues.

Final Thoughts

A logging system is not about looking professional. It is about reducing uncertainty in production.

Early on, the most important things are fields, levels, key events, request correlation, sensitive data control, and centralized search.

A good logging system passes a simple test: when a user says “it failed just now,” you can find that request within minutes, see the steps it went through, identify the likely cause, and know what to fix next.

Next, we continue with production stability: How to Do Error Monitoring.