Key takeaway
Choose sources and investigation goals first, then set transport, field mapping, retention, access and cost controls to fit that purpose.
The path from source to search
A centralized logging system does more than gather files. In a typical pipeline, a source creates an event, an agent or API transports it, an intake layer validates the format, a parser extracts fields, processing can add metadata and trace context, an index or archive stores the data, and search and alerts make it useful to operators. Track records that are lost, delayed or duplicated at each stage.
Source and transport
An application, OS, web server, network device or cloud service, sent through an agent, syslog or API.
Ingest and processing
Recognize the format, extract fields and preserve time and source; add metadata only where needed.
Storage and use
Set frequent-search storage, lower-cost archive, queries and alert routing for the intended use.
Parsing and normalization
A parser extracts fields such as time, level, source, HTTP status and message from raw entries. Structured formats such as JSON and Windows Event Log may already expose fields; free text and CSV may need rules, delimiters or pattern-based parsing. Validate representative records: time zones, empty values, escaping and format changes can break field mapping. Keep the original record permitted by your data policy so parsing changes can be checked later; exclude secrets and unnecessary personal data.
Identify the format
W3C, JSON/JSONL, CSV, syslog and Windows Event Log have different structures.
Stabilize fields
Align field names, types and time conventions across sources.
Test the parser
Recheck sample records when fields or formats change.
Search, analysis and correlation
Start a search with a time range, service, event type and outcome. Correlate rising repeated errors, a failed dependency and a release time; do not treat one record as proof of cause. In the Elastic Stack, Elastic Agent or Filebeat can ship data, Logstash can handle more complex processing, Elasticsearch can index and search it, and Kibana can present queries and dashboards. Graylog offers search, pipeline rules and dashboards; a metrics system such as Prometheus does not replace event logs. Product capabilities depend on version and configuration.
Retention, integrity and service choice
Retention depends on purpose, internal policy, applicable obligations and investigation needs; there is no single duration for every organization. Account for hot-index cost, archive retrieval time, data location, access rights, export, transport encryption and protection against alteration. When evaluating a logging service, test supported sources, query language, volume and retention pricing, alert integrations and data export at contract end. Log management covers collection, storage and search. SIEM adds rules and other analytics for security detection and investigation; specific product capabilities can overlap.
Access and integrity
Limit log administrator access, track changes and deletion, and protect data in transit.
Measure volume and cost
Test source volume, indexing, retention, searches and archive retrieval with representative data.
Document retention
Give each retention and deletion rule an owner and purpose; a platform feature alone does not establish compliance.
Control file growth
Log rotation replaces or truncates an active file by time or size and preserves earlier records; compression and deletion are separate policies. On Linux, logrotate supports daily, weekly and size-based criteria. If an application keeps writing to an open file, a rename may affect rotation, so check its supported reopen signal or service mechanism. For Windows Event Log, configure maximum size, overwrite behavior and archiving through Event Viewer or Group Policy according to the owner’s plan. Rotation does not replace a central archive or retention policy.
When should rotation be time or size based?
Time suits routine daily handling; a size threshold suits variable high volume. Both can be used. Align thresholds with retention and recovery needs.
Does rotation guarantee no lost records?
No. Check how the application opens and writes files, rotation races and archive backups.