A primary challenge in centralized log collection lies in the intricacy of transporting logs from their source to the backend logging system. This process varies in complexity, impacting the capabilities of the centralized logging system in executing functions like alerting SOC analysts or engineers through rules, reports, and dashboards.
The initial phase of the log journey involves ensuring that the source (e.g., server) is appropriately instrumented to generate logs. Following this, a component known as a "log shipper" transports the logs to the centralized logging system, where, despite some tongue-in-cheek skepticism about their magical potential, various useful actions can be performed (though it's acknowledged that there are limitations to what can be achieved with logs).
This may cause confusion, as the log shipper is frequently associated with the instrumentation of source logs since it essentially functions as a gatekeeper determining which logs from the source are sent to centralized storage. However, it's important to note that the source log instrumentation and log shipping are distinct functions, even though they are often intertwined.
System log instrumentation can be intricate, with varying nuances between Linux, Windows, network devices, and application log instrumentation. Detailed coverage of these specific topics is planned for future articles.
Here are several common log shippers:
Beats (Used with Elastic):
Includes Filebeat, Winlogbeat, Auditbeat, and others.
Website: Elastic Beats
Logstash:
Website: Elastic Logstash
Snare:
Website: Snare Solutions
Vector (Acquired by DataDog):
Website: Vector
Splunk Forwarder/Heavy Forwarder:
Used for sending logs to Splunk.
Documentation: Splunk Forwarder
ArcSight Smart Connector:
Used for sending logs to ArcSight.
Website: ArcSight Smart Connector
One of the initial considerations when it comes to log shippers is choosing between an agent-based or an agent-less configuration. In an agent-based setup, the log shipper functions as a distinct software process on the system where it collects logs. In certain scenarios, the choice might be constrained by the systems you are logging from and the specific log types you aim to collect.
The key inquiries regarding agent and agentless log shipping include:
1. System Compatibility:
Can the system you are logging from accommodate the installation of log shippers?
Does the system inherently support native log shipping?
2. Control Over Log injest Pipeline:
Does opting for agentless log shipping provide the necessary control over the log injest pipeline?
3. Administrative Considerations:
Are system administrators willing to support additional software deployments across all systems under their purview?
Does the system you are logging from support the installation of log shippers?
For specific log sources, particularly network equipment such as routers, switches, and firewalls with operating systems that don't allow the installation of a standalone log shipper, a common alternative is available. These devices generally support sending logs to a central location through remote syslog. Therefore, if network logging is required from devices like firewalls, routers, and switches, it's likely necessary to have a dedicated server with a log shipper to receive and process the logs from these network devices.
Does the system you are logging from support native log shipping?
Historically, Windows systems lacked native capabilities to send their logs directly, in contrast to Linux systems and other network devices that could utilize "Remote Syslog" through rsyslog for native log shipping. To collect logs into the log database or SIEM, organizations had to install and manage a log shipper/agent on all Windows systems. Each individual Windows log shipper sent collected logs to the backend log database or SIEM, either directly or through an intermediary log shipper, provided the Windows log shipper/agent could convert logs to syslog. While some specialized log shipper tools could remotely log directly into Windows systems, these solutions were often suboptimal, relying on weak SMBv1-based logons and offering limited granularity in the types of logs that could be collected (e.g., all security logs and/or all system logs).
Several years ago, Windows introduced "Windows Event Forwarding" (WEF), enabling Windows systems to utilize native remoting to transport logs from end sources to a centralized Windows Log Collector. While a log shipper is still required on the WEF server, the introduction of WEF significantly diminishes the number of log shipper/agents that need deployment and management.
My main point about Windows logging is that it doesn't support rsyslog. Linux servers, on the other hand, offer more options with rsyslog, as most Linux systems come with the rsyslog package by default.
Does opting for agentless log shipping provide the necessary control over the log injest pipeline?
While agent-based log shippers come with the drawback of deploying agents across various locations, they offer advantages such as enhanced granularity (e.g., log file monitoring) from the end system to the back-end logging system. Additionally, the end-system agent serves as a point for limiting unnecessary logs, aiding in controlling licensing and storage costs.
Are your system administrators willing to manage additional software deployments across all systems under their responsibility?
In case system administrators, for valid reasons (none come to mind at the moment), are hesitant to install log shipper software on their systems, an alternative approach could involve setting up an external centralized log shipper to process logs. The sys admins would then simply be directed to point their logs to this log shipper listener over the network.
Returning to the question of why Logging Systems/SIEMs may fail, particularly in terms of controlling instrumentation and the log pipeline:
While the diagram below illustrates the central log database system and the SOC, the success of the entire system hinges on other critical areas. These areas demand a high level of expertise in network/system architecture and administration. If the logging system/SIEM lacks recognition as an enterprise application and fails to secure executive support that mandates collaboration with these critical areas, the logging system/SIEM project is likely to fail. In some instances, success might still be achievable if SOC engineers possess the necessary skills and connections within these domains to facilitate the required work.
Comments