Network Detection Response (NDR) - Sources of Network Evidence and Capture Points

brencronin

17 minutes ago14 min read

When threat actors compromise a system, they often take steps to evade detection, such as disabling Endpoint Detection and Response (EDR) tools, auditing mechanisms, or other monitoring capabilities. In such scenarios, Network Detection and Response (NDR) tools like zeek become an invaluable part of a defender’s arsenal, helping to shift the balance in favor of the defenders.

Rob Joyce, in his talk Disrupting Nation State Hackers, emphasized the importance of comprehensive network monitoring:

"Great, you’ve got logs, it’ll tell you that you’ve been had. Enable those logs. Look at those logs. I’ll tell you one of our worst nightmares is that out-of-band network tap that really is capturing all the data, understanding anomalous behavior going on, and somebody’s paying attention to it." -Rob Joice 'Disrupting Nation State Hackers' https://www.usenix.org/sites/default/files/conference/protected-files/engima2016_transcript_joyce_v2.pdf

However, before traffic can be analyzed by an NDR system, it must first be captured. This process, while essential, can present challenges, particularly for cybersecurity analysts who may lack a strong background in network engineering.

This article explores common methods for capturing traffic to feed into NDR sensors, including the pros and cons of each approach. Different traffic capture solutions offer unique advantages; however, their availability can be limited by factors such as cost, implementation timelines, or suitability for rapidly addressing immediate traffic capture needs. We will cover the following traffic capture techniques:

Network Taps
Traffic Brokers
Switched Port Analyzer (SPAN)/Mirror Ports
Server Based Traffic Capture
Cloud-Based Traffic Capture

By understanding these methods and their implications, cybersecurity professionals can implement effective traffic capture strategies to maximize the value of their NDR solutions.

Network Taps

A network tap (Test Access Point) is a hardware device used for monitoring network traffic. It operates by duplicating data traveling across a network link and forwarding the copy to monitoring or security tools such as Network Detection and Response (NDR) solutions like Zeek sensors., Intrusion Detection Systems (IDS), or full packet capture sensors.

Network taps are often deployed at strategic network chokepoints, such as the connection to a perimeter firewall, where they can capture the most significant volume of network traffic. They are installed inline between two network devices: one side of the network connection plugs into one port on the tap, while the other side plugs into another tap port. Internally, the tap duplicates the traffic and sends the copy to an additional port, which connects to your cybersecurity sensor or monitoring tool.

Here’s an example of a basic network tap setup. Popular network tap vendors include Dualcom and also Keysight which is known for their high-quality Ixia taps.

While most network taps offer shunting support to maintain traffic flow during certain failures, they still come with some challenges:

Single Point of Failure -Since taps are inline devices, a complete failure of the tap can result in a network outage.
Unfiltered Traffic Duplication - Taps duplicate all traffic to the monitor port without any filtering or configuration options, which may overwhelm the connected monitoring tool receiving the data from the tap.
Limited Connectivity - Achieving full visibility of network traffic often requires tapping multiple connections. However, many smaller physical taps have limitations on the number of network connections they can handle, both for inbound traffic from network links and duplicated traffic to multiple cybersecurity tools.

Traffic Brokers

A network traffic broker, like Gigamon or cPacket, is a specialized solution designed to optimize, manage, and deliver network traffic from multiple sources to various monitoring, security, and network tools. The traffic broker acts as an intermediary layer between the network infrastructure and the tools that analyze network data.

In addition to supporting multiple inbound network connections and duplicating traffic to multiple monitoring ports (i.e., "cyber tool" ports), network traffic brokers offer a powerful feature: the ability to filter traffic sent to monitoring tools.

For instance, if an organization wants to send only DNS traffic to a specialized DNS analysis tool, the broker can be configured to duplicate only that specific traffic to the monitoring tool. This capability is highly beneficial for managing costs, particularly when licensing for cybersecurity tools is based on traffic ingestion rates.

Traffic brokers also support routing inline traffic through tools that require active interaction with the traffic. Examples include an IDS/IPS that blocks malicious traffic or a proxy that terminates and regenerates sessions as the source. In more complex implementations, inline traffic can be routed through multiple tools via the traffic broker, enabling a sequential or layered approach to traffic inspection and handling. This flexibility makes traffic brokers an invaluable asset to have complex visibility and control over network traffic in sophisticated network security architectures.

These advanced features make traffic brokers significantly more versatile and efficient than basic network taps, though they are also considerably more expensive.

Switched Port Analyzer (SPAN)/Mirror Ports

A SPAN port (Switched Port Analyzer), also known as a mirror port, is a feature on a network switch that duplicates traffic from one or more source ports or VLANs to a designated monitoring port. This allows the duplicated traffic to be analyzed by tools like packet analyzers (e.g., Wireshark), Intrusion Detection Systems (IDS), or Network Detection and Response (NDR) solutions.

One reason SPAN ports are widely used for copying traffic to monitoring tools is their accessibility and convenience. SPAN functionality is a standard feature available on most network switches, allowing traffic to be mirrored from existing network connections. Utilizing existing network devices eliminates the need for additional taps or traffic brokers, reducing costs. Additionally, SPAN ports can often be configured without rerouting network connections, minimizing deployment risks and avoiding potential network outages.

RSPAN

RSPAN, or Remote Switched Port Analyzer, is a network monitoring tool that allows users to monitor traffic from multiple sources across multiple switches. RSPAN works by mirroring traffic from source ports into a dedicated VLAN, which is then trunked to other switches in the network. One of the most pertinent use cases of RSPAN is where the sensor that is getting the mirrored traffic is not on the same switch as the traffic that is being mirrored.

When implementing RSPAN (Remote Switched Port Analyzer), it’s important to consider whether your switched network is using Spanning Tree Protocol (STP). Certain vendor RSPAN implementations have been known to contain bugs where the flooding of RSPAN traffic through trunk ports triggers multiple reconvergences of the Spanning Tree. This can result in significant network instability. Proper testing and validation are essential to prevent such issues.

ERSPAN

Encapsulated remote SPAN (ERSPAN) brings generic routing encapsulation (GRE) for all captured traffic and allows it to be extended across Layer 3 domains.

Network tap, traffic broker, and SPAN placement location considerations

To ensure effective network security, it's critical to analyze all network traffic using security tools, particularly traffic entering or leaving your organizational boundaries. Any traffic not captured by network sensors creates network blind spots, which can leave your organization vulnerable. This makes it a priority to capture traffic at network choke points, where most of the organization's traffic flows.

Issues with traffic capture beyond perimeter firewalls

Another key consideration is the impact of network traffic encryption on traffic capture. Many organizations use perimeter firewalls or routers to encrypt traffic destined for external locations. If traffic is captured on the encrypted side of these connections, the monitoring sensor will only see the firewall or router as the source and destination. This issue was uncommon in the past because WAN circuits were primarily SONET-based (e.g., T1, DS3, OCX), with SONET cards integrated directly into perimeter routers or firewalls. However, with the widespread adoption of Ethernet WAN circuits, it's now common to have a switch located outside the perimeter router or firewall, which readily supports traffic capture through SPAN/Mirror ports.

Capturing traffic at this point renders encrypted data unreadable, severely limiting the sensor's ability to conduct meaningful analysis. Analyzing encrypted client-server traffic, such as HTTPS, is already challenging. When designing your deployment, you should avoid restricting your own visibility by placing traffic capture beyond your encryption points, such as a network firewall or router. Strategic placement of traffic capture ensures maximum visibility and enhances your ability to analyze network activity effectively.

Options for inline visibility to encrypted network traffic

Captured encrypted client-server traffic can be routed to an SSL intercept device, which terminates the encrypted connection. This process decrypts the traffic, enabling in-depth inspection and analysis of its contents before re-encrypting it and forwarding it to its destination. This approach provides enhanced visibility into encrypted traffic while maintaining secure communication.

The process full encrypted traffic analysis begins with traffic being routed through an intercept device that decrypts it. The decrypted traffic is then forwarded to an inline cybersecurity sensor for analysis. Once analyzed, the traffic is sent back to the intercept device, which proxies the session, re-encrypts the traffic, and forwards it to its final destination. This approach ensures both comprehensive traffic inspection and the preservation of secure communication.

With traffic brokers there is also the option to route the unencrypted traffic, intercepted traffic, back into the traffic broker to be routed through other inline security tools and back in and out of the traffic broker to the intercept device to be re-encrypted before its set on its way.

Analyzing encryption metadata

Another valuable aspect of network traffic capture in addressing the encrypted traffic visibility challenge lies in network analysis protocols like Zeek. These tools do not decrypt the data but instead capture metadata about the encrypted session. This metadata includes details about how the session is established and the type of encryption used. Such insights can reveal patterns and anomalies linked to malicious behavior and threat actor activity, offering an additional layer of detection without requiring decryption.

Traffic capture through firewall capability

Most Next-Generation Firewalls (NGFWs) include the capability to capture network traffic and save it in Packet Capture (PCAP) file format. This feature allows administrators to specify capture criteria, such as source and destination IP addresses or port numbers, to focus on relevant traffic.

Captured PCAP files can be downloaded and analyzed using tools like Wireshark for detailed examination. Additionally, these files are often stored offline in either flat file storage systems or specialized platforms like Arkime, which are designed for efficient storage and analysis of large-scale packet captures.

Server Based Traffic Capture Options

Hosts offer specific capabilities for capturing network traffic, with one of the most common being the use of TCPDUMP. This powerful utility, available on most Linux distributions, enables the capture of network traffic and saves it as PCAP files for analysis.

Using TCPDUMP for server-based network traffic capture into PCAP files is an essential capability in your network traffic capture toolkit. This approach provides flexibility for situations where other methods, such as network taps, traffic brokers, or SPAN ports, are unavailable or impractical due to network design constraints. Additionally, PCAP files captured with TCPDUMP can be directly analyzed or converted into network analysis files using protocols like Zeek. However, relying on server-based TCPDUMP traffic captures as a long-term solution has notable limitations and considerations.

Challenges with Using TCPDUMP for Server Traffic Captures

While TCPDUMP is a powerful tool for capturing network traffic, several challenges arise when using it for continuous monitoring, particularly in cybersecurity contexts. Originally designed for network diagnostics and troubleshooting, TCPDUMP was intended for temporary use rather than continuous traffic analysis. However, as cybersecurity demands evolved to require persistent traffic monitoring, there are several key issues with continuous PCAP-based captures.

Impact on Server Performance

Running TCPDUMP continuously on a server can significantly impact its performance. Servers not optimized for traffic capture may experience degraded functionality, affecting their primary organizational roles. This limitation is why specialized appliances, purpose-built for traffic capture and analysis, are often used in enterprise environments.

PCAP File Storage and Management

Continuous traffic captures generate massive amounts of data, placing strain on storage systems. On servers, this can consume valuable storage resources, further hindering their primary functions. Solutions often involve offloading PCAP files to long-term storage systems, which requires additional system administration and infrastructure investment.

Single Server Capture

When conducting traffic analysis using PCAP on servers, the capture is limited to the traffic of a single server. To monitor traffic across multiple servers, each server must run its own instance of TCPDUMP and generate individual PCAP files simultaneously. This decentralized approach can quickly become resource-intensive and complex to manage.

In contrast, capturing traffic at network choke points, such as perimeter firewalls, switches, or routers, provides a centralized solution. By monitoring these strategic points, you can capture traffic to and from multiple servers in one location, simplifying data collection and analysis while providing broader visibility into network activity.

Challenges in Searching Traffic Captures

Analyzing and searching through large or multiple PCAP files is also complex and time-consuming:

Tools like Wireshark (GUI) https://www.wireshark.org/ and TShark (command-line) https://linux.die.net/man/1/tshark are useful but struggle with extensive datasets.
Merging multiple PCAP files for consolidated analysis can result in cumbersome and unwieldy datasets.

To address these challenges, solutions like Arkime https://arkime.com/ provide a web-based interface for searching large volumes of PCAP data. However, even with such tools, the inherently unstructured nature of PCAP data can limit efficiency.

The Value of Network Analysis Protocols versus Full packet capture

Protocols like Zeek https://zeek.org/ (formerly Bro) offer a more structured approach to traffic analysis. Zeek processes PCAP data to extract key protocol fields, storing them in a format that is more searchable and compact than raw packet captures. This approach Significantly reduces storage requirements by focusing on actionable network traffic data, and greatly enhances the effectiveness of network traffic analysis, enabling more effective network traffic analysis and detections.

BPF Filters and Traffic Capture

When discussing TCPDUMP and network traffic capture, it’s crucial to understand the role of Berkeley Packet Filter (BPF) in filtering traffic during data ingestion. This capability is particularly important in scenarios where network sensors may be overwhelmed by high volumes of traffic.

Traffic Filtering Challenges

Some traffic capture technologies, such as network taps and SPAN/mirror ports, do not have built-in filtering capabilities. As a result, all traffic is sent directly to network sensors for capture and analysis. If this traffic isn’t filtered at the point of capture, sensors can become overloaded, leading to performance degradation or missed packets.

In contrast, traffic brokers often provide robust filtering options, allowing administrators to limit the data sent to sensors based on predefined criteria. For systems without this feature, filtering must occur on ingestion by the sensor itself, typically using BPF. BPF is a highly efficient packet filtering mechanism that enables selective traffic capture and filtering based on user-defined criteria.

Linux Host-Based Traffic Filtering with iptables --tee

When filtering network traffic for capture, whether through host-based TCPDUMP or mirroring traffic off a server, the Linux iptables --tee option offers a versatile solution. This feature is inspired by the concept of a "tee" in plumbing, where a "tee" fitting splits a single stream into two outputs. Similarly, the iptables --tee option duplicates network packets, allowing for simultaneous routing to their original destination and an additional specified interface or monitoring tool. The --tee option in iptables enables packet mirroring in Linux. When configured, packets matching a specific rule are duplicated. One copy continues to its intended destination. The other copy is forwarded to a specified interface or monitoring server. For example, to monitor traffic entering eth0 and duplicate it to a monitoring server connected to eth1, you can use the following command:

iptables -A PREROUTING -t mangle -i eth0 -j TEE --gateway 192.168.1.100

-A PREROUTING - Adds a rule to the PREROUTING chain, ensuring the rule applies to packets before routing decisions are made. The --tee option is commonly used in the PREROUTING or POSTROUTING chains to access packets before or after routing.

-t mangle - Specifies the mangle table, typically used for advanced packet modifications.

-i eth0 - Matches incoming traffic on interface eth0.

-j TEE - Specifies the TEE target, enabling packet duplication.

--gateway 192.168.1.100 - Directs the mirrored traffic to a gateway or monitoring server with the specified IP address.

Virtual Machines Traffic Capture

There are several traffic capture scenarios worth considering, particularly when dealing with virtual machines on hypervisors. In these environments, the two primary traffic capture options still apply: capturing traffic outside the hypervisor at the physical network connection and capturing it within the virtual machine using TCPDUMP. Both of these methods have been discussed earlier.

However, hypervisors introduce an additional factor, the virtual switch. Virtual machines within a hypervisor often communicate through a shared or separate virtual switch, which manages network access. In some cases, capturing traffic externally from the hypervisor may not be feasible, and capturing traffic within each VM may not be desirable.

To address this, certain hypervisors provide built-in traffic capture tools. For example, VMware ESXi offers pktcap-uw, and Windows offers pktmon which are both utilities that functions similarly to TCPDUMP but operates at the virtual NIC and virtual switch levels, enabling multiple traffic capture in the virtualization environment without requiring network traffic to exit the hypervisor to perform traffic copture.

End User System Traffic Capture

A common challenge in network traffic capture and analysis arises when monitoring remote or small office users. Deploying a traditional traffic capture device on a home Wi-Fi network is impractical, and running full packet captures on a user's laptop is often not feasible. However, alternative methods exist to collect valuable network traffic data for analysis.

One effective approach is using Sysmon, a system monitoring utility that can be deployed as an agent on the user's system. Once installed, Sysmon can be configured to log Event ID 3, which records network connection data. This log provides critical details such as:

Process & User Information associated with the network connection
Network Protocol
Source Details - IP, Hostname, Port, and Port Name
Destination Details - IP, Hostname, Port, and Port Name

To centralize this data for further analysis, logs can be forwarded from the user's system to a SIEM using agents like Winlogbeat, Elastic Agent, or similar log forwarding tools.

Another option for remote network traffic collection and analysis is leveraging Microsoft Defender for Endpoint (MDE), which provides built-in network telemetry through its DeviceNetworkEvents table. This table consistently captures key network connection data, including:

Source & Destination: IP addresses, Ports
Process Information: Names, IDs, Parent Process details
Command Lines & Process Hashes, and more

In 2022, Microsoft partnered with Zeek to enhance MDE’s network telemetry by integrating Zeek-formatted protocol metadata. As a result, the DeviceNetworkEvents table includes an ActionType field, which reflects the protocol being inspected, such as:

DnsConnectionInspected
FtpConnectionInspected
HttpConnectionInspected, and more

This integration enriches network telemetry by providing additional protocol-specific metadata. For example, HTTP traffic logs can include values like HTTP method, response body length, and user-agent string, offering deeper visibility into endpoint network activity.

Another option for network traffic collection in these scenarios is leveraging a zero-trust client such as Microsoft Global Secure Access, Zscaler Client Connector, or Cloudflare WARP with Digital Experience Monitoring (DEX). These solutions are widely adopted due to their security and access control benefits.

A key advantage of these secure access tools is their ability to decrypt and inspect encrypted traffic before forwarding it to its destination, providing visibility into protocols like DNS, HTTPS, and other encrypted communications.

There are typically three traffic collection options with these solutions:

Cloud-based Network Traffic Logs - These logs, collected from the cloud firewall/proxy, provide details such as source/destination IP and port, FQDN, traffic type, policy rules applied, and bytes sent/received.
Policy-Based PCAP Generation - Some cloud services allow the firewall/proxy to generate packet captures (PCAPs) for specific traffic flows based on defined rules.
Client-Side Packet Capture - Certain zero-trust clients support on-demand packet captures from the endpoint itself, which can be initiated remotely for deeper traffic analysis.

Understanding Your Traffic Capture and Analysis Strategy

What is your approach to traffic capture and network analysis? Clarifying this is essential for ensuring effective visibility and response to network events. Consider the following questions to evaluate your traffic capture & analysis solution:

Traffic Capture Methodology:

Is your traffic capture system only capturing traffic in PCAP files that are only searched when troubleshooting or investigating an incident?
Does your traffic capture system capture traffic and analyze traffic in real time with Intrusion Detection System (IDS) capabilities such as Suricata?
Does your traffic capture system capture traffic and convert the traffic into a specialized network analysis format like Zeek?
Can the system do both network traffic capture, network analysis (IDS, and Zeek), and capture PCAPs based on specific conditions? Often referred to as 'Smart PCAP' because only targeted network traffic is fully collected into PCAPs thereby limiting the amount of data collected.
Does your traffic capture support injest filtering?

Integration with Analysis Tools:

Does your traffic capture system only collect data, requiring you to ingest the PCAP files into another platform for analysis?
Or does your system perform both traffic capture and perform analysis, like a Zeek/IDS sensor that parses and evaluates traffic on the fly?

Extent of On-Sensor Analysis:

If your solution involves sensors that handle both capture and analysis, how much processing occurs directly on the sensor?
Are alerts generated and insights provided on the sensor itself, or is the network analysis data forwarded to centralized system, like SIEM, for further evaluation, or is there a combination of both types of analysis and alerting?