Elastic - Overview
When considering data storage in Elasticsearch, it's common to think of relational databases. However, Elasticsearch data storage (e.g., indices) is not exactly the same as storage in a relational database. Nevertheless, analogies can be drawn between Elasticsearch indices and relational databases to enhance your understanding of Elasticsearch.
Think of an Elasticsearch cluster as a database that can contain many indices, similar to how a relational database contains tables. Within each index, you have many documents, which are akin to rows and columns in a relational database table.
RDBMS: Databases => Tables => Columns/Rows
Elasticsearch: Clusters => Indices => Shards => Documents with key-value pairs
Understanding the internals of an Elasticsearch index
When users want to store data (or documents) on Elasticsearch, they do so in an index. An index on Elasticsearch is a location to store and organize related documents. They don't all have to be the same type of data, but they generally have to be related to one another. In the SQL world, an index would be comparable to a database containing multiple tables (where each table is designed for a single type of data).
An index is made up of primary shards. Primary shards can be replicated into replica shards to achieve high availability. Each shard is an instance of a Lucene index with the ability to handle indexing and search requests. The primary shard can handle both read and write requests, while replica shards are read-only. When a document is indexed into Elasticsearch, it is indexed by the primary shard before being replicated to the replica shard. The indexing request is only acknowledged once the replica shard has been successfully updated, ensuring read consistency across the Elasticsearch cluster.
Indexing is the action of writing documents into an Elasticsearch index. Elasticsearch will index individual document fields to make them searchable as part of the indexing request.
Inside an index
Elasticsearch documents are JSON objects that are stored in indices. In the SQL world, a document represents a row in a database table. JSON documents are complex data structures and contain key/value pairs. Keys are generally strings, while values can be nested objects, arrays, or data types such as datetimes, geo_points, IP addresses, and more.
When a document is indexed into Elasticsearch, it is stored in the source field. The following additional system fields are also added to each document: -The name of the index the document is stored in is indicated by the index field.
-An index-wide unique identifier for the document is stored in the _id field.
Index mappings
"All the fields in a document need to be mapped to a data type in Elasticsearch. Mappings specify the data type for each field and also determine how the field should be indexed and analyzed for search. Mappings are like schemas when defining tables in a SQL database. Mappings can be declared explicitly or generated dynamically.
Index templates
An index template on Elasticsearch is a blueprint for index settings and mappings. It is common to distribute a data source across multiple indices on Elasticsearch. For instance, a logging platform might choose to implement an indexing strategy for two of its data sources. There are three indices for the first data source and two indices for the second, as shown here:
firewall-logs-10.12.2020
firewall-logs-11.12.2020
firewall-logs-12.12.2020
password-resets-11.2020
password-resets-12.2020
Firewall logs are written to a daily index, while password reset logs are written to a monthly index because of their lower volume. Managing index mappings and settings consistently across the indices can be error-prone when done manually. It might also be preferable to have ETL clients such as Logstash automatically create indices based on event dates, without knowing about the mappings and settings to be used.
Index templates can be automatically applied to new indices based on the name of the index. To create an index template for all indices starting with the firewall-logs string, run the following code:"
Athick, Asjad. Getting Started with Elastic Stack 8.0: Run powerful and scalable data platforms to search, observe, and secure your organization (pp. 154-155). Packt Publishing. Kindle Edition.
Elasticsearch - Shards
????
Elasticsearch - Data Streams
10 key concepts in elastic
Example elastic ECS
"To achieve multi-tenancy, there are a few more steps to take to segregate the logs across multiple indices for the different customers:
If the data is directly shipped to Elasticsearch, you can simply use a special pattern in naming the sensors to help you distinguish which tenant the data belongs to (e.g., *-<customer-Id>).
The name of the sensor will be ingested under the observer.name field of ECS, and a custom ingest pipeline can be then used to read this value and set the appropriate index name for the event.
If the data is first sent to a different Kafka topic for each customer, then you can also use the name of this topic from the event metadata to place the data on the appropriate indices as well."
Kibana - Spaces
"Kibana spaces can be thought of as separate instances of Kibana, where each space contains all the dashboards, detections and alerts, and machine learning jobs for one customer. Those spaces are accessible via role-based access control to achieve the required level of segregation. Granular levels of access control are provided by the document and field-level security features, which can help enforce robust access for compliance with strict security and privacy standards."
data Viewers???
Elastic 8 book git hub page
Elastic Security Overview - Log Sources
The diagram below from Elastic displays the Elastic security product (e.g. Elastic Security)
The Elastic Security product 1st starts on the foundational base of Elastic of getting data into Elastic. This is shown by the various types of log sources and that data being sent into Elastic.
It is important to understand these sources because this serves as the instrumentation and telemetry for data into the SIEM as illustrated by the SOC Triad.
Elastic Integrations
Log Source - NGFW
Fortigate Firewalls
Fortigate divides logs into major types:
Traffic
Event
UTM
UTM stands for Unified Threat Management and has several major subtypes.
IPS
Example Fortigate UTM logs.
Palo Alto Firewalls
Juniper Firewalls
Log Source - Server OS
Windows and Linux servers running Elastic agent. These are controlled by policy, but currently its different than the elastic security endpoint agent which is similar to an EDR (Elastic purchased a company called Endgame).
Log Source - Endpoint Detection Response (EDR) -
Microsoft Defender
Log Source - Cloud Logs - Azure
Log Source - Network Detection Response (NDR) - zeek/corelight
Zeek NDR from Corelight systems can be injested into elastic in three ways:
Direct zeek data injest via API
Logstash zeek data injest
Elastic agent zeek data injest
Direct zeek data injest via API
This is not a HEC injest.
Logstash zeek data injest
Elastic agent zeek data injest
Log Source - VMware - esxi
Log Source - Application - VMware
Log Source - Secrets Managers
Keeper
1password
Log Source - Application
Workday
Log Source - Other Sources
For certain log sources there exists no integration with elastic. But of the log source can export alert logs via CEF syslog the elastic CEF integration can be used to injest the log data into elastic. File beat CEF module
Some log sources with no specific elastic integration but export logs in syslog CEF format:
Tenable Identity exposure
AC Hunter
Log Source - Elastic Endpoint Security Agent
Elastic acquired an endpoint security product called Endgame security. Elastic acquired Endgame in 2019 https://en.wikipedia.org/wiki/Endgame,_Inc. . Subsequently Elastic has the Endgame cybersecurity detection engine into its Elastic Agent product and refers to it as Elastic security Agent. The same Elastic Agent that colelcts logs from the host also performs the EDR like security function, and would only need to be enabled by policy.
???example policy picture???
When enabling the Edgame Functionality additional telemetry from the host it is running on comes to the SIEM and there are several different rules that alert on this telemetry. In many cases you will still see the term 'Endgame' referenced in the rules.
Custom TCP Pipelines
Enriching Data Concepts
Enrichment at KQL query
Enrichment at Injest
Enrich at query
Runtime fields when dashboards are queried
GEO IP with beats (mainly an ECS thing)
Elastic SIEM Overview - Detection Engine
The Elastic Security app in Kibana is employed for overseeing the Detection engine, Cases, and Timeline, in addition to administering hosts running Endpoint Security.
The Detection engine automatically searches for suspicious host and network activity via the following:
Detection rules involve conducting periodic searches on data (Elasticsearch indices) sent from your log sources (e.g., intrumentation telemtry) to identify any suspicious events. Upon discovering a suspicious event, a detection alert is then generated.
Machine learning jobs automatically detect anomalies in host and network events, providing anomaly scores per host that can be utilized in conjunction with detection rules.
KQL Queries and Aggregations which allow free search of the logs.
Timeline: Workspace for investigating alerts and events. Timelines use queries and filters to drill down into events related to a specific incident. Timeline templates are attached to rules and use predefined queries when alerts are investigated. Timelines can be saved and shared with others, as well as attached to Cases.
Elastic Detection Rules
Elastic comes with several built in Detection rules. Create detection rules:
Building Custom elastic Rules
Aggregation Rules
zeek ecs rules
detecting password info stealers (mimikats, lazagna, seatbelt)
Indicator match rules
ndicator match rules can be used to detect and alert on indicators of compromise (IOC) in your environment. An IOC refers to evidence of malicious activity that can be used to identify whether a breach occurred and the severity and extent of the compromise. The indicator match detection type can be configured to work with any type of indicator dataset. A threat intelligence module is available to ingest IOC feeds from a range of open source APIs and intelligence platforms. URL Abuse is a web-based malicious URL sharing platform and is a useful feed to detect potentially malicious network destinations. Configure Filebeat or Elastic Agent to ingest URL feeds into your SIEM. The threatintel.yml file can be used as a reference when onboarding this dataset. Configure an indicator match rule using the url-ioc-match.ndjson file as a reference. The url.full source field can be mapped with threatintel.indicator.url.full under the indicator mapping section. The rule should generate alerts if any activity to a malicious URL is found in your logs or telemetry as shown.
Athick, Asjad. Getting Started with Elastic Stack 8.0: Run powerful and scalable data platforms to search, observe, and secure your organization (p. 548). Packt Publishing. Kindle Edition.
Tuning Elastic rules
Elastic rule exceptions
Exceptions: Reduce noise and the number of false positives. Exceptions are associated with rules and prevent alerts when an exception’s conditions are met. Value lists contain source event values that can be used as part of an exception’s conditions. When Elastic Endpoint Security is installed on your hosts, you can add malware exceptions directly to the endpoint from the Security app.
The impact of lists on rules
Elastic Machine Learning
Machine learning Note
"Machine learning jobs look back and analyse two weeks of historical data prior to the time they are enabled. After jobs are enabled, they continuously analyse incoming data. When jobs are stopped and restarted within the two week timeframe, previously analysed data is not processed again."
Kibana Info
Kibana Query Language (KQL)
Dashboards and Lens
Cases: An internal system for opening, tracking, and sharing security issues directly in the Security app. Cases can be integrated with external ticketing systems.
Administration: View and manage hosts running Endpoint Security."
"Timeline: Workspace for investigating alerts and events. Timelines use queries and filters to drill down into events related to a specific incident. Timeline templates are attached to rules and use predefined queries when alerts are investigated. Timelines can be saved and shared with others, as well as attached to Cases.
Cases: An internal system for opening, tracking, and sharing security issues directly in the Security app. Cases can be integrated with external ticketing systems.
Administration: View and manage hosts running Endpoint Security."
Query performance
I would recommend starting with the event.xxxxx fields listed https://www.elastic.co/guide/en/ecs/current/ecs-category-field-values-reference.html . I use event.kind, event.module, event.dataset, event.category, and event.type a lot when trying to limit logs to the ones that I want.For example, event.kind: event (rather than event.kind: alert for example) will focus only on events.
Then event.module or event.dataset to narrow down the log source that you want to see. For example, for Fortigate:
event.module: fortinet
event.dataset: fortinet_fortigate.log < within an event.module, there can be multiple event.dataset's for a log source. Fortigate has only one I believe.
Then if for example, you're looking for network events only:
event.category: network
Creating Cases
Elastic Systems Administration
Elastic sizing
elastic sizing notes
Elastic system auditing
There are two places where you can enable the collection of Audit logs in your Elastic environment: https://www.elastic.co/guide/en/elasticsearch/reference/8.9/enable-audit-logging.html https://www.elastic.co/guide/en/kibana/current/xpack-security-audit-logging.html
Audit logs are disabled by default in kibana.yml. Use the Kibana audit logs in conjunction with Elasticsearch audit logging to get a holistic view of all security related events. Kibana defers to the Elasticsearch security model for authentication, data index authorization, and features that are driven by cluster-wide privileges. For more information on enabling audit logging in Elasticsearch, refer to Auditing security events.
Kibana audit log functionality now also captures some event coming from the Fleet server activity:
Standard analyzer
References
Elastic online book: https://github.com/PacktPublishing/Getting-Started-with-Elastic-Stack-8.0
elastic pod cast: https://open.spotify.com/show/6DLhSV2P7aBQemyUcTVZKE?si=149e999e631b4068&nd=1&dlsi=4fa994fdd1a54eb7&linkId=528519094
Comentários