Monitoring Kubernetes Nodes for Security Events using Wazuh
As innovation and adoption of Kubernetes increases, the threat landscape also increases. Security professionals need to protect assets by adopting practices to prevent, detect, and respond to cyber threats while meeting strict compliance requirements. One of these requirements is to secure and monitor each layer of the cloud native stack, which includes the host OS Kubernetes runs on.
Wazuh can be used for this purpose. It is a free, open source host-based intrusion detection system (HIDS). It provides intrusion detection for most operating systems, including Linux, macOS and Windows. Wazuh provides a security solution capable of monitoring your infrastructure, detecting threats and poorly configured applications. Wazuh is a fork of a popular HIDS known as OSSEC. Its main components are the Wazuh manager, the Elastic stack, and the Wazuh agent.
Wazuh Installation
You can find the Wazuh installation steps here. There are multiple installation methods available like manual install, Ansible, Docker and Kubernetes, choose the one which best fits you. The Docker method is the quickest way to get it up and running for testing purposes.
Overview - Wazuh Modules
Wazuh has multiple modules which can help you with various aspects of security monitoring of your environment. To list a few:
- Security logging
- Configuration assessment
- Vulnerability detection
- File Integrity Monitoring (FIM)
- Active response
- Regulatory compliance
- Cloud security monitoring
- Container security monitoring
Let’s take a closer look at some of these capabilities.
Security logging
The Wazuh agent monitors and sends the relevant security events to the Wazuh manager. Wazuh uses a ruleset to detect attacks, intrusions configuration problems, malware, system anomalies or security policy violations. OSSEC provides an out-of-the-box set of rules that Wazuh updates and augments, to increase Wazuh detection capabilities. You can find the Wazuh ruleset in this GitHub repository.
Rules
The Wazuh agent running in the endpoints ship the OS and application logs to the Wazuh manager. The manager uses decoders to identify and parse the incoming logs. This activity is similar to using Logstash or Fluentd to parse logs. It then analyzes the data using its rules. Here is an example of a rule.
<rule id="35004" level="5">
<if_sid>35002</if_sid>
<id>^401</id>
<description>Squid: Unauthorized: Failed attempt to access </description>
<description>authorization-required file or directory.</description>
<group>pci_dss_10.2.4,pci_dss_10.2.5,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.7,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
</rule>
<rule id="35005" level="5">
<if_sid>35002</if_sid>
<id>^403</id>
<description>Squid: Forbidden: Attempt to access forbidden file </description>
<description>or directory.</description>
<group>pci_dss_10.2.4,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.7,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
</rule>
The rules can be found on all the endpoints at var/ossec/etc/rules. They can be centrally managed using the Wazuh manager too, more details on that are available in the Centralized configuration section of the documentation. When these events are matched using the rule, an alert log is sent to Elasticsearch to the wazuh-alerts-* index for storage and visualization. To further get notified in real-time on Slack or any other messaging service, we will need to use a plugin on top of Elasticsearch like ElastAlert or Open Distro’s alerting plugin. By using this module you can utilize Wazuh to continuously monitor your endpoints.
Security Configuration Assessment (SCA)
It is important to know what your environment looks like and which software versions are used. Using this module we can gain insights on the current configuration of the endpoints. This helps to reduce the attack surface and ultimately control the risks. This module can also help us with policy management and auditing.
There are multiple Wazuh integrations that perform configuration assessment scans including OpenSCAP and CIS-CAT and more recently the Security Configuration Assessment (SCA). The SCA was created by the Wazuh development team to overcome limitations that were inherent to the other integrations, to name a few:
- The OpenSCAP integration is only available on Linux hosts, not Windows agents.
- The CIS-CAT tool is proprietary software which requires an external license for its use. Each check comprises some metadata information, a description of the purpose of the check, and its logical description (fields condition and rules). On its metadata, it can contain an optional compliance field used to specify if the check is relevant to any compliance specifications, and to which. Most of the Wazuh policies, especially CIS policies, already have their CIS and PCI-DSS controls mapped. See an example below. This can be configured in the
<agent-installation-folder>/ruleset/sca.
- id: 3064
title: "Ensure IPv6 default deny firewall policy"
description: "A default deny all policy on connections ensures that any unconfigured network usage will be rejected."
rationale: "With a default accept policy the firewall will accept any packet that is not configured to be denied. It is easier to white list acceptable usage than to black list unacceptable usage."
remediation: "Run the following commands to implement a default DROP policy: # ip6tables -P INPUT DROP # ip6tables -P OUTPUT DROP # ip6tables -P FORWARD DROP. Notes: Changing firewall settings while connected over network can result in being locked out of the system. Remediation will only affect the active system firewall, be sure to configure the default policy in your firewall management to apply on boot as well."
compliance:
- cis: ["3.5.2.1"]
- cis_csc: ["9.4"]
condition: all
rules:
- 'c:ip6tables -L -> r:^Chain INPUT && r:policy DROP'
- 'c:ip6tables -L -> r:^Chain FORWARD && r:policy DROP'
- 'c:ip6tables -L -> r:^Chain OUTPUT && r:policy DROP'
You can view the current configuration status of the nodes within the SCA tab in the Wazuh agents page. In the overview section, you can also see the overall status of the checks. Here you can see the default SCA rules which have been loaded for Debian Linux. 
You can open one of these policies and get more information on the checks. 
File Integrity Monitoring
Wazuh’s File integrity monitoring (FIM) system watches selected files and triggering alerts when these files are modified. The component responsible for this task is called syscheck. This component stores the cryptographic checksum and other attributes of a known good file or Windows registry key and regularly compares it to the current file being used by the system, watching for changes. The configuration for FIM is on the ossec.conf file.
<syscheck>
<disabled>no</disabled>
<!-- Frequency that syscheck is executed default every 12 hours -->
<frequency>43200</frequency>
<scan_on_start>yes</scan_on_start>
<!-- Directories to check (perform all possible verifications) -->
<directories>/etc,/usr/bin,/usr/sbin</directories>
<directories>/bin,/sbin,/boot</directories>
</syscheck>
You can view the monitored files in the Integrity monitoring section within the agents tab. 
{
"_index": "wazuh-alerts-3.x-2020.12.04",
"_type": "_doc",
"_id": "UyzSLHYBdtSAXlv4qwjh",
"_version": 1,
"_score": null,
"_source": {
"syscheck": {
"uname_after": "root",
"mtime_after": "2020-11-23T18:08:58",
"size_after": "8388256",
"gid_after": "0",
"path": "/boot/vmlinuz-4.15.0-126-generic",
"sha1_after": "b088e428e48a7610c99933362e0742dd1e548623",
"gname_after": "root",
"uid_after": "0",
"perm_after": "rw-------",
"event": "added",
"md5_after": "4a72b7431b14cdf750696a073d08e3c6",
"sha256_after": "27525c53ec38a7257411bfbb8f7cad31d3f3899b52bd5c52a9db3da0fefc9cbe",
"inode_after": 7098
},
"input": {
"type": "log"
},
"agent": {
"ip": "159.65.144.151",
"name": "k8s-master",
"id": "002"
},
"manager": {
"name": "wazuh-manager"
},
"rule": {
"firedtimes": 1,
"mail": false,
"level": 5,
"pci_dss": [
"11.5"
],
"hipaa": [
"164.312.c.1",
"164.312.c.2"
],
"description": "File added to the system.",
"groups": [
"ossec",
"syscheck"
],
"id": "554",
"nist_800_53": [
"SI.7"
],
"gpg13": [
"4.11"
],
"gdpr": [
"II_5.1.f"
]
},
"location": "syscheck",
"decoder": {
"name": "syscheck_new_entry"
},
"id": "1607069771.18714",
"full_log": "File '/boot/vmlinuz-4.15.0-126-generic' added\nMode: scheduled\n",
"timestamp": "2020-12-04T08:16:11.383+0000"
},
"fields": {
"syscheck.mtime_after": [
"2020-11-23T18:08:58.000Z"
],
"timestamp": [
"2020-12-04T08:16:11.383Z"
]
},
"sort": [
1607069771383
]
}
From the above FIM event, we can gather information on which user did what changes to the file, the hash before the change and after the change, the file permissions, uid which did the change.
Monitoring Linux files using the Audit system
While syscheck’s basic functionality tells us which file has been modified, it does not have the context of the process which did the change. Linux has the Audit subsystem for this purpose. This is similar to what Sysmon does for Windows. Wazuh uses the Audit subsystem in the who-data monitoring module to get the information about who made the changes in a monitored directory. These changes produce audit events that are processed by syscheck and reported to the manager. This will need the Linux Audit subsystem to be installed on the endpoints to work.
{
"_index": "wazuh-alerts-3.x-2020.12.04",
"_type": "_doc",
"_id": "FCzeLHYBdtSAXlv4jxIT",
"_version": 1,
"_score": null,
"_source": {
"syscheck": {
"size_before": "1594",
"uname_after": "root",
"mtime_after": "2020-12-04T08:29:09",
"inode_before": 64411,
"size_after": "1647",
"gid_after": "0",
"md5_before": "a2cdfd55c855b8df2d90522925d96899",
"sha256_before": "21b4241e69e206b6225b2bd6bac31053694ccfdea914b099346e74497e686226",
"mtime_before": "2020-11-26T14:59:49",
"path": "/etc/passwd",
"sha1_after": "eab1943351469518f31008b13904d639a05e7d73",
"changed_attributes": [
"size",
"mtime",
"inode",
"md5",
"sha1",
"sha256"
],
"gname_after": "root",
"audit": {
"process": {
"name": "/usr/sbin/useradd",
"id": "17995",
"ppid": "14616"
},
"login_user": {
"name": "ubuntu",
"id": "1000"
},
"effective_user": {
"name": "root",
"id": "0"
},
"user": {
"name": "root",
"id": "0"
},
"group": {
"name": "root",
"id": "0"
}
},
"uid_after": "0",
"perm_after": "rw-r--r--",
"event": "modified",
"md5_after": "45a37421f31f20a0429aed09850e3972",
"sha1_before": "3975c54b401de8485c31d0b43c26f9cd4d5799f2",
"sha256_after": "05227c5a9d80f7d8d79ad84b13873dd185435be2bbface138a184edeb2ba9c2b",
"inode_after": 2537
},
"input": {
"type": "log"
},
"agent": {
"ip": "159.89.167.59",
"name": "k8s-node-2",
"id": "001"
},
"manager": {
"name": "wazuh-manager"
},
"rule": {
"firedtimes": 5,
"mail": false,
"level": 7,
"pci_dss": [
"11.5"
],
"hipaa": [
"164.312.c.1",
"164.312.c.2"
],
"description": "Integrity checksum changed.",
"groups": [
"ossec",
"syscheck"
],
"id": "550",
"nist_800_53": [
"SI.7"
],
"gpg13": [
"4.11"
],
"gdpr": [
"II_5.1.f"
]
},
"location": "syscheck",
"decoder": {
"name": "syscheck_integrity_changed"
},
"id": "1607070549.63722",
"full_log": "File '/etc/passwd' modified\nMode: whodata\nChanged attributes: size,mtime,inode,md5,sha1,sha256\nSize changed from '1594' to '1647'\nOld modification time was: '1606402789', now it is '1607070549'\nOld inode was: '64411', now it is '2537'\nOld md5sum was: 'a2cdfd55c855b8df2d90522925d96899'\nNew md5sum is : '45a37421f31f20a0429aed09850e3972'\nOld sha1sum was: '3975c54b401de8485c31d0b43c26f9cd4d5799f2'\nNew sha1sum is : 'eab1943351469518f31008b13904d639a05e7d73'\nOld sha256sum was: '21b4241e69e206b6225b2bd6bac31053694ccfdea914b099346e74497e686226'\nNew sha256sum is : '05227c5a9d80f7d8d79ad84b13873dd185435be2bbface138a184edeb2ba9c2b'\n",
"timestamp": "2020-12-04T08:29:09.256+0000"
},
"fields": {
"syscheck.mtime_after": [
"2020-12-04T08:29:09.000Z"
],
"syscheck.mtime_before": [
"2020-11-26T14:59:49.000Z"
],
"timestamp": [
"2020-12-04T08:29:09.256Z"
]
},
"sort": [
1607070549256
]
}
By using this module we get further information on the changed attributes, the process which executed the change, the process’s parent ID which can give us further visibility in our environments in case of a security incident.
Conclusion
We have seen how Wazuh can help us in the security monitoring of Linux environments using some of its capabilities. Wazuh is a good open source HIDS systems available for Linux environments in particular. But there are some caveats here, the configuration and management of the endpoints will take some time and effort to run in production. If you are interested in running Wazuh in production make sure you test it extensively in a test environment.
I’d love to hear your comments and/or feedback - let’s start a conversation on Twitter @infracloudio.
Looking for help with cloud native security consulting & implementation? do check out our capabilities how we’re helping startups & enterprises as an cloud native consulting & services provider.
References
Adopt DevSecOps Faster with InfraCloud's Expertise
Posts You Might Like
