Using ELK Stack To Manage AWS Cloud Security

Using ELK Stack To Manage AWS Cloud Security

AWS Security: The Shared Responsibility Model

Most major public cloud providers including, Amazon Web Services (AWS), follow a “Shared Responsibility Model” for security and compliance. This means that security and compliance tasks are a shared responsibility between the cloud platform and the cloud customer. This applies for regulatory compliance such as HIPAA, PCI DSS, and FedRAMP, but also cybersecurity frameworks such as NIST, ISO, and SOC. AWS customers are able to take advantage of established AWS certifications and security programs to jump-start compliance efforts. Cloud providers implement certain physical security protections, but customers are responsible for building secure solutions with these cloud services.


Why Collect Security Events?

Most cybersecurity frameworks and regulatory standards have set requirements for organizations to implement solutions around audit logging, backups and disaster recovery (DR), vulnerability scanning, intrusion detection systems (IDS), and firewall/networking. Organizations may implement these solutions internally or turn to a number of different vendors.

Since managing large cloud environments and multiple security solutions can be difficult, organizations often turn to SIEM solutions logging solutions such as the ELK stack. The “ELK stack” is based on Elastic’s opensource solutions Elasticsearch, Logstash, Kibana. Organizations can deploy the ELK stack themselves or turn to hosted offerings of ELK such as Elastic Cloud, AWS Elasticsearch, and ELK can help organizations better view and manage overall security events and security operations. Using a log aggregation solution makes it easier for organizations to oversee security events, service availability, and potential suspicious activity.


What To Collect From AWS?

Amazon Web Services provides many different cloud services, from traditional virtual machines and data storage to serverless and container-based workloads. In order to gather insight into AWS cloud environments, organizations may consider using Logstash/ELK stack and AWS log information including:

VPC Flow LogsAWS allows organizations to collect account activity (such as user logins) as well as region activity related to individual cloud services in the regions.

Permissions and IAM LogsOrganizations may monitor changes to permissions and events that occur related to Amazon Identity and Access Management (IAM).

Cloud Service Access LogsAccess logs can be collected from individual cloud services such as S3, RDS, and Redshift. This allows team to see individual service queries and access.

System LogsOrganizations using EC2 instances or E containers may collect logs operating system logs (syslog, etc).

Intrusion Detection Logs Organizations may collect suspicious access attempts for cloud services from intrusion detection systems (IDS).

Vulnerability Scanning LogsOrganizations may collect information related to operating system vulnerabilities and manage this information in connection to an organization patching schedule.

Cloud Configuration Logs Organizations may collect logs related to changes in overall cloud configuration, cloud resources being created, or general configuration management and orchestration.

elk stack aws

Once this information is collected in Logstash, security teams can build reports and visualizations and evaluate the security and compliance state for the organization. Security teams may work with DevOps staff and other team members to resolve security issues.

Sending AWS Data To Logstash/ELK

There are several ways to connect Logstash and AWS. Teams may send AWS cloud service logs to Logstash and may configure system specific logging for EC2 instance and other systems. Cloud information can be aggregated and delivered to Logstash or other SIEM solutions through the following approaches:

  1. Teams may enable AWS Elasticsearch and start collecting logs by streaming Cloudwatch streams and other service streams right into AWS.
  2. Teams may deliver logs to hosted ELK/Logstash services via Amazon Cloudwatch and AWS Cloudtrail or a variety of other outputs. Organizations may consider structuring and collecting logs in Cloudwatch, and using AWS Lambda or other services to send logs over to hosted ELK services.


Gathering Cloud Compliance Events with Dash

Collecting cloud security events in ELK is a good step towards visualizing security issues, but this information, does not give organizations the full picture into their state of compliance. Security teams must gather, assess, and determine how events in their cloud environment affect compliance with regulatory standards such as HIPAA/HITECH, PCI DSS, FedRAMP, as well as cybersecurity frameworks such as the NIST CSF, ISO 27001 and SOC.

The Dash Compliance Automation Platform makes it easy for organizations to gather regulatory compliance events for AWS and the public cloud. Security teams can configure Dash security policies, gather compliance information, and stream compliant events to Logstash and other SIEMs. Companies can collect cloud security issues that are mapped to HIPAA, NIST CSF, and other regulatory standards and compliance frameworks. Security issues include issues such as:

Publicly Available S3 Buckets

HIPAA: 164.312(a)(1) – Access Control

NIST CSF: NIST SP 800-53, NIST SP 800-63, NIST SP 800-21, NIST SP 800-34, FIPS 140-2

ISO: 9.1.1, 9.4.1, 9.6.1, 12.1.3


Unencrypted EBS Volumes

HIPAA: 164.312(a)(2)(iv) – Encryption and Decryption

NIST CSF: NIST SP 800-63, NIST SP 800-21, NIST SP 800-34, FIPS 140-2, NIST SP 800-12

ISO:  8.5.1, 8.7.4, 10.3.1, 10.3.2, 10.3.3, 12.1.6


RDS cluster configured with only one availability zone (AZ)

HIPAA: 164.310(a)(2)(i) Contingency Operations

NIST CSF: NIST SP 800-18 Rev 1

ISO: 7.2.2, 11.1.1, 11.1.3, 12.1.3, 4.1.7, 7.2.3, 7.2.4, 8.1.1


IAM AssumeRole is misconfigured

HIPAA: 164.312(c)(2) Mechanism to Authenticate Electronic Protected Health Information,

  164.312(d) Person or Entity Authentication

NIST CSF: NIST SP 800-14, NIST SP 800-53 Rev 4, NIST SP 800-106, NIST SP 800-107

ISO: 10.2.3, 8.1.6


With Dash, organizations can simplify their security operations and compliance management and have instant insight into their cloud security and compliance programs.


Maintaining AWS Security and Compliance With Logstash

After gathering AWS security and compliance information in Logstash, organizations must manage security operations (SecOps) over time. Security teams should develop processes for identifying and resolving security issues. Organizations may also turn to Dash for setting security standards and policies that fit the organization.

Determine types of issue Determine what type of security events will require response and remediation.

Set security policiesSet policies for security and compliance standards that will be upheld by the security team (IE. disaster recovery policies, vulnerability scanning policies).

Create rolesAssign roles to staff related to assessing and resolving security and compliance events.

Create processes for creating security and DevOps ticketsConnect security events into organization workflow and ticketing, so teams can resolve security events.

Continuously assess security and complianceContinuously view security events and respond to security events within your organization. Teams should access security policies on a frequent basis.

Following these steps will enable teams better streamline the security process. Teams should create policies that fit into their staff structure and technologies. Learn how Dash can be used in coordination with a logging solution or SIEM to manage security policies and conduct continuous compliance monitoring.


Add Dash Compliance Stream To Logstash