SIEM Solution Using Native AWS Tools

For most companies, implementing good security practices includes a combination of Log Management, Log Retention, Event Alerting, & Correlation. There are many Security Information and Event Management (SIEM) solutions available from a variety of companies that address these concerns when hosting in Amazon Web Services (AWS).  As your Security and DevOps teams evaluate these 3rd-party vendors, consider using some native AWS tools to solve for most of these requirements.

Amazon has a great blog that we used as a foundation to build our solution. In a separate article, Amazon provides some more examples of events such as failed console logins & API authorization failures. Even with these additional examples, addressing system log management & log retention requires a bit more engineering.

Implementation

At ReachForce, part of our entire SIEM solution is deployed via a CloudFormation stack. This stack aggregates events from operating systems, applications and infrastructure into CloudWatch Logs. Additionally, there are filters, coupled with CloudWatch Alarms, that signal to an SNS topic (Email & SMS) when a pattern is matched or a threshold is breached.

In the example CloudFormation stack below, we define the Log Group, the Metric Filter and the Alarms. Notice that retention period for the Log Groups is set to 365 days for compliance reasons.  

Note: For demonstration purposes, we have put the Log Groups in the stack together with the filters. Doing this in production can be dangerous because your Log Groups and logs will be deleted if the stack is deleted. To prevent this, you can use termination protection, an IAM policy, and/or create the Log Groups separately.

AWSTemplateFormatVersion: '2010-09-09'
Description: Cloudwatch logs filters and alarms
Parameters:
InvalidFilterPattern:
Description: Auth filter pattern
Type: String
Default: invalid
AlarmNotificationTopic:
Description: SNS Topic - change this Account ID and SNS Topic Name when ready
Type: String
Default: "arn:aws:sns:us-east-1:ACCOUNT_ID:SNS_TOPIC_NAME"
Resources:
# Configure Log Groups
CloudTrailLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /siem/cloudtrail
RetentionInDays: 365
MessagesLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /siem/messages
RetentionInDays: 365
AuthLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /siem/auth
RetentionInDays: 365
AuditLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /siem/audit
RetentionInDays: 365
SyslogLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /siem/syslog
RetentionInDays: 365
SecureLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /siem/secure
RetentionInDays: 365
WindowsLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /siem/windows
RetentionInDays: 365

# /var/log/messages
MessagesMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: MessagesLogGroup
FilterPattern: !Ref InvalidFilterPattern
MetricTransformations:
- MetricValue: 3
MetricNamespace: LogMetrics
MetricName: SIEM_Messages
MessagesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: /var/log/messages Alarm
AlarmName: /siem/messages Alarm
MetricName: SIEM_Messages
Namespace: LogMetrics
Statistic: Sum
Threshold: 3
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching

# /var/log/auth.log
AuthMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: AuthLogGroup
FilterPattern: maximum authentication attempts exceeded
MetricTransformations:
- MetricValue: 3
MetricNamespace: LogMetrics
MetricName: SIEM_Auth
AuthAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: monitoring of /var/log/auth.log
AlarmName: /siem/auth Alarm
MetricName: SIEM_Auth
Namespace: LogMetrics
Statistic: Sum
Threshold: 3
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching

# /var/log/audit/audit.log
AuditMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: AuditLogGroup
FilterPattern: !Ref InvalidFilterPattern
MetricTransformations:
- MetricValue: 3
MetricNamespace: LogMetrics
MetricName: SIEM_Audit
AuditAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: monitoring of /var/log/audit
AlarmName: /siem/audit Alarm
MetricName: SIEM_Audit
Namespace: LogMetrics
Statistic: Sum
Threshold: 3
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching

# /var/log/syslog
SyslogMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: SyslogLogGroup
FilterPattern: !Ref InvalidFilterPattern
MetricTransformations:
- MetricValue: 3
MetricNamespace: LogMetrics
MetricName: SIEM_Syslog
SyslogAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: monitoring of /var/log/syslog
AlarmName: /siem/syslog Alarm
MetricName: SIEM_Syslog
Namespace: LogMetrics
Statistic: Sum
Threshold: 3
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching

# /var/log/secure
SecureMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: SecureLogGroup
FilterPattern: invalid user
MetricTransformations:
- MetricValue: 3
MetricNamespace: LogMetrics
MetricName: SIEM_Secure
SecureAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: monitoring of /var/log/secure
AlarmName: /siem/secure Alarm
MetricName: SIEM_Secure
Namespace: LogMetrics
Statistic: Sum
Threshold: 3
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching

# Windows
WindowsMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: WindowsLogGroup
FilterPattern: error # you'll want more specific rules here
MetricTransformations:
- MetricValue: 3
MetricNamespace: LogMetrics
MetricName: SIEM_Windows
WindowsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: monitoring of windows
AlarmName: /siem/windows Alarm
MetricName: SIEM_Windows
Namespace: LogMetrics
Statistic: Sum
Threshold: 2
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching

# AWS/Infrastructure Stuff
ModifyCloudTrailFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.eventName = CreateTrail) || ($.eventName = UpdateTrail) || ($.eventName = DeleteTrail) || ($.eventName = StartLogging) || ($.eventName = StopLogging) }"
MetricTransformations:
- MetricValue: 1
MetricNamespace: LogMetrics
MetricName: SIEM_Modify_CloudTrail
ModifyCloudTrailAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has modified cloudtrail
AlarmName: SIEM CloudTrail Alarm
MetricName: SIEM_Modify_CloudTrail
Namespace: LogMetrics
Statistic: Sum
Threshold: 1
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
FaileConsoleLoginsFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.eventName = ConsoleLogin) && ($.errorMessage = \"Failed authentication\") }"
MetricTransformations:
- MetricValue: 3
MetricNamespace: LogMetrics
MetricName: SIEM_Failed_Logins_to_AWS_Console
FaileConsoleLoginsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has modified failed login several times
AlarmName: SIEM Failed Logins Alarm
MetricName: SIEM_Failed_Logins_to_AWS_Console
Namespace: LogMetrics
Statistic: Sum
Threshold: 3
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
APIAuthFailuresFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.errorCode = \"*UnauthorizedOperation\") || ($.errorCode = \"AccessDenied*\") }"
MetricTransformations:
- MetricValue: 3
MetricNamespace: LogMetrics
MetricName: SIEM_API_Authorization_Failures
APIAuthFailuresAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: Multiple API authentication failures
AlarmName: SIEM API Auth Failures Alarm
MetricName: SIEM_API_Authorization_Failures
Namespace: LogMetrics
Statistic: Sum
Threshold: 3
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
ModifySecurityGroupsFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.eventName = AuthorizeSecurityGroupIngress) || ($.eventName = AuthorizeSecurityGroupEgress) || ($.eventName = RevokeSecurityGroupIngress) || ($.eventName = RevokeSecurityGroupEgress) || ($.eventName = CreateSecurityGroup) || ($.eventName = DeleteSecurityGroup) }"
MetricTransformations:
- MetricValue: 1
MetricNamespace: LogMetrics
MetricName: SIEM_Create_or_Modify_Security_Groups
ModifySecurityGroupsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has modified a security groups
AlarmName: SIEM SecurityGroups Alarm
MetricName: SIEM_Create_or_Modify_Security_Groups
Namespace: LogMetrics
Statistic: Sum
Threshold: 1
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
ModifyGatewayFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.eventName = CreateCustomerGateway) || ($.eventName = DeleteCustomerGateway) || ($.eventName = AttachInternetGateway) || ($.eventName = CreateInternetGateway) || ($.eventName = DeleteInternetGateway) || ($.eventName = DetachInternetGateway) }"
MetricTransformations:
- MetricValue: 1
MetricNamespace: LogMetrics
MetricName: SIEM_Network_Gateway_Modification
ModifyGatewayAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has modified a network gateway
AlarmName: SIEM Gateway Modification Alarm
MetricName: SIEM_Network_Gateway_Modification
Namespace: LogMetrics
Statistic: Sum
Threshold: 1
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
PutBucketACLFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.eventName = PutBucketAcl) }"
MetricTransformations:
- MetricValue: 1
MetricNamespace: LogMetrics
MetricName: SIEM_Bucket_ACL_Modification
PutBucketACLAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has modified an S3 bucket's Access Control List - Could be exposed to public
AlarmName: SIEM Bucket ACL Alarm
MetricName: SIEM_Bucket_ACL_Modification
Namespace: LogMetrics
Statistic: Sum
Threshold: 1
Period: '60'
EvaluationPeriods: '1'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
ACLMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.eventName = CreateNetworkAcl) || ($.eventName = CreateNetworkAclEntry) || ($.eventName = DeleteNetworkAcl) || ($.eventName = DeleteNetworkAclEntry) || ($.eventName = ReplaceNetworkAclEntry) || ($.eventName = ReplaceNetworkAclAssociation) }"
MetricTransformations:
- MetricValue: 1
MetricNamespace: LogMetrics
MetricName: SIEM_Network_ACL_Modification
ACLAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has modified a network access control list
AlarmName: SIEM Network ACL Alarm
MetricName: SIEM_Network_ACL_Modification
Namespace: LogMetrics
Statistic: Sum
Threshold: 1
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
ModifyVPCFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.eventName = CreateVpc) || ($.eventName = DeleteVpc) || ($.eventName = ModifyVpcAttribute) || ($.eventName = AcceptVpcPeeringConnection) || ($.eventName = CreateVpcPeeringConnection) || ($.eventName = DeleteVpcPeeringConnection) || ($.eventName = RejectVpcPeeringConnection) || ($.eventName = AttachClassicLinkVpc) || ($.eventName = DetachClassicLinkVpc) || ($.eventName = DisableVpcClassicLink) || ($.eventName = EnableVpcClassicLink) }"
MetricTransformations:
- MetricValue: 1
MetricNamespace: LogMetrics
MetricName: SIEM_VPC_Modifications
ModifyVPCAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has modified a VPC
AlarmName: SIEM VPC Modification Alarm
MetricName: SIEM_VPC_Modifications
Namespace: LogMetrics
Statistic: Sum
Threshold: 1
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
IAMPolicyFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{($.eventName=DeleteGroupPolicy)||($.eventName=DeleteRolePolicy)||($.eventName=DeleteUserPolicy)||($.eventName=PutGroupPolicy)||($.eventName=PutRolePolicy)||($.eventName=PutUserPolicy)||($.eventName=CreatePolicy)||($.eventName=DeletePolicy)||($.eventName=CreatePolicyVersion)||($.eventName=DeletePolicyVersion)||($.eventName=AttachRolePolicy)||($.eventName=DetachRolePolicy)||($.eventName=AttachUserPolicy)||($.eventName=DetachUserPolicy)||($.eventName=AttachGroupPolicy)||($.eventName=DetachGroupPolicy)}"
MetricTransformations:
- MetricValue: 1
MetricNamespace: LogMetrics
MetricName: SIEM_IAM_Policy_Changes
IAMPolicyAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has modified IAM policies
AlarmName: SIEM IAM Policy Changes Alarm
MetricName: SIEM_IAM_Policy_Changes
Namespace: LogMetrics
Statistic: Sum
Threshold: 1
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
LargeInstancesMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: CloudTrailLogGroup
FilterPattern: "{ ($.eventName = RunInstances) && (($.requestParameters.instanceType = *.8xlarge) || ($.requestParameters.instanceType = *.4xlarge)) }"
MetricTransformations:
- MetricValue: 2
MetricNamespace: LogMetrics
MetricName: SIEM_8XLarge_Instances_Started
LargeInstancesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: A User has started more than 2 8XL instances
AlarmName: SIEM Large Instances Alarm
MetricName: SIEM_8XLarge_Instances_Started
Namespace: LogMetrics
Statistic: Sum
Threshold: 2
Period: '60'
EvaluationPeriods: '2'
ActionsEnabled: True
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- Ref: AlarmNotificationTopic
TreatMissingData: notBreaching
Deploy the CloudFormation stack

Use the AWS CLI or the AWS Console to deploy the stack:

aws cloudformation create-stack --enable-termination-protection --stack-name siem-monitoring-alerting --template-body file://./siem.yaml --capabilities CAPABILITY_IAM

Once the stack is deployed, and you see a “Create Complete” in the Status Section, be sure to check that the Termination Protection is set to ‘Enabled’.

Monitor Infrastructure Events

Next, you can use the AWS Console to configure CloudTrail to deliver logs to CloudWatch Logs. Edit the CloudTrail so that the CloudWatch Logs Log Group name matches the one from the above stack (/siem/cloudtrail).

 

CloudTrail CloudWatch Logs Log Group Configure CloudTrail Logs

 

Now that the Log Groups are configured, CloudTrail events should be showing up in the new ‘/siem/cloudtrail’ Log Group.

Configure the EC2 Instances

In order to correlate the infrastructure events with system and application logs we need to add the AWS Logs agent to each ec2 instance. We created an AWS Logs config file for each flavor of Linux and stored it in an S3 bucket. Here are two examples:

[general]
state_file = /var/awslogs/state/agent-state
#Ubuntu
[/var/log/syslog]
datetime_format = %b %d %H:%M:%S
file = /var/log/syslog
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /siem/syslog
[/var/log/auth]
datetime_format = %b %d %H:%M:%S
file = /var/log/auth.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /siem/auth

[general]
state_file = /var/awslogs/state/agent-state
[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /var/log/messages
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /siem/messages
[/var/log/secure]
datetime_format = %b %d %H:%M:%S
file = /var/log/secure
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /siem/secure
[/var/log/audit]
datetime_format = %b %d %H:%M:%S
file = /var/log/audit/audit.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /siem/audit

Other examples can be found in the awslogs github repo.

Next, we wrote a simple bash script to install the AWS CloudWatch Logs agent on each instance:

Ubuntu:

ssh -q -oStrictHostKeyChecking=no $user@$ip -i ~/.ssh/$key.pem 'curl -s https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O; sudo python ./awslogs-agent-setup.py -n --region=$region --configfile=https://s3.amazonaws.com/${bucket}/ubuntu_awslogs.conf; sudo service awslogs restart'

Amazon Linux:

ssh -q -oStrictHostKeyChecking=no $user@$ip -i ~/.ssh/$key.pem 'curl -s https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O; sudo python ./awslogs-agent-setup.py -n --region=$region --configfile="https://s3.amazonaws.com/${bucket}/amzn_awslogs.conf; sudo service awslogs restart'

Or

sudo yum install -y awslogs; sudo aws s3 cp s3://$bucket/amzn_awslogs.conf /etc/awslogs/; sudo service awslogs restart

You could also use AWS Systems Manager instead of logging into each EC2 Instance. Once the AWS CloudWatch Logs agent is installed, operating system logs should now be forwarded to their respective CloudWatch Logs Log Group.  

Analysis & Correlation

When searching for specific messages in the AWS Console, be sure not to select a particular stream, since you’ll want to search across streams.

 

Searching CloudWatch Logs Searching CloudWatch Logs

 

Note: Searching for events in CloudWatch Logs can be a bit cumbersome. Thankfully, Jorge Bastida has an elegant solution for searching and correlating CloudWatch Logs from the command line.

Maintaining the Posture

To deny accidental (or malicious) deletion of the logs, we attached a policy to the users’ IAM group.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "DenyAuditLogs", "Effect": "Deny", "Action": [ "logs:Delete*" ], "Resource": [ "arn:aws:logs:us-east-1::log-group:/siem*", ] } ] }

And finally, to ensure that all of our instances are compliant, we wrote a Python Lambda function that verifies all instances are logging to CloudWatch Logs.

CloudWatch Verify Lambda

Conclusion

If you are looking for a budget-friendly, automated way to monitor and alert on your AWS infrastructure, try out the above solution. You can update the CloudFormation stack with new filters and triggers as your SIEM needs change. Additionally, check out AWS Config and AWS Trust Advisor to help firm up your security stance.  

General

Click me

Recent Posts

Subscribe to the Blog