Automate Linux & Windows Security Patches Using AWS Systems Manager Patch Manager

One of the issues I had to tackle at work was keeping servers up-to-date with the most recent security patches. This is not easy to do; manually, when it comes to environments with a large number of servers. In order to resolve this issue, I decided to use AWS Systems Manager which allows me to view and control our infrastructure in AWS.

AWS Systems Manager (SSM) allows you to view operational data from multiple AWS services and automate operational tasks. To find out more about this service Click Here. In this article we will go through the process of setting up the prerequisites as well as configuring SSM.

First, you need to make sure that all of your EC2 instances have the SSM Agent installed and running.

Note: Here is how you can install the SSM Agent for both Linux and Windows EC2 instances. That said, the SSM Agent is preinstalled, by default, on instance created from Amazon Machine Images [Source].

Since I also want to receive Email Alerts whenever failures occur, I will also use the AWS SNS service to get Status Notifications.

Here are the steps I went through in order to automate the process.

IAM Policies:

First, we have to create the IAM Policies. In this case, I have copied the policies from the AWS Managed ones, just in case I needed to edit them in the future.

omardulaimi/ssm-autopatching-IAM-policies

IAM Roles:

The AWS Services (SSM and SNS) that we will use are going to require that you create the IAM Roles below:

AutoPatching-SSM-Service-Role and attach the policy AutoPatchingPolicy-SSM-Custom to it. 
AutoPatching-SNS-Service-Role and attach the policy AutoPatchingPolicy-SNS-Custom to it.

You will also need to update the Trust Entities for the above roles by adding ssm.amazonaws.com and ec2.amazonaws.com.

Auto-Patching Logs:

It is important to keep logs when auto-patching using SSM. This will make it easier to troubleshoot any issue that may occur in the future. That said, you can store the logs in an S3 bucket. In this example, I have created the S3 bucket: autopatching-logs-medium-example

You may want to place the bucket in the same region where the instances reside and make sure that Encryption at Rest is enabled and that the bucket is Private.

I also recommend creating a Bucket Life Cycle Rule in order to transition the files to Glacier or delete them after a period of time to save costs.

Instance(s) Tagging & IAM Role:

Another important factor is tagging. In order for SSM to know which group of instances to patch, each EC2 instance needs to have a ‘Patch Group’ tag. In this example, I am using the following tags, however you can customize this however you like.

Then you have to add the following policies to Instance IAM Role:

AutoPatchingPolicy-S3-Instance-Custom
AutoPatchingPolicy-SSM-Instance-Custom

Keep in mind that you have to update the Trust Entities for the above roles by adding ssm.amazonaws.com and ec2.amazonaws.com.

Notifications Setup and Configuration:

Now we are going to configure SNS to send out a notification every time Auto-Patching fails. In the AWS Console, navigate to SNS >> Topics >> Create Topic and enter the following information:

Name: AutoPatching-Notification
Display Name: MyAccount-AutoPatching-Notification

Then, click on Create Subscription and provide the following information:

Topic ARN: Select the one you've just created
Protocol: Email
Endpoint: AlertsEmailAddress@Company.com

Check your inbox and follow the verification steps to confirm your subscription.

Create a Patch Baseline

A Patch Baseline defines which patches are approved for installation on your instances. You can specify approved or rejected patches one by one. You can also create auto-approval rules to specify that certain types of updates (for example, critical updates) should be automatically approved. The rejected list overrides both the rules and the approve list [Source]. In order to create and configure a Patch Baseline, in the AWS Console, go to SSM >> Patch Manager >> View predefined patch baseline >> Patch Baseline.

Name: ProdWebApp1-Ubuntu
Operating System: Select the Operating System

Rules: The rules will be different for Windows vs Linux.

Windows Rules:
--------------  
Product: All. 
Classification: Critical Updates & Security Updates. 
Severity: Critical & Important.
 
Linux Rules: 
--------------
Product: All. 
Classification: Security. 
Severity: Critical & Important.

Ubuntu Rules:  
--------------
Product: All.   
Classification: All.   
Severity: Required & Important.

Click Create Patch Baseline.

For each of the new patch baselines go to Actions >> Modify Patch Group and associate the instances ‘Patch Groups’ respectively. For Linux Patch Baseline, add the Linux patch group tags (i.e., ProdWebApp1-Ubuntu or StagingWebApp2-AWSLinux1).

Configure a Patch Schedule

Select the desired Patch Baseline and click on Configure Patching. Choose Select Patch Group and then select the desired patch group from the drop-down list.

You can create a new or select an existing Maintenance Window and configure the interval of which the patching will occur.

Note: Before you apply the patches to your Production Environment, you should test them on a Staging Environment first.

Schedule: This is where you schedule how often you want the patch to run.

Patch Developments Environment every 1st Wednesday of every month:
cron(0 0 4 ? 1/1 WED#1 *) 
Patch Staging Environments every 2nd Wednesday of every month:
cron(0 0 4 ? 1/1 WED#2 *)
Patch Production Environments every 3rd Wed of every month:
cron(0 0 4 ? 1/1 WED#3 *)

Provide the Maintenance Window Name: ProdWebApp1-Ubuntu

You will also need to choose the Patching Operation; which in this case, you will need to select Scan and Install.

This will create a new Maintenance Window which you will need to configure. In order to do so, you will need to navigate to SSM >> Maintenance Windows >> Select the Maintenance Window (in this example it is ProdWebApp1-Ubuntu)>> Edit.

You will need to the following:

Unregistered Targets: Uncheck this box.
Duration: 2 hours (You can increase this to meet your requirements)

Click on Save Changes then navigate to the Tasks tab under the same maintenance window, select the Window Task ID and Edit. Under the ‘Maintenance window task details’ scroll down to the ‘IAM service role’ section, and select the custom IAM Role (AutoPatching-SSM-Service-Role) we created for Systems Manager.

Under the ‘Output Options’, check the box to Enable Writing to S3, and specify the S3 bucket we created earlier: autopatching-logs-medium-example

You may want to specify the ‘S3 Key Prefix’ to keep things organized. This will create a subfolder, were logs will be stored for each environment.

And finally, we need to enable ‘SNS Notifications’, select the IAM Role ‘AutoPatching-SNS-Service-Role’, and specify the ‘SNS Topic ARN’ that we created earlier. Then select the following ‘Event Type(s)’:

Timed Out
Cancelled
Failed

You can configure SNS to send a notification on a per instance basis, however I chose a summary instead. As for the ‘Parameters’ section, I have selected ‘Install’ as the ‘Operation’. Save the changes by clicking on Edit Run Command Task.

Once the SSM starts patching the instances you can view the progress in the ‘Run Command’ section:

In the ‘Run Command’ console, you have the ability to monitor the progress of the process and find out which instances were successfully patched and which ones have failed. In my case, I chose to get notified for ‘All’ events.

Here is the notification email I have received while the maintenance was ‘In Progress’ and another when it was a ‘success’.

And the logs were stored in the S3 bucket we created for this tutorial.

Now all you have to do is monitor the ‘Run Command’ history for failures, while SSM is performing security patches across your entire environment. Keep in mind that this is not an ideal solution for environments where EC2 Instances are part of an Auto-Scaling Groups.

If this article has helped you solve a problem, please consider sharing it and following me on Medium as I will be posting more articles in the future. Also, feel free to connect with me on LinkedIn.

Stay safe and thanks for coming by.