In our previous blog posts covering our in-house Security Orchestration, Automation & Response (SOAR) application, we described the basic building blocks for the system and the steps we took to orchestrate interactions between a number of security tools. While building orchestration, we already introduced a significant amount of Automation powering features like threat intelligence collection and event polling. In this blog post, we continue to expand on automation and discuss how automation features allow our Security Operations team to achieve goals like manually reviewing fewer events and reducing the need to manually contact users to confirm alerts.
Building playbooks to identify benign activity
All security analysts have an internal set of “playbooks” deep in their brain that allows them to determine whether something is clearly a threat, totally benign or warrants further investigation. As security teams grow, they tend to go one step further and write standard operating procedures containing concrete investigation and response steps. This helps soften the blow of senior analyst attrition, speeds up onboarding of new analysts, and should in theory ensure that every alert of a particular type is investigated in the same way. One of the most important steps in formalizing these processes is identifying reliable and useful sources of information. Having automated threat intelligence gathering, as discussed in the previous blog post, we had already done much of that work.
With appropriate and reliable information on hand, the next step in building a playbook is determining what that information will look like in the benign case, and how it will be different in the malicious case. For example, an analyst might decide that an unusual first-time network connection from a company-issued laptop to an IP address known to belong to an authorized service like Google Drive is not something to worry about. Conversely, if that same alert were to occur and reference an IP address flagged as malicious or physically located in a country on Appian’s list of prohibited countries for business, it would be highly concerning. Sometimes contextual information about who is performing an activity can also be critical - one probably ought to be more concerned about unknown software being installed on the computer of a finance department employee as opposed to someone in IT whose responsibilities include testing new software.
By combining solid data and a good understanding of its features, analysts can start writing up instructions on how to handle events, such as the following simple playbook for investigating an anomalous network connection:
Codifying threat analysis to dismiss benign alerts
Security analysts tend to face repetitive tasks such as performing the above sequence of steps on dozens or hundreds of near-identical events per day. This was a huge opportunity for automation. We had already automated steps 1 and 2, so couldn’t we just feed the underlying data to some code and make these decisions automatically? Cue the Rule Engine. The Rule Engine is a combination of Appian interfaces, process models, Custom Data Types, and a purpose-built Appian plug-in designed to ingest the raw event data and its corresponding enrichment data, evaluate analyst-defined rules against the data, and trigger Process Models to take automated actions in accordance with matching rules. In the Rule Engine, the checks above would be defined like this:
After the rule is approved in production, every event matching the specified conditions is automatically marked as resolved shortly after ingestion. Being able to go from documenting and manually following a process to codifying the analysis process with automation is a tremendous help to our Security Operations team, allowing us to:
The best part? While I have talked about “codifying”, the Rule Engine is fully configurable by analysts directly in the same Appian Site where they review alerts, meaning that it is a business application analyzing data and requires no designer access to Appian, programming skills or even Appian application design expertise. Only a high-level understanding of JSON is needed for configuring advanced rules.
Leveraging ChatOps to collect information from users
Automated dismissal helped us reduce the number of alerts requiring human review by our analyst team by over 50%, and freed up time to instead investigate things that may represent legitimate threats or issues within our IT environments. In order to further optimize repetitive tasks, we started identifying other frequently repeated workflows. One very common analyst task is contacting a person who owns a resource or is implicated in an alert to confirm whether what they were doing was expected and authorized. Examples of this include using administrator credentials to log in to a service, changing cloud infrastructure resources, or even just accessing the corporate VPN from an unexpected geolocation.
The manual version of this workflow is tedious. An analyst starts by manually typing up an email or chat message to the user, then responds to their follow-up questions, and finally copies what the user said to the SOAR application in order to close out an event as expected activity. Sometimes, these messages may be sent to users hours after the activity, since the event may not be immediately picked up for review. To add more delay, they may be reaching out to a user who is located in a different timezone and takes hours to respond. If an activity was actually unusual and malicious, the earlier it is broadcast, the better. The combination of time sensitivity and tedium made this a great candidate for automation.
We initially thought about using email to mimic the common notifications from consumer services about security alerts such as logins from a new device, but emails are easy to ignore and everyone already gets too many. Email also would have made it more difficult to support requesting additional information as a self-service feature. Instead, we decided to solve the problem using ChatOps. The end result was a chatbot that can be used to manually send ad-hoc tasks to chat channels shared by Information Security and other technical departments, or paired with the rule engine to automatically trigger interactive task notifications for certain events:
Beyond being able to confirm or escalate events on the channel, users can query information using the bot, and provide context that is automatically stored into the SOAR application for recordkeeping and analyst review as part of the context of the alert.
For certain events where context is always required, such as the unusual activity of accessing AWS root accounts, the rule engine can use data from the event to automatically build a task, along with supporting features like rate limiting tasks when multiple events of the same type are triggered simultaneously. After all, we are in the business of having all users click less rather than more. To set this up, analysts can just build a rule, after which Appian will take care of everything to get the chat alert posted, and record responses. The system can also automatically mark an event as confirmed upon task completion, so that an analyst can easily review the response and finalize its closure with one click if everything checks out.
By automating issuing tasks to users, we unlocked further efficiency gains for Security Operations while also improving our security posture:
Reducing noise with automation
The automation discussed in this post is critical to ensuring security for internal corporate users and our customers as we grow the footprint of Appian Cloud and support overall company growth. By reducing the number of repetitive tasks and benign alerts analysts have to contend with, we can also support a happier and less overloaded analyst workforce and thus a more alert Security Operations Center (pun intended). In the next and final post in our series, we will discuss the automation for Response to those events that require us to take action.
If you are a current or aspiring InfoSec professional looking for a great place to practice your trade, be sure to check out our open positions! We are currently hiring across all teams. In Security Operations, we have openings from a new grad Information Security Analyst all the way to an experienced SOC Manager. We are also looking for an Appian Developer to work on SOAR and many other security-focused Appian projects, as well as many other open positions in all security disciplines.