In our previous blog posts covering our in-house Security Orchestration, Automation & Response (SOAR) application, we described the basic building blocks for the system and the steps we took to orchestrate interactions between a number of security tools. While building orchestration, we already introduced a significant amount of Automation powering features like threat intelligence collection and event polling. In this blog post, we continue to expand on automation and discuss how automation features allow our Security Operations team to achieve goals like manually reviewing fewer events and reducing the need to manually contact users to confirm alerts.
Building playbooks to identify benign activity
All security analysts have an internal set of “playbooks” deep in their brain that allows them to determine whether something is clearly a threat, totally benign or warrants further investigation. As security teams grow, they tend to go one step further and write standard operating procedures containing concrete investigation and response steps. This helps soften the blow of senior analyst attrition, speeds up onboarding of new analysts, and should in theory ensure that every alert of a particular type is investigated in the same way. One of the most important steps in formalizing these processes is identifying reliable and useful sources of information. Having automated threat intelligence gathering, as discussed in the previous blog post, we had already done much of that work.
With appropriate and reliable information on hand, the next step in building a playbook is determining what that information will look like in the benign case, and how it will be different in the malicious case. For example, an analyst might decide that an unusual first-time network connection from a company-issued laptop to an IP address known to belong to an authorized service like Google Drive is not something to worry about. Conversely, if that same alert were to occur and reference an IP address flagged as malicious or physically located in a country on Appian’s list of prohibited countries for business, it would be highly concerning. Sometimes contextual information about who is performing an activity can also be critical - one probably ought to be more concerned about unknown software being installed on the computer of a finance department employee as opposed to someone in IT whose responsibilities include testing new software.
By combining solid data and a good understanding of its features, analysts can start writing up instructions on how to handle events, such as the following simple playbook for investigating an anomalous network connection:
- Query for the IP on GreyNoise
- Query for the IP on VirusTotal
- Check if VirusTotal thinks the IP is benign
- Check if GreyNoise recognizes the IP as belonging to a known tech service used within the enterprise
- Check if the data transfer was not an unusually large one, such as under 100 megabytes
- If 3, 4 and 5 are all true, the alert is safe to ignore
Codifying threat analysis to dismiss benign alerts
Security analysts tend to face repetitive tasks such as performing the above sequence of steps on dozens or hundreds of near-identical events per day. This was a huge opportunity for automation. We had already automated steps 1 and 2, so couldn’t we just feed the underlying data to some code and make these decisions automatically? Cue the Rule Engine. The Rule Engine is a combination of Appian interfaces, process models, Custom Data Types, and a purpose-built Appian plug-in designed to ingest the raw event data and its corresponding enrichment data, evaluate analyst-defined rules against the data, and trigger Process Models to take automated actions in accordance with matching rules. In the Rule Engine, the checks above would be defined like this:
After the rule is approved in production, every event matching the specified conditions is automatically marked as resolved shortly after ingestion. Being able to go from documenting and manually following a process to codifying the analysis process with automation is a tremendous help to our Security Operations team, allowing us to:
- Spend less time investigating alerts that turn out to be benign.
- Guarantee that every incident that is automatically dismissed as benign goes through the same rigorous investigation steps, even at 5 pm on a Friday.
- Enforce review and approval of new and modified auto-dismissal rules to ensure only thoughtful patterns of verification are deployed to production.
- Have a less chaotic alert queue, allowing unusual events to stand out.
- Understand and report on the percentage of alerts identified as benign so that we can tune our detection tools to reduce the number of irrelevant alerts.
The best part? While I have talked about “codifying”, the Rule Engine is fully configurable by analysts directly in the same Appian Site where they review alerts, meaning that it is a business application analyzing data and requires no designer access to Appian, programming skills or even Appian application design expertise. Only a high-level understanding of JSON is needed for configuring advanced rules.
Leveraging ChatOps to collect information from users
Automated dismissal helped us reduce the number of alerts requiring human review by our analyst team by over 50%, and freed up time to instead investigate things that may represent legitimate threats or issues within our IT environments. In order to further optimize repetitive tasks, we started identifying other frequently repeated workflows. One very common analyst task is contacting a person who owns a resource or is implicated in an alert to confirm whether what they were doing was expected and authorized. Examples of this include using administrator credentials to log in to a service, changing cloud infrastructure resources, or even just accessing the corporate VPN from an unexpected geolocation.
The manual version of this workflow is tedious. An analyst starts by manually typing up an email or chat message to the user, then responds to their follow-up questions, and finally copies what the user said to the SOAR application in order to close out an event as expected activity. Sometimes, these messages may be sent to users hours after the activity, since the event may not be immediately picked up for review. To add more delay, they may be reaching out to a user who is located in a different timezone and takes hours to respond. If an activity was actually unusual and malicious, the earlier it is broadcast, the better. The combination of time sensitivity and tedium made this a great candidate for automation.
We initially thought about using email to mimic the common notifications from consumer services about security alerts such as logins from a new device, but emails are easy to ignore and everyone already gets too many. Email also would have made it more difficult to support requesting additional information as a self-service feature. Instead, we decided to solve the problem using ChatOps. The end result was a chatbot that can be used to manually send ad-hoc tasks to chat channels shared by Information Security and other technical departments, or paired with the rule engine to automatically trigger interactive task notifications for certain events:
Beyond being able to confirm or escalate events on the channel, users can query information using the bot, and provide context that is automatically stored into the SOAR application for recordkeeping and analyst review as part of the context of the alert.
For certain events where context is always required, such as the unusual activity of accessing AWS root accounts, the rule engine can use data from the event to automatically build a task, along with supporting features like rate limiting tasks when multiple events of the same type are triggered simultaneously. After all, we are in the business of having all users click less rather than more. To set this up, analysts can just build a rule, after which Appian will take care of everything to get the chat alert posted, and record responses. The system can also automatically mark an event as confirmed upon task completion, so that an analyst can easily review the response and finalize its closure with one click if everything checks out.
By automating issuing tasks to users, we unlocked further efficiency gains for Security Operations while also improving our security posture:
- Saved time and reduced human error by eliminating the need to copy and paste information between emails and chats and the SOAR application.
- Enabled self-service and eliminated bottlenecks in cross-timezone communications by dramatically reducing the number of questions that a human needs to answer.
- Reduced response time, sometimes by hours, by having the system automatically generate tasks as soon as an event matching a rule is processed, rather than whenever the event is reviewed by a human.
- Embraced transparency by moving event confirmations from direct conversations between analysts and administrators to departmental or team channels. This both helps analysts have visibility into events other analysts have been handling and reduces the possibility that a compromised account can be used in malicious ways and simultaneously communicate with security saying everything is expected without other team members noticing.
Reducing noise with automation
The automation discussed in this post is critical to ensuring security for internal corporate users and our customers as we grow the footprint of Appian Cloud and support overall company growth. By reducing the number of repetitive tasks and benign alerts analysts have to contend with, we can also support a happier and less overloaded analyst workforce and thus a more alert Security Operations Center (pun intended). In the next and final post in our series, we will discuss the automation for Response to those events that require us to take action.
If you are a current or aspiring InfoSec professional looking for a great place to practice your trade, be sure to check out our open positions! We are currently hiring across all teams. In Security Operations, we have openings from a new grad Information Security Analyst all the way to an experienced SOC Manager. We are also looking for an Appian Developer to work on SOAR and many other security-focused Appian projects, as well as many other open positions in all security disciplines.
