Here at Appian, we have experienced tremendous growth in the past few years by helping our customers solve their most important business problems faster. We do this by providing a low-code platform that brings together humans, systems, and most recently, robots in support of any mission. As we’ve grown, we’ve added many productivity tools and endpoints to our global IT environment for employees to do their jobs, and AWS instances to the Appian Cloud for new customers to develop and run their applications on. Additional compliance frameworks, cutting-edge technologies, and a strong focus on ensuring the security of our corporate environment, development infrastructure, and the Appian Cloud have further expanded the portfolio of custom tools and off-the-shelf software that we use to monitor our security posture.
The problem: security efforts lead to alert fatigue.
What do all of these tools have in common? They generate alerts. Lots of them. Some time ago, we had a small application for managing actual security incidents, but every single alert was sent to the small team of security analysts through chat or email or was exclusively viewable in the tool’s dashboard. This approach was not scalable and had a number of drawbacks, even for a small team: New analysts couldn’t easily review steps that were taken to resolve past instances of a recurring alert. It was hard to know who was responsible for investigating an alert. And the best (and only) way to find metrics about how many alerts were received over time was for a senior analyst to search their Gmail inbox, which may have just resulted in “1–50 of many.”
This sea of colored labels in security analysts’ inboxes, information silos, and dashboards with blinking lights would have been fertile ground for alert fatigue and a surefire way to cause great analysts to leave or miss something important at the least opportune time. In fact, the Sumo Logic 2020 State of SecOps and Automation Report found that 99% of 427 senior IT security leaders at mid- to large-size companies said high alert volume caused problems for security teams, and 83% said their staff encountered alert fatigue.
The solution: adopting a SOAR platform.
Instead of sitting around and waiting until things became unbearable, Appian decided to tackle the problem head on. This is the first in a series of four blog posts that will cover our company’s journey of going from messy inboxes, disparate dashboards, and endless swivel chairing to an end-to-end Security Orchestration, Automation, and Response (SOAR) platform:
- Security: Protecting data and information and mitigating risk of unauthorized information access.
- Orchestration: Bringing together numerous disparate alert sources and tools into one platform.
- Automation: Adding workflows that enrich alerts with threat intelligence, automatically close known false positives, and reduce the amount of manual work analysts perform.
- Response: Performing immediate remediation actions for security events that require a response.
To build or to buy?
Because Appian is not the only company to have encountered alert fatigue while working to strengthen security, there is a set of products geared toward finding a solution. SOAR is a burgeoning market in the security industry, however, many SOAR products are sold by large vendors whose solutions promote vendor lock-in and offer limited integration options. The alternative to this, of course, is spending the time and effort to build everything from scratch using lots of custom code.
After a little shopping around and a few demos, we went back to the drawing board. We realized we already had a platform that we love to use to run our business, so, we decided to grab a hearty serving of dogfood and build exactly what we wanted using our own product—the Appian Low-Code Platform.
Using the Appian Low-Code Platform for SOAR.
How we went about this is a real testament to the power of low-code development. Instead of commissioning a team of developers to build a platform over many months, we were able to tackle this challenge with a team of . . . one developer. That’s right, one guy with a few years of Appian experience and a pretty good handle on APIs and databases. Within weeks, the initial version went live, offering security event management, automated alert evaluation rules, and metrics in production. As we’ve continued to expand the application with integrations and capabilities for advanced automation, new features are going from idea to production in days or weeks, not months.
Defining the building blocks.
In order to get started taming the mess of security alerts our analysts were receiving, we had to first figure out what alerts existed in their inboxes and distill them to their essence. This meant creating a unified data schema that could take in security events, indicators of compromise, and enrichment data from dozens of different APIs, internal databases, and event providers. We wound up normalizing the core security event data types into the following core building blocks:
- Event: An alert that we receive from a monitoring tool, or the abnormal results of an automated periodic query the application itself runs when acting as a detection engine. Events contain times, sources, and other data about the event in a standard format we can use for statistics and tools.
- Indicator: This is an observable value, such as an IP address, username, hostname, or cloud instance ID. Each event can have as many of these as needed.
- Query: This is a unit of raw threat intelligence fetched from a tool, such as VirusTotal or GreyNoise. Each indicator can have as many of these as there are integrations.
As we moved from merely collecting security events in one place to adding case management and automation, we added more blocks. These will be discussed further in the next three blog posts in this series.
The underlying technology stack for this initial version was simple. Each building block is represented in a MariaDB database as a table. The Appian instance uses custom data types to represent these tables, Process Models to write and update the data in them, and integrations and web APIs to ingest data from event sources for processing. The data is shown to end users in interfaces built using SAIL (the Appian-patented low-code UI language) and Appian Records:
In addition to building a UI for reviewing security events, one of the first features built was a list of events that analysts can filter to review based on type, time, provider, and many other grouping fields. Beyond search and viewing individual events, this dashboard allows analysts to view entity-level breakdowns and statistics for the frequency of events:
Quick wins.
Defining just the three core data types and building a relatively simple application for storing, searching, and viewing events from multiple sources helped us do the following:
- Collect new security events in one central place instead of having analysts “swivel chair” between different tools and dashboards.
- Identify trends and patterns in security events.
- Create reusable components to allow for integrating any new event source by just writing a parser to parse its data into the event schema format.
- Hard code the event storage workflow to escalate every critical event to our analysts via chat as soon as it is processed.
With a solid foundation in place, the next step was enabling our analysts to go from viewing events and metrics in the platform to being able to actively manage and work on events within the platform. More on that in the next blog post on Orchestration.
If you too want to take world-class automation software and put it to work for security use cases, be sure to check out our open positions, including an opening for an Information Security Application Engineer to work on SOAR and many other security-focused Appian projects.
