A Data Driven Comparison of Open Source Adversary Emulation Tools

Cyber threats pose a significant risk to organizations, and threat actors come up with more sophisticated adversary techniques every day. Thus, organizations should continuously assess and improve their security posture, which is vital for business continuity and maintaining reputation. 

While the security posture can be assessed with various methods, not many are as effective as adversary emulations to show how security controls stand against a real cyber threat. Since the cyber threat landscape is vast and adversary techniques are too many to handle manually, security professionals developed automated adversary emulation tools to help their red teaming activities.

Open-source automated adversary emulation tools are a great alternative for security teams starting to adopt automated adversary emulation to their toolset. In this blog post, we compared and evaluated four popular and actively maintained open-source automated adversary emulation tools.

SANS Whitepaper: How Breach and Attack Simulation Works

Open Source Adversary Emulation Tools

1. MITRE Caldera

2.Atomic Red Team

3. Infection Monkey

4. Stratus Red Team

5. DumpsterFire

6. Metta

7. Red Team Automation (RTA)

MITRE Caldera

MITRE Caldera is an open-source automated adversary emulation framework introduced in 2016 and released in 2019.  Initially, Caldera emulated techniques that adversaries use after gaining initial access. Later versions expanded ATT&CK coverage and added initial access techniques with limited scope. 

Some of the key aspects of MITRE Caldera are:

• Autonomous red-team engagements
  • atomic ATT&CK TTPs
  • attack campaigns
• Post-compromise adversary emulation
• Customizable adversary scenarios
• Customizable plugins
• Autonomous incident-response
• pre-defined defender profiles
• Manual red-team engagements

In its default configuration, the Caldera framework comes with 527 different procedures of ATT&CK techniques, including the Atomic Red library. These procedures cover post-compromise ATT&CK techniques and very few initial access techniques.

Although the adversary emulations are executed automatically, users must plan and select which emulations will be executed by Caldera beforehand.

Atomic Red Team

Atomic Red Team library is an open-source library of adversary procedures designed to test security controls' detection capabilities. The library was introduced in 2017 by Red Canary, and it is currently maintained by the cybersecurity community worldwide. Atomic Red Team is made of atomic tests that emulate individual adversary techniques and may contain multiple procedures for a given technique. The library is accompanied by a framework called Invoke-AtomicRedTeam. The Invoke-AtomicRedTeam framework allows users to run atomic tests individually and sequentially. Also, the library can be migrated and used in other frameworks, such as MITRE Caldera.

Each technique in the library is mapped to the MITRE ATT&CK framework, and many of them include multiple procedures to emulate adversary actions. Currently, Atomic Red Team covers 261 ATT&CK techniques and consists of 1225 Atomic Tests. Since these are individual tests, they do not emulate any specific attack campaign unless an operator chains multiple atomic tests to do so.

Although the Invoke-AtomicRedTeam framework does not include automation in its default configuration, we included Atomic Red Team in this list because it is a widely used library for assessing security controls against individual ATT&CK techniques.

Infection Monkey

Infection Monkey is an open-source breach and attack simulation (BAS) tool that was introduced in 2015 by Guardicore. Since it is a BAS tool, Infection Monkey has more features in addition to adversary emulation, such as ransomware and zero trust assessment.

Unlike other open-source tools, Infection Monkey prioritizes breaching a target and infecting the entire network by moving laterally from host to host. Also, Infection Monkey does not follow a specific attack scenario in its default configuration. Developers of Infection Monkey describe the tool as a monkey that runs amok in the target network aptly named Monkey Island. After starting adversary emulation, each agent (aka monkey) tries all adversary techniques available in its library unless an operator limits its capabilities.

Infection Monkey emulates

• vulnerability exploitation techniques
• brute force attacks
• post-compromise adversary techniques
• ransomware attacks with limited scope

Stratus Red Team

Stratus Red Team is an open-source adversary emulation tool for cloud environments. Stratus Red Team is described as the Atomic Red Team for the cloud by Datadog Security Labs, the originator of the tool. 

Stratus Red Team emulates adversary techniques from MITRE ATT&CK for Cloud Matrix. A binary named Stratus is used to emulate adversary techniques. Similar to Invoke-AtomicRedTeam, Stratus does not provide automation in its default configuration. Stratus Red Team is able to test various cloud environments such as AWS, Azure, GCP, and Kubernetes. Currently, Stratus Red Team consists of 40 atomic tests that cover the following adversary tactics.

• Initial Access
• Execution
• Persistence
• Privilege Escalation
• Defense Evasion
• Credential Access
• Discovery
• Exfiltration

Honorable Mentions

• DumpsterFire

DumpsterFire was introduced in 2017 by Joe Gervais as an open-source tool that replicates security events to test and validate security controls. DumpsterFire aims to simulate a wide range of adversaries, including insider threats, non-technical threat actors, and sophisticated attackers. This simulation can be used for purple teaming, tabletop exercises, and SIEM configuration validation.DumpsterFire has 50 security incidents in its repository; however, these test scenarios were created more than five years ago and may not represent the current threat landscape.

• Metta

Metta is an open-source adversarial simulation tool announced by Uber in March 2018. Metta aims to test and validate the detection capabilities of hosts and networks. The project runs adversary actions described in YAML format. Each adversary action is mapped to the MITRE ATT&CK framework, and security teams can chain different adversary actions to create adversary scenarios to be tested.

Metta has 65 adversary actions in its repository, and this very limited library has not been updated since 2018.

• Red Team Automation

Red Team Automation (RTA) is an open-source framework of scripts that allows security teams to assess their detection capabilities with test scenarios modeled after MITRE ATT&CK. Similar to Atomic Red Team, security teams can chain multiple test scenarios using RTA. However, RTA does not provide automation. RTA was introduced in 2018 with 43 test scenarios but has not been maintained since then.

Evaluating and Comparing Open-Source Adversary Emulation Tools

Every open-source adversary tool has its advantages and disadvantages over other tools. For a fair evaluation and comparison, we need to lay down some objective criteria. These criteria that adversary emulation tools should have are:

• Emulation scope:Adversary techniques can be divided into three categories in terms of the scope of the emulation. Adversary emulation tools should cover a large scope of emulations to represent the cyber threat landscape.

Pre-compromise techniques: ATT&CK techniques classified under Reconnaissance, Resource Development, and Initial Access are considered pre-compromise techniques. Adversary emulation tools' scope often focuses on Initial Access techniques since Reconnaissance and Resource Development techniques are often irrelevant to technical security controls.

Post-compromise techniques: ATT&CK techniques that are used after gaining initial access to the target network are considered post-compromise techniques.

Attack campaigns: Attack campaigns are constructed by multiple ATT&CK techniques to imitate malicious actions of cyber threat actors.

• Update frequency: Emulation tools depend on a repository of emulations. Since new cyber threats and vulnerabilities emerge every day, the repository should be up-to-date and relevant to the current threat landscape.

• Automation: Adversary emulations should be autonomous or automated so that security teams can run emulations effortlessly.

• Emulation customization: Security teams might want to customize emulations and create a better representation of their cyber threat landscape. Adversary emulation tools should allow security teams to add new emulations or chain existing ones.

• Mitigation insights: After an adversary emulation, the next step for security teams is to mitigate the identified security gaps. Adversary emulation tools should provide mitigation insights and ease security teams' workload.

Under these criteria, a comparison between the four popular open-source adversary emulation tools is given below.


Here are some notable features and drawbacks of these open-source tools:

• MITRE Caldera

• Highlights: Caldera allows users to configure the framework and build plugins for their needs. 

• Drawbacks: ACaldera is a complicated tool and requires skilled operators for better utilization of its capabilities.

• Atomic Red Team

• Highlights: Atomic Red Team is the most used library for atomic tests.

• Drawbacks: Atomic Red Team does not provide automation capabilities by default.

• Infection Monkey

• Highlights: Infection Monkey has impressive lateral movement capabilities.

• Drawbacks: Sophisticated adversaries such as APTs and state-sponsored threat actors aim to be as stealthy as possible. However, Infection Monkey attempts to test every technique available in its library and creates a lot of noise.

• Stratus Red Team

• Highlights: There are not many open-source adversary emulation tools available in cloud environments, and Stratus Red Team covers that gap. Also, it is easy to use and easy to deploy.

• Drawbacks: Emulation scope and supported environments are very limited. 

Open-source adversary emulation tools are a great starting point for security teams looking to adopt adversary emulation to assess their security posture. Because there is not a silver bullet solution among these tools that address the needs of a large network, security teams need to invest time and resources to modify any open-source tool to improve test coverage. Organizations should look for enterprise adversary emulation tools that can go beyond limited network and atomic tests for a complete and thorough assessment.

The Definitive Guide for Choosing the Right BAS Tool