Six Stages of Dealing with a Global Security Incident

A Brief History of the SolarWinds Breach

This case is a tale of a legitimate software that has become a lethal weapon in the hands of threat actors. It all started with the FireEye report published on December 8th. This report discloses that a “highly sophisticated state-sponsored adversary” broke into FireEye’s network and stole FireEye Red Team’s tools [1]. Even though this incident caused a great sensation from the first moment, later it turned out to be only a small part of a much larger breach. On December 13th, FireEye issued a new report revealing that a threat actor leveraged the SolarWinds supply chain to compromise multiple global victims and dubbed this attack campaign as UNC2452 [2]. On the same day, Microsoft also published a blog post about the SolarWinds breach [3]. A malicious domain name, avsvmcloud[.]com, was used by attackers for command and control activities. Microsoft reconfigured the domain to serve as a “killswitch” that would prevent the malware from continuing to operate under some circumstances [4].

SolarWinds published a Security Advisory outlining the SolarWinds Orion Platform supply-chain compromise and associated defensive measures [5]. According to SolarWinds SEC Filing issued on December 14th [6], the malicious code was embedded within the Orion products and existed in updates released between March and June 2020. On December 15th, it was revealed that the victims include U.S. government agencies such as the Department of Commerce, Department of the Treasury, Department of Homeland Security (DHS), National Institutes of Health (NIH), and State Department. On December 21st, it was exposed that some technology companies such as Cisco, Intel, Nvidia, Deloitte, VMware, and Belkin had installed the infected SolarWinds Orion software [7]. Until now, the names of nearly a hundred victim organizations have been published in different sources. It seems that the number of victims will increase even more.

Executive Summary

Dozens of blog posts, analyses, and news articles are published about the SolarWinds incident. In this blog, we look from a different perspective and explain what the Security Operation Center (SOC) teams could do to handle the SolarWinds incident or other similar cases. Picus Labs Blue Team prepared a process that includes six stages depicted in the below graphic. If you have enough resources, you can complete the process in a shorter time by running these steps in parallel instead of doing them consecutively.

• The first stage is the preparation stage, and it starts with gaining an understanding of the case. In this phase, Indicators of Compromise (IOCs) and Tactics and Techniques and Procedures (TTPs) are collected, and a response plan is created. 
• The second stage is mitigation through preventive security controls. In this stage, IOCs are applied to the preventive security controls, signature sets of cybersecurity products are updated, and signatures relevant to the incident are enabled. In this section of the blog, you can find the vendor-specific (Check Point, Cisco, Forcepoint, Fortinet, McAfee, Palo Alto Networks, and Snort) network prevention signatures to defend against Sunburst threats.
• Following the mitigation stage, it is necessary to understand the possible impacts this case might have on your systems. For this purpose, a compromise assessment must be performed to confirm that your infrastructure is free from ongoing compromises like the SolarWinds breach. 
• The fourth stage is threat hunting. Unlike the compromise assessment, threat hunting is a hypothesis-driven process that aims to search for cyber threats in a network proactively. If a compromise indicator is identified in a system during the compromise assessment or threat hunting process, you need to immediately start an incident response procedure.
• Writing, validating, and deploying detection rules are a part of the detection engineering stage. Picus Labs Blue Team provides detection rules for IBM QRadar, Splunk, Micro Focus ArcSight, Elastic, and VMware Carbon Black products to defend against Sunburst threats.
• Continuous Security Validation is a necessary process for proactive defense against cyberattacks. It validates your cyber resilience by continuously assessing security controls with real cyberattacks in production environments, measuring the assessed products’ effectiveness, and improving their effectiveness by applying actionable mitigation suggestions.

In the final section of the blog, we briefly discuss the lessons learned from the SolarWinds case.

1. Preparation

1.1 Understanding the Case

The first step is understanding the case. In brief, attackers embedded their malicious payload on a legitimate component of the SolarWinds Orion Platform software in the SolarWinds incident. This component is a DLL library, SolarWinds.Orion.Core.BusinessLayer.dll. FireEye named the backdoored version of the DLL file as SUNBURST [2]. The SUNBURST backdoor delivers different payloads, such as a previously unseen memory-only dropper dubbed TEARDROP [2]. The TEARDROP dropper deploys an infamous post-compromise tool, Cobalt Strike Beacon. 

1.2 Collecting IOCs

The below resource list provides published IOCs from different sources. You can prepare an IOC list of your own for further investigations like threat hunting. New IOCs are posted every day, so keeping up with the news on this case is very important.

1.3 Determining TTPs

You can find the tactics, techniques, and procedures (TTPs) used in the SolarWinds breach in our previous blog post. In the mentioned post, we analyzed the tactics, techniques, and procedures utilized by threat actors of the SolarWinds incident to understand their attack methods and the impact of this breach.

1.4 Creating a Response Plan

So now you have understood the case and collected technical information. But you don’t know if your systems are compromised or not. Moreover, you don’t know if your defense technologies, processes, and people are ready for this threat. Now it is time to make a response plan and take action.

2. Mitigating Preventive Security Controls

2.1 Apply IOCs to Preventive Security Products

To prevent and detect related threats in the future, you can add IOCs you collected from published sources to your security products, such as web proxies, firewalls, and IPS devices. However, keep in mind that these IoCs can easily be changed by adversaries. 

2.2 Update and Apply Signatures to Security Products

Picus collaborates with network security vendors through its Technical Alliance Partnership (TAP) program. Picus Labs Blue Team simulates the threats developed by Picus Labs Red Team against TAP vendor environments, analyzes and maps the results to the right signatures to eliminate F/P issues. Finally, they add these verified signatures to the Picus Mitigation Library to provide Picus’ customers immediate mitigation suggestions for their security controls.

In this section, you will find the vendor-specific network prevention signatures to defend against Sunburst threats. The vendor, engine, and signature details are given in the below table. We recommend that you first activate the relevant signatures of your prevention products in your test environments. Then, activate them in detection mode in your production environment and put them into prevention mode after making sure that there is no F/P situation. If your security product is not on the list below, you should update the product’s signature database immediately.

2.3 Other Suggestions

Solarwinds announced the upgrade procedure of its software on its official website [8]. Microsoft, FireEye, and GoDaddy collaborated to create a kill switch for the SunBurst backdoor distributed in the SolarWinds hack [9].

3. Compromise Assessment

You can conduct compromise assessments on your systems by using the YARA rules released by FireEye [10]. To utilize YARA rules, you can use an open-source Yara scanning tool or an enterprise product and distribute it to the endpoints on your systems, then add the rules and get the results. Moreover, you can use IoCs included in Yara rules and search them in your SIEM environment.

You can use free tools to scan YARA rules on your systems. Three of these tools are shared below as an example:

• Yara Endpoint [11]
• Spyre [12]
• Loki [13]

You can also access the list of products that support YARA from VirusTotal [8] and use their features if you have these products in your systems.

If you detect any suspicious activity or pattern during this investigation, you need to immediately start an incident response procedure.

4. Threat Hunting

4.1 Visibility Health Check

Before starting the hunting process, you need to check your log visibility and duration. For example, if you don’t have firewall logs in your SIEM, you can’t perform a hunting query using related IP addresses in the IOC list. And also, if you are storing only the last month’s logs in your SIEM, you don’t need to search for these IOCs beyond a month.

After this health check operation, you can define a hunting scope. If there is any gap in visibility, you should fix these gaps for further investigations. You can also check the SIGMA rules published in the Picus Labs Github repository to find out suggested endpoint log configurations.

4.2 Threat Hunting with IOCs

You have the IOC list you prepared before the response plan, so now you can start a threat hunting process by using these IOCs. You need to write IOC search queries and perform them on your SIEM and EDR products, starting from at least nine months back.

4.3 Threat Hunting with TTPs

You started a hunting process with IOCs, but as you know, adversaries can easily change them. To perform effective and well-defined hunting, you need to understand the TTPs of this case. And you already prepared a TTP list before the response plan. 

You can write your own SIEM and EDR queries to hunt and detect these TTPs and/or you can use SIGMA or SIEM/EDR specific rules published in Picus Labs Github repository. Finally, you need to perform these queries on your SIEM and EDR products starting from at least nine months back, since we are sure that this incident started no later than March 2020.

Similar to the compromise assessment stage, if you detect any suspicious activity or pattern during this investigation, you need to start an incident response procedure immediately.

5. Detection Engineering

Picus is working with SIEM and EDR vendors through its Technical Alliance Partnership program. Picus Labs Blue Team develops SIEM (IBM QRadar, Splunk, Micro Focus ArcSight) and EDR (VMware Carbon Black) specific detection rules based on the SIGMA rules defined in the chapter above. Following the development, the Picus Blue Team tests and analyzes the results on each vendor environment and finalizes the rules with the specific product’s query language. Finally, they add these verified rules to the Picus Mitigation Library to provide immediate mitigation suggestions to Picus’ customers for their security controls.

In this section, you can find the IBM QRadar, Splunk, Micro Focus ArcSight, Elastic, and VMware Carbon Black rules to defend against Sunburst threats. You can test and deploy these rules to your detection products following the hunting process. The rule names mapped with each vendor can be found in the below table. If you are looking for detailed information about these contents, you can check out Picus Labs’ Github repository.

6. Continuous Security Control Validation

6.1 Adversary Emulation

If we know the specific TTPs used by adversaries in a cyberattack, we can follow the Adversary Emulation Process proposed by MITRE ATT&CK [14]. That is initiated by gathering the threat intel, extracting techniques, developing tools, and running attack simulations to emulate the adversary. This part can be run by your red-team working closely with you. If they are not available or are too busy, then the blue-team can either run this adversary emulation process by herself or leverage Breach and Attack Simulation (BAS) products. BAS is a set of technologies developed to challenge cybersecurity controls using real threat scenarios and reveal defense gaps and detection shortcomings before an attack or an incident takes place. 

Adversary Emulation Process recommended by MITRE ATT&CK [14]

6.2 Prevention Validation

A typical enterprise uses a combination of 30+ different prevention and detection technologies. Investing in these well-known security solutions, without the option of measuring their effectiveness in real-life situations, often creates a false sense of safety. Studies show that companies achieve 53% prevention efficacy [15] due to reasons that include limited resources and the absence of expertise required to fine-tune sophisticated security technologies. Therefore, the effectiveness of preventive security devices such as NGFW, IPS, WAF, Sandbox, and other tools against cyberattacks must be measured to identify technology, people, and process-related gaps for prevention. 

6.3 Log Validation

It is crucial to identify security events that were detected (or prevented) by security controls but not seen in SIEM platforms. Such situations indicate that log mechanisms may be missing, have not been activated, or are not working as expected. Configuration errors, software bugs, expired licenses, end of life APIs, heavy traffic load, software agent related issues, and some other factors on IT assets and security controls may be the root cause of these situations. Proactively identifying these shortcomings is key for ensuring a healthy log mechanism and that no alerting gaps occur due to unidentified security events in the future.

6.4 Alert Validation

Security analysts rely on alerts to detect and potentially respond to malicious events. Each alert needs to be investigated either manually or by an automation system. Based on this investigation, in a process called alert triage, the analyst should decide if the alert should be escalated for further triage as an incident or be dropped. 

Quality and scope of detection rules are the most critical factors in efficient alert management. In case detection rules are missing, are narrow in scope, or are poorly written; this means no alerts will be generated, malicious activities will go undetected and create a significant level of cyber risk. If the detection rules’ scope is too broad with no precise intent, the number of false positives will increase, causing alert fatigue. There needs to be a proactive alert validation process to identify redundant and obsolete rules in addition to incomplete and ambiguous use cases. Only then, the number of alerts will decrease, making sure that security analysts receive only relevant alerts. 

Lessons Learned

• This case has emphasized the importance of teamwork. Not only the SOC team, but all security teams need to work in collaboration to determine what is done correctly and what needs to be.
• The most important part of illuminating a security incident is visibility, which can be achieved by not only purchasing and installing new products but also maintaining, updating, and tuning them continuously. Moreover, the collected logs on all networks and endpoints must be properly stored for analysis. Therefore, validating effective log collection is crucial to response incidents.
• Once visibility is established, it is necessary that the collected data is analyzed by skilled cybersecurity analysts. It is crucial to make a skill-set analysis of employees to determine the skill-gaps. Then, the analyst should be supported with the right training to close the skill-gap. Tabletop exercises are important to foster efficient teamwork. These exercises validate your response to cybersecurity incidents. You can automate tabletop exercises with Breach and Attack Simulation (BAS) products.
• Evaluate whether you are getting the right information at the right time from your cyber threat intelligence (CTI) sources. It is crucial to make the right investments for CTI. Threat intelligence should not only be IoC (Indicators of Compromise) based but also include information and TTPs (Tactics, Techniques and Procedures) in order to shed light on events.
• In order to deal with cyber incidents with the least damage possible, it is necessary to respond as soon as possible. Incident response plans and procedures must be prepared and implemented ahead of time to shorten the response time. Breach and Attack Simulation (BAS) products can be used in the implementation of IR plans and playbooks.
• Apply continuous purple teaming, regular threat emulation, and cyber drill exercises to improve cyber resilience.

References

[1] FireEye, “Unauthorized Access of FireEye Red Team Tools.” [Online]. Available: https://www.fireeye.com/blog/threat-research/2020/12/unauthorized-access-of-fireeye-red-team-tools.html.

[2] FireEye, “Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor.” [Online]. Available: https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html.

[3] msrc, “Customer Guidance on Recent Nation-State Cyber Attacks.” [Online]. Available: https://msrc-blog.microsoft.com/2020/12/13/customer-guidance-on-recent-nation-state-cyber-attacks/.

[4] “Krebs on Security.” [Online]. Available: https://krebsonsecurity.com/2020/12/malicious-domain-in-solarwinds-hack-turned-into-killswitch/.

[5] “Security Advisory | SolarWinds” [Online]. Available: https://www.solarwinds.com/securityadvisory.

[6] “Inline XBRL Viewer.” [Online]. Available: https://www.sec.gov/ix?doc=/Archives/edgar/data/1739942/000162828020017451/swi-20201214.htm.

[7] R. M. Kevin Poulsen and D. Volz, “Solarwinds Hack Victims: From Tech Companies to a Hospital and University,” The Wall Street Journal, wsj.com, 21-Dec-2020 [Online]. Available: https://www.wsj.com/articles/solarwinds-hack-victims-from-tech-companies-to-a-hospital-and-university-11608548402.

[8] “Security Advisory | SolarWinds” [Online]. Available: https://www.solarwinds.com/securityadvisory.

[9] L. Abrams, “FireEye, Microsoft create kill switch for SolarWinds backdoor,” BleepingComputer, 16-Dec-2020. [Online]. Available: https://www.bleepingcomputer.com/news/security/fireeye-microsoft-create-kill-switch-for-solarwinds-backdoor/.

[10] fireeye, “fireeye/sunburst_countermeasures.” [Online]. Available: https://github.com/fireeye/sunburst_countermeasures.

[11] Yara-Rules, “Yara-Rules/yara-endpoint.” [Online]. Available: https://github.com/Yara-Rules/yara-endpoint.

[12] spyre-project, “spyre-project/spyre.” [Online]. Available: https://github.com/spyre-project/spyre.

[13] Neo23x, “Neo23x0/Loki.” [Online]. Available: https://github.com/Neo23x0/Loki.

[14] B. Strom, “Getting Started with ATT&CK: Adversary Emulation and Red Teaming,” MITRE ATT&CK®, 17-Jul-2019. [Online]. Available: https://medium.com/mitre-attack/getting-started-with-attack-red-29f074ccf7e3. [Accessed: 31-Dec-2020]

[15] “IBM Study: More Than Half of Organizations with Cybersecurity Incident Response Plans Fail to Test Them.” [Online]. Available: https://newsroom.ibm.com/2019-04-11-IBM-Study-More-Than-Half-of-Organizations-with-Cybersecurity-Incident-Response-Plans-Fail-to-Test-Them.