How To Check For Malware in CICD Pipeline
Organizations have begun to greatly embrace DevOps and the CI/CD pipeline, and all of the benefits that it provides, to accelerate bringing their software applications to production. This has been a great improvement over other legacy development methodologies, such as agile and waterfall, and has allowed for continuous improvements to deployed software based on real world testing and feedback.
Application Security has traditionally been an afterthought of development or seen as a roadblock to getting to final deployment. Similar to DevOps adoption, organizations have realized that the earlier they address security in the software development life cycle, the better the outcomes. Simply put, DevSecOps is DevOps with security built-in, right from the start. This “shift-left” mentality takes security considerations into account as soon as the development process begins. DevSecOps ultimately saves time and money by identifying and resolving security vulnerabilities as early as possible in the development lifecycle.
Current approaches to DevSecOps in the CI/CD pipeline include:
Vulnerability Scanning – technologies like SAST (Static Application Security Testing) and DAST (Dynamic Application Security Testing) help identify security vulnerabilities in code automatically
Application Runtime Protection – technologies like RASP (Runtime Application Self-Protection) help secure and monitor applications in their deployed environment
Cloud Service Providers – major CSP’s like AWS and Azure, provide technologies to help secure applications, such as web application firewalls, security logging & auditing, and encryption services
Standards & Policies – clearly laid out security standards allow development teams to account for application security earlier in the development process
Software Composition Analysis – SCA tools used to detect open-source components used in software with known vulnerabilities or licensing issues
Penetration Testing – manual checking of security vulnerabilities on deployed applications
Container & Service Management – orchestration tools such as Kubernetes and Docker provide significant advantages in automating typical security tasks
However, there is one critical part missing from current DevSecOps approaches – validation that malware is not being inserted into our code throughout its development, deployment, and distribution. The CI/CD pipeline can be a complicated beast for many software shops, and the opportunities for an attacker to compromise one component of it and insert malicious code into the legitimate application are numerous.
There have been multiple attacks over the last few years where this exact attack has occurred, where software is compromised in its development and deployment process along its CI/CD pipeline. In 2015, more than 39 iOS apps found in the app store were affected by a sinister strain of malware, called XcodeGhost. This malware originated from a malicious Xcode (Apple’s official tool for developing apps for iOS) installer being circulated on the internet. Any app developed with this version of Xcode would contain the malware and would even bypass Apple’s stringent app store review process. In 2017, PyPI, the official module repository for the Python programming language was compromised. Several well-known Python packages were published under slightly modified names and contained malicious functionality. As soon as the malicious packages were installed, the developers we’re compromised, along with any applications they developed that used the malicious packages. In 2019, Asus, one of the world’s largest computer makers, was pushing out malicious software updates to its users. It was later identified that this malware was inserted by an attacker who had compromised Asus’s servers responsible for normal software updates and distribution. The reputational damage experienced by Asus was crippling.
In all of these attacks, one thing is common: attackers can manipulate an application and insert backdoors at any point during its development and deployment. Malicious software modules, such as the PyPI attack, can affect an application from the moment development begins. A malicious compiler or build server can insert malware, as seen in the XcodeGhost attack. Attackers can modify software deployed in all environments, including production and patching environments, as seen in the Asus attack. Ultimately, organizations must ensure that developed applications are free of malicious code, across all of their development and distribution points.
How to check for malware in the CI/CD pipeline?
Application security teams must no longer be focused solely on application-level vulnerabilities, such as those covered by the CWE project by MITRE, but also on the potential that malware can be embedded in those applications at any point across its CI/CD pipeline.
Since malware can be embedded at any step of the CI/CD pipeline, our malware prevention & detection efforts must take place at every possible point of tampering. Hardening of the DevOps pipeline, and maintaining strong security practices such as multi-factor authentication are critical steps in this process. However, as with any form of cyber defense, a well-motivated attacker will always have the advantage, and we must prepare for the eventuality that our defenses may be circumvented. We must focus not solely on prevention, but also heavily on detection & response capabilities.
The difficulty arises with how to practically, effectively, and accurately detect this malware. Malware, by nature, is just normal software, written with malicious intent. If you looked at two pieces of seemingly identical software, one being a legitimate file encryption utility, and one being a piece of ransomware, you’d likely see significant code overlap, as both hackers and legitimate developers rely on the same underlying API calls to implement their functionality.
Luckily, there is a very straightforward way to understand if a binary has been modified from its known ‘good state’: if its cryptographic hash changes. Hashing is a one-way function, that is, a function which is practically infeasible to invert. It is the only method of technically verifying with significant confidence the data integrity for a binary.
For example, if we know that the SHA-256 hash of our legitimate software is “f0e9beba…”, if at any point we observe that software’s hash changes to something else on its way to the end-user, then we know at least one byte in the binary has been modified, and potentially malicious code has been inserted. If this occurs, an incident response process must be conducted to understand what code changes were made, how those changes were made, and how to prevent any further tampering.
CodeHunter for the CI/CD pipeline.
The CodeHunter platform is a revolutionary way to fully and automatically compute the behavior of software to identify exactly what it does, without having to manually reverse-engineer it. Using our patented behavior computation technology, CodeHunter automatically answers the question: “What does this software do, and is it secure?”
Behavior computation has picked up where sandboxing, dynamic analysis, and machine learning have left off, and is resistant to evasion techniques targeted at those threat detection methods. Since CodeHunter computes the full behavior of software as executed by the CPU, any obfuscation, spaghetti logic, dead-end code, or other evasion techniques are rendered useless, and the software’s true capabilities are left exposed. All of this is achieved without requiring access to source code, as CodeHunter can fully and automatically reverse engineer compiled binaries using our patented behavior computation approach.
Organizations rely on CodeHunter to assess if their developed applications are free of malware across the entirety of their DevOps pipeline. CodeHunter can accurately, and quickly identify if developed code exhibits any malicious functionality, before it impacts your applications end-users. Behavior computation picks up where other application security tools leave off and validates that malware is not being inserted into applications across their development and deployment lifecycle. Using CodeHunter as part of the CI/CD pipeline, organizations can:
Ensure that the code they are developing, deploying, and distributing has not been maliciously tampered with
Deliver software to end-users with confidence that it is clean and free of threats
Protect their brand and reputation from a damaging attack on their legitimate software