container vulnerability scanning – Cyber Security Kings

With so many cloud container security tools to choose from, it becomes difficult to know which is best? As a seasoned Cloud Security Architect, I regularly end up trying to find the best tool for my clients. Best Container Security Tools? Container security tools scan docker container images for security vulnerabilities, malware, configuration issues to deviations in expected behaviour using the latest threat intelligence and machine learning during the CI/CD pipeline build, in the registry and at runtime. The 6 best container security tools are: Twistlock AquaSec Qualys Layered Insight BlackDuck OpsSight Tenable.io Container Security Trend Micro Cloud One™ Container Security In the following part of the article, I'm going to provide information about each of these tools. The following information about these tools is only a partial assessment and it is highly recommended the criteria in the section 'Choosing the right tool' is used, to form a more valid opinion of each of tools. #1. Twistlock (now Prisma Cloud) Twistlock is now part of Palo Alto's Prisma Cloud offering and is one of the leading container security scanning solutions. Containers Twistlock can be installed as a side car container to monitor other containers in the following container hosting services: AWS[1] Azure[2] Google Cloud Platform Kubernetes [1] For AWS Fargate, the vulnerabilities will not be scanned in real-time as the solution will not be given the privileges to be able to do the vulnerability scanning. Vulnerability scanning will need to be done at the registry level, AWS Elastic Container Registry (ECR) on a periodic basis. [2]Azure Container Instances does not allow privileged side car containers, so registry scanning will be required for real-time scanning of vulnerabilities. Twistlock is able to integrate with the following common container registries: AWS ECR Azure Container Registry Google Container Registry Sonatype Nexus Other registries with suitable APIs can also be integrated with Twistlock, allowing Twistlock to query the image information and then be able to compare this against known vulnerabilities and issues. Serverless Functions The following cloud vendor serverless functions are supported: AWS Lambda Azure Functions Google Functions Container Image Support Twistlock supports both the standard container images and the RedHat images, providing accurate vulnerability assessment scores for both types of these images. Runtime Application Self Protection (RASP) Support Twistlock's Runtime Application Self Protection (RASP) is provided as embedded security ensuring containers and serverless functions run as they are designed to run, with any deviations such as suspicious processes, suspicious outbound network connections being blocked or at the very least being alerted upon. For AWS Fargate and other managed container solutions, RASP is important to ensure if any malicious event tries to escape outside Malware Scanning Twistlock has malware scanning capabilities built in and will scan images for any malware as part of any scanning done on the container images. It's important to note with the RASP functionality, additional protection against malware is also provided, as malware will be stopped in its tracks (if RASP blocking is set) if the malware tries to do something outside of the expected behaviour profile. Automation & Machine Learning Support Twistlock uses artificial intelligence, machine learning to be more exact, to profile expected behaviours to create models, which can be enforced at run-time and any deviations reported with the option to block these types of events. Threat Intelligence Twistlock uses the vulnerability and threat feeds with real-time updates, allowing organisations to stay up to date with the latest CVEs, exploits and threats. With Twistlock now being part of Palo Alto, additional threat intelligence will be made available within Twistlock from Palo Alto, gathered from its extensive range of other security products. Threat Reporting Twistlock has comprehensive reporting capabilities, allowing you to dig deep into what vulnerabilities and issues have been found. When I first came across Twistlock they were still using the version 2 of the CVSS and in my initial evaluation with them, I flagged this up as a concern. They were already working on this as an update and within a month, we had a version working with version 3 of CVSS. Continuous Compliance Twistlock has the ability to use preconfigured and configurable security policies to maintain continuous compliance. Compliance with CIS benchmarks, NIST, HIPAA to PCI standards are available out of the box. Additional functionality Twistlock has built in support for: Secrets Management Virtual Machines Container Network Application Firewall (CNAF) Licensing Licensed as commercially available tool, requires a yearly subscription. #2. AquaSec AquaSec is another leader in container security scanning solutions and a direct competitor with Twistlock. Containers AquaSec is installed as a side car container that monitors other containers on the same host and can be used with the following cloud container hosting services: AWS Azure Google Cloud Platform Kubernetes For environments where side car containers can't be deployed like AWS Fargate and Azure Container Images, the AquaSec MicroEnforcer tool can be used. AquaSec is able to integrate with the following common container registries: AWS ECR Azure Container Registry Google Container Registry Sonatype Nexus AquaSec can also integrate with other registries as long as they have a compatible APIs which allows AquaSec to get the container image information so, this can be assessed against current vulnerabilities and anomalies. Serverless Functions The following cloud vendor serverless functions are supported by AquaSec: AWS Lambda Azure Functions Google Functions Container Image Support AquaSec supports both the standard container images and the RedHat images, providing accurate vulnerability assessment scores for both types of these images. Runtime Application Self Protection (RASP) Support AquaSec Runtime Application Self Protection (RASP) is called AquaSec MicroEnforcer and this provides security for containers and serverless functions to ensure they run as they are designed to run. Any deviations such as suspicious processes, suspicious outbound network connections being blocked or at the very least being alerted upon. For AWS Fargate and other managed container solutions, AquaSec MicroEnforcer is important to ensure if any malicious event tries to escape outside Malware Scanning AquaSec will scan for malware as part of the image scanning process. Malware contamination signs, such as actively scanning the network (host scanning, port scanning) or trying to access bad reputation IP addresses as part of command and control, can be picked up by AquaSec tools. Cyber intelligence feeds provide up to date information on IP addresses with bad reputation classification. Automation & Machine Learning Support AquaSec uses machine learning to build container profiles of expected behaviours, these profiles can be enforced at run-time with any deviations triggering alerts. Profile policy can be set to just alert or block any deviations in expected behaviour. Threat Intelligence AquaSec uses the vulnerability and threat feeds with real-time updates, allowing organisations to stay up to date with the latest CVEs, exploits and threats. Threat Reporting AquaSec has comprehensive reporting capabilities including its own AquaScore, allowing a comprehensive scoring to understand the severity of any vulnerabilities and issues. Continuous Compliance CIS benchmarks, NIST, HIPAA to PCI standards are available for compliance checking. Additional functionality AquaSec has built in support for: Environment variables encryption (secrets) Microservices-level Firewall Licensing Licensed as commercially available tool, requires a yearly subscription. #3. Qualys Layered Insight Qualys acquired the Layered Insight product in 2019 and when I was at Black Hat that year in Las Vegas, I spent some quality time talking to Qualys people about their new acquisition. Layered Insight can provide container image vulnerability scanning and compliance checking. Behaviour profiles can also be created using machine learning capabilities with monitoring done at run-time to watch for any deviations from expected behaviours. #4. Blackduck OpsSight BlackDuck OpsSight can scan container images for open source vulnerabilities and compare findings against its own Black Duck Security Advisories (BDSAs). These advisories include remediation guidance. Policies can be created and enforced to ensure only container images meeting the standards defined in the policies are allowed to operate, with any container images not meeting these standards, flagged up as alerts. #5. Tenable.io Container Security Tenable.io® Container Security checks the security of container images by checking for vulnerabilities and malware, with the benefit of checking for any policy violations. #6. Trend Micro Cloud One™ Container Security The Trend Micro solution can detect malware, along with vulnerabilities and for best practice in security, the solution can also detect any secrets, keys or passwords hard coded inside images. Compliance checking against PCI-DSS, HIPAA to NIST 800-190 to name a few standards and regulations. This solution can use the Snyk source code vulnerability database allowing early detection of vulnerabilities in open source dependencies. Machine learning capabilities are included and threat intelligence from the Trend Micro's Smart Protection Network. Choosing the right tool I'm not going to recommend any container security tool, as most do a good job and one brand of container security tool might be right for one organisation but may not right for another. However, I will look at the considerations required for choosing a container security tool, as detailed below. To find the right cloud container security tool for your environment, a thorough investigation is required using the following criteria: Containers only or Serverless Functions too? Technical Capabilities Operational Aspects Support Enterprise Security Integration Design Additional functionality No Silver Bullet The tips below could help in creating acceptance criteria to make sure the cloud container security tool selected meets the requirements, functionally as well as from a security perspective. 1. Containers only or Serverless Functions too? Some cloud container security tools can work better with serverless functions (also known as Function as a Service - FaaS), others with containers and then there are those security scanning solutions that will work well with both serverless functions and containers. Is the cloud container security tool required for container scanning or for serverless function scanning or for both? Answering what will the cloud container security tool be used for is the most important starting step. Getting this wrong will mean the security posture ends up being put at risk, as the cloud container security tool could fail in finding vulnerabilities, anomalies and malware when it's used in a different context from how it was evaluated. Serverless function security Finding out how well the cloud security tools work for Serverless deployments, also known as Function as a Service (FaaS) is a must if any of the following services are being used: AWS Lambda; Azure Functions; or Google Cloud Functions. Cloud security tools from the likes of Aquasec and Twistlock (as well as others) can provide security testing for these types of serverless functions deployments. Container security Finding out how well the cloud container security tools work with containerised deployments like Docker is a must, as container images provide a large attack surface for any would-be hacker. Using any of the following solutions will require some form of cloud container security tool to ensure security isn't overlooked: AWS Elastic Container Service (ECS); AWS Fargate; AWS Elastic Kubernetes Service (EKS); Azure Kubernetes Service (AKS); Azure Container Instances (ACI); Google Container Engine (GCE); Oracle Kubernetes Engine (OKE); Kubernetes; to OpenShift Container Platform (OCP). Some of these container orchestration services may have their own native scanning solution. Azure and Qualys have partnered to provide vulnerability scanning functionality, some of which is still in preview. 2. Technical Capabilities In any cloud container security tool assessment, the qualifying technical aspects need to be determined and understood. This will form part of the overall acceptance criteria, allowing for easier comparison between different security scanning tools. It's important to actually check out how these cloud container security tools work with Serverless and Container deployments. Serverless Functions By testing sample, serverless functions deployments as part of any tool selection process will ensure the cloud security tool can work effectively. Below I've listed some common criteria to consider. (i). Does the cloud security tool return back alerts for anomalies, such as vulnerabilities? The basics of security testing must be covered by any cloud security tool tested. (ii). Does the cloud security tool return back alerts when trying to run unauthorised commands? A vulnerability causes shelling out from a python script in AWS Lambda and tries to run some commands, this will need to be picked up by the cloud security tool. (iii). Does the cloud security tool being tested return too many false positives? Too many false positive alerts could mean the cloud security tool isn't able to distinguish effectively between real alerts and those that are not real. (iv). Does the cloud security tool have the granularity to deal with serverless functions? For example, can the cloud security tool distinguish between uname -n as a safe bash command and trying to shell out from python as bad. From experience, I've found testing in this area is essential just to make sure the security coverage works as expected. I've also found this can also be an opportunity to test how good the cloud security tool vendors support is, should fixes be required during the assessment. Believe me, I've had this happen a few times where the cloud security tool has not done what's expected and a fix has been required. (v). Can the cloud security tool block serverless functions from running as well as alerting on them running incorrectly? Being able to block malicious serverless functions and malicious code within normal-looking serverless functions needs to be tested. Blocking can be useful in stopping malicious code from propagating itself. (vi). Does the cloud security tool work well with the language used in the code? Python and Node.js are generally a given with most of the cloud security tools and they merrily analyse away but there are a whole host of other languages you can use in serverless functions. Then there are frameworks which have wrappers, from Django wrapped with Zappa and Laravel PHP wrapped with Vapour, can these framework deployments still be effectively scanned? (vii). Does the learning mode accurately determine behaviours and report on any deviation? A good cloud security tool should have the ability to learn a behaviour and then alert on any deviation from that behaviour. So a serverless function running to make mortgage payment calculations has had its behaviour monitored by the cloud security tool. This functions behaviour changes after a few days and the function starts to try to run a different set of functions than expected. This change in behaviour would need to be flagged up as this deviation could be the sign of an attacker at work. Containers By testing sample, containerised deployments as part of any tool selection process will ensure the cloud container security tool can work effectively. Below I've listed some common criteria to consider. (i). Can the cloud container security tool run benchmarks for compliance? Generally, for compliance CIS, NIST or any other common benchmark can be used and the tool must be able to report back on the compliance status when these benchmarks are run. (ii). Can the cloud container security tool use out of the box security policies? Many tools will come with a set of security policies to provide additional security checks. For example, there may be a policy that can alert on secrets like passwords and tokens being embedded in images. So if an image has embedded database access credentials these will be picked up the cloud container security tool and alerted. (iii). Can the cloud container security tool detect and isolate malware? Malware can creep into images, inadvertently through the choice of image and dependency repositories used for libraries. Any tool assessed must be capable of finding malware such as crypto-mining malware. (iv). Can the cloud container security tool work with different image types to determine vulnerability scores? From experience, some tools could have a problem with Redhat images and incorrectly report on scoring. (v). Can the cloud container security tool integrate with the corporate identity provider (IdP)? Managing the cloud container security tool is an important part of any evaluation and identity control provides scope to lock down access and controls. (vi). Can the cloud container security tool integrate with Security Incident and Event Management (SIEM) tools? This is important especially considering there will be a lot of logs generated from all the serverless and container scanning. Relying on the cloud container security tools dashboard probably won't give you the level of cover a SIEM combined with SOC service can. Then there's round the clock coverage all year long unless the security scanning tool dashboard will be managed around the clock, integration into a SOC team will be important. Whilst it may not be possible to check the actual SIEM integration, the interfaces to integrate can be determined as being satisfactory or not, generally, if the tool can dump a Syslog and it can be delivered to the SIEM then it should be ok. Custom parsing development may be required for the SIEM to appreciate the log content but being able to get the logs is an excellent starting point. 3. Operational Aspects I would also look at how the cloud container security tool operates. (i). Does the cloud container security tool run as a privileged container or as a sidecar container using normal privileges? Privileged containers can be a problem for security as they tend to run with root privileges and if compromised not only give access to the container but potentially also the other containers on the host and the host the container is running on. (ii). Does the cloud container security tool work with the container orchestration platform? AWS Fargate, in particular, is locked down, meaning the standard way to use a cloud container security tool won't work because the cloud container security tool will be running in a restricted environment. So the cloud container security tool will not be possible with containers running in AWS Fargate, this isn't a show stopper, it just means the cloud container security tool will need to be done in the image registry at the AWS Elastic Container Registry (ECR) level. However, by using a sidecar approach, the cloud container security tool can run as a sidecar container and use the 'VolumeFrom' feature in a non-privileged mode to do scanning of other containers. Albeit this won't be vulnerability scanning but Run-time Application Security Protection is more commonly known as RASP, to detect any anomalies whilst the container is in use. Does Runtime Application Self Protection (RASP) stop rogue commands from running? RASP can be used for detecting anomalies in images, such as the running of commands requiring an elevation of privilege. In the real world, if this was happening, it could be a sign of an attacker at work. (iii). Can the cloud container security tool identify development tools to build images? So if development is done using Java, can the cloud container security tool check there's no Java Development Kit plus dependencies in the image heading for production? One of my own pet hates is Fat Jars, these are files full of multiple libraries and dependencies that aren't all required. These increase the attack surface as you end up with many more components each with their own vulnerabilities. (iv). Can the cloud container security tool determine images based on tagging, signing or notarizing depending on the security requirement? Keeping non-production container images out of production environments is a MUST and the cloud container security tools must be able to at a minimum have policies that can be configured to allow PRODUCTION tagged images only, into production environments as part of the Continuous Delivery (CD) in a CI/CD pipeline. (v). Can the cloud container security tool integrate with the pipeline? Does the tool have a plugin to use with Jenkins or if AWS CodePipeline is being used, how well does the tool integrate? Will you need to write some code to be able to pull and parse information from cloud container security tools to be able to integrate the vulnerability scanning in the pipeline? (vi). How will the cloud container security tool score vulnerabilities? The cloud container security tool should be able to give an indication of the severity of any vulnerability found. This may simply be: LOW; MEDIUM; IMPORTANT (or HIGH) or CRITICAL. Or the cloud container security tool may use some form of proprietary scoring like AquaSec does, where they have their own scoring called AquaScore. Any proprietary scoring will need to validate against the Common Vulnerability Scoring System (CVSS). Some vendors will also use different scoring criteria, for example, if you are planning to use Redhat images for the container images then RedHat may give different vulnerability scores online to their image vulnerabilities. So a Firefox vulnerability may be marked Critical in the associated Common Vulnerability and Exposures (CVE) note but Redhat may mark it as low. Simply because the CVE reflects the vulnerability impact across the operating system environments, including windows, where Firefox use on Windows with the vulnerability is particularly dangerous but Redhat will only view the vulnerability on their own operating system Redhat, so a lower possible impact. It's important to consider the CVSS version is accurate to your security needs, as CVSS version 3 is widely acknowledged as being the standard to go with. CVSS version 2 was criticised by vendors for not being complete enough. So some of the vendor's criticisms were acknowledged and improvements made in the new version. CVSS version 3 includes additional metrics including User Interaction (UI) and Privileges Required (PR) to name a few. Once a vulnerability is analysed with the metrics from CVSS version 3, they give a better scoring of the risk. I've seen some security scanning tools still using CVSS version 2 and this is unacceptable in my opinion for any projects as CVSS version 3 has deeper risk assessment criteria and CVSS version 2. Any project with internet-facing workloads must use CVSS version 3 vulnerability risk scoring. A vulnerability classed as Medium for example by using CVSS version 2 could be an Important or Critical class of vulnerability when assessed using CVSS version 3, as CVSS 2 doesn't factor as many metrics as CVSS version 3. This Medium vulnerability could be exploitable using no user interaction and no privileges, making it exceptionally dangerous, more dangerous than a Medium rating using CVSS version 2. 4. Support In any assessment, it's imperative the support of the cloud container security tool is also considered, as having great cloud container security tools without a decent operating model diminishes the effectiveness of the cloud container security tool. I regularly review Operating Models which contain some of the following details. (i). Who's going to run the cloud container security tool? Generally, this could be DevOps team but it could also be automated, so once the tool is deployed after stringent testing, there's no human involvement. However many facets of the cloud container security tools will not necessarily be enabled (blocking vs alerting) and some manual intervention will need to be scoped out. (ii). Who's going to be alerted? Even with the cloud container security tool doing its job effectively, picking up on anomalies and alerting but without the right people to follow up on the alerts in a timely fashion, the security scanning tools isn't going to be effective at lowering the risk exposure. Having a dedicated team like a Security Operations Centre (SOC) is the ideal. Suitably qualified people who know how to deal with the alerts generated.# I've seen many organisations firmly believe their DevOps teams will provide the operational capabilities for security scanning tooling. This usually ends up being problematic as the DevOps teams simply don't have the experience from an operational capacity to be effective. Does the cloud container security tool bring up to many false positives and therefore require more support effort to distinguish between what matters and what's not so important? (iii). Incident management Is the current incident management team sufficiently skilled to deal with these cloud container security tools? These cloud container security tools could be providing alerts on services the organisation doesn't currently use, such as serverless functions and containers. How are incident managers going to know what assets are deployed, as most are dynamic in the cloud? The containers spin up and then once they have finished, they spin down, so keeping track of what's what from an incident response can be challenging. (iv). Vulnerability Management Vulnerabilities are inevitable and having an effective plan to deal with them is vital. Any operating model should list how to deal with vulnerabilities found in containers and serverless functions. It's easy to say, a policy will be defined on the cloud container security tool that if any critical vulnerabilities are found, the container is blocked from running. Whilst this is fine for non-production workloads, stopping a container in a production environment could have a serious impact on the business. So a sensible approach is needed, with agreement from management and the business owner. Equifax is a good example of how having sufficient security scanning tooling without an effective operating model that led to a massive breach. Where the time taken to fix a known vulnerability exceeded the time deemed acceptable, allowing hackers to be able to exploit the vulnerability and gain access to the Equifax systems. 5. Integration How any cloud container security tool integrates into an organisation's Enterprise Security is essential. I generally look at how the cloud container security tool under evaluation can integrate with the organisation's Identity Provider and it's Security Incident & Event Management (SIEM) tools. Identity Providers I find it risky using cloud container security tools that don't integrate with an organisations identity provider. This can become an operation nightmare especially if the cloud container security tool uses its own built-in identity provider. This can affect Identity Governance and Administration (IGA) making it a chore to keep track of joiners, movers and leavers. My golden rule for identity is to minimize on identity providers so having one identity provider such as Microsoft AD for all tools from Jira, Confluence, Jenkins, AWS to the SAST, DAST, and security scanning tools IS A MUST. SIEM Integration capabilities with SIEM solutions are important as many SIEM tools have their own behaviour analysis and threat intelligence. They also provide the capability to store logs for a prolonged period which is essential for forensics. Log tampering in cloud environments is difficult to control, as someone always ends up with access to the storage area where the logs are kept and some cloud providers to offer log integrity from signing but I always find it better to ship logs back to the organisations SIEM. Where they can be stored for as long as needed. To be able to integrate with a SIEM, the cloud container security tool needs to be able to output logs in a manner usable by the SIEM tool. In most cloud container security tools I've seen they generally have the capability to output Syslog. Then either the SIEM tool can read the logs from Syslog or requires a parser to be developed for the SIEM tool be able to interpret the log data coming across. It's not particularly complicated and the criteria of being able to output to Syslog are sufficient in most cases. 6. Design I'd highly recommend creating an interim design for any cloud container security tool as part of the evaluation process. This allows stakeholders including security to see not only how the different pieces of the cloud container security tool fit together but how the tool will operate. I like to see the placement of the cloud container security tools components including: Dashboards Agents (pipelines, containers, hosts) Integrations (Active Directory, SIEM) Network topology The cloud container security tool will probably need access out to receive threat intelligence and some tools may even send metadata out for analysis in their cloud service to determine the vulnerabilities and vulnerability scoring. Sending data out in this way for analysis might not sit well with an organisations risk profile, so getting a 'heads up' on how the tool communicates is easily determines during the interim design. Tool placement Placement of the cloud container security tool can impact delivery, that is, placing the vulnerability checking later in the pipeline when the Continuous Deployment is done to a staging environment, will slow down delivery. When compared to discovering vulnerabilities earlier on, as in the Continuous Integration part of the pipeline. Therefore any analysis of the cloud container security tool under evaluation must be able to work without impeding delivery whilst still retaining its security functionality. Vulnerabilities found later instead of earlier have a significant cost impact too, as the delays in fixing impact the delivery. I would always check to see if the cloud container security tool can be leveraged as a first gate so developers themselves can get an indication of whether the Docker images, libraries and dependencies they are going to use are not full of vulnerabilities. Otherwise, these will be unearthed too far down the pipeline and require either the risk to be accepted for the vulnerabilities to go into production (not a good idea, not even with a remediation plan, or time and money to fix. Fixing in development always costs less than to fix it once it's 'live'.Any cloud container security tool shouldn't necessarily slow down delivery in the way that it works whereby it takes an age to scan each container image. 7. Additional functionality A lot of the cloud container security tools and solutions provide additional functionality including secrets management to Cloud-Native Firewalls proving mesh-like functionality. The organisations I've worked for, intend to have their own solutions for this additional functionality, but there's no harm in evaluating this functionality if your organisation doesn't have an alternative. 8. No Silver bullet It's important to remember cloud container security tools aren't a silver bullet when it comes to security nor is just having a robust DevSecOps function, as these will not eradicate a large proportion of security risk. Cloud container security tools need to be considered as part of the organisation's whole security posture, with other facets such as a secure SDLC, threat modelling, security requirements, security policies being equally important. Without the other facets of security in place, it becomes a pointless exercise just relying on just cloud container security tools. Need help? If you need some advice just connect send me a message. I can provide security strategy, security assessment and compliance consultancy. I have done this for many organisations large and small. One of the areas I focus on is DevSecOps including secure SDLC, SAST, DAST, Threat Modelling and CI/CD pipeline security. I provide security strategy, security assessment and compliance reviews for many organisations. There are many other cloud container security tools available too. I spoke with Trend Micro when I was a BlackHat in Las Vegas last year and they were looking at bringing in their own cloud container security tool. I look forward to working with their new tool soon. Related Questions: SonarQube vs Veracode vs Fortify which one is better? Structured acceptance criteria will need to be developed to determine which one of these SAST tools is appropriate for Static Code Analysis Testing. Which Cyber Security Automation Security tools are required? There are a lot of Cyber Security automation tools from automated continuous compliance, to actual automation frameworks such as Terraform, Chef to Puppet. Disclaimer: I am in no way affiliated with, or endorsed or work for any of the organisations mentioned in this article.