Skip to main content

Report Finds OpenSSF Scorecards Are Highly Effective Measures to Assess Project Security

By October 20, 2022Blog
state of the software supply chain sonatype

By Stephen Magill, VP of Product Innovation, Sonatype

This year’s eighth annual State of the Software Supply Chain Report by Sonatype found massive growth in open source demand, growing at 30-35%, and in supply of new components, growing by 33%. The report also found that attacks targeting the software supply chain have increased both in frequency and complexity – with a 633% increase in software supply chain attacks since last year. And the challenges go beyond malicious attacks, as the research revealed that 96% of open source Java downloads were of components with known-vulnerabilities even when a safer version was available. This indicates that open source consumers are either not tracking vulnerability or not prioritizing vulnerability when choosing component versions.

Identifying Vulnerable Projects

One lever that could help control vulnerability is to choose projects that are less likely to have vulnerabilities discovered in them. To see if this is possible, Sonatype used machine learning to evaluate the connection between the OpenSSF Security Scorecard checks and vulnerability information. Because OpenSSF publishes the individual checks that feed into their Scorecard Score, Sonatype was able to test how well a model based solely on these software development best practices could correctly identify projects with known vulnerabilities. The model was able to successfully label projects as vulnerable or not with 78% accuracy based just on Scorecard data, indicating that Scorecard checks provide a very useful vulnerability signal.

Elements-most-useful-for-identifying-vulnerable-projects

Sonatype researchers were also able to determine which practices in Security Scorecard are most important, as seen in Figure 2.2. Not surprisingly, code review emerges as the most important factor. Code review has long been identified as a high-impact practice that can substantially improve code quality. Having binaries checked into the repository was the second-most important factor associated with vulnerability. Binaries provide an attack path, decrease transparency, and reduce auditability of code. Pinning dependencies was the third most important factor, hinting at the importance of dependency management in maintaining secure software. Branch protection, which enables a formal approval process for code changes and pairs well with code review, was the fourth most important factor.

Dependency Management

Projects adopting the practices set out by the OpenSSF in its Security Score, including adopting a dependency update tool that ensures rapid updating of vulnerable dependencies, will improve their project’s security and the security of the open source projects that depend on them. Dependency management is critical, because Sonatype’s research revealed that about 6 out of every 7 vulnerabilities affecting projects come from transitive dependencies.

To account for the importance of dependency management, Sonatype incorporated Mean Time To Update, a measure of dependency management hygiene into the model. The researcher team then used this model to generate a score for each project. This score, ranging from 1 (lowest) to 10 (highest) has been released as the Sonatype Safety Rating and is published in OSS Index and Maven Central. In testing, the rating was 86% accurate for identifying vulnerable projects, and now Sonatype is soliciting community feedback to further improve the metric with real world data.

Report Recommendations

Sonatype recommends, since most vulnerabilities arise from transitive dependencies, to carefully consider every library you use. Favor ones with smaller dependency trees. Look for projects that are quick to update when new versions of their dependencies are released (low MTTU). Minimizing the total number of dependencies and maintaining low update times for your own project’s dependencies are two critical factors for reducing the risk of transitive vulnerabilities.

This advice holds true not just for consumers of open source components but also open source maintainers. Projects adopting the practices set out by the OpenSSF in its Security Score, including adopting a dependency update tool that ensures rapid updating of vulnerable dependencies, will improve their project’s security and the security of the open source projects that depend on them.

Read the full State of the Software Supply Chain Report here.