Debates about open source software quickly develop a religious dimension. And nothing is more likely to set off an argument of inquisitional proportions than accusations of shortcomings in an area as important as security. Two recent events highlighted the issue of the security of Open Source Software (OSS): one was a report claiming that open source developers are failing to achieve the necessary standards. The other was a suggestion that Linux kernel developers may have tried to cover up security vulnerabilities. However, when it comes to ensuring the quality of software from a security standpoint, is there really any difference between open source and closed source?
When the Linux 2.6.25.10 stable kernel was released it came with the announcement that “any users of the 2.6.25 kernel series are STRONGLY encouraged to upgrade to this release”.1 Yet no specific mention was made of any security fixes. This lead to accusations that security fixes were being deliberately obscured in the commit log by using benign descriptions that avoided mentioning vulnerabilities.
The ensuing debate had some suggesting that this amounted to a policy of hiding security vulnerabilities. The facts, as ever, are not so simple, but it made the kernel developers a soft target for those who would criticise them – and by implication the whole process of open source development – for a lack of rigour, and perhaps standards, in approaching this important issue.
Fortify's findings
In July 2008, Fortify Software released its report, 'Open Source Security Study: How are open source development communities embracing security best practices?'.2 It concluded that they're not.
It's an ever more important issue because of the widespread take-up of OSS. A CIO.com survey found more than half of organisations using OSS in some capacity.3 Another 10 per cent plan to do so in the next year and nearly half view OSS on an equal footing with proprietary software. Gartner estimates that, by 2012, 80 per cent or more of commercial software packages will contain some element of open source software.
Fortify tested 11 Java-based projects using its Fortify SCA static analysis tool. It found vulnerabilities in all of them, and also found that, rather than having fewer problems, subsequent versions actually had more. (The report did not suggest how this compares with proprietary software.) The report extrapolates from this to suggest that these projects lack a 'secure development process'.
Fortify also takes issue with open source projects for not providing programmers with the necessary access to security expertise (it offers the free Java Open Review project as one means of fixing that). And it says that these projects do not take sufficient advantage of security tools – such as Fortify SCA.
The report recommends that government and commercial organisations should view open source software warily because of the allegedly high risks involved. “Risk analysis and code review should be performed on any open source code running in business-critical applications, and these processes should be repeated before new versions of open source components are approved for use.”
Coverity report
Fortify's findings contrast somewhat with those from Coverity. The latter started its Scan site with backing from the US Department of Homeland Security, as part of the US Government's Open Source Hardening Project.
Like Fortify, Coverity uses its own static source analysis tool, Coverity Prevent, to search for vulnerabilities. But the most recent Scan report, published in May 2008, draws from two years' worth of data, analysing 55 million lines of code across 250 projects, including the larger ones such as Linux and Apache.4
The report concludes that “the quality and security of open source software is improving – researchers at the Scan site observed a 16 per cent reduction in static analysis defect density over the last two years, which reflects the elimination of more than 8,500 individual defects.”
What kind of software?
It's important to determine precisely what kind of open source project we're talking about. A quick search of Sourceforge.net shows more than 130,000 projects. A Joint Information Systems Committee (JISC) report showed that fewer than 5 per cent of them are sustained and that more than 90 per cent have only one contributor. Yet this is not the kind of software that is being used in corporate or government environments. The CIO.com survey's respondents were using Linux (78 per cent), back-end databases and web servers (74 per cent), software development tools (61 per cent) and desktop applications such as OpenOffice.org (29 per cent).
Some people still have a lingering image of open source as the domain of semi-religious geeks, operating in small, loosely knit bands. The implication is always that open source doesn't involve the commitment and rigorous processes of mainstream, commercial software houses.
This is wrong for a very good reason. Open source is mainstream now. Just look at the kinds of enterprise that are both using and developing OSS – names like Google, eBay and Facebook.
The distinction between open source and commercial no longer holds up. The OSS vendors are commercial enterprises. Major Linux distributors such as Red Hat offer the same kinds of support, services and assurances of continued development as proprietary software vendors.
On the whole, both open source and proprietary vendors are using the same development practices, tools and even, sometimes, the same programmers.
“As a commercial company that manages multiple open source projects, we apply best practices in the development of those projects,” says Mike Guiterman, open source marketing director for Sourcefire, which produces Snort and ClamAV. “They have very mature software development lifecycles where we QA and security check our technology. For example, our regression suite has in excess of 100 million automated tests. We also use outside code auditors, and both open source and commercial testing tools to verify the quality and security of our codebase.”
Quality review
Typically, open source projects of all sizes are overseen by project managers who are very careful about who is allowed to check in code on the CVS. Submissions that come in from the wider community are likely to go through a review for both quality and security.
A variety of tools and processes are used, too, which may include regression tests, system tests, code scanning (simple pattern matching, or static-analysis), public bug tracking, security architecture review and penetration testing.
Programmers also exploit open management tools – bug trackers like Bugzilla, source code revision management tools like SVN and automatic build tools such as Buildbot. These tools are so well known and trusted that many proprietary software vendors use them too.
Open source developers may be wary of some formalised procedures and standards, such as ISO 9000. But that's true of many proprietary software developers. And as Charlie Hull, managing director of Lemur Consulting points out: “The trouble with formal quality control procedures, such as ISO 9000, is they can very easily become fantastic systems for writing down, very carefully, all about the bad code you're creating.”
Given the similarity in development processes, one might expect both open and closed source development efforts to have the same level of quality when it comes to security.
“In my experience, the fact that something is an open source project doesn’t tell you much about security,” says Jeff Williams, chair of the OWASP Foundation. “Just this week, I reviewed an enterprise intranet application for a financial organisation that handles sensitive data. That one application has thousands of egregious flaws, far worse than most open projects.”
Security depends more on the software design. However, OSS advocates claim that the initial coding tends to be of a higher quality to start with. “Developers have to earn their right to contribute. Therefore programmers tend to be of high quality,” says Simon Heron, Internet security analyst at Network Box. And as the code is available for anyone to peruse, programmers feel the need to do their best work.
Check it yourself
If the code is critical to your business and security is a major concern, with OSS you have the option of scrutinising and testing it yourself. You can even get involved in the whole development process. “Anyone can follow the OWASP Code Review and OWASP Testing Guide to generate their own assurance,” says Williams.
Fortify asserted that organisations adopting OSS should carry out risk assessment and code reviews. But perhaps the point is that this is possible with OSS where it isn't with proprietary software. That allows organisations to reduce risk by applying any processes for ensuring application security they may have developed themselves – a not uncommon situation in the banking sector, for example.
By monitoring the development of an OSS project, joining mailing lists, seeing how quickly issues are fixed, how often releases are made and so on, users can verify how well the project is being managed.
Best of both worlds
Major OSS projects, like MySQL or Snort, benefit from both the controlled environment of a commercial software house and the community aspect of open source.
Even after going through internal testing and QA, the software still has to gain the approval of that community. “If you look at SnortSP, we put out alpha code a year ago for review,” says Guiterman. “We've had three beta releases of the code to date, and we're not going to production until later this year.”
The results of code reviews and tests are publicly available, so that if something is broken, everyone knows about it. That's why OSS typically gets fixed fast. Users can even fix it themselves. With proprietary software they are at the mercy of the vendor.
Developing best practice & standards
There are no formal standards for application security, but some organisations are working towards developing guidelines, best practices and tools.
OWASP has over 70 secure development projects hosted on its website, including guidelines, best practices, SDLC discussions and tools for testing and scanning code.
Last November, the SANS Institute's Secure Programming Council announced a standard designed to allow programmers to demonstrate that they have the ability to write secure code.5 The Institute for Secure and Open Methodolgies (ISECOM) offers a peer-reviewed methodology for performing security tests and metrics.6 And the oCert project is a group of experienced open source developers that is providing security support to OSS projects.7
Ultimately, organisations looking for invulnerable software are going to be out of luck. Whether they choose open or closed source software is most likely to be a licensing decision. But at least with OSS they get a chance to see for themselves the efforts that are being made.
References
1. There's a good summary of the Linux kernel debate here: http://lwn.net/Articles/290227/
2. Fortify report: www.fortify.com/ossreport.html
3. CIO report: http://cio.com/article/375916/Open_Source_is_Entering_the_Enterprise_Mainstream_Survey_Shows
4. Coverity Report: http://scan.coverity.com/report/
5. SANS Institute: www.sans.org
6. ISECOM: http://www.isecom.org/osstmm/
7. oCert: http://ocert.org
