Heartbleed: NSA's Bloodletting of Vulnerabilities
By Ann-Marie de Veer
Saturday 19 April 2014
The notion that the OpenSSL bug called Heartbleed is somehow analogous to a medical condition of the heart is not simply a misplaced metaphor but the hallmark of further nefarious activities by the NSA and GCHQ.
Of course, in animals, including humans, there is no such state as heartbleed: the closest analogy is a mitral valve prolapse, a condition where the mitral valve in the heart does not close correctly and allows blood to flow backwards on each contraction, often called mitral regurgitation. Not a loss of blood as such, more of an internal malfunction within the pulmonary system.
While in contrast, the OpenSSL bug did cause a loss of blood, or more precisely data. That is, chunks of memory previously requested and freed, but not cleared by OpenSSL in its "heartbeat" function with a client computer, could be exploited by an attacker in sending a malformed heartbeat request allowing them (the attacker) to recover the uncleared data in memory. The system bled up to 64 Kb of server memory at a time on each request.
Moving on from the semantics of the issue, whose relevance will be become clear later, the initial report by the Guardian was quickly followed by others in the MSM. We soon discovered the author of the bug was Robin Seggelmann, who documented the proposed new standard for OpenSSL in February 2012. We also learnt that Seggelmann's coding work was actually submitted on 1 January 2012 by Stephen Henson, a code reviewer at OpenSSL, for inclusion in its next release in March 2013. It was also Henson who submitted the patch for the software on 7 April 2014.
In effect, OpenSSL had provided a publicly available audit trail for review.
Similarly, The Sydney Mornig Herald (SMH) provided an audit trail of its own:
- All times are in US Pacific Daylight Time
- Friday, March 21 or before - Neel Mehta of Google Security discovers Heartbleed vulnerability.
- Friday, March 21 10.23 - Bodo Moeller and Adam Langley of Google commit a patch for the flaw (This is according to the timestamp on the patch file Google created and later sent to OpenSSL, which OpenSSL forwarded to Red Hat and others). The patch is then progressively applied to Google services/servers across the globe.
- Monday, March 31 or before - Someone tells content distribution network CloudFlare about Heartbleed and they patch against it. CloudFlare later boasts on its blog about how they were able to protect their clients before many others. CloudFlare chief executive officer Matthew Prince would not tell Fairfax how his company found out about the flaw early. "I think the most accurate reporting of events with regard to the disclosure process, to the extent I know them, was written by Danny over at the [Wall Street Journal]," he says. The article says CloudFlare was notified of the bug the week before last and made the recommended fix "after signing a non-disclosure agreement".
- Tuesday, April 1 - Google Security notifies "OpenSSL team members" about the flaw it has found in OpenSSL, which later becomes known as "Heartbleed", Mark Cox at OpenSSL says on social network Google Plus.
- Tuesday, April 1 04:09 - "OpenSSL team members" forward Google's email to OpenSSL's "core team members". Cox at OpenSSL says the following on Google Plus: "Original plan was to push [a fix] that week, but it was postponed until April 9 to give time for proper processes." Google tells OpenSSL, according to Cox, that they had "notified some infrastructure providers under embargo". Cox says OpenSSL does not have the names of providers Google told or the dates they were told. Google declined to tell Fairfax which partners it had told. "We aren't commenting on when or who was given a heads up," a Google spokesman said.
- Wednesday, April 2 ~23:30 - Finnish IT security testing firm Codenomicon separately discovers the same bug that Neel Mehta of Google found in OpenSSL. A source inside the company gives Fairfax the time it was found as 09:30 EEST April 3, which converts to 23:30 PDT, April 2.
- Thursday, April 3 04:30 - Codenomicon notifies the National Cyber Security Centre Finland (NCSC-FI) about its discovery of the OpenSSL bug. Codenomicon tells Fairfax in a statement that they're not willing to say whether they disclosed the bug to others. "We have strict [non-disclosure agreements] which do not allow us to discuss any customer engagements. Therefore, we do not want to weigh in on the disclosure debate," a company spokeswoman says. A source inside the company later tells Fairfax: "Our customers were not notified. They first learned about it after OpenSSL went public with the information."
- Friday, April 4 - Content distribution network Akamai patches its servers. They initially say OpenSSL told them about bug but the OpenSSL core team denies this in an email interview with Fairfax. Akamai updates its blog after the denial - prompted by Fairfax - and Akamai's blog now says an individual in the OpenSSL community told them. Akamai's chief security officer, Andy Ellis, tells Fairfax: "We've amended the blog to specific [sic] a member of the community; but we aren't going to disclose our source." It's well known a number of OpenSSL community members work for companies in the tech sector that could be connected to Akamai.
- Friday, April 4 - Rumours begin to swirl in open source community about a bug existing in OpenSSL, according to one security person at a Linux distribution Fairfax spoke to. No details were apparent so it was ignored by most.
- Saturday, April 5 15:13 - Codenomicon purchases the Heartbleed.com domain name, where it later publishes information about the security flaw.
- Saturday, April 5 16:51 - OpenSSL (not public at this point) publishes this (since taken offline) to its Git repository.
- Sunday, April 6 02:30 - The National Cyber Security Centre Finland asks the CERT Coordination Centre (CERT/CC) in America to be allocated a common vulnerabilites exposure (CVE) number "on a critical OpenSSL issue" without disclosing what exactly the bug is. CERT/CC is located at the Software Engineering Institute, a US government funded research centre operated by Carnegie Mellon University. The centre was created in in 1988 at DARPA's direction in response to the Morris worm incident.
- Sunday, April 6 ~22:56 - Mark Cox of OpenSSL (who also works for Red Hat and was on holiday) notifies Linux distribution Red Hat about the Heartbleed bug and authorises them to share details of the vulnerability on behalf of OpenSSL to other Linux operating system distributions.
- Sunday, April 6 22.56 - Huzaifa Sidhpurwala (who works for Red Hat) adds a (then private) bug to Red Hat's bugzilla.
- Sunday, April 6 23.10 - Huzaifa Sidhpurwala sends an email about the bug to a private Linux distribution mailing list with no details about Heartbleed but an offer to request them privately under embargo. Sidhpurwala says in the email that the issue would be made public on April 9. Cox of OpenSSL says on Google Plus: "No details of the issue are given: just affected versions [of OpenSSL]. Vendors are told to contact Red Hat for the full advisory under embargo."
- Sunday, April 6 ~23:10 - A number of people on the private mailing list ask Sidhpurwala, who lives in India, for details about the bug. Sidhpurwala gives details of the issue, advisory, and patch to the operating system vendors that replied under embargo. Those who got a response included SuSE (Monday, April 7 at 01:15), Debian (01:16), FreeBSD (01:49) and AltLinux (03:00). “Some other [operating system] vendors replied but [Red Hat] did not give details in time before the issue was public," Cox said. Sidhpurwala was asleep during the time the other operating system vendors requested details. "Some of them mailed during my night time. I saw these emails the next day, and it was pointless to answer them at that time, since the issue was already public," Sidhpurwala says. Those who attempted to ask and were left without a response included Ubuntu (asked at 04:30), Gentoo (07:14) and Chromium (09:15), says Cox.
- Prior to Monday, April 7 or early April 7 - Facebook gets a heads up, people familiar with matter tell the Wall Street Journal. Facebook say after the disclosure: "We added protections for Facebook’s implementation of OpenSSL before this issue was publicly disclosed, and we're continuing to monitor the situation closely."
- Monday, April 7 08.19 - The National Cyber Security Centre Finland reports Codenomicon's OpenSSL "Heartbleed" bug to OpenSSL core team members Ben Laurie (who works for Google) and Mark Cox (Red Hat) via encrypted email.
- Monday, April 7 09.11 - The encrypted email is forwarded to the OpenSSL core team members, who then decide, according to Cox, that "the coincidence of the two finds of the same issue at the same time increases the risk while this issue remained unpatched. OpenSSL therefore released updated packages [later] that day."
- Monday, April 7 09:53 - A fix for the OpenSSL Heartbleed bug is committed to OpenSSL's Git repository (at this point private). Confirmed by Red Hat employee: "At this point it was private."
- Monday, April 7 10:21:29 - A new OpenSSL version is uploaded to OpenSSL's web server with the filename "openssl-1.0.1g.tgz".
- Monday, April 7 10:27 - OpenSSL publishes a Heatbleed security advisory on its website (website metadata shows time as 10:27 PDT).
- Monday, April 7 10:49 - OpenSSL issues a Heartbleed advisory via its mailing list. It takes time to get around.
- Monday, April 7 11:00 - CloudFlare posts a blog entry about the bug.
- Monday, April 7 12:23 - CloudFlare tweets about its blog post.
- Monday, April 7 12:37 - Google's Neel Mehta comes out of Twitter hiding to tweet about the OpenSSL flaw.
- Monday, April 7 13:13 - Codenomicon tweets they found bug too and link to their Heartbleed.com website.
- Monday, April 7 ~13:13 - Most of the world finds out about the issue through heartbleed.com.
- Monday, April 7 15:01 - Ubuntu comes out with patch.
- Monday, April 7 23.45 - The National Cyber Security Centre Finland issues a security advisory on its website in Finnish.
- Monday, April 8 ~00:45 - The National Cyber Security Centre Finland issues a security advisory on its website in English.
- Tuesday, April 9 - A Red Hat technical administrator for cloud security, Kurt Seifried, says in a public mailing list that Red Hat and OpenSSL tried to coordinate disclosure. But Seifried says things "blew up" when Codenomicon reported the bug too. "My understanding is that OpenSSL made this public due to additional reports. I suspect it boiled down to 'Group A found this flaw, reported it, and has a reproducer, and now Group B found the same thing independently and also has a reproducer. Chances are the bad guys do as well so better to let everyone know the barn door is open now rather than wait 2 more days'. But there may be other factors I'm not aware [of],” Seifried says.
- Wednesday, April 9 - A Debian developer, Yves-Alexis Perez, says on the same mailing list: "I think we would have managed to handle it properly if the embargo didn't break."
- Wednesday, April 9 - Facebook and Microsoft donate $US15,000 to Neel Mehta via the Internet Bug Bounty program for finding the OpenSSL bug. Mehta gives the funds to the Freedom of the Press Foundation.
There is little doubt that the above timeline demonstrates corporate America at work, or more accurately, how corporate America does not work.
The SMH continued by acknowledging three unanswered questions:
- Who knew of heatbleed prior to release? (sic)
- Who knew hours before public release?
- Who didn't know until public release?
While the latter two questions are of use in quantifying the ramifications of the timing of the release, the former is much more significant. In a short, but revealing Bloomberg report followed by a longer one at the Washington Post and yet another in the Guardian, the spectre of the NSA, and by default the other four members of ECHELON , abusing this security flaw for nefarious purposes became apparent. The New York Times followed up, these reports quoting officials of the administration who said that Obama had carved an exception for a clear national security or law enforcement need'. Thus, if there is a flaw in either the hardware or software of equipment connected, or not connected, to the Internet we will exploit it under the umbrella of national security and law enforcement, or until we get caught.
Enter Neel Mehta of Google Security who is said to have discovered the vulnerability on, or before the 21 March closely followed by a Finnish company, Codenomicon, on the 2 April.
Had the NSA been caught in exploiting a data leak, or was this a carefully planned release of a known vulnerability by the agency to avoid further censure by the global internet community?
Interestingly, the MSM gave us some clues on this point: they particularly emphasised the independent discovery of the bug by the Finnish company but failed to acknowledge the statistical chances of such an event as being infinitesimal. They also failed to inform the reader that the CEO of Codenomicon, David Chartier, is an American, that the company employs a little over one hundred personnel (unlike the NSA who have over a thousand staff working in this field alone), it has offices in Saratoga, California (US), Singapore and Hong Kong (China) and the US representative, Steve Hayes, is well connected in Silicon Valley.
In other words, we have been exploiting this vulnerability for years but we had best arrange for an orderly fix to the bug in case we are outed by the Snowden documents and while there is a good chance that we can claim plausible deniability. You guys over there at Google need to "find" this bug, flag it up the chain of responsibility but don't get too involved yourself. You must use one of our state sponsored affiliates, preferably outwith ECHELON, to take the lead and make sure there is absolutely no chance of fallout landing on either us or corporate America.
Zelda says: You may remove the leeches now.
- Evil brings men together.