Is the FBI trying to kill the use of "cloud" email services like Google's Gmail or Microsoft's Office 365 by local law enforcement agencies? That's what the City of Los Angeles says. This article is a follow-up to my earlier Seeking Alpha piece on Google’s ambitions in enterprise software.
Google (GOOG) and the City of Los Angeles have finally agreed to pull the plug on their two year struggle to deploy Google Apps for Government (GAFG) at the Los Angeles Police Department. In a document published last week, city officials blamed the failure on FBI information security rules which, they allege, make it impossible to deploy cloud email services like Gmail to law enforcement agencies. If this assessment is correct, local police and sheriff departments across the country who are considering Google Apps or similar cloud services may be in for an unpleasant surprise. The FBI has not yet responded to my inquiry on this matter. In the meantime, 13,000 LAPD employees will be sticking with their existing GroupWise email system, while 17,000 other city employees will stay on Gmail. The agreement – expected to be signed by Google, its reseller CSC and the city – marks a somewhat ignominious closing chapter to the Mountain View search engine’s most highly touted public sector deployment of Gmail.
The deal is all the more remarkable for its financial aspect. Google will not only reimburse Los Angeles for any charges incurred so far by LAPD users and lower the price it charges for LA’s other users, but will actually pay the city up to $350,000 per year for the life of the contract to cover the cost of maintaining GroupWise at LAPD. This works out to a de facto $20 per year discount on the already rock bottom price of approximately $40 per user per year that Google is charging the city’s other departments. If the city renews the contract through 2014 as expected, Google’s total outlay could reach $1.4 million. Google’s implementation partner CSC will also take a haircut to the tune of $250,000 on its upfront integration fee of $830,000.
Although these large financial payouts indicate that Google and CSC have accepted responsibility for the failure, the city’s official posture – perhaps a face-saving gesture for the city’s IT staff and its vendors – is that the fault lies with the FBI computer security rules governing access to the national Criminal Justice Information System database (CJIS). The Bureau imposes strict regulations on all local law enforcement computer systems and personnel who have access to criminal history information contained in CJIS. The regulations cover both direct access to CJIS and secondary dissemination of CJIS-derived information, such as routine email messages that police departments circulate internally. The regulations also apply to outside IT contractors who provide services to law enforcement agencies, and this appears to be the source of the trouble in Los Angeles.
Neither Google nor the city has disclosed exactly which CJIS requirements GAFG failed to meet. The FBI demands 128-bit or better encryption of CJIS-derived information. So-called “at rest” (i.e. storage-based) encryption is not a publicly announced feature of GAFG, but LA’s Chief Technical Officer Randi Levin has indicated that Google meets this requirement. Interestingly enough, Office 365, the cloud email service of Google’s rival Microsoft, does not currently offer at rest encryption, and would therefore also fail to meet CJIS requirements. However, LAPD’s existing on premises email server, Novell GroupWise, does meet the FBI’s encryption standard, as do comparable systems such as IBM Lotus Notes and Microsoft Exchange.
Google’s problem with CJIS may concern the FBI requirement that IT contractor personnel pass criminal background checks and sign a document known as the FBI Security Addendum. The city says that Google has confirmed it is unable to meet this requirement, but does not say why. However, analyst firm Gartner reported in July that some of Google’s support staff with access to GAFG servers are based in Europe. The FBI doesn’t explicitly mandate that support personnel be located in the U.S., but European law may make it difficult for Google to force its European employees to submit to screening (including fingerprinting) by U.S. authorities.
The issue of exactly what constitutes “access” to servers containing CJIS information is complex. Partly in response to early concerns over CJIS rules voiced by LA, Google in 2010 launched Google Apps for Government as a specialized version of Google Apps. Unlike the standard version, GAFG’s servers are located exclusively in the U.S. and are physically separate from servers used for non-government customers. However, according to an LAPD report, GAFG servers are accessible by other Google server administration employees. This apparent loophole in Google’s internal security policy may be the source of its difficulty in complying with the FBI’s requirements, although this has not yet been confirmed by Google or the FBI.
It's rather shameless of Los Angeles and Google to blame their troubles on the FBI by claiming that, in the words of city officials, “CJIS regulations are currently incompatible with cloud computing”. Google and CSC did tell the city last year that they couldn’t meet CJIS requirements because the FBI had changed them after the LA contract was signed. But the version of the FBI’s CJIS policy document in force at the time of the contract clearly stated that IT contractor staffers must sign the Security Addendum and submit to background checks.
It is true that the current Google and Microsoft (MSFT) cloud email products fail to meet CJIS requirements. But this is because the vendors have chosen not to comply for reasons of their own, not because CJIS is inherently incompatible with cloud computing. The truth is that the current generation of cloud email products are still immature compared to the established server products. The vendors can fix this problem if they want to. But adding such high-end features as CJIS compliance is an expensive option that fits the traditional enterprise software business model better than Google's ad-supported consumer model.
The bottom line is that Los Angeles and its vendors, in their rush to get the deal done, failed to perform adequate due diligence. At the time of the contract, LA officials didn’t realize that they would be the first big city to deploy Google apps, because they mistakenly believed that Washington DC had already successfully completed a large deployment. LA CTO Levin later admitted that she had "misinterpreted the extent" of the Washington implementation. Google’s sales team, who surely were in a position to know the truth, can be faulted for allowing this “misinterpretation” to occur. But at the end of the day, it’s the job of IT leadership to evaluate the credibility of vendor claims.
What happens next? At this point the answer is probably “not much”. Google and CSC are offering Los Angeles a substantial financial compensation for the trouble and cost overruns caused by the failed deployment at LAPD. Los Angeles CTO Levin has stated that the city is satisfied with Gmail’s performance in the city’s other departments. LAPD has said it will stay with GroupWise for the time being. The original motivation for the 2009 decision to go with Google Apps was the antiquated state of the GroupWise system then in use. Gmail offered much greater storage capacity per user and, as a cloud-based service with an aggressively low price, it was expected to be much cheaper. Going forward, LAPD has the option of upgrading GroupWise, switching to a competing on premises or hosted email server such as Lotus Notes or Exchange, or – just possibly – waiting until one of the cloud email providers releases a service that complies with the FBI’s CJIS requirements.
Google’s stumble in Los Angeles is not likely to be a fatal setback to the company’s ambitious plans to push its consumer email and collaboration products into the enterprise. But the failure is an expensive lesson for the company, as well as for its integration partner CSC and for the City of Los Angeles. Enterprise IT professionals know all too well that large IT projects can be a source of unexpected cost and risk. Perhaps Google will learn from this experience that serving the complex needs of the enterprise IT customer is not the same as serving millions of eyeballs to advertisers with free consumer web offerings.
Note: Official City of Los Angeles documents referred to in this article can be found on the LA City Clerk web site (Contract number C-116359).
Disclosure: I have no positions in any stocks mentioned, and no plans to initiate any positions within the next 72 hours.