Summary
Is Google (GOOG), the world’s leading online advertising firm, really committed to becoming a major player in enterprise software? The answer is unclear. In this article we look at the reasons for the stalled deployment of Gmail at one of Google’s largest public sector accounts to date, the city of Los Angeles. We conclude that while Gmail is a first-rate consumer email product, Google does not presently have the right products to be a significant player in enterprise software.
Google Apps in Los Angeles: How an ambitious project went wrong
Probably no other government cloud email deployment has received more attention than the city of Los Angeles’ ongoing attempt to migrate some 30,000 users of an antiquated on-premises email system (based on Novell GroupWise) to Gmail. What makes this case exceptionally interesting is the rich paper trail of official and leaked documents that detail almost every step of the project’s progress – or lack thereof.
When Los Angeles CTO Randi Levin pitched Google’s three-year $7.2 million proposal to the City Council back in 2009, it was greeted with enthusiasm. Going from GroupWise to Google Apps would allow the city to repurpose 92 of the city’s servers and cancel hundreds of thousands of dollars worth of GroupWise licenses. Longer term, the city also hoped to cut back drastically on its Microsoft Office licenses. The move would also save nearly $150,000 per year in power and cooling costs, and would free up the 16 full-time positions devoted to supporting the old system. While the shift from the GroupWise Windows client software to Gmail’s web interface did entail some loss of functionality for the city’s email users, the latter would benefit from a huge gain in email storage space, which would soar from one to 25 gigabytes per user. All this would be delivered for less than $50 per user per year, not counting one-time implementation, training and migration costs.
The city’s original plan was to move all users from GroupWise to Gmail by December 31, 2009. But this proved to be overly optimistic. Today, nearly two years later, 13,000 of the city’s 30,000 users are still on GroupWise. The plan also called for 80% of the city’s users to migrate from Microsoft Office to Google Docs for basic tasks such as word processing, presentations and spreadsheets. However, this migration, which was to have taken place over two years, now appears to be on indefinite hold, and there is real doubt as to whether the basic email migration itself will ever be completed. Both the rapid deployment of Gmail (allowing the early termination of the city’s GroupWise licenses) and the gradual shift away from Microsoft Office were key elements in the cost savings the city expected to achieve from the project. Without them, much of the city’s business case for the migration evaporates.
The main stumbling block for Gmail in Los Angeles has been security. On the eve of the scheduled migration of its 13,000 employees in the spring of 2010, the Los Angeles Police Department announced that a long list of promised improvements to Gmail’s security had not been completed and that as a consequence the department could not move forward with the migration. This decision was the first in a series of increasingly serious setbacks that have turned the LA deployment from a shining example of public sector cloud computing (one that is still touted on Google’s web site) into an embarrassing debacle.
What were the LAPD requirements that Google couldn’t meet? Some involved missing features such as email receipts (which have since been added) or the integration of Gmail with other IT systems. But the most important requirement concerned the security of the back-end servers used to implement Gmail in Google’s data centers. During initial negotiations, the city told Google it would require these servers to be physically separated from servers used by non-government customers, and in 2010 Google duly rolled out its Google Apps for Government edition, which did indeed isolate all servers used to deliver Gmail and Calendar services to government customers in the United States. However, Google did not isolate the servers used for its Postini eDiscovery service. But for LAPD, eDiscovery (used for the department’s mandatory email retention policies) was indispensable. Without the assurance that these servers would also be physically segregated, the department put its migration on hold.
But there was more. LAPD, like all police departments, is an intensive user of State and Federal criminal records databases. Access to these systems is tightly controlled by Department of Justice protocols supervised by the FBI. In addition to restrictions on system access and extensive training and audit requirements, employees of government contractors involved in the administration of IT systems through which such sensitive data transits must pass stringent criminal background checks (including the submission of fingerprints). The prime systems integrator on the contract, CSC, assured the city that meeting these requirements would not be a problem. But in October 2010, just a week before the promised deadline, Google unexpectedly told CSC it would not be able to meet the requirements after all. Just what the problem was has not been publicly disclosed. But according to Gartner, some of Google’s support personnel with access to the Apps for Government servers are based in Europe. Gartner also reports that Google hopes to meet the FBI’s requirements by the first quarter of 2012 – more than two years after the contract was first signed. But it now appears increasingly unlikely that LAPD will agree to restart its long delayed migration to Gmail.
“These failures are wholly unacceptable…”
Reading the documents published on the city’s web site or leaked to the press, we witness a progressive breakdown in communication between the city and its vendors. Consider the following:
- In July 2009, the City Administrative Officer, in a report evaluating the proposed contract with Google, optimistically wrote that: “The shift in control over the City's e-mail and office applications is mitigated by service level agreements guaranteeing a high level of service from Google.”
- By the following spring, however, signs of trouble began to appear. In April 2010 the City Administrative Officer informed the City Council that: “[M]any of the departments [have] expressed concerns about both the performance and the functionality of the new system.”
- Ominously, in July 2010, LAPD informed the City Administrative Officer that: “At this time, a definitive timeline for LAPD migration cannot be provided.”
- Then, in November 2010, the delays erupted into a full-fledged crisis of confidence between the city and CSC/Google, leading CTO Levin to dispatch a formal Notice of Deficiencies to the contractor that bluntly expressed the city’s disappointment: “These failures are wholly unacceptable to the City of Los Angeles. CSC and Google have repeatedly committed to meet particular deliverables on specific dates, only to reveal, at the last minute, that the set deliverables/dates will not be met. CSC and Google’s behavior goes beyond a mere failure to communicate in a timely manner, and instead, on several occasions, has risen to the level of misrepresentation.”
- Finally, in August 2011, Levin sent a proposed contract amendment to CSC stipulating that “without completion and compliance with all City security requirements, the LAPD and other law enforcement employees will be unable to migrate to Google Apps”. The CTO also demanded that Google pay the costs of keeping GroupWise in service for more than a year beyond its planned retirement.
There is something about the LA Gmail project that is depressingly familiar to students of large IT projects, especially those involving complex enterprise software implementations such as ERP. It is a shockingly frequent occurrence for such projects to experience cost overruns, functionality shortfalls and even catastrophic “pull-the-plug-and-walk-away” failures. Something about the nature of these huge projects apparently creates skewed incentives for the teams that manage them, causing the leaders to systematically overpromise and underdeliver. The Los Angeles CTO in particular can be faulted for her failure to obtain buy-in from LAPD – the city agency with the largest number of users and the most sensitive security requirements – before steamrolling the project through the City Council.
Even in the absence of litigation or an embarrassing cancellation of the contract, the Los Angeles Gmail contract has turned into a money-losing disaster for Google. The city’s demand that Google pay the cost of keeping GroupWise in service beyond its expected retirement will likely cost the firm several million dollars, and will certainly wipe out any profit Google may have anticipated from the contract.
Consumer vs. Enterprise
Why did Google fall short in Los Angeles? There is no doubt that Gmail is an excellent product. Since its launch in 2004, it has consistently been the most innovative and talked-about of the free consumer email services. In the view of many analysts (including this author), Gmail’s polished interface is much superior to Microsoft’s competing Office 365 email service.
But Gmail is not an enterprise product. It was conceived as a no-frills consumer product. It was designed to be extremely cheap to operate, so that it could be given away free to millions of consumers, whose numerous eyeballs would drive a torrent of keyword-triggered advertising revenue into Google’s coffers. It was never intended as a product that could meet the endlessly expanding feature requirements and even more daunting security needs of large enterprises or government agencies.
In truth, the most important selling point of Google Apps for large customers is simply its unbeatably low price – just $50 per user per year (even less with volume discounts). This is why cash-strapped government agencies have been eager to embrace these apps.
Enterprise email products like Microsoft Exchange and IBM Lotus Notes offer a vast and sometimes bewildering array of features and options, some delivered by the vendors themselves, but many provided by third party partners. For example, numerous products allow Exchange and Notes customers to exercise extremely fine-grained control over who can do what with the information contained in corporate email. Add-ons exist that prevent unauthorized users from forwarding, printing, transferring to a USB key or even opening sensitive email attachments without proper credentials. Similar products are beginning to appear for Gmail, but to date they are less numerous and more rudimentary.
Even more importantly, Exchange and Notes do not stand on their own within their parent organizations. Rather, they are part of vast, multi-billion dollar enterprise software businesses. Enterprise software is a portfolio business because this is the only way to amortize the extremely high costs of developing and selling such complex products to the world’s very diverse population of large and mid-sized organizations. In enterprise software, the name of the game is to get big or get acquired. While Google has been assiduously building its portfolio of apps, a business unit that only generates a few hundred million dollars per year in revenue and probably no profit doesn’t have the scale to thrive alongside the giants.
Finally, there is the issue of Google’s insistence that all cloud applications must be delivered solely from its own data centers. The economic logic behind the rise of cloud computing is compelling: it’s more efficient to concentrate expensive IT resources in a few big data centers managed by specialists and shared by many users, rather than distribute them over lots of redundant smaller facilities run by the users themselves. But not every case fits this centralized template. Some user needs are so specialized or so sensitive – for example, LAPD’s security requirements – that the added cost of a private cloud is justified. The leading enterprise cloud players – IBM, Microsoft, Oracle, VMware – all propose a mix of private and shared cloud solutions. All of them, that is, except Google, whose rejection of hybrid solutions appears motivated more by ideology than by a pragmatic desire to solve customer problems.
The bottom line
Despite the negatives, Google’s disruptive entry into the market has been a very good thing for enterprise email, because it has educated users about the benefits of cloud computing and is forcing the established leaders to hustle. It has prodded market leader Microsoft to respond with Office 365 and a multi-tenant cloud version of Exchange. It has even awakened slumbering giant IBM, which in recent years appeared to lose interest in an email market it once dominated, but has now launched a cloud email service of its own (LotusLive Notes).
Users of all sizes benefit by having Google hold the leaders’ feet to the fire. But enterprise customers with the most demanding needs might want to take a careful look at the Los Angeles case before they sign on for Google’s cloud. And investors should not expect this particular product to make any measurable contribution to Google’s bottom line.
Disclosure: I have no positions in any stocks mentioned, and no plans to initiate any positions within the next 72 hours.
