The Time Has Come to Regulate Search Engine Marketing and SEO

TechCrunch editor's note: The following post was written by a well known executive at one of the largest sites on the Internet. The author has requested to remain anonymous - not for dramatic effect, but because of the backlash he would receive from the SEO industry and possibly Google itself. He also doesn’t want his company associated with the post.

He is starting a discussion on the need for government regulation of the organic and paid search policies of Google, which maintains a commanding lead in search market share today. Or at least transparency in how search results are determined. There is clearly growing frustration on the constantly changing “border policies” that are created and enforced by Google and other search engines. It is a fascinating read.

Imagine, if you will, that the entire Internet is contained within a single continent. That continent is filled with countries, states and cities. Each jurisdiction is autonomous, relying on visitors to cross on to their turf to engage in commerce. Now, imagine if the only way to get into this continent involved just two methods: SEO and SEM. Let’s further imagine that the borders to this continent were controlled by a single company. Let’s also hold that the rules for search engine optimization listings and search engine marketing were not only defined but were completely controlled at the whim of this single company. Of course, we all realize that word-of-mouth marketing and viral marketing also contribute to traffic to individual websites. That said, the primary methodology for all users to reach any individual website destination is still search, of either paid or organic listings.

Or suppose the paradigm is the streets of Los Angeles. Let’s imagine that in order to enter the city you had to pass through a single gate. And once you entered that gate, the streets you were or were not allowed to go down — and thus the businesses you were or were not allowed to visit — could be randomly blocked from your access. Blocked to a point where you might not even know they exist; whatever streets were available for you to traverse were in essence the only streets you knew where business could be transacted.

Whatever the scenario, it’s unsettlingly close to the situation that prevails today in search. It’s now conventional wisdom that search engine optimization, representing the organic result sets on any search query, is more voodoo than science. Through an uncontrolled set of factors search engines determine which listings appear at the top and bottom of any individual query. In addition, consumer behavior dictates the top three results on any search page are all that matter. If you happen to own an online business, unless you exist within those top three, the amount of individual traffic you will obtain from organic listings is very, very low. As the proprietor of that business you may hire search engine optimization companies to assist in increasing your rankings on organic results, with or without success. And at any one time, the controller of these borders (that is, the search engine itself) can change and manipulate those rules – and that can substantially decrease or destroy all organic traffic coming to your website, without notice and without your knowledge.

Search engine marketing now faces a similar challenge. Although anyone can open an account to buy paid search listings, the rules on each account are arbitrary. Accounts can be shut down at any time, without notice to the website owner. In other words, if you haven’t successfully obtained enough traffic to your site from organic listings and you decide to rely on paid search, you still face a situation where regardless of how much you bid per click you may or may not show up at the top of the paid results. That’s because paid results that are displayed on any query are not only determined based on the price the buyer is willing to pay. Unlike other auctions that are completely priced-based, these results are determined and sequenced not only on price but also on quality of advertising and click-through volume. For example, if company A was willing to pay $1 per click on a certain term and company B was willing to pay 10 cents but company B’s ad generated ten times as many clicks as company A’s, the yield to the search engine would be identical between the two.

The second factor is that the search engine can, at any time, determine that either company A or company B may or may not buy traffic within its index. And without notifying the company and with no path toward recourse and statement, the search engine can disable the paid search account from either business. Returning to the continent metaphor, this ends up looking quite a bit like free trade. Various businesses (call them sellers), operating within this continent, wish to conduct business with the rest of the world (that is, the population of buyers). The border — which in this case is the search engine — thus has complete control of who can transact and how often. And at its discretion, the search engine can decide to increase, decrease or completely disable access between buyers and sellers. Because search is the dominant methodology for consumers to find what they are looking for, whether a product or a service, the unilateral control that search engines wield enables them to control billions upon billions of dollars of consumer spend every year. It also gives them the ability to completely determine which companies become more successful — or less so.

The situation we face today is unique. Due to Google’s dominance — and the fact that it controls such an enormous amount of consumer behavior through paid and organic search listings – the company in essence governs commerce on the web. And any company that falls out of favor with Google, whether for reasons of bad practice or simple disagreement, can find itself at risk of going out of business.

This system also benefits the few in a host of other ways. Because the rules of organic and paid search change frequently – and remain undefined — agencies and other traffic brokers can win big; through their experience, they’re capable of reverse-engineering these rules. This means that, as this market matures, individual businesses have a diminishing chance to actually compete and gain search market share. That, in turn, puts them in a position of not only needing to hire an agency in order to find any traffic, but also making it more expensive overall to build businesses on the web.

I’ve worked with many businesses who feel they are playing in Google’s world — behaviors from product decisions to marketing strategies rely completely on appeasing these undocumented and often mystical Google desires. I’ve seen companies choose to not work with Google’s competitors for fear that by building those relationships, they’re damaging the ability to be indexed properly on Google and are anxious that result sets will be compromised. Many likewise believe that by having a monetization relationship through Google, they will somehow achieve higher quality listings through organic search. I’ve also witnessed companies who, in addition to using Google for monetization, have preferred relationships with purchasing traffic through Google Adwords. By supporting this dual relationship, they appear to want to live by two sets of rules – those that exist within the Adwords marketplace and those that apply to the Adsense product. And because they’re walking on both sides of the (Google) street, they feel they have a strategic advantage — as though the Adwords product will enable them to acquire traffic at both a lower cost and with a looser rule set than their competitors.

Here’s where the parallel to free trade breaks down. There are no perfect paradigms looking at free trade and import/export laws that exactly define or address this challenge. Neither would a secret relationship between the government and the search engines solve the problem. The only real solution is disclosure. Transparency. Those traffic generators that use rule-based algorithms to determine result sets must publicly disclose their methodologies. That is the means by which all businesses can compete freely in the organic and paid search marketplaces. If we lived in a world where Google didn’t hold sway over such a significant portion of consumer behavior, this kind of regulation wouldn’t be necessary. The market would be self-correcting, and we could trust the individual decisions of a healthy and competitive search industry. Regrettably, due to search dominance, the industry can’t be left to its own devices.

Original post

Comments

[A Barrel Full:]

One of the things that worries me about google is the fact that they believe themselves to be a force for good. That makes them more dangerous than those of us who know we are flawed.

I am not sure whether we could devise a decent regulatory system, but it is worrying that one company holds so much power.

Jul 13 09:29 AM

[wavelength:]

Almost impossible to know what to say. I could not disagree more - either with the demonization of Google, or with the notion that the Internet become another site of big Govt regulation and quasi-nationalization. I operate a small website where the majority of my webpages rank on Google's first page. I do not pay for favors from Google and I do not hire expensive SEO analysts. I follow basic SEO optimization techniques that are widely publicized and freely available, and I write GREAT CONTENT - it is this original and quality content content that Google rewards. The writer may resent that a micro business can exploit Google's bias towards quality content in order to compete so effectively with larger players, but I think this is the essence of the free market's promise. At a time when so many economic and personal freedoms are under assult it is remarkable that the writer has time to sulk about search engines. Google are self-interested, and at times may be a little heavy handed or smug. They are hardly the evil empire though. The analogy with physical space is deeply flawed, but congratulations on such a provocative article.

Jul 13 09:43 AM

[Jeff Borack:]

This article only considers the website managers point of view. Google is popular because users (the people searching for information) believe the search results or accurate or useful. Creating SEO rules or expecting google to show the public how they rank results would certainly allow websites to game the system and attract more users at least for a while. But will people searching for information benefit?

Using your "streets of LA" example, imagine you've never been to LA before and you want a good cup of coffee. Google is the guy you ask for directions, and he gives you a list of the top 10 best coffee places using an extremely sophisticated algorithm (and google maps it for you too). If google gave bad advice, you could always ask someone else. Google has competition despite popular belief. If google was legally forced to follow a specific algorithm or was forced to explain his algorithm to the public, do you think that would make his recommendations better or worse? The coffee houses might benefit, but not the drinkers.

Jul 13 09:49 AM

[Adam Sharp:]

Wow Techcrunch. Who is this mystery man? CEO of Family Dollar? He obviously knows nothing about technology:

"Those traffic generators that use rule-based algorithms to determine result sets must publicly disclose their methodologies. That is the means by which all businesses can compete freely in the organic and paid search marketplaces."

Is that all he wants? For Google to disclose how their top-secret, industry-leading algorithm works? Sure... they'll comply with that.

Jul 13 10:10 AM

[Stock Lobster:]

Those who defend Google's monopoly do so because they have not been affected yet. There are too many reports amongst webmasters of Google using it's position to sandbox sites whose political content they disagree with, for me to feel entirely comfortable with them lately. The author of this post is correct imho, and it is a shame he feels he needs to hide his name, lest he experience a backlash from Google. That alone should be a tip off that Google has grown too big. Competition is healthy, and I for one am starting to use Bing now, just to help support diversity.

Jul 13 10:14 AM

[Economyst:]

Have you ever noticed that the products that are marketed the most heavily are not necessarily the best. The marketing costs are added to the cost of your good or service.

Why can't businesses spend more on customer service and building quality products and grow a business organically instead of reading something from a marketing shill.

I agree with wavelength - concentrate on quality and you will get dividends it's just that it takes time which in today's world is seen in a negative light. Those companies who are built on massive marketing once they try to cut their marketing spend realise that their sales fall as they just have not concentrated in the right areas.

Nothing speaks more than word of mouth recommendations from someone you know. If you want anonymous customers trawling the net for the cheapest offer then you won't build a sustainable business from that.

Jul 13 10:36 AM

[Snitzer:]

Search engines are a net good, although there seems to be more ads returned than legitimate hits. Example: Search for "How to screw in a left handed widget" and you'll be forced to wade through a slew of commercial returns exhorting you to buy Left Handed Widgets before you get to any useable knowledge of Widgets.

Jul 13 10:38 AM

[mineralt:]

What planet is this author from? Is it a planet where everything is 100% fair? What a crybaby. If you don't like Google spend more money advertising your URLs on mainstream media, so that users type them in from memory. I'm sure the MSM will appreciate the patronage.

By the way, it's hardly voodoo.

Jul 13 11:15 AM

[Alan Young:]

Although the article is interesting in that it informed me about something I didn't know, this is a stupid idea.

I have no problem with government regulation of markets that can really hurt consumers, including financial markets. But --search engines?? First, there's no monopoly; Metacrawler checks 5 different SEs (used to be about 15, but that's the way everything is going these days). Second, I haven't heard anything about results actually being excluded; if the user is too lazy to look beyond the top 3 results, that's the user's fault.

And third, if Google's business model is so flawed, the author should invest in a better business model and go compete with them. You want the government to do *what??*

Jul 13 11:56 AM

[Wizler:]

ASking to give trade secrets rather than innovate a better solution. No wonder the author wants to be anonymous

Jul 13 12:36 PM

[aldousd:]

So somehow it's the governments job to monitor and 'level the playing field' in light of the fact that too many people like to type in www.google.com instead of www.bing.com in their browsers to search. Wow. Now I've seen everything. So basically, a company can't charge what it will fetch, cause that's not fair? But, instead, anyone, who fancies themselves a player in a particular market, has somehow now a right to come in without having to find a real reason to be better.. they just get in the door since... well they want it! Fantastic. Next thing you know they'll start giving out free money.

Jul 13 01:08 PM

[just.a.guy:]

If a company's business is entirely dependent upon being found via online (Google) searches, perhaps that company should invest in alternate means of generating awareness. There are many media choices out there.

If, however, such a business cannot cost effectively and profitably generate awareness and sales through other means, then its owner/operator will learn a valuable and possibly expensive lesson about building a business that depends upon the kindness of strangers.

Jul 13 05:12 PM

[holden:]

I am not in favor of Google. In fact, their crawler is not allowed to index my website, both by using robots.txt and blocking google's IP address range via .htaccess.

Having said that, I see no reason why Google's policies and practices should be any of the gov's business. Businesses which do not want to spend money on advertising in newspapers and billboards should not complain that Google is not listing them high enough. Businesses with boring or uninformative websites should not complain either. Most of the traffic on my website comes from customer referrals who recommend my services to their friends. Providing good services means more referrals, and more clients.

The day Google starts screwing with the results to make nice with a business or political organization will be the start of their decline. It will not happen overnight, but the users will migrate to another engine which provides better results. If you do not believe me, recall that in the 90s the undisputed leader among search engines was altavista. Google offered a better product and the users migrated. It can and will happen again.

Jul 13 05:32 PM

[Moon Kil Woong:]

Usually those companies, especially virtual monopolies or oligopolies, that say they work in the public's interest and have nothing but good intentions (like the Federal Reserve and Microsoft) simply don't want to be regulated and want complete authority to be as capricious, self serving, and unfair as they like.

Google is not alone. That is the nature of monopolies. That's why we have something called anti-trust. Break Google and you really should also break Microsoft. After all, Microsoft, IBM, and the Wintel dulopoly started us all on the course of allowing tech monopolies and bullies to rule the computing market. We should be smart enough not to let it continue.

Good point and argument Tech Crunch. I hope someone covers how crooked and unfair domain listing is next.

Jul 13 10:02 PM

[Gregman2:]

It's also important to consider that SEO is actually a subset of a broader sales, marketing, business development strategy for most businesses. Buyers seek optimal value, and search ranking is part of being found both locally and for e-commerce, but it's just one peice of the mix.

Jul 14 01:02 PM