eBay: The Ads That Only Worked in the Spreadsheet

What happened

eBay was a household name buying search ads on its own name and on ordinary product terms, and the reporting said the money was working. In spring 2012 it switched brand ads off at Yahoo and MSN while continuing to buy them on Google, so Google acted as a control. A naive before-and-after showed clicks down 5.6%; measured against the control, only 0.53% of clicks were actually lost. The bigger test then turned off non-brand keywords across about 30% of the country, and while sales the ad platform attributed to paid search collapsed by more than 72%, total sales barely moved.

The case, start to finish

The dashboard was not slightly optimistic. It had the wrong sign.

A number everybody trusted

eBay is a marketplace: buyers and sellers transact, and eBay takes a fee. It holds no inventory, so its two levers are the supply of listings and the demand of buyers, and buyers are bought, in large part, through search advertising. That makes the value of a search ad not a marketing question but a question about the price of the company's main input.

In 2012 the reporting said that money was working spectacularly. Measured the standard way, with a simple before-and-after comparison, the return on ad spend was 4,173%. Add controls for time and geography and it still read 1,632%, which is close to the method the ad platform's own help pages described for computing search return. Nobody here was being careless. They were computing the number the whole industry computed, and computing it carefully.

Who clicks an ad for the word 'eBay'

Attribution software credits a sale to an ad if the buyer clicked that ad shortly before buying. At eBay the window was 24 hours. Now picture the person typing the word eBay into a search box. They have already decided where they are going. The ad did not create that intention. It stood in front of it and charged admission.

So in spring 2012 the company switched off ads on its own brand searches at two search engines while continuing to buy them at a third, which left that third engine acting as a control for anything seasonal. A naive before-and-after showed clicks down 5.6%. Compared properly against the control, only 0.53% of clicks were lost. 99.5% of them came back through the free listing, which had been sitting directly beneath the paid one the entire time. In organic search terms, the company had been paying for traffic it was already going to get.

The bigger test, and what it costs to run one

The brand result is easy to dismiss as a special case, so eBay ran the harder version. Between April and July 2012 non-brand keywords were switched off in about 30% of the country: 68 test markets went dark and 142 stayed on as controls, matched so that the two groups had historically moved together. That design is the whole method. Anything national, a season, a competitor, a news cycle, hits both groups equally and cancels out.

In the dark markets, the sales the ad platform attributed to paid search fell by more than 72%. Total sales barely moved. The estimated contribution of the entire program was 0.66% of sales, and even that could not be distinguished from zero. Expressed as a return, the experiment said −63%, in a range running from −124% to −3%. A follow-up test that July, plus a further test in Germany that preserved a control group, reproduced the finding, which is what turns a surprising result into a method.

Notice what running this costs. Switching real ads off across a third of the country loses real revenue if the ads work. That is precisely why it is worth doing: the test only hurts you when the answer is good news.

The part that is easy to over-apply

The wrong lesson is that search ads do not work. The study found the largest effect among people who had never bought on eBay, and effects near zero among people buying more than three times a year. The ads did work. They worked on strangers, and almost all the money was being spent on regulars.

eBay is also a household name whose free listing sits at the top of its own results page, so when the brand ad disappeared there was something to catch the click. A young brand nobody searches by name has no such safety net, and its brand ads may be genuinely incremental. These tests are hard to run well, too. The effect you are hunting is small relative to ordinary sales variation, so an underpowered test returns a range wide enough to be consistent with almost anything, and that is how a real finding gets waved away.

The habit that transfers is one sentence long. Attribution reports which ad happened to be nearby when a sale occurred; whether the sale needed the ad is a different question, and only a holdout group answers it. Before you defend the budget on your biggest channel, switch it off somewhere chosen at random, leave it on elsewhere, and compare total sales rather than attributed ones.

Timeline

• Before the tests eBay is a household name buying search ads on its own name and on ordinary product terms. The reporting says the money is working: measured the standard way, a simple before-and-after comparison shows a return on investment of 4,173%.
• Spring 2012 The company switches off paid ads on eBay-brand searches at Yahoo! and MSN, while continuing to buy them on Google, so Google acts as a control for anything seasonal.
• The brand result A naive before-and-after on MSN shows click volume down 5.6%. Compared properly against the Google control, only 0.53% of clicks are lost, and 99.5% come back through the free organic listing. The company had been paying for traffic it was already going to get.
• Apr–Jul 2012 The bigger test: non-brand keywords are switched off in about 30% of the country. 68 test markets go dark, 142 remain on as controls, matched so the two groups had historically moved together.
• The non-brand result Sales the ad platform ATTRIBUTED to paid search collapse by more than 72% in the switched-off markets. Total sales barely move. The entire paid-search programme is estimated to add 0.66% to sales, and that estimate cannot be distinguished from zero.
• Jul 2012 A follow-up test on Google, plus a further test in Germany that preserved a control group, reproduce the brand finding. This is now a repeatable measurement method, not a one-off result.
• 2014–2015 The work is published. Its central estimate: the experimental return on that spending was −63%, with a range from −124% to −3%. The 4,173% was not a small overestimate. It had the wrong sign.

You're in the owner's chair

You run marketing at a large, well-known online marketplace. Your attribution dashboard reports a return of over 1,600% on paid search: every dollar in, many dollars of sales credited back. Finance wants to move 20% of that budget somewhere else, and you have a week to respond. What do you do?

• Cut the budget 20% across the board and watch what happens to sales
• Switch ads off in a random third of the country and compare TOTAL sales
• Defend the budget with the dashboard — it is the same method the whole industry uses

One shutdown, two measurements

• Sales the attribution report credited to paid search, and how far they fell where ads went off: 72 % of sales
• Sales that actually stopped happening: 0.66 % of sales

Both figures are from the same experiment, April–July 2012. The first bar is what the dashboard reported. The second is what the randomised comparison against 142 control markets measured, and even that 0.66% could not be distinguished from zero.

Business model

eBay is a marketplace: buyers and sellers transact, eBay takes a fee. Because it does not hold inventory, its two big levers are supply of listings and demand of buyers, and demand is bought, in large part, through search. That makes 'how much is a search ad worth?' not a marketing question but a question about the cost of the company's core input.

Revenue model

Transaction fees on goods sold. What matters for this case is what sits on the other side of the ledger: paid search was one of the largest lines in the acquisition budget, justified by a dashboard number that everyone in the industry computes the same way and almost nobody had ever tested.

Cost structure

A search ad costs money every time somebody clicks it. The click is certain. The question is whether the sale that followed was CAUSED by the click, or merely reported next to it. The paper makes the difference concrete: eliminating brand ads lost 0.5% of all clicks to the site, which was about 1.5% of the PAID clicks, and the rest of that paid traffic simply walked in through the free door instead.

Strategic challenge

Attribution software counts a sale as caused by an ad if the buyer clicked that ad shortly beforehand, at eBay within 24 hours. But think about who clicks a search ad for 'eBay'. It is somebody who already decided to go to eBay and typed the name into a search box. The ad did not create that intent; it stood in front of it and charged admission. The measurement was not slightly optimistic. It was systematically counting the company's most loyal customers as its most expensive conversions.

Key decision

Rather than argue about the dashboard, eBay turned the ads OFF for a randomly chosen part of the country and left them on everywhere else, then compared total sales, not attributed sales, between the two groups. That is the whole method. It costs real revenue if the ads work, which is exactly why it is worth doing: the test only hurts you when the answer is good news.

What worked

Two things. First, the design: switching off ads in 68 markets and keeping 142 as controls means anything that hits the whole country, whether a season, a competitor, or a news cycle, hits both groups equally and cancels out. Second, the segmentation. The paper found the largest effect among people who had NEVER bought on eBay, and effects near zero for people buying more than three times a year. Ads did work; they worked on strangers. Almost all the money was being spent on regulars.

What failed

The industry-standard calculation, applied honestly by capable people. The paper reproduces it: 4,173% return with no controls, and 1,632% once you add time and geography controls, and it notes that this is close to the method Google's own help pages described for computing AdWords return. Adding controls did not fix the bias. It just made a wrong number look more rigorous.

Risk factors

This result is easy to over-generalise, and doing so is its own expensive mistake. eBay is a household name with a dominant organic listing: when its brand ad disappeared, the free result was sitting right underneath. A young brand nobody searches by name has no such safety net, and its brand ads may be genuinely incremental. The experiment also measures SHORT-TERM sales; it says nothing about long-run brand building. And such tests are hard to run well: the effect you are hunting is small relative to normal sales variation, so an underpowered test returns a confidence interval wide enough to be consistent with almost anything, which is how a real finding gets waved away.

Lesson summary

Attribution tells you which ad was standing nearby when a sale happened. It cannot tell you whether the sale would have happened anyway, and for a well-known brand the answer is usually yes. The only instrument that separates the two is a holdout: switch the spending off somewhere random, leave it on elsewhere, and compare TOTAL sales. Run that test on your largest channel before you defend its budget, because the eBay numbers moved from +4,173% to −63% and nothing about the advertising changed except how it was measured.

Key data

• 99.5% (0.53% lost, versus a Google control) Clicks recovered free when brand ads were switched off
• 5.6% of clicks lost Naive before-and-after estimate of the same shutdown
• 68 test markets, 142 control markets, Apr–Jul 2012 Non-brand test size
• Down more than 72% Attributed sales in the switched-off markets
• 0.66% of sales — not distinguishable from zero Actual contribution of the whole paid-search programme
• 4,173% (raw) · 1,632% (with controls) · −63% (experiment) Return on investment, three ways of measuring
• −124% to −3% Confidence range on the experimental estimate

Sources & basis

The company here is real and named, and nothing about it was invented to make the story land. The list below is where each fact came from — public filings, court records, published reporting — so you can open a source and check it against the sentence that used it.

1. Tom Blake, Chris Nosko and Steven Tadelis, "Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment", NBER Working Paper 20171 (May 2014) — abstract and paper listing View source ↗
2. Full working paper (PDF) — the brand-keyword substitution result (99.5% of clicks retained; 5.6% naive pre-post), the 68 test / 142 control market design, attributed sales falling over 72%, the 0.66% estimated contribution, and the ROI table showing 4,173%, 1,632% and −63% with a [−124%, −3%] confidence interval View source ↗
3. Published version: Econometrica, volume 83, number 1 (January 2015), pages 155–174