Bigger data isn't necessarily better

Sometimes it's hard being a statistician. Sometimes a long established statistical concept jars with your audience and no matter how hard you try to explain it in plain terms, you can see in the audience's eyes that they don't really believe you. Those suspicious eyes staring back at you are fairly sure you're pulling some shenanigans to get out of working harder, or to wring an answer from the data that isn't really there. What you're saying just feels wrong.

Explaining sampling can be like that, particularly when you're dealing with online data that comes in huge volumes and fighting against a tidal wave of 'Big Data' PR.

The audience's thinking goes...

More data is just better, because more of a good thing is always better.

More data must be more accurate, more robust.

More impressive.

Then a statistician says, "We only need 10% of your file to get you all the answers that you need".

And rather than sounding like an efficient, cost effective analysis, it feels disappointing.

"You only need a spoonful of soup to know what the whole bowl tastes like"

A common question from non-statisticians is to ask, "Overall, I have five million advert views [or search advert clicks, or people living in the North East of England, or whatever], so how big does my sample size need to be?"

Which sounds like a sensible question, but it's wrong.

Statisticians call that overall views number the "Universe" or "Population". It's the group from which you're going to draw your sample.

Once your population is bigger than about twenty thousand, it makes no difference at all to the size of the sample that you need. If you say that you've got one hundred million online advert views, and ask how big your sample needs to be, then the answer is exactly the same as if you had fifty million views. Or two hundred million.

Which probably sounds like statistical shenanigans again.

Think about it like this. I've got lots of ping-pong balls in a really big box and I tell you that some are red and some are white and they've all been thoroughly mixed.You can draw balls from the box one at a time until you're happy to tell me what proportion of each colour you think is in the box. How many ping pong balls do you want to draw?

Seriously, pause and have a think, how many do you want to draw? It's a really big box and you'll be counting ping pong balls for a week if you check them all.

Let's start with ten. You draw ten balls and get four red and six white.

Is the overall proportion in the box 60/40 in favour of white? It might be, but you're not really sure. Ten isn't very many to check.

You pull another ten and this time you get five more of each colour. Now you've got eleven white and nine red. Happy to tell me what's in the box yet? No?

Let's keep drawing all the way up to 100 ping pong balls.

Now you've got 47 whites and 53 reds. The proportion seems like it's close to 50/50, but is it exactly 50/50 in the rest of the box?

Every time you draw more ping-pong balls, you get a bit more sure of your result. But have you noticed that we haven't mentioned once how many balls are in the box in total; only that it was a big box? It's because it doesn't matter.

As long as the population is "big" and we draw balls at random, it doesn't matter how big it is.

Here's how your confidence in the result changes as you draw more ping-pong balls from the box:

The bigger your sample, the better your accuracy, but beyond a certain size - say 5,000 - your result is highly accurate and having an even bigger sample doesn't make very much difference.

"But!", say the objectors, "Online, data is basically free and we can use the whole dataset, so we should!"

And that's true, up to a point. Data storage is so cheap it's close to free, but data processing isn't. A large part of the cost is in your own time - you can wait ten minutes for a results dashboard to refresh, or you can sample the data, wait thirty seconds and get the same answer. It's your choice, but personally I like faster.

Outside the digital world, storage is still cheap, but data collection can get really expensive.

The TV industry in the UK is constantly beaten with a stick based on the fact that TV audience figures are estimated using a sample of 'only' 5,100 homes. It costs a lot to put tracking boxes into homes and this number has been arrived at very carefully, by very well trained statisticians. It's just enough to measure TV audiences with high accuracy, without wasting money.

In fairness, The BARB TV audience panel is challenged by a proliferation of tiny satellite TV channels - because sometimes nobody at all out of those 5,100 homes is watching them - and by Sky AdSmart, which delivers different adverts to individual homes. It may need to adapt using new technology and grow to cope, but nobody is seriously suggesting tracking what everybody in the UK watches on TV, at all times, on all devices. That would be ridiculous.

I'll be blunt. Any online data specialist who uses the 5,100 home sample to beat 'old fashioned' TV viewing figures, doesn't know what they're talking about.

Sampling is an incredibly useful tool and sometimes more isn't better, it's just more. More time to wait, more computer processing power, more cost and more difficulty getting to the same answer.

.

Are we branding? Or selling?

If you've worked in advertising for a while, you'll have come across the question of whether it's ok to compromise the 'creative vision' of a 'brand' ad, by sullying it with practical things like phone numbers and web addresses.

Being a data-type person and so not the sort to think that the majority of TV ads have a great deal of 'creative vision', I don't really understand this question, but it's a question that comes up a lot.

I wrote some time ago about adverts, which don't have the product that they're advertising in them and ads which don't tell you how to buy the product, fall into pretty much the same category. The challenge of making an ad is to make something that holds people's attention, so that you can talk to them about a product. Holding people's attention for 30" without the discipline of showing the product, is easy. It's a music video. You just show whatever you want and anyone can do that, even me.

So we need an ad with the product in it. A comment on the post I linked to above, summed it up perfectly: "Can you describe the advertisement without mentioning the product?" Keep that one in mind as you watch an ad break tonight - it's scary how often the answer is, "yes you can and by the way, what was the product?"


OK, Mr. Super-Brand so now you've communicated your product to me and I want to buy it. What do I do next? Oh, you've gone.

If it's not blazingly obvious what to do next then you need to tell me, because your ad wasn't that gripping and I'm very, very lazy. Give me a phone number, give me a web address, tell me where to buy your product. Do not make me work hard to buy you.

Apple can get away with 'pure' brand ads that have no direction afterwards (but that always, always, have the product in them), because everybody knows where to buy Apple's stuff. Most brands aren't Apple.

If a creative agency tells you that their advert won't work so well if it includes your phone number, or retail stores, or web address, or (God forbid) the product that you sell, then they're not doing their job properly. Their job is to include all of those things and still make the ad interesting enough that people will pay attention. That's hard. It's why creative agencies cost money.

Here's something that's caught my eye recently; Samsung have taken to sticking their logo in the corner of the screen for virtually the entire duration of a TV spot.

In a laptop ad, this makes perfect sense because let's face it, all laptops look the same. Without that logo, you're asking somebody to really pay close attention to the ad in order to realise whose laptop they're looking at.

Why don't many, many more advertisers do this?

TV channels do it. Their only purpose is to entertain and they still stick their logo in the corner of the screen!

But the advertisers who pay to show their products on those TV channels, don't do it. Why on earth not? The only reason can be due to a feeling that it would make the ad look 'cheap'. Putting your logo on the screen isn't subtle. Maybe we're afraid to be caught actually doing marketing, so we pretend not to.

Me? In my 30" spot, I'd want my product front and centre, my brand logo next to it and a dirty great web address on the screen at the end. When you say that would compromise the 'quality' of the ad, are you saying my brand looks cheap? You might want to try another argument.

A clash of broadcasting worlds

Did you watch Felix Baumgartner's record breaking jump yesterday? It was amazing.

Watching live on the Red Bull branded wesite at www.redbullstratos.com, I thought we could be seeing a new era in live broadcasting. After all, if you own the content, then why get somebody else to broadcast it for you? Run your own station, for as long as you need it, on the web.

Why would Neil Armstrong's moon landing be broadcast on TV in an era of live streaming?

There is a very good reason.

Red Bull's stream (via YouTube) attracted an audience of eight million. That would be very impressive if it was a UK audience, but it isn't. It's a global audience.

Eight million globally is, frankly, a bit crap. It's a YouTube record, but that's not the point. ITV were showing an hour long Coronation Street special at the same time as Felix was hoping he'd packed his 'chute properly and that did 6.25m, just in the UK.

So why do you want a regular TV broadcaster for your content? Simple. Audience reach. It's the same reason advertisers want TV ads, even if they've truly bought into social media.

I'm assuming one of two things happened with Red Bull Stratos. Either Google bunged Red Bull some fairly serious cash for the exclusive rights to Live stream via YouTube, or regular broadcasters just weren't interested, because they couldn't be given a predictable prime-time slot when the jump would take place. "Sometime in the next week if the weather's ok" doesn't really work for an ITV scheduler.

In one (fabulous) event, we've got the best and worst of new and old media. Only old media could have got that footage in front of its true potential audience. But TV is too inflexible to make the scheduling work for an event as unpredictable as the Stratos project.

In the UK at least, it's a shame we didn't have a dedicated digital TV channel that could be activated on short notice and then trailed on a major network. Press the red button to watch a nutter jump from space. The technology is there and it worked really well during the olympics. For a moon landing type event (Mars landing?) in 2012, I'm betting that's what would happen.

Unfortunately, the broadcaster with that capability is the BBC. With Red Bull logos everywhere? Never going to happen.

We're not quite living in the future yet. If you missed the footage yesterday because you didn't have one eye on Twitter, then here's the Austrian with the big cahones in all his high altitude parachutey glory. Enjoy.

Apparently, TV isn't dead after all

Saw a good article today on TV effectiveness from Advertising Age, which brings together a few different studies showing that the demise of TV might have been overstated.

Lets have a look at the central claim that TV is as 'Effective as Ever'.

MMA has run a meta-analysis across marketing mix models dating from the early 1990s and concluded that the effectiveness of TV per Rating Point (GRP) is not dropping. They also found that 1/3 of brand-name Google search queries are actually being driven by off-line media spend - so some of that online ROI is actually (partly at least) offline ROI.

Interesting study, but should we be surprised by the result? I don't know about you, but I'm not.

I was asked by a media planner not long ago to show that TV was still right for their 'youth' brand as the client was pushing hard to swap TV budget for online (the question is always asked this way round, rather than 'could my client be right?')

Two things jumped out straight away. One, it wasn't really a youth brand - the client had put far too much emphasis on a segmentation - and two, the best way to reach a young audience quickly is still TV. Young people have embraced new technologies faster than older demographics, but in the UK at least, the single media consumption activity that they do the most is to watch TV. A quick TGI run confirms it, but I won't post the chart up here as its not my company's blog.

The second reason why it's not surprising that TV is holding up, is the key part of the effectiveness statement. Effective at what? Selling product. Measured how? Per GRP.

So if you put the same amount of TV in the top, you get the same sales out the bottom. This could change if the reach of TV had dropped dramatically, so a lot fewer people were watching a lot more TV. It hasn't. Alternatively, you'd get less sales per GRP if TV ads were being watched, but had somehow mysteriously stopped working. These studies say not.

Before you get excited, do TV GRPs for your target audience cost more or less than they did in 1990?

What the MMA study hints at, and where analysts are starting to find new questions, is how you can best use new media to amplify TV, not to replace it.


Footnote:
Another study mentioned in the article states that

"more than half the advertisers (16 of 29) in the study still lost money by running their TV ads"

Including a long term effect? It would be good to know.

When does a creative wear out?

Media analysts get this question a lot. I certainly do. How long can you run a TV creative, before it 'wears out' and is no longer effective?
Still, I've never seen a reliable piece of modelling work that showed a creative had worn out and wasn't generating sales any more.

Us analysts don't like this sort of question. It's not answerable with a nice, billable, 6-8 week econometrics project. It's also generally asked in a way that implies there should be a general rule for this sort of thing. If somebody showed reliably that for a few advertisers the creative was definitely starting to wear out after 18months, that would be it. 18 months set in stone and wheeled out time and again in presentations.

If anyone reading is looking for a rule of thumb, I didn't just say it was 18 months!

Lets be honest, it depends on the creative and what you use it for. Pepsi's new Superbowl ad is going to get old quickly (ineffective? maybe, maybe not.)



This monstrosity from America is old almost instantly and can't sink any lower after that. Bet it still works though.