Jacqui Smith needs to have a word with some Clubcard analysts

The UK government seems to be easily seduced by IT companies offering all encompassing database solutions to its problems. Contactpoint goes live today, holding the details of every child under 18 in the country and designed to prevent cases of child abuse slipping through the cracks chasms between different government agencies.

The NHS computerisation scheme is four years late after five years work and will cost £12.7bn if it's ever finished. That has very little to do with marketing, but you've got to really believe in a database to pay £12.7bn for it.

The government also aims to release the Interception Modernisation Programme by 2012. This is a big one, holding details of all electronic communications between individuals. Like having a huge itemised phone bill that includes emails, texts and telephone calls. Baroness Miller asked a question in the Lords concerning,

"three billion emails - that is 35,000 every second - 18 million internet connections and 57 billion text messages a year."

Disregarding the - frankly, vast - civil liberties implications of this database, would it be useful? Marketing analysts like big databases too, so what have we learned?

When a marketing analyst analyses a huge database, they're not looking for individuals. They're looking for patterns of behaviour among groups, that might be useful for targeting adverts.

The legendary nappies and beer example (diapers and beer if you're American) states that database analysts working for a major retailer noticed nappies and beer were often sold together. The story goes that young male parents often buy nappies on a Friday night and pick up a pack of beer too, so cross marketing these two products is extremely effective. Take your pick on which retailer - Wal-mart, Tesco, ASDA - it's not actually true.

The story works because it's an example of the sorts of things that marketing database analysts might look for. More realistically, analysts would profile different customer segments so that appropriate offers can be targeted towards them. What offers might appeal to a 25 year old mother of two who buys a lot of ready meals?

When you try to pick individuals out of the database, the whole thing falls apart. As well as being useless for marketing purposes, the 'outliers' (© Malcolm Gladwell) are, well, weird.

Some analyst friends working on a customer loyalty database, used to run a routine that looked for fraudulent activity. It picked out the people who were earning unfeasibly large numbers of rewards and flagged them for investigation.
As well as a few charlatans, it found a chap who was quite legitimately routing all his utility bills through the loyalty scheme and was quite probably their perfect customer, a member of the Saudi royal family with an unfeasbily large shopping bill and a football supporters' club where all the members were earning rewards together for fundraising.

What the goverment will find out if they ever get this thing off the ground, is that there are a lot of people in the UK with perfectly good reasons to send all sorts of suspicious looking emails. When the price of being flagged is a police interview (for starters) it really would be worth talking to the people who've been doing this stuff for ages - have a word with the analysts at Tesco Clubcard.