This is something I've been itching to write for a while, and I'm not sure I've got my analogies and descriptions right yet, but the gist is right more or less right (perhaps less than I'd like), the long tail sucks, viva le long tail...
The long tail - or rather Zipf's Law - has been getting a lot of
publicity recently, which is nice in a way, but also suprising in
others. I first came across the long tail 8 years ago when I started
work at the Janet Web Cache Service in a variety of papers from the 3rd
web caching workshop (which had been held in Manchester just before I
started working there). The one thing that struck me was the fact that
Zipf's law is a fundamental aspect of human behaviour (as fundamental
As a result, when people started putting catalogues on the internet people, whole new communities - business communities - started to see the effects of the long tail of human behaviour on their profits. Intuitively we all understand the long tail - we know that we want things that suit us, that match our desires.
However the reason why "hits" and "hit culture" took off is simple
to explain - we all want quality. We don't want to choose from
20,000,000 things - this for me is the nightmare of the record store
(online or real). Ask me what music I like and I'll say "good stuff".
Don't ask me if it's garage or rap or indie, I don't know. I don't care
about that subdivision, that label. I like cool stuff. I can point at
stuff by Queen, stuff by Rob Dougan, Pictures at an Exhibition, the
theme tune from twin peaks, Ernie the Milkman, Lilly the Pink, and so
on. I don't classify my likes. As a result music stores suck for me (of
almost all kinds).
Expanding that choice by even double, let alone 10 fold or 20 fold,
leaves me cold. I'm aware that this makes it more likely that it is
_possible_ for me to find something I'll like, but my ability to find
it decreases as that choice size increases. This is Mooer's law in
action. The usefulness of the system to me decreases as the amount of
I'm not alone. (I might be one datapoint, but I'm not arrogant
enough to assume on a planet with 6 billion people to believe I'm that
different from other people)
That's why top tens are good. It's why if I'm going on a long flight
at an airport, I'm pleased that they have a top ten (or a variety of
top tens). It's why I'm pleased that shops tend to operate on a
principle of "survival of the fittest". If a book is good, it's likely
to stay on the shelves (through restocking). If it's not, it's likely
to disappear or get covered in dust. The smaller the shelf space, the
more important those decisions become. Too much dust and you go out of
business. Sure, the books available are less likely to be a good fit
for me, but they're also more likely to be closer to the head than the
tail (The tail being where the likelihood of it sucking for me
This for me is the real issue. Why do you see zipf distributions?
Because by and large the values as to what is good are shared by many
people, we do tend to have similar likes on some levels to other
people. That is why places like Amazon are particularly good, they
don't just operate a long tail - every online bookstore does that -
they allow people to gain insight into what's going on in that long
tail. Similarly, Google News has the ability to look at what thousands
of journalists worldwide have chosen to write about and chosen to
publish. They then allow you to look at the head of that snake, by
time, date, a search or a combination of all of these.
These services make the head & body of the long tail visible,
which in essence is what we all want anyway, though personal to us.
The long tail exists because our tastes all subtley differ, and we all want quality. What you think is terrible, I might think is great. I still remember seeing Spawn at the cinema, how much fun I found it, and how good a film I find it, and yet, I'm still to find another person who agrees. The phrase "so bad it's good" is a cliche, and with good reason. If I say "The Matrix" however, you find lots of people agreeing that think its a cool film.
As a result, the head of the snake is useful. The head of the snake
is a means of navigating yourself to content that lots of other people
who may share similar tastes to you think is good for some reason. If
you make the place to choose from attractive to a wide audience who
choose from the wide variety of content, then the head of that snake
will be attractive to that wide audience. And that's why hit culture
took off. As long as everyone was choosing from the same pot and the
reporting on that pot was accurate, then the top 10, top 40, top 100
was useful. That's why the top 40 in your local supermarket might be
more relevant to you than a general top 40.
The real interesting aspect of things like recommendation engines is
that they're personalising this snake. They're turning the snake into a
hydra, and each head is a real user.
However, the interesting point is this: caching makes sense. Caching
to be effective has to identify the head of the snake. By identifying
the head of the snake, but still making available the tail the cache is
useful, but provides a time benefit to the user and cost benefit to the
provider with regard to the content. What does this mean in the context
of a long tail? It means that small stores can exist, and can stock a
wide variety of useful content, and can even use simple heuristics to
make money. This is essentially what web caching does after all.
And why does caching make sense? It identifies the head & body
of the snake, allowing you take advantage of the fact that the head and
body have equal business or bandwidth value as the entirety of the
tail, which is a choice set, many, many, many, many, times larger.
Now, I'm not a business person (by choice), but I'm savvy enough to
realise this: if a web cache (fixed amount of choice of storage) can
cope with the vagaries of an effectively infinite choice zipf
distribution, and still turn a profit (ie be worth running), surely the
same can be true of a business. You don't have to say "we'll stock
everything", merely being able to get everything, and be able to serve
the high quality stuff (as chosen by that audience) is sufficient.
Furthermore, it's entirely likely that given a sufficiently "good"
recommendation engine, that the amount you stock can be kept small.
The paradox of the long tail is this: you don't need a tail to take
advantage of it, a virtual tail is sufficient - as long as you're
willing to change your body and head to match the whims and desires of
those choosing. If you can provide insight into that long tail, and
shift content into a local store - and turn that tail into a body &
head, then you increase the value of your proposition to the audience,
and they will move your store along the long tail of online stores
further towards the head, than the tail of online stores.
After all, if you could go into a store on the high street and say "give me something cool to listen to", and they did, and every time you went there not only did they give you something cool, but it got cheaper with time, surely you'd go back? You'd stop caring about the size of the tail, as long as you could get at it.