A great piece in The Guardian on Freenet software and the idea of the 'Deep Web'.

'The dark side of the internet'

In the 'deep web', Freenet software allows users complete anonymity as they share viruses, criminal contacts and child pornography

Andy Beckett

The Guardian, Thursday 26 November 2009

Freenet means controversial information does not need to be stored in physical data havens such as this one, Sealand. Photograph: Kim Gilmour/Alamy

Fourteen years ago, a pasty Irish teenager with a flair for inventions arrived at Edinburgh University to study artificial intelligence and computer science. For his thesis project, Ian Clarke created "a Distributed, Decentralised Information Storage and Retrieval System", or, as a less precise person might put it, a revolutionary new way for people to use the internet without detection. By downloading Clarke's software, which he intended to distribute for free, anyone could chat online, or read or set up a website, or share files, with almost complete anonymity.

"It seemed so obvious that that was what the net was supposed to be about – freedom to communicate," Clarke says now. "But [back then] in the late 90s that simply wasn't the case. The internet could be monitored more quickly, more comprehensively, more cheaply than more old-fashioned communications systems like the mail." His pioneering software was intended to change that.

His tutors were not bowled over. "I would say the response was a bit lukewarm. They gave me a B. They thought the project was a bit wacky … they said, 'You didn't cite enough prior work.'"

Undaunted, in 2000 Clarke publicly released his software, now more appealingly called Freenet. Nine years on, he has lost count of how many people are using it: "At least 2m copies have been downloaded from the website, primarily in Europe and the US. The website is blocked in [authoritarian] countries like China so there, people tend to get Freenet from friends." Last year Clarke produced an improved version: it hides not only the identities of Freenet users but also, in any online environment, the fact that someone is using Freenet at all.

Installing the software takes barely a couple of minutes and requires minimal computer skills. You find the Freenet website, read a few terse instructions, and answer a few questions ("How much security do you need?" … "NORMAL: I live in a relatively free country" or "MAXIMUM: I intend to access information that could get me arrested, imprisoned, or worse"). Then you enter a previously hidden online world. In utilitarian type and bald capsule descriptions, an official Freenet index lists the hundreds of "freesites" available: "Iran News", "Horny Kate", "The Terrorist's Handbook: A practical guide to explosives and other things of interests to terrorists", "How To Spot A Pedophile [sic]", "Freenet Warez Portal: The source for pirate copies of books, games, movies, music, software, TV series and more", "Arson Around With Auntie: A how-to guide on arson attacks for animal rights activists". There is material written in Russian, Spanish, Dutch, Polish and Italian. There is English-language material from America and Thailand, from Argentina and Japan. There are disconcerting blogs ("Welcome to my first Freenet site. I'm not here because of kiddie porn … [but] I might post some images of naked women") and legally dubious political revelations. There is all the teeming life of the everyday internet, but rendered a little stranger and more intense. One of the Freenet bloggers sums up the difference: "If you're reading this now, then you're on the darkweb."

The modern internet is often thought of as a miracle of openness – its global reach, its outflanking of censors, its seemingly all-seeing search engines. "Many many users think that when they search on Google they're getting all the web pages," says Anand Rajaraman, co-founder of Kosmix, one of a new generation of post-Google search engine companies. But Rajaraman knows different. "I think it's a very small fraction of the deep web which search engines are bringing to the surface. I don't know, to be honest, what fraction. No one has a really good estimate of how big the deep web is. Five hundred times as big as the surface web is the only estimate I know."

Unfathomable and mysterious

"The darkweb"; "the deep web"; beneath "the surface web" – the metaphors alone make the internet feel suddenly more unfathomable and mysterious. Other terms circulate among those in the know: "darknet", "invisible web", "dark address space", "murky address space", "dirty address space". Not all these phrases mean the same thing. While a "darknet" is an online network such as Freenet that is concealed from non-users, with all the potential for transgressive behaviour that implies, much of "the deep web", spooky as it sounds, consists of unremarkable consumer and research data that is beyond the reach of search engines. "Dark address space" often refers to internet addresses that, for purely technical reasons, have simply stopped working.

And yet, in a sense, they are all part of the same picture: beyond the confines of most people's online lives, there is a vast other internet out there, used by millions but largely ignored by the media and properly understood by only a few computer scientists. How was it created? What exactly happens in it? And does it represent the future of life online or the past?

Michael K Bergman, an American academic and entrepreneur, is one of the foremost authorities on this other internet. In the late 90s he undertook research to try to gauge its scale. "I remember saying to my staff, 'It's probably two or three times bigger than the regular web,"' he remembers. "But the vastness of the deep web . . . completely took my breath away. We kept turning over rocks and discovering things."

In 2001 he published a paper on the deep web that is still regularly cited today. "The deep web is currently 400 to 550 times larger than the commonly defined world wide web," he wrote. "The deep web is the fastest growing category of new information on the internet … The value of deep web content is immeasurable … internet searches are searching only 0.03% … of the [total web] pages available."

In the eight years since, use of the internet has been utterly transformed in many ways, but improvements in search technology by Google, Kosmix and others have only begun to plumb the deep web. "A hidden web [search] engine that's going to have everything – that's not quite practical," says Professor Juliana Freire of the University of Utah, who is leading a deep web search project called Deep Peep. "It's not actually feasible to index the whole deep web. There's just too much data."

But sheer scale is not the only problem. "When we've crawled [searched] several sites, we've gotten blocked," says Freire. "You can actually come up with ways that make it impossible for anyone [searching] to grab all your data." Sometimes the motivation is commercial – "people have spent a lot of time and money building, say, a database of used cars for sale, and don't want you to be able to copy their site"; and sometimes privacy is sought for other reasons. "There's a well-known crime syndicate called the Russian Business Network (RBN)," says Craig Labovitz, chief scientist at Arbor Networks, a leading online security firm, "and they're always jumping around the internet, grabbing bits of [disused] address space, sending out millions of spam emails from there, and then quickly disconnecting."

The RBN also rents temporary websites to other criminals for online identity theft, child pornography and releasing computer viruses. The internet has been infamous for such activities for decades; what has been less understood until recently was how the increasingly complex geography of the internet has aided them. "In 2000 dark and murky address space was a bit of a novelty," says Labovitz. "This is now an entrenched part of the daily life of the internet." Defunct online companies; technical errors and failures; disputes between internet service providers; abandoned addresses once used by the US military in the earliest days of the internet – all these have left the online landscape scattered with derelict or forgotten properties, perfect for illicit exploitation, sometimes for only a few seconds before they are returned to disuse. How easy is it to take over a dark address? "I don't think my mother could do it," says Labovitz. "But it just takes a PC and a connection. The internet has been largely built on trust."

Open or closed?

In fact, the internet has always been driven as much by a desire for secrecy as a desire for transparency. The network was the joint creation of the US defence department and the American counterculture – the WELL, one of the first and most influential online communities, was a spinoff from hippy bible the Whole Earth Catalog – and both groups had reasons to build hidden or semi-hidden online environments as well as open ones. "Strong encryption [code-writing] developed in parallel with the internet," says Danny O'Brien, an activist with the Electronic Frontier Foundation, a long-established pressure group for online privacy.

There are still secretive parts of the internet where this unlikely alliance between hairy libertarians and the cloak-and-dagger military endures. The Onion Router, or Tor, is an American volunteer-run project that offers free software to those seeking anonymous online communication, like a more respectable version of Freenet. Tor's users, according to its website, include US secret service "field agents" and "law enforcement officers . . . Tor allows officials to surf questionable websites and services without leaving tell-tale tracks," but also "activists and whistleblowers", for example "environmental groups [who] are increasingly falling under surveillance in the US under laws meant to protect against terrorism". Tor, in short, is used both by the American state and by some of its fiercest opponents. On the hidden internet, political life can be as labyrinthine as in a novel by Thomas Pynchon.

The hollow legs of Sealand

The often furtive, anarchic quality of life online struck some observers decades ago. In 1975, only half a dozen years after the internet was created, the science-fiction author John Brunner wrote of "so many worms and counter-worms loose in the data-net" in his influential novel The Shockwave Rider. By the 80s "data havens", at first physical then online locations where sensitive computerised information could be concealed, were established in discreet jurisdictions such as Caribbean tax havens. In 2000 an American internet startup called HavenCo set up a much more provocative data haven, in a former second world war sea fort just outside British territorial waters off the Suffolk coast, which since the 60s had housed an eccentric independent "principality" called Sealand. HavenCo announced that it would store any data unless it concerned terrorism or child pornography, on servers built into the hollow legs of Sealand as they extended beneath the waves. A better metaphor for the hidden depths of the internet was hard to imagine.

In 2007 the highly successful Swedish filesharing website The Pirate Bay – the downloading of music and films for free being another booming darknet enterprise – announced its intention to buy Sealand. The plan has come to nothing so far, and last year it was reported that HavenCo had ceased operation, but in truth the need for physical data havens is probably diminishing. Services such as Tor and Freenet perform the same function electronically; and in a sense, even the "open" internet, as online privacy-seekers sometimes slightly contemptuously refer to it, has increasingly become a place for concealment: people posting and blogging under pseudonyms, people walling off their online lives from prying eyes on social networking websites.

"The more people do everything online, the more there's going to be bits of your life that you don't want to be part of your public online persona," says O'Brien. A spokesman for the Police Central e-crime Unit [PCeU] at the Metropolitan Police points out that many internet secrets hide in plain sight: "A lot of internet criminal activity is on online forums that are not hidden, you just have to know where to find them. Like paedophile websites: people who use them might go to an innocent-looking website with a picture of flowers, click on the 18th flower, arrive on another innocent-looking website, click something there, and so on." The paedophile ring convicted this autumn and currently awaiting sentence for offences involving Little Ted's nursery in Plymouth met on Facebook. Such secret criminal networks are not purely a product of the digital age: codes and slang and pathways known only to initiates were granting access to illicit worlds long before the internet.

To libertarians such as O'Brien and Clarke the hidden internet, however you define it, is constantly under threat from restrictive governments and corporations. Its freedoms, they say, must be defended absolutely. "Child pornography does exist on Freenet," says Clarke. "But it exists all over the web, in the post . . . At Freenet we could establish a virus to destroy any child pornography on Freenet – we could implement that technically. But then whoever has the key [to that filtering software] becomes a target. Suddenly we'd start getting served copyright notices; anything suspect on Freenet, we'd get pressure to shut it down. To modify Freenet would be the end of Freenet."

Always recorded

According to the police, for criminal users of services such as Freenet, the end is coming anyway. The PCeU spokesman says, "The anonymity things, there are ways to get round them, and we do get round them. When you use the internet, something's always recorded somewhere. It's a question of identifying who is holding that information." Don't the police find their investigations obstructed by the libertarian culture of so much life online? "No, people tend to be co-operative."

The internet, for all its anarchy, is becoming steadily more commercialised; as internet service providers, for example, become larger and more profit-driven, the spokesman suggests, it is increasingly in their interests to accept a degree of policing. "There has been an increasing centralisation," Ian Clarke acknowledges regretfully.

Meanwhile the search engine companies are restlessly looking for paths into the deep web and the other sections of the internet currently denied to them. "There's a deep implication for privacy," says Anand Rajaraman of Kosmix. "Tonnes and tonnes of stuff out there on the deep web has what I call security through obscurity. But security through obscurity is actually a false security. You [the average internet user] can't find something, but the bad guys can find it if they try hard enough."

As Kosmix and other search engines improve, he says, they will make the internet truly transparent: "You will be on the same level playing field as the bad guys." The internet as a sort of electronic panopticon, everything on it unforgivingly visible and retrievable – suddenly its current murky depths seem in some ways preferable.

Ten years ago Tim Berners-Lee, the British computer scientist credited with inventing the web, wrote: "I have a dream for the web in which computers become capable of analysing all the data on the web – the content, links, and transactions between people … A 'Semantic Web', which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines." Yet this "semantic web" remains the stuff of knotty computer science papers rather than a reality.

"It's really been the holy grail for 30 years," says Bergman. One obstacle, he continues, is that the internet continues to expand in unpredictable and messy surges. "The boundaries of what the web is have become much more blurred. Is Twitter part of the web or part of something else? Now the web, in a sense, is just everything. In 1998, the NEC laboratory at Princeton published a paper on the size of the internet. Who could get something like that published now? You can't talk about how big the internet is. Because what is the metric?"

Gold Rush

It seems likely that the internet will remain in its Gold Rush phase for some time yet. And in the crevices and corners of its slightly thrown-together structures, darknets and other private online environments will continue to flourish. They can be inspiring places to spend time in, full of dissidents and eccentrics and the internet's original freewheeling spirit. But a darknet is not always somewhere for the squeamish.

On Freenet, there is a currently a "freesite" which makes allegations against supposed paedophiles, complete with names, photographs, extensive details of their lives online, and partial home addresses. In much smaller type underneath runs the disclaimer: "The material contained in this freesite is hearsay . . . It is not admissable in court proceedings and would certainly not reach the burden of proof requirement of a criminal trial." For the time being, when I'm wandering around online, I may stick to Google.

A world of hits

Reuters

NOVEMBER 20th saw the return of an old phenomenon: the sold-out cinema. “New Moon”, a tale of vampires, werewolves and the women who love them, earned more in a single day at the American box office than any film in history. The record may not stand for long: next month “Avatar”, a three-dimensional action movie thick with special effects, will be released (see picture). This film’s production budget is reportedly $230m, which would make it one of the most expensive movies ever made. “Avatar” will be a great disappointment if its worldwide ticket sales fail to exceed $500m. Yet it is a reflection of how things are changing in the media business that such an outcome is unlikely.

There has never been so much choice in entertainment. Last year 610 films were released in America, up from 471 in 1999. Cable and satellite television are growing quickly, supplying more channels to more people across the world. More than half of all pay-television subscribers now live in the Asia-Pacific region. Online video is exploding: every minute about 20 hours’ worth of content is added to YouTube. The internet has greatly expanded choice in music and books. Yet the ever-increasing supply of content tailored to every taste seems not to have dented the appeal of the blockbuster. Quite the opposite.

This is not what was predicted by one of the most influential business books of the past few years. In “The Long Tail”, Chris Anderson, editor-in-chief of Wired, a technology magazine (and before that a journalist at The Economist), argued that demand for media was moving inexorably from the head of the distribution curve to the tail. That is, the few products that sell a lot were losing market share to the great many that sell modestly. By cutting storage and distribution costs, the internet was overturning the tyranny of the shop shelf, which had limited consumers’ choices. And, by developing software that analysed and predicted consumers’ tastes, companies like Amazon were encouraging people to wallow in esoterica. Such companies did not just supply niche markets—they helped create them.

“The Long Tail” set off a lively debate. Professors at Harvard Business School questioned whether many companies can profit from selling a little of a great many things. The supply of obscure films and music seems to be growing faster than people are discovering them. Harvard’s Anita Elberse argued in an article last year that only a foolish firm would shift its focus away from the mass market. People in the media business, who have to back their judgments with money, have a different view. In a sense, they say, both Mr Anderson and his detractors are right. At the same time, both are missing the real story.

“Both the hits and the tail are doing well,” says Jeff Bewkes, the head of Time Warner, an American media giant. Audiences are at once fragmenting into niches and consolidating around blockbusters. Of course, media consumption has not risen much over the years, so something must be losing out. That something is the almost but not quite popular content that occupies the middle ground between blockbusters and niches. The stuff that people used to watch or listen to largely because there was little else on is increasingly being ignored.

Take American broadcast television. This is an industry in decline, albeit from an immensely profitable peak. The “big four” networks (ABC, CBS, NBC and Fox) have steadily lost viewers over the years as eyeballs have wandered to cable networks and, to a lesser extent, online video. Yet the top programmes are holding up well. In the 2000-01 season the most popular show on broadcast television, “Survivor”, was watched by 17.8m households on average. The leader in 2008-09, the Wednesday edition of “American Idol”, drew 16.5m. Less popular shows—fairly funny sitcoms and cheap reality programmes such as “COPS”—suffered far worse. Indeed, the further you look down the prime-time rankings, the more audiences have eroded (see chart 1).

The fact that the biggest shows continue to draw audiences nearly as big as their predecessors did may not sound impressive. But it has great commercial consequences. As broadcast television has lost audience share, its salesmen have convinced advertisers to pay more and more to reach a given number of viewers. They get away with this because broadcast television is still unchallenged as a mass medium. No other venue, including the internet, can guarantee an audience of many millions on a single evening. So a show that reaches 10m Americans today is worth a lot more than a show that reached 10m at the beginning of this decade. Simon Cowell, the star judge on “American Idol”, reportedly renewed his contract earlier this year for more than $100m over three seasons. He is probably worth it.

Or take music. Like broadcast television, recorded music is a troubled business. Sales of CD albums are declining thanks to illegal file-sharing and the rise of digital download services such as iTunes, which allow listeners to pluck out the best tracks. Yet hit albums can still sell well. In Britain, where album sales in all formats have declined by 18% since the 2004 peak, those of albums occupying the number-one spot have increased slightly (see chart 2). A recent analysis by Billboard, a trade magazine, found a similar trend in America. There, sales had declined across the board, but the hits were holding up best. Albums ranked between 300 and 400 suffered the greatest proportionate losses.

One possible reason is that the profile of music buyers has changed. Young fans, who are more likely to follow up-and-coming guitar bands and dance music, are highly likely to download music illegally. In 2008 Britons in their 40s spent more on pop and rock music than teenagers or people in their 20s, according to TNS, a market-research firm. As young people with more unusual tastes abandon music shops altogether, the market becomes skewed towards safe, established pop acts. One of last year’s biggest sellers was the cast recording of “Mamma Mia!”, which features songs by ABBA, a fizzy 1970s pop group. But this is not the whole story.

Offer music fans a virtually infinite choice of songs free of charge, and they will still gravitate to hits. That has been the experience of We7, a music-streaming service based in London which has 2.5m users. Only 22% of We7’s 4m songs are streamed in any given week, says Steve Purdham, who founded the company. The top 100 artists account for more than half of all streams. Users of Spotify, another ad-supported music service, are similarly unadventurous. Will Page of PRS for Music, which collects royalties for British songwriters, calculates that the most popular 5% of tracks on Spotify account for 80% of all streams. He is counting only the 3m tracks that were streamed at least once between February and July. Another 1.5m were not touched at all.

“People want to share the same culture,” explains Roger Faxon, head of EMI Music Publishing. Music is an intensely social medium, most enjoyable when it is discussed and shared with friends. Because choice in music—and, to an extent, other media—is collective as well as individual, it is hardly surprising that people cluster around popular products. And Mr Faxon cites another reason, having to do with the advent of online piracy. File-sharing has made virtually all music available for nothing. Yet it has not altogether stopped people from buying. Even enthusiastic pirates will still buy CDs of music that they love, in order to get the cover art or simply to express their devotion. And what people love, it turns out, are hits.

Although you might expect people who seek out obscure products to derive more pleasure from their discoveries than those who simply trudge off to see the occasional blockbuster, the opposite is true. Tom Tan and Serguei Netessine of Wharton Business School have analysed reviews on Netflix, a popular American outfit that dispatches DVDs by post and asks subscribers to rate the films they have rented. They find that blockbusters get better ratings from the people who have watched them than more obscure ones do. Even the critically loathed “Transformers: Revenge of the Fallen” is awarded four stars out of five. Ms Elberse of Harvard Business School has found the same of ratings on Quickflix, the Australian equivalent of Netflix.

Perhaps the best explanation of why this might be so was offered in 1963. In “Formal Theories of Mass Behaviour”, William McPhee noted that a disproportionate share of the audience for a hit was made up of people who consumed few products of that type. (Many other studies have since reached the same conclusion.) A lot of the people who read a bestselling novel, for example, do not read much other fiction. By contrast, the audience for an obscure novel is largely composed of people who read a lot. That means the least popular books are judged by people who have the highest standards, while the most popular are judged by people who literally do not know any better. An American who read just one book this year was disproportionately likely to have read “The Lost Symbol”, by Dan Brown. He almost certainly liked it.

This explains why bestselling books, or blockbuster films, occasionally seem to grow not just more quickly than products which are merely very popular, but also in a wholly different way. As a media product moves from the pool of frequent consumers into the ocean of occasional consumers, the prevailing attitude to it—what Hollywood folk call word of mouth—can become less critical. The hit is carried along by a wave of ill-informed goodwill.

These days it may travel far. Blockbuster films, long among the most international of media products, are now more so. The leading studios have beefed up their foreign-sales arms and learned how to market films that open at roughly the same time all over the world. Sony has done particularly well this year, pulling in more than $1.6 billion in ticket sales from outside America, a record for the studio. Big films such as “2012” and “Angels and Demons” have earned roughly twice as much in foreign cinemas as in American ones.

Hollywood seems to have exported the blockbuster model, too. Anil Arjun, chief executive of Reliance MediaWorks, reckons ticket sales for the top five films in India grew by 250% between 2004 and 2008. Growing professionalism and a multiplex building boom have lifted the market as a whole, but the biggest films are rising most quickly. In addition, the best Hindi films increasingly serve expatriates in London and California’s Bay Area.

“New Moon”, newly enthroned

Blockbusters are also reaching people in more ways. Peter Chernin, who recently stepped down from overseeing News Corporation’s film and television business, notes that technology does not just expand the range of products available. It also gives people much greater choice in how they consume the most popular ones. No longer must people go to a cinema or a video shop if they want to see a popular film. They can get hold of it as a video-on-demand, download it or stream it. “Hits are going to be the single biggest beneficiary of technology,” Mr Chernin reckons.

In short, just because people have more choice does not mean they will opt for more obscure entertainments. That is especially clear in the book trade. A study of the Australian market by Nielsen, a research firm, found that the number of titles bought each year (measured by ISBNs) has risen dramatically, from about 275,000 in 2004 to almost 450,000 in 2007. Niche titles selling fewer than 1,000 copies each accounted for nearly all the growth in variety. Yet their market share fell. In Britain, sales of the ten bestselling books increased from 3.4m to 6m between 1998 and 2008.

The bestsellers are gaining in part because of a change in the retail marketplace that affects other media too. High-street bookshops, which carry a reasonable selection of what publishers call “mid-list” and “back-list” titles—that is, modestly popular and somewhat out-of-date books—are struggling. Borders UK, owner of a British chain, is reportedly seeking a buyer for its stores. Such shops are being displaced by online retailers, which offer vast selections of obscure titles, and also by supermarkets, which sell a tiny selection of hugely popular books.

In newspapers, too, the leading outfits are faring best. The three biggest-selling American publications—the New York Times, USA Today and the Wall Street Journal—have all held on to subscribers better than the large metropolitan papers that comprise the second tier. That market also has a thriving long tail in the shape of small-town papers. This has less to do with consumers’ tastes than with advertising. The metropolitan papers are suffering because the classified advertisements on which they rely have collapsed under pressure from free listings websites such as Craigslist. Small-town papers also depend on classified ads, but they face less online competition. The nationals rely more on display advertising.

What is a media company to do? As sales become ever more concentrated, it is becoming both more urgent and harder to establish a foothold near the top of the market. A book or film that fails to attract a mass audience tumbles quickly into the depressed middle. To avoid this fate, should a company spread its development and marketing budget over lots of products, hoping that one or two catch on, or should it bet on just a few? The problem is especially acute in businesses like music, where money is tighter than ever and even the hits are not quite as solid as they used to be.

The joke answer, proffered by several executives asked these questions, is that media companies should simply churn out hits. This is less of a joke if a firm knows in advance what will prove popular. Of all creative media enterprises, Hollywood is the most confident of its ability to predict demand. Film studios carry out rigorous audience research and adjust production and marketing budgets according to the size of the group they are targeting. The studios have learned that stars are much less reliable generators of profits than films based on known characters and stories. That is why, in August, Disney agreed to pay $4 billion for Marvel Entertainment, a veteran comic-book and media firm that had filed for bankruptcy protection in the mid-1990s.

Above all, Hollywood has learned that bigger is better. Although small films can do astonishingly well (the latest is “Paranormal Activity”, a cheap thriller that has sold more than $100m-worth of cinema tickets in America alone), they do not do so at all dependably. SNL Kagan, a research firm, calculates that between 2004 and 2008 films costing more than $100m to produce consistently returned greater profits to the big studios than cheaper films did. With DVD sales slumping in the recession and outside financing hard to obtain, the leading studios are cutting back their output of films. But the cuts are concentrated at the bottom end. Studios have shut down or neglected their divisions that specialise in distributing low- and middle-budget films. None has sounded a retreat from big-budget blockbusters.

Hollywood is one of America’s most stable industries, with the same number of big studios now (six) as in the early 20th century. As both production budgets and risks soar it seems less likely than ever that an outsider will break into the club. On the contrary: the club may eventually shrink. The trend towards blockbusters is likely to suit the most successful studios, with the deepest pockets, the best marketing departments and the greatest ability to wring revenues from a hit by selling DVDs, television rights and toys.

Of course, not everything can be big. Complex political films; violent cartoons; English folk music—none of these things is likely to produce queues around the block. And a great many hoped-for hits fail: every media company puts out many more misses than hits. What to do with them? One answer, says Mr Bewkes, is to take advantage of the protective power of brands. Television programmes, in particular, do not compete for audiences on a completely featureless playing field. They are shown on channels that attract distinct audiences with different expectations. In the right context, a middling show can survive. “You let the strong brand carry the medium product,” Mr Bewkes explains.

Another way of rescuing less popular stuff is to charge more for it. In many media businesses it is an accepted principle that such products cost more. Specialist magazines are often expensive. Less popular books cost more than bestsellers, because the latter are discounted. Even television has a successful model for charging more for shows that a smallish group of people feel passionately about, in the form of premium subscription channels like HBO. Yet, with the exception of live screenings of opera and sport, film tickets all cost the same. This is one thing that could change.

But nobody knows quite what to do. The old-media world of limited choice, in which any product that was not too objectionable was guaranteed a decent audience, was a comfortable place. Pleasing a customer who can choose from several hundred films and television programmes even without getting up from the sofa, by contrast, is an unnerving prospect.

Via The Economist.

A fantastic interview (from the 2009 Edinburgh TV festival) with David Simon, the creator of 'The Wire' - arguably the best TV show of all time :)

A presentation that Associate Professor Michael Wesch gave at this year's Personal Democracy Forum in the US.

A great piece from Graham Linehan on the 'value' of Twitter.

***

Newspaper editors!  A small request?

Ahem...

Please stop allowing your writers, who have no little to no idea of what they speak, to submit pieces on Twitter and social media. It's embarrassing. You're allowing them to embarrass themselves. Jackie Ashley's piece in the Guardian last week was a classic example of the condition. She writes maybe a thousand words on Twitter and makes it painfully clear that she simply does not know what it is.

Like many people, Jackie Ashley knows how to make her PC do the one thing it needs to do to enable her to make a living.  But beyond, in her case, firing up Microsoft Word, she's at a loss. I would imagine that when Jackie "surfs the net" , she has to wait patiently for each page to load because she doesn't know about tabbed browsing yet. I would also guess that she doesn't know how to edit a wiki...or set up a simple website...or write and save her work online...and that's fine!  I have a few blind spots myself! For instance, I don't know how to use Photoshop at even the most basic level. So in the same way you would not let me write about Photoshop, I would ask that you not let Jackie write about Twitter .

I would in fact go even further and say that if a journalist doesn't know how to do any of the things mentioned above,  he or she is no longer capable of writing about the modern world. And if they can't keep up, for heaven's sake keep them out of the way.

Like a lot of Web 2.0, Twitter is as good as the people on it, and uninformed pieces like Jackie's continue to fool otherwise bright people into thinking that it's some sort of "I'm having a sandwich" announcement service. Certainly, there are people on Twitter who might use it this way...and more power to them if it makes them happy... but there are also journalists, scientists, humorists, magazines, newspapers, authors,  and Samuel Johnson. There are Mums, Dads, soldiers, doctors, nurses, firemen, base jumpers, astronauts, old people, young people, builders, boxers, cops and at least one tank (don't ask). And what are they all doing? Sorry, what are we all doing? We are sharing links to thought-provoking articles, we are making each other laugh, we are keeping each other up to speed on current events...we are communicating with each other on a platform that encourages good manners, that rewards us when we're interesting and lightly smacks our hand when we're not. For the first time in history, the human race is having a global conversation, and despite all our differences, we actually seem to be getting on quite well.

Meanwhile, the dying newspaper industry sits on the sidelines and sneers.

My experience of Twitter is slightly slanted, because of my limited, bogus celebrity , so I won't go into the many ways that the service has enriched my life because I'm unrepresentative.  The one thing I will say is that it has made it very easy for me to do polls. So last week I asked the Twittersphere a question (The Ignoranti is my term, a collective noun for the pundit class, and the #whatTwitterdidforme hashtag is...well, do we really have to keep explaining that?). Here are just a few of the thousands of replies.

(I realise that the following sounds like an infomercial. But you'll notice that no-one's trying to sell you anything. One of the things the Ignoranti likes to sneer about, in their confused way, is Twitter's seeming inability to make money, as if that remains a useful way of assessing value after the diddling we've recently received from The Masters of The Economic Universe.)

If you want to hear any of the stories behind these tweets, join Twitter and speak to the people I've named. It's incredibly easy to talk to them once you get past the site's admittedly arcane entry point (why has no-one made Twitter easier to understand for new users yet?)*

Of course, many people replied with the words "not a damn thing", and that's OK too, because the beauty of Twitter is that it is only as useful as the person who is using it wants it to be. It is such a simple and flexible service that everyone who uses it does so in a different way. Not only that, but it's a meritocracy. Not only that, but Karma seems to have something to do with it. If you use it for good, you will be rewarded, if you use it for evil, you will be blocked. As a result, it's leading to some remarkably civil conversations between ideological enemies. If the inventors of Twitter never win a Nobel Prize, they wuz robbed. Because as far as I'm concerned, they have enabled us all to take a major evolutionary step at a crucial moment. At a time when the human race faces not one but several extinction threats, we suddenly get the ability to talk to one another.

Ladies and Gentlemen, we are having The Conversation.

(Via http://glinner.posterous.com/the-conversation-23)

A great piece from the Design Observer Group on the future of attention.

(Via http://observatory.designobserver.com/entry.html?entry=10297)

Web 2.0 ... The Machine is Us/ing Us. An amazing video done by Assistant Professor Michael Wesch at the University of Kansas. I know everyone has seen this before (including me many times!) but I couldn't resist not posting it :-)

Another great video by Associate Professor Michael Wesch.

A great video that captures the big changes we're seeing as a result of social media. Some good facts and figures as well!