When the poet laureate Andrew Motion penned a birthday rap to Prince William last year, the reaction was a mixture of surprise and embarrassment. Motion's verses were declared unworthy of the genre. They also remained firmly on the printed page. Unlike literary poetry, rapping is an oral discipline which lives or dies by the microphone. By definition, this wasn't a rap at all. More curious, though, was the incredulity with which Motion's choice of prosody was received. After all, hip hop is a mainstream phenomenon which has dominated the charts and high street fashion for as long as most teenagers have been alive (even farmers in Devon have swapped their overalls for Adidas trainers and puffa jackets). What actually lay behind the reaction to Motion's gesture was a notion of authenticity. Any old Joe can rap, but licence is only granted to residents of the street - to those toting an easy association with criminality and violence.
For a genre that is 25 years old this year, hip hop has little to show for its maturity. While its influence has stretched into the shires and beyond, walk down any megastore hip hop aisle and scowling back at you is a line-up of the same kind of hardmen as a decade ago. Numbers may have burgeoned (there are now believed to be over 100 hip hop millionaires in the world) as has the body count, but the lifestyles, platitudes and contradictions represented by protagonists of the culture have, if anything, grown narrower and more impossible. Repetitive images of material excess and recidivism continue to dominate the commercial rap market, and while production techniques have evolved to become the most sophisticated in pop music, rapping itself - the essence of hip hop culture - has not developed in at least a decade. As the ideas have dried up, celebrities and industry investors have been forced to promote the most sensational aspects of the culture. Even loyal fans are now claiming that hip hop's message to the disenfranchised is one of confusion and self-destruction. For a musical form that once claimed to offer meaning, and even hope, this must spell the end.
Which is a shame, considering hip hop's beginnings. The history of rap is as eclectic as America itself. Its influences range from the tall tales of Chicago blues singers to the sing-song declarations of Mississippi riverboat men; from wild west folk songs to the intoned rhetoric of Martin Luther King and Muhammad Ali; from skipping rope rhymes and gospel to the syncopated rhythms of jazz, the pick 'n' mix culture of creole and even the lewder compositions of Cole Porter. Still regarded as rap's essential ancestor, however, is the toast. Toasting is the African-American folk art of rhythmic narrative, a tradition probably passed down by nomadic African griots who would wander from village to village offering musical renditions of local history and mythology. Their early 20th-century American descendants told rhyming tales of exaggerated heroism, in which the black man, in contrast to his usual lot, won the day.
A well-known toast is about the eponymous hero Shine, reputedly the only black man working aboard the Titanic (in fact, records show that there were no black passengers or crew). Informing a disbelieving captain that the ship had sprung a leak, Shine went back below and began to think/That, umph, this big bad muthafucka is bound to sink./Shine said, it's fish in the ocean, crabs in the sea/But it's one time you good cool white people ain't gone bullshit me. Before the other passengers end up in the drink, Shine swims to New York where, surrounded by whores and flush with money, he lives out his days drunk in a Manhattan bar. Shine's immodesty is heavily ironic, half-mocking of the boastful, entrepreneurial America which excludes him, yet half-yearning for the impossible fantasy (out-talking, out-drinking, out-sexing the doomed white man) in which he is engaged.
The story of rap follows the migration of these attitudes - mockery, self-preservation and pride - from the inner city to the commercial mainstream. By the early 1970s, toasts were finding their way on to records. James Brown stole them wholesale, while Jalal Nuriddin's collective, the Last Poets, stuck theirs into pop songs. Radio DJs devised toast-like routines to segue songs, often rhyming well into the second verse. But it was the Caribbean-style sound systems, plugged illegally into the lampposts of the Bronx, that facilitated the development of what became rap. These tangles of wires and amplifiers invoked spontaneous street gatherings at which DJs and bystanders would rhyme to the beat. It was at the so-called "block parties" of the late 1970s that hip hop's celebrated innovator, DJ Kool Herc, hung out. Formally known as Clive Campbell, the Jamaican-born Herc had one of the most powerful sound systems in the neighbourhood, with a funk and soul record collection to match. The story goes that in a flash of inspiration Herc decided to put copies of the same record on to both turntables. By crossfading between decks he could play the best bit of the record - usually a percussive breakdown - for as long as he liked. In this way the breakbeat was born. And so were the B-boys, or breakdancers, who during this musical peak would spin their best moves.
The accident of scratching occurred when failed attempts to cue up a disc resulted in the squealing crunch of diamond on vinyl. The miracle of needle-dropping - landing the stylus in exactly the right spot - arose in the same way. But the most significant development was the rise of the MC, or "mic controller." Until then, the man with the mic simply jollied the party along, like at a wedding disco. When Grandmaster Flash, arguably the most nimble-fingered of all the Bronx DJs, magnanimously moved his decks to one side, it allowed his posse, the Furious Five, to take centre stage, rapping in formation. The crowds were impressed and these prototype MCs became ghetto stars. Colouring in this local scene were the graffiti artists, who provided a chaotic backdrop to hip hop's most innovative, and least commercial, era.
But soon after, it was time for business. An ex-soul singer called Sylvia Robinson founded the first rap label, Sugar Hill Records, and in 1979 released what is now widely acclaimed as the first ever rap record: "Rapper's Delight," by the Sugar Hill Gang. Although the raps and bass line were pilfered, Robinson's creation was an instant hit. A rush of new labels eager to exploit New York talent quickly followed - Paul Winley Records, Enjoy, Spring, and eventually Tommy Boy and Def Jam - creating a flourishing underground movement.
Its cast was multiethnic, bringing African, Caribbean, European and Latin culture to the mix. One of the landmark productions of the time was Afrika Bambaataa's "Planet Rock" (1982). By stitching Kraftwerk samples together, producer Arthur Baker introduced an electro (and German) sound to the genre, helping it depart from its funky roots. Another was Grandmaster Flash and Melle Mel's "The Message" (1982), which ditched light-hearted party raps for race and politics, and became the first ghetto classic. Following these "old school" days came the explosive delivery of Run-DMC and the Beastie Boys; the provocative content of NWA and 2 Live Crew; the dazzling production aesthetics of Bomb Squad and Dr Dre; the smart business practices of Russell Simmons; the race politics of KRS-One and Public Enemy - all helping to build the multibillion dollar industry of today. Last October, Billboard magazine announced that for the first time in its history all the top ten acts in the Hot 100 were black. And most were rapping.
So at a moment when black music is celebrating its greatest achievement, why does hip hop appear creatively impoverished? As an eclectic musical form, it was always inclined to racial and musical crossover. The video of its first American top ten record, the 1986 Run-DMC/Aerosmith collaboration "Walk This Way," had the two black/white rival groups tearing down an on-stage dividing wall in a symbolic act of desegregation. Similar acts such as Public Enemy and Ice T appealed to thousands of white college students eager to shock their parents, while the jazzy tones of Jungle Brothers and De La Soul attracted more passive audiences. Yet for most of its lifespan, hip hop remained, with all its associated violence, poverty and politics, a subversive sound, frightening off MTV's potential advertisers. In the subsequent process of becoming the ultimate symbol of the mainstream, it has - like all countercultural music - lost its meaning.
For a period it looked like rap would break this depressing rule, achieving mass popularity without blunting its creative edge. Because hip hop was generated out of a cut and paste process - lifting and looping sections of rhythm from obscure vinyl, or sampling street noises and media jingles - it pre-empted the marvels of digital music production, which allow modern producers to layer hundreds of different samples at a click of a mouse. When, in the mid-1990s, this technology became widely available, the marriage with hip hop (between software and genre) seemed made in heaven. Grassroots audiences and pop consumers were momentarily united, as the street joined hands with the market. But it wasn't long before producers such as Timbaland and the Neptunes were feeding couch-bound America bagloads of ear candy. Despite breaking new ground, the faltering ideas of their acts rendered the final mix technically astonishing but ultimately empty.
Rap has a problem with originality. Its habit of salvaging old hit records in the service of the new (for those who can afford copyright clearance) means that it has a limited repertoire of material to plunder. At some theoretical point in this retroactive process, the whole history of rock 'n' roll gets exhausted into pastiche. Unlike rock music - which itself has been on the ropes since the 1990s - rap can't look forward to original compositions. A recent album by New York rapper Nas, for instance, credits samples on nine out of ten tracks. It is mechanical reproduction gone mad.
Rap at its best is capable of amazing effects. To stand in spitting distance of a freestyling MC can be a thrilling experience. Any skilled rapper will rock his whole body as he takes on the character of his subject matter, twisting his features and splaying his fingers as feeling demands. But the key to the performance is vocal delivery. In an essay last year in the Hudson Review, the poet (and head of America's National Endowment for the Arts) Dana Gioia deconstructed rap's prosody, holding up its rhythmic vitality as a contrast to the weakness of free verse. Gioia argued that rap represents a reconnection with fundamental principles of rhythm that literary poets - in their effete self-consciousness - have long since abandoned: "Rap characteristically uses the four-stress, accentual line that has been the most common meter for spoken popular poetry in English from Anglo-Saxon verse... to Rudyard Kipling." Gioia described how the stress meter, freed from the visual scanning of the written word, permits as many syllables per line as the rapper requires (provided the number of stresses remains constant). This is why, with a superimposed beat, MCs can accelerate to the verbal velocity of a livestock auctioneer. And, linking rap to the African-American tradition is its deployment of the impulsive, improvised rhythms of jazz, which impose themselves on either side of the downbeat.
As with meter, so with rhyme: rap briefly reinvigorated our sense of aural pleasure. But Gioia doesn't acknowledge the logical conclusion of his comparison. Just as free verse emerged in literary poetry because the possibilities of rhyme were exhausted, so it is with the rhymes of rap - with the added disadvantage that rap is far more limited in its often crude couplings, exhausting itself after 25 years rather than 2,500. The history of all art tells us that no sooner do we grow accustomed to one form than we begin searching for another. And it is unlikely that rap can extend itself much beyond the rhyming couplet.
The predominace of rap can be attributed to its increasing acceptability and loss of shock value. Like all sensationalism, it is subject to the law of diminishing returns. Expletives like muthafucker (since described as the "Oedipal now"), nigga, bitch and ho (whore) are overused and banal, while gangsterisms like the drive-by shooting are as hackneyed as the spaghetti western quickdraw. Nowadays, parents dismiss such outrages with a tut, if not openly enthusing about the lyrics of Eminem. Such profanity once drew attention to ghetto life, forcing conservatives, policymakers and critics into sociological debate; now hip hop is more likely to suffer admission into the American literary canon.
Underpinning this cultural acceptance is hip hop's aptitude for enterprise. In contrast to rock music's anti-commercial gestures and fear of "selling out," rappers passionately embrace the entrepreneurial spirit. The career of the average hip hop artist now includes boardroom duties for the various subsidiary businesses they own. Hip hop moguls such as Russell Simmons (co-founder of Def Jam records), P Diddy and Master P have developed their own clothing lines, while last year the business artists Nelly and Ice T launched hip hop styled energy drinks (the former running into trouble after crassly naming his tonic "Pimp Juice"). Sneakers, colognes, ringtones, sportswear and entertainment companies have taken pop merchandise out of the venue ticket office and straight to the mall. The result is a booming hip hop economy. Perhaps, in a country where wealth is status, this is America's best chance yet to dismantle the ghetto.
The reality, however, is that most of these businesses depend on the mutating fashions of a capricious mass market. The number of truly successful companies is relatively small - many are owned by major labels like Universal and BMG - and the wealth gap between the super-rich and those on whom their influence is exercised has never been so gaping. Such material excess, combined with showmanship and conceit, is for large sections of the community a poisonous admixture, and has led merely to attention-seeking vulgarity. Hip hop has never been short on irony, yet an unbounded display of wealth, an impossible dream of gold, diamonds, thrones, canes, whores and mansions, is now permanently woven into the hip hop aesthetic. For those sick of wealth envy, this is good reason to turn off the radio.
Compared to its American counterpart, British hip hop is a cottage industry. Most acts struggle to sell more than 3,000 units, and major label signings are rare. Back in the 1980s, the story was different. Monie Love, the Cookie Crew and Derek B and other British acts were popular enough even to stand a chance in America. But the boom in dance music dashed all hopes that Britain would become a hip hop nation - 1989's so-called "second summer of love" lured thousands of potential fans away from rap towards the self-abandonment of acid house. During the course of the 1990s, inward-looking innovators in Britain continued pillaging US hip hop imports for their own styles, creating ragga and techno hybrids like jungle and drum & bass. The latest stage in this development is UK garage, which now dominates the country's urban music - hip hop being just one element in its hard mix of R&B, reggae and techno. There are dedicated hip hop purists who, tucked deep inside the housing estates of inner-city England (rap has failed to emerge from Wales or Scotland), are still making records. Supported by a network of independent labels, these self-producing rappers court a dwindling audience. British hip hop DJs such as Radio 1's Tim Westwood and 1Xtra's Rodney P & Skitz prefer corporate-backed American stars.
To make matters worse, the British scene is divided by colour: the white guys (Taskforce, Braintax, Wolftown Committee) and the black (57th Dynasty, Roots Manuva, Ty, Lewis Parker). Additionally, both camps suffer from the embarrassing problem of accent. Regional variations in pronunciation stand out, and for audiences familiar with the tones of the Bronx or Compton, a muthafucking Brummee can, like the poet laureate, seem hopelessly inappropriate. For years, British rap acts have tried to overcome the problem. In the 1980s, London Posse's MC Roddie Rok (now Rodney P of the BBC's black music station 1Xtra) dropped his American accent after supporting Big Audio Dynamite at a gig in New York. Confused New York rappers convinced him that being himself was the only way forward, and so from that day on he rapped out his yardie slang in broad Lambeth cockney. (French rappers, too, have occasionally succeeded, famously MC Solaar with "Bouge De L? .") Yet impersonators such as Derek B, who unscrupulously mimicked an American accent in his 1986 hit Rock the Beat, have always reaped the most rewards. Almost all the hip hop bought in this country is of US origin.
But the deleterious effects of hip hop culture are felt on both sides of the Atlantic: the arguments over its racism, misogyny, homophobia, political nihilism and problem with gun culture are well known. On the one hand are the defenders - the liberal press and the hip hop community itself, who claim that rap is just a fiction, and that a line from "Fuck Tha Police" has no more chance of turning young black kids into cop killers than an episode of ER will lead them into medical school. They suggest, too, that it is a form of reportage, reality music from the American slums and trailer parks. In this way artists such as Eminem have, through a confessional style of rap, relayed back to the people who matter tales of white trash culture and the junk diet upon which it subsists.
On the other hand we have the censors and traducers. There are the moralising ladies of US politics, Tipper Gore and Lynne Cheney, whose efforts to silence the profanity led to the ubiquitous Parental Advisory warning - the "Tipper sticker." There is the politician and frequent litigant C Delores Tucker, who persuaded Time Warner to drop its interests in Death Row Records; and there is Bill O'Reilly, the Fox News commentator, whose outrage is disseminated through the most watched cable news show in America. But perhaps the wittiest of the hip hop haters is black jazz critic Stanley Crouch, who once described rappers as "monkey-moving, gold-chain wearing, illiteracy-spouting, penis-pulling, sullen, combative buffoons." This confusing alliance of conservatives, well-meaning busybodies and insider critics is united by the conviction that rap music destroys lives.
That the gangster urge has become the most popular expression of discontent among black communities is unquestionable. Its endless appetite for self-gratification, its self-destructive and inward nature contribute nothing to the community. High-profile feuds, like that between rival rappers Tupac Shakur and the Notorious BIG, which resulted in the death of both, are played out in countless copycat disputes. Reports of neglected teenagers living out the gangster dream are frequent. For many boys, the intended consequences of rap's playful fantasies are cruelly reversed - resulting in disempowerment rather than opportunity. While life isn't easy for many young black people, you have to ask if rap music and the lifestyle it preaches are the best expression of protest they have. For community leaders and inner-city residents, the convoluted, self-referential politics of race and identity now mix with debates about role models and stereotypes.
While many cite hip hop as an alternative, semi-political movement seeking to address such problems, it is difficult to find any coherent manifesto or hard reasoning. Hip hop's different factions are often at cross purposes, with artists espousing contradictory values. Penitent icons like Nelly hold a gun in one hand and a donation to a children's literacy programme in the other. In the old school days, Afrika Bambaataa and other icons were more constructive, teaching emancipation from the ghetto through knowledge and non-violence, but contemporary rap artists - the ones that are still alive - have only the distance between them and the local jail as their moral ground.
The best-known hip hop slogan - "keep it real" - appears on the surface to be a call for solidarity. Scratch a little deeper and the words "don't change" emerge. By "keepin' it real," prescribed tribal codes are maintained. In the early days of rap, when to the delight of partygoers Grandmaster Flash's wheels of steel were at full tilt, innovation in music, dress and language were the order of the day. The terms "hip hop" and "rap" had barely been coined, and many people were more inclined to creative pursuits than perpetuating a cultish identity. "Keeping it real" hinders change, both social and musical, and contributes to hip hop's current catatonic state.
The final chapter in the story of hip hop is Eminem. Since his debut in 1995, the white rapper has come to dominate American music culture. His most recent album, The Eminem Show, sold 7.4m copies, making it the bestselling album of 2002. Growing up in a black neighbourhood in Detroit, Eminem suffered all the damaging events of a ghetto upbringing, including parental separation, financial hardship and drug abuse. Although technically white trailer trash, he naturally took on all the characteristics of his black neighbours, emerging as a white negro (accusations of fakery would beset him in his early days of fame). Eminem perfected the art of rapping, gaining his credentials through regular battles at local freestyling competitions. Recognising his lyrical gift, Dr Dre, the celebrated producer and one-time member of NWA, took him on as a prot?g?. And after one false start, Eminem's first major release, The Real Slim Shady LP, pushed him into the hip hop superleague. As an explicit, trailer park comedy of intoxication, pornography and self-harm, its themes resonated with white American and British teenagers suffering from similar personality disorders. Eminem's skill is in simultaneously treating his audience as co-conspirators and targets, and his three albums have unearthed a white middle America as problematic as the ghetto and as wilfully destructive as any Schwarzenegger movie.
Eminem's ability to pull all aspects of his life into his work - infamy and mental dysfunction being favourites - makes him both a diarist and a commentator, and proof that rap can still be inventive. But only by breaking rules. Consequently, Eminem must confront a growing number of enemies in the hip hop industry itself. Rap magazine the Source, which describes itself as "the magazine of hip hop culture and politics," is voicing critics' concern that Eminem might be "leading hip hop down a dangerous path for the culture, similar to the impact of Elvis on the death of African-American contributions... in rock'n'roll." A recent issue features a "special report on the state of emergency in hip hop." Eminem has extended the subject matter, and racial boundaries, of hip hop. But the comparison with Elvis may be apt. After Eminem, there is no going back. How much can rap and its audience change before the music becomes something fundamentally different? And change is the only option left for a form once built on innovation, but now characterised by self-limiting dogma and paucity of ideas.