"I don't mind Miliband, but..."

On Wednesday afternoon, shortly before it was formally announced that David Miliband would not be standing for the shadow cabinet, David and Louise Miliband came out of their Primrose Hill home for a doorstep photocall in front of a scrum of photographers. David small talked with some of the journalists but remained tight-lipped on the subject they were there to ask him about. He kept his arms tightly by his side. “If you get a shot of me with my arm raised you'll make it look like I'm waving goodbye,” he joked, but no one was laughing.

The last time I was outside the Miliband house was a balmy Sunday evening in July 2009. I'd received a tip off that the newly resigned James Purnell had been spotted going inside the Miliband home together with Peter Mandelson. The Daily Mail sent a photographer to join my stakeout in the hope of capturing what might well have been a "Granita-type" moment in the Labour party's history. We waited until 2am and no one came out. Had the three men been plotting inside, the presence of a journalist and a paparazza no doubt persuaded them to stay in until after we had given up and gone home.

Back then, had Miliband chosen to resign together with Purnell, Gordon Brown would have been dealt a mortal blow. A vote of no confidence would have led to a leadership election and Miliband would have probably become the leader of his party and the country. But Brown would not have gone quietly, and the Labour party may well have lost the general election by an even bigger margin than it did in May. Miliband would have the ignominy of being the shortest serving British prime minister in history, and his political career would be over. No, better to wait until after the election when Brown would be gone and his path to leadership of his party would be clear. And so it would have gone, had it not been for his pesky little brother.

Yet already the victorious Ed looks like he's in for a bumpy ride. It took a couple of years as Labour leader before Neil Kinnock was given his “Welsh windbag” sobriquet by Fleet Street. By contrast, Ed Miliband has already been dubbed “Red Ed” by much of the media, and the country's journalists are in an apparent competition to see who can come up with the most disparaging description of his “pancake lips,” “bog brush hair” and “strangely nasal voice.” A cruel gag in the early 1990s went along the following lines: “I actually don't mind Kinnock...it's just her husband I can't stand.” Even though David has stepped out of politics for the time being, the “I don't mind Miliband...” joke seems inevitable.