In April last year, the British ambassador to Afghanistan invited me for a picnic in the Panjshir. I was told to report to the British Embassy in Kabul at 7am. Threading my way through a slalom of checkpoints and blast walls surrounding the British mission, I arrived to find the boyish figure of Our Man in Kabul, Sir Sherard Cowper-Coles, being hustled into a convoy of heavily armoured Range Rovers by his ever-present phalanx of bodyguards, walkie-talkies crackling and assault rifles primed.

For Cowper-Coles, it was a farewell trip at the very end of his three years at the helm of Britain's diplomatic effort in Afghanistan. For me, it was the second day of a research trip for a book I am writing about the first Afghan war. That war ended in 1842 with an entire British army being destroyed by poorly equipped tribesmen. The current situation is not as bad, but as our convoy drove through the scrappy roundabouts of central Kabul, I sat listening to Cowper-Coles's assessment of a campaign that was showing every sign of ending with as few political gains as Britain's three previous Afghan wars.

The evidence of failure lay all around us. Kabul remains one of the poorest capitals in the world. The US has poured around $80bn into Afghanistan, but almost all of it has disappeared into defence and security, and the roads of Kabul remain more rutted than those in the smallest provincial towns of Pakistan. There is no street lighting and apparently no rubbish collection. Less still is there security. The newspapers sometimes give the impression that Helmand is a frontline, separating Karzai's Afghanistan from the border areas ruled by the Taliban. In reality, the Taliban controls more than 75% of the country and Karzai's government holds just 29 out of 121 key strategic districts.

As we drove through the villages and towns of the Shomali Plain, Cowper-Coles's security turned on their jamming devices to block the call signals that might let off roadside bombs. No cars were allowed to come level with the ambassadorial SUV for fear of suicide bombers. Only when we reached the Panjshir, the Tajik anti-Taliban heartland, did things relax slightly. We visited the bleak domed tomb of Ahmad Shah Masood, on the crest of the valley, then had an oddly English picnic in the drizzle, with rugs and cucumber sandwiches and plastic cups of chardonnay. If you ignored the litter of wrecked Soviet APCs and downed helicopter-gunships, it could almost have been the Cotswolds.

Cowper-Coles seemed genuinely sad to be leaving Afghanistan but he was frank in his assessment. All was not lost; there were still opportunities. But without a clearer political vision, the sacrifices being made by British soldiers were doomed to terminate in embarrassing withdrawal, with Afghanistan yet again left in tribal chaos and quite possibly ruled by the same government that the war was originally fought to overthrow.

He was equally frank in his dispatches to David Miliband in London; then later, on his return to Whitehall, to William Hague and David Cameron. He paid the price of his honesty. Despite being regarded as one of the most brilliant diplomats of his generation, he was blocked from getting the senior position he hoped for  in Paris, Washington or Delhi  and resigned from the Foreign Office.

But the FCO's loss is our gain, as it has allowed Cowper-Coles to write a characteristically frank memoir of his time in Afghanistan. Cables From Kabul is unquestionably the most important record yet published of the diplomatic wrangling that has accompanied the slow military encirclement of western forces in Afghanistan. It is also the best account I have read of how post-colonial colonialism actually works. As with Imperial Life in the Emerald City, Rajiv Chandrasekaran's now classic account of Yankee incompetence in Baghdad, Cables From Kabul is at its best exposing the mixture of arrogance, over-confidence and rudderless dithering that has defined the conflict. And it is all the more remarkable for being written by a sceptical senior insider.

During his first week in Kabul in 2007, Cowper-Coles goes to see the American ambassador, nominally his most important ally, only to discover to his horror that "Chemical Bill" Wood is planning unilaterally to spray the Helmand poppy crop with weedkiller, without telling the Karzai government, despite its known opposition; "Chemical Bill" tells Sherard that he is counting on British support. Later, both men are summoned to Karzai's palace and encouraged to help rescue one of the president's friends who is besieged in a Helmand village by the Taliban. Karzai's palace, though, doesn't seem to contain a single map and no one can find the village on the only one available: an old RAF chart Cowper-Coles unearths from his briefcase. "This," he comments, "was how the supreme command of the war operated." On another occasion, £70m of taxpayers' money is wasted widening airstrips in Kandahar for Tornado jets that were never needed.

Cowper-Coles writes well about the difficulties of playing second fiddle in what was primarily an American war. He is also revealing about the difficulties of managing the gung-ho military: "An army that is willing to fight and die must, almost by definition, be hugely optimistic, unquenchably enthusiastic... and, ideally, not too imaginative." Over and over again, he is told by military briefers that progress is being made "though challenges remain". The reality was almost always grimmer.

The main problem was the lack of a clear strategic plan. Having blundered in, the west found it had unwittingly taken sides in the complex Afghan civil war that has been running since the 1970s, siding with the north against the south, town against country, secularism against Islam, Tajiks against Pashtuns. We installed a government and trained an army that in many ways discriminated against the Pashtuns. It is the largest ethnic group in the country yet, under Karzai, Pashtuns from the south make up only 3% of the Afghan National Army. Not surprisingly, almost all Pashtuns supported the insurgency.

Cowper-Coles urged his American counterparts to see that ultimately the only possible solution was political. Military action and counterinsurgency were only of value if there was a clear overall plan. When I met him in Kabul, the Americans were still opposing the negotiations with the Taliban that he believed to be inevitable; he described Richard Holbrooke , his main opponent in this matter, as "a bull who brought his own china shop wherever he went".

Just before his death late last year, Holbrooke came around to Cowper-Coles's way of thinking, but arguably when it was too late; the Taliban were then so strong, even in the north of the country, that by 2011 they had little incentive to negotiate.

What the future now holds is anyone's guess. Karzai could hold on after western withdrawal, like Najibullah after the Russian retreat. The Taliban could roll over the country as the Vietcong did in Vietnam. There may be a return to the civil war that destroyed Afghanistan prior to the rise of the Taliban. Maybe it will be China's turn to try and conquer a country full of the mineral resources it so badly craves. But there can be no doubt that when historians look back on the current fiasco, with its tragic missed opportunities, Cables From Kabul will be remembered as one of the best and most well-informed accounts of how Britain lost its fourth Afghan war.