Afghanistan, Jonathan Steele writes in his new book, is full of ghosts. There are the ghosts of Alexander the Great, Genghis Khan, and their numerous successors  invaders who were able to conquer the place, but could not hold it. "Ghosts" is the word the Russians used for the mujahideen guerrillas who merged with the peaceful population during the day and struck during the night  who fought the Russians in the 1980s and are fighting us today.

Steele has covered events in Afghanistan for many years, and he skewers with palpable glee the myths and half-truths that are peddled by politicians, generals, official spokesmen, and too many commentators. Perhaps the most damaging in his eyes is the myth that the destruction of the Taliban would eliminate the threat from the Arab terrorists who felled the twin towers in New York. That myth has made it all but impossible for us to construct a sensible strategy for Afghanistan.

And so he now writes with increasing despair about the failing efforts of the United States and its allies to build a viable Afghan state out of the physical, institutional and human rubble left behind by three decades of civil war and foreign intervention. A central theme is "the similarity of the Soviet and US interventions and the lessons that ought to be learned". In spite of failing to defeat the mujahideen, Steele writes, the Soviet leadership "had the sense to change course".

When the Russians invaded, Afghanistan had a fairly coherent state apparatus, built up since the end of the 19th century, and an economy which, though very backward, was beginning to develop thanks to American and above all Soviet aid. Most modern Afghan leaders were reformers, but their attempts to change Afghan society, especially when it came to the treatment of women, often met with bloody resistance.

The Afghan communists were reformers too. But they were a small and murderously fractious clique, and they killed their opponents in very large numbers. They soon faced violent uprisings, especially in the countryside.

The Russians invaded in December 1979 not as part of a plan for the world victory of communism, as our propaganda had it, but to protect the security of their southern frontier against Islamic unrest and American intrigue. Senior Soviet generals and officials had warned against getting mixed up in an Afghan civil war. They pointed to the unhappy precedent of Vietnam. Their leaders went ahead nevertheless, as much as anything because they could not think of a better alternative. Change the Afghan leader, train up his army and his police, and then  they hoped  the Soviet troops would be able to leave in a matter of months. It did not happen. The Russians were soon sucked into just the quagmire that the sceptics had feared. Before long they were looking for a way out.

The man who eventually did it was Mikhail Gorbachev. By the time he came to power in 1985, Gorbachev firmly believed that the issue could not be settled by force. He was determined to cut the Gordian knot through negotiation. He was supported by his political colleagues and by his generals, who did not try to argue that they could achieve "victory" if only they were given more troops and more time. He told the Afghan government that his soldiers would be out within 18 months. It still took three years of difficult negotiation before the Soviet army marched away, undefeated on the battlefield but profoundly demoralised by nine years of useless war. And though the Russians left their man Najibullah behind them, they did not bequeath him a wider settlement that would command the support of his enemies among the mujahideen and in Pakistan. The consequence was a bloody civil war, which ended in the victory of the Taliban.

Steele contrasts this record with that of President Obama who, unlike Gorbachev, came to power believing that his war in Afghanistan was a "necessary" war. He thought  at least at first  that if the Afghans adopted western standards of good governance and financial probity they would, by some alchemy, no longer be willing to give cover to the Taliban. Instead of looking for ways to negotiate with the insurgents, he listened to his generals, who persuaded him that they could beat the Taliban if they were given the tools to finish the job.

That has not happened. Despite tired claims of progress by official spokesmen, the Taliban continue to advance. The Afghan government remains unconvincing, hamstrung by arrogant foreign advice, corrupted by an overdose of foreign money. Steele believes that the only way forward is for the Americans to change course, like the Russians did, to commit themselves wholeheartedly to a far-reaching settlement between Kabul and all its enemies, under which the foreign armies can withdraw in good order. The Afghans, helped by their foreign friends, can then devote themselves to rebuilding their shattered country in peace.

That is an ambitious objective. It seems more likely that the Americans and their allies will eventually leave as the Russians did, without the broad settlement that alone can ensure that the civil war does not resume. As we leave, we will doubtless explain away our failure by blaming the ungrateful Afghans for their unwillingness to adopt our western ways. And meanwhile both Afghans and foreigners will continue to die in another useless war.

Sir Rodric Braithwaite is the author of Afgantsy: The Russians in Afghanistan, 1979-89 (Profile).