There is nothing inevitable about the way things have turned out in Afghanistan. Afghans do not appreciate foreign troops on their soil more than any other people, but in 2001, after two decades of war and five years of brutal Taliban misrule, they were open to whatever the Americans and their British allies had to offer. We had to work hard to make them hate us.

The disasters of the Bush era are all too well known. After invading Afghanistan in pursuit of al-Qaida, US forces were ordered into Iraq, leaving Afghanistan to languish and fester. It was the greatest strategic folly in modern times.

Barack Obama's Afghan war was meant to be different. Starting in early 2009, the strategy was subject to root-and-branch review. Troop numbers were doubled, ambitions were lowered, and a "civilian surge" was decided on, involving the dispatch of thousands of experts in everything from governance to counter-narcotics to fruit cultivation, in an attempt to show Afghans the benefits of siding with the Hamid Karzai government.

Three years on, there is a growing threat of civil war when western combat troops leave in 2014. The best hope is that the Afghan government and national security forces can somehow hold Kabul and most points north and west, in a country that is likely to be violent and chaotic for the foreseeable future. There are no convincing reasons to assume that the undoubted gains in security and development of the past three years will survive the transition for even a few months.

Little America: The War Within the War for Afghanistan is a detailed account of Obama's efforts to cope with the blunders made before he took power and to create the conditions for a sustainable US withdrawal. Having witnessed the bloodstained farce of the Iraq occupation, recounted in his previous book, Imperial Life in the Emerald City, Rajiv Chandrasekaran wondered whether, "confronted with a stubborn insurgency in a primitive land, officials in Washington could adapt? Could we wage a good war?"

Despite the best intentions, some remarkable people and enormous sacrifice, his conclusion is a resounding "no", on both counts. The roots of failure are many and various. A counter-insurgency strategy (focusing more on protecting people than killing Taliban) was only tried eight years into the conflict, after war fatigue had set in, and it was consequently under-resourced and underfunded. The US Marines were successful in bringing peace to districts that had suffered murderous levels of violence under British control. But the cost was absurdly out of proportion. The US spent more each year keeping Marine battalions in the Helmand districts of Nawa and Garmser than it gave to the whole of Egypt in military and civilian assistance, as an investment in Middle East peace. And there is no guarantee that the advances in these thinly populated outposts will survive the Marines' departure.

"Where we went, we made a difference. But not next door," one official told Chandrasekaran. "The surge worked locally, but it did not have the nationwide effect that was advertised."

One of the striking themes to emerge from Little America is the degree of insubordination running through the US military from the highest levels down. When Obama approved the 30,000-strong surge in late 2009, he made it clear to the top brass it was not for a "fully resourced counter-insurgency or nation building, but a narrower approach tied more tightly to the core goal of disrupting, dismantling, and eventually defeating al-Qaida". The generals carried on with their hugely ambitious counter-insurgency strategy regardless. When Chandrasekaran asked an aide to General David Petraeus, former US and Nato commander in Afghanistan and now head of the CIA, about Obama's marching orders, he was told: "We didn't pay much attention to that memo."

Similarly, the US Marines in Helmand in 2009 answered only to their own chain of command, not to Nato headquarters in Kabul. One of the reasons there was such a disproportionate concentration of troops in Helmand was that the Marines wanted an area of contiguous operations, which soon became known as "Marinestan" due to its fierce sense of autonomy.

The infantry who marched into Kandahar, meanwhile, had no time for the counter-insurgency doctrine (known as "Coin" to the US military) of their commanders, focusing instead on killing "bad guys". When a junior officer pinned up some thoughts about Coin by the then Nato commander, General Stanley McChrystal, a fiercely anti-Coin infantry colonel had it taken down and ordered the officer reassigned.

Chandrasekaran is at his observant best when chronicling the absurdities of the State Department bureaucracy as it went to war in Kabul. The idea of the "civilian surge" had been eagerly embraced not only by the Obama administration but by Britain's foreign secretary, David Miliband. The fighting would count for little, they argued, if the people liberated from the Taliban could not be provided with governance, law and order, and jobs.

The theory was admirable, but the execution was lamentable. By late 2010 two-thirds of the US government civilians in Afghanistan were in Kabul "to feed the mushrooming bureaucracy at the embassy and  USAID" (the US foreign assistance agency). They were for the most part restricted to the US embassy compound for health and safety reasons. Many were content to count the days until their departure. But Chandrasekaran tells the story of an idealistic volunteer called Summer Coish, who had to resort to subterfuge to escape the compound walls and actually go out into Kabul to meet some Afghans. Coish worked furiously throughout her time in Afghanistan, often getting only a few hours' sleep at night, but when she tried to think of ways she had helped the Afghan  or the American  people she concluded: "I couldn't come up with anything worthwhile that I had done."

Most of the people involved in the civilian surge in Kabul got cabin fever, passing their days watching videos or succumbing to alcohol in the embassy's Duck and Cover bar. The clinic doled out increasing quantities of anti-depressants. The British faced similar problems of becoming disconnected from Afghan reality behind the barbed wire and concrete perimeter of their Helmand base of Lashkar Gah. Chandrasekaran recounts the anger of a US Marine colonel at finding that, while his troops were in the heat of battle, the Brits were putting on a "Las Vegas Pimps and Hos" bash.

USAID comes under particular scrutiny for incompetence and doctrinal rigidity. Although there was plentiful evidence that cotton was a realistic alternative crop to poppies, USAID insisted it was unfeasible  it had got its arithmetic wrong in calculating the costs involved. When the mistake was pointed out, officials said they could not back cotton cultivation that would compete with American growers. The overall result of such narrow-mindedness, Chandrasekaran argues, was "a gross failure to capitalise on security improvements, paid for by the lives and limbs of American troops, to build a sustainable economy of the farmers of southern Afghanistan".

A substantial chunk of Little America focuses on honourable exceptions to the blinkered norm, such as the intrepid State Department representative in Garmser district, Carter Malkasian, who taught himself Pashto and stayed much longer than the standard one-year tour, ignoring the security strictures; he travelled around in an Afghan pick-up, "a potential firing offence", in an effort to build trust.

It often appears, however, that as soon as an Afghan local leader would begin to gain respect in his district, Karzai would replace him with a hand-picked crony from Kabul. The constitution agreed by the victors in 2001 centralised power in Karzai's hands, and he used it to build up networks of patronage, often at the expense of local people.

Such was the venality of the government in many places that the formerly despised Taliban looked good by comparison. The form of law and order they offered, dispensed by mullahs on motorcycles, may have been rough, but in the absence of a government alternative and in the presence of predatory police and corrupt officials, it was at least something.

Of the many problems with Coin, this remained the most serious. The whole idea was to win control of territory so that Afghan government representatives could move in, but if these representatives made themselves more hated than the Taliban, the whole strategy was self-defeating.

By current standards, Chandrasekaran is an optimist in that he believes that the Kabul government and the Afghan army will just about hang together and avoid either an all-out civil war or a Taliban reconquest in 2014. But, he argues, such basic achievements could have been achieved at much less cost in blood and treasure by a lighter western footprint that could be sustained for longer. "To the American military, more troops meant more security," Chandrasekaran writes. "To many Afghans, it was just the opposite: more troops meant more insecurity. If the foreign forces weren't here, they maintained, the insurgents wouldn't be seeding the roads with explosives."

To tell his story, Chandrasekaran waded through canals under fire with the Marines, supped with warlords, toured the battlefield with commanders and talked politics with leaders in Washington and Kabul. All his central characters are American; the British, the other allies and even the Afghans have mostly walk-on roles. The book is effectively a critique of US policy and performance, and he makes a nuanced and convincing case. "For years, we dwelled on the limitations of the Afghans," he concludes. "We should have focused on ours."

In the spring of 2002, the American army arrived in force in Afghanistan. Its main base was at the old airfield at Bagram, a short drive north of Kabul. By early summer, serried ranks of tents had gone up in the fine dust, a basic field hospital had been established and the all-important PX opened  where soldiers could buy Hershey bars, Hot Tamales candy, R&B compact discs and Bibles in "tactical" camouflage covers. In the evening, during the month I spent at the base, I went running on the airstrip, past the helicopters, the transport planes, the artillery park and the compound where prisoners were being kept. All around the perimeter, as dusk turned to night, newly arrived US troops tested their weapons, sending streams of tracer fire into the gathering darkness.

Over the following years, the base at Bagram grew rapidly. The old runway was renovated and extended, the makeshift jail converted to one of the most notorious detention facilities run for suspected al-Qaida and Taliban captives, power plants built and vast dining halls opened.

The tents were replaced by prefabricated and then solid concrete barracks. On one recent visit, I got my dinner from Burger King and, when waiting for a helicopter the next morning, watched Tropic Thunder, the spoof Vietnam war film, projected on the wall of a hangar. The old air base had become a small town of around 10,000 inhabitants, a Little America in the middle of an Afghan plain.

Little America is an apt title for this comprehensive, perceptive and detailed work. It is the second book by Rajiv Chandrasekaran, among the best of those brave, clever, and assiduous reporters that American journalism seems to produce in such happy abundance. His first, Imperial Life in the Emerald City, was a justly praised description of the follies, excesses, naivety and negligence of the Americans who tried to run Iraq with such disastrous results between 2003 and 2004.

In this new work he looks at the American experience in Afghanistan, focusing on the 2009 to 2011 period. That his book is already effectively a history  albeit a very contemporary history  says much about the rapidity with which the aims and methods of the west have changed in that tragic country over the last years. Little America is focused on the decision by President Obama to send around 30,000 new troops into Afghanistan in 2009 and its consequences. This "surge", modelled in part on a similar but more successful exercise in Iraq, is now over. Whether it worked or not is one of the many themes explored by Chandrasekaran in this book. The CIA apparently decided it had, at best, led to a stalemate.

"Where we went, we made a difference. But not next door," one official told the author. "The surge worked locally, but it did not have the nationwide effect that was advertised."

But the ultimate answer is that the debate is no longer relevant. Largely due to factors which have little to do with Afghanistan itself  the state of the American economy, general war fatigue in the US and elsewhere, the death of Osama bin Laden last year, the fragmentation and regionalisation of al-Qaida  the international project in the country is now winding down. In 2009 I asked the then US ambassador in Kabul if, as America was a democracy, any one leader could really promise eternal support. "Can I say we will be committed for the full life of the sun? No," he answered. "But short of that, yes, we will be here." That wasn't true.

As Chandrasekaran observes, it costs $1m to keep one American service member in Afghanistan for a year. The annual bill for the war last year was more than $100bn. Was achieving a marginally less bad outcome in Afghanistan worth the expense, administration officials wondered? With other pressing security challenges  Iran, North Korea, and the political upheaval in the Middle East  was it prudent to be committing so many resources to stabilising remote Afghan villages? The United States was spending more each year to keep marine battalions in two single remote districts in the deep south of the country than it was providing the entire nation of Egypt in military and development assistance, Chandrasekaran points out.

The war in Afghanistan started with goals that were purely to do with US security  the need to eradicate the bases from which the 11 September attacks had been launched. We have come full circle. The only debate now is whether what is left behind is enough to prevent a reconstituted al-Qaida gaining some kind of foothold. In the intervening period, the western project in Afghanistan was something much more ambitious, involving the creation of a prosperous, stable democracy with significantly improved human rights for women and minorities. The contrast with the modest aims of today is, at the very least, unedifying. There are stronger adjectives that could be used.

So what went wrong? First of all, winning in Afghanistan, however defined, was never going to be easy. The international effort there came at the end of a long series of increasingly ambitious  and even sometimes successful  "liberal humanitarian" interventions. It was a period of extraordinary hubris  and great trauma after the 9/11 attacks. Together this bred a toxic mix of astonishing overconfidence and deep insecurity. Then came the distraction of Iraq.

Chandrasekaran suggests  and some military historians might disagree  that Afghanistan is "by far the most complicated war the USA has ever prosecuted". The superlative here might be challenged by many historians but that the conflict was a tough challenge is without doubt.

America, however, could not meet it, he argues. Chandrasekaran's indictment is savage. "Too few generals recognised that surging forces could be counterproductive Too few soldiers were ordered to leave their air-conditioned bases and live among the people in fly-infested villages. Too few diplomats invested the effort to understand the languages and cultures of the places in which they were stationed. Too few development experts were interested in anything other than making a buck. Too few officials in Washington were willing to assume the risks necessary to forge a lasting peace Generals and diplomats were too ambitious and arrogant. Uniformed and civilian bureaucracies were rife with internal rivalries. Our development experts were inept. Our leaders were distracted." The result, inevitably, was that "the good war turned bad".

And if, after reading this, any Europeans are feeling smug, they shouldn't. Where they are mentioned at all, allies are portrayed as incompetent, feckless, cowardly or complacent. The British, whose ineffectual attempts to secure Helmand province and appalling relations with both local communities and the Americans are described in painful detail, come out particularly badly. Though not, it must be said, as badly as the Pakistani security establishment, whose strategically critical support, passive and active, for the insurgents was apparently not fully understood by US military planners until only very recently.

There are now thousands of books on this most recent Afghan conflict, ranging from airport thrillers to specialist study. Many books are extremely good, some are very bad. Little America is powerful and important and should be read by anyone interested in this ongoing and deeply depressing war, particularly those waiting for helicopters or eating Hot Tamales in Bagram.