Despite recently declassified materials, historians of the FBI remain painfully hostage to the fragmentary records that survived "the routine destruction of FBI files" typifying J Edgar Hoover's 48-year, secrecy-obsessed directorship and especially "the bonfire of his personal files after his death" in 1972. Because deliberate cover-ups naturally excite scurrilous conjectures, suspicious students of the bureau have reacted to Hoover's well-oiled system for the undetectable destruction of government archives by mimicking his tendency to assume the worst, often on the basis of hearsay evidence, about the target of an investigation.

Author of a celebrated account of serial blundering and incompetence at the CIA, Tim Weiner is commendably impervious to this familiar temptation, refusing to replicate Hoover's paranoid style. Although Hoover unscrupulously exploited stories of homosexuality to discredit or silence political adversaries, Weiner begins his latest book, based in part on the freshly declassified documents, by dismissing the racy gossip that Hoover himself was "a tyrant in a tutu". And although one of Hoover's chief deputies testified to his boss's smouldering bigotry ("He hated liberalism, he hated blacks, he hated Jews  he had this great long list of hates"), Weiner is more lenient in his own assessment, assuring us that Hoover "was not a monster" and concluding, in an effort at balance, that the man's "knowledge was enormous, though his mind was narrow".

Leaving Hoover's alleged personality disorders to Hollywood scriptwriters, Weiner focuses his efforts instead on refuting a series of misperceptions about the FBI that a lasting cult of secrecy has allowed to crystallise in the public mind.

The first of these misunderstandings is that the FBI, after 9/11, had to be dragged reluctantly from its traditional law-enforcement approach towards an unprecedented preventive mission, aimed at stopping in advance those who might possibly harm the country. There is nothing especially novel, it turns out, about the preventive mission assigned to the bureau in the war on terror. From its very origins, the FBI devoted the lion's share of its resources towards averting future harms, not towards solving past crimes, which remains a largely state and local function. As one-time FBI associate director Mark Felt (aka "Deep Throat") explained in the mid-1970s, the mission of the bureau has consistently been "to stop violence before it happens" by targeting individuals deemed dangerous by unaccountable bureaucrats on the basis of undisclosed evidence.

Contrary to the comic-strip image of the G-man gunning down gangsters in the 30s, Hoover's fixation was the communist menace, a "malignant and evil way of life" with which red professors and preachers, union activists, movie-makers, civil rights leaders and student protesters threatened to infect the American population. While warning, after the second world war, that the Soviets might be secretly plotting nuclear terrorism inside the US, his real concern was to prevent secret armies of disloyal citizens from luring their countrymen into a noxiously anti-American way of life. Rather than fighting crime, Hoover, even after he severed his cloakroom ties to the malodorous politics of McCarthyism, kept busy "creating the political culture of the cold war".

Refreshingly, Weiner also dismantles the myth of an imperial presidency, its powers supposedly swollen by real and imaginary threats. Permanent national-security bureaucracies, such as the FBI, make ephemerally sitting presidents seem not imperial but astonishingly easy to dominate, manipulate and deceive. After Truman announced that he wanted "no Gestapo or secret police" and objected to the FBI's "dabbling in sex-life scandals and plain blackmail", for instance, Hoover "all but declared war on the White House".

The shocking "insubordination" of intelligence agencies towards the presidents they nominally serve is facilitated by the former's closely held possession of embarrassing information. Illegally obtained evidence of reputation-ruining antics permitted Hoover to defy the Kennedys without risking his job, just as it allowed him to charm (rather than merely threaten) presidents from FDR to LBJ with voyeuristic morsels about political rivals culled from his confidential trove. The way disgruntled FBI agents toppled Nixon's would-be imperial presidency by leaking secrets to the press illustrates again how a formal hierarchy of authority can be informally capsized.

But perhaps the deepest reason why presidents have a hard time controlling their national security bureaucracies involves the preventive mission itself. The performance of routine law enforcement bodies can be evaluated by the percentage of committed crimes they have successfully solved. Moreover, their ability to indict and convict depends on their compliance with rules and procedures set by elected officials. By contrast, intelligence agencies cannot be disciplined in this way; nor is any clear method available for evaluating their performance or measuring, in retrospect, the number and gravity of attacks that they have successfully foiled.

Hoover's "alliance" with President Truman's "strongest political enemies in Congress" also illustrates the inadequacy of schoolbook accounts of the American separation of powers. Rather than the legislative and executive branches checking and balancing each other for the sake of the public, what we find is an agency officially located within the executive but that repeatedly colluded with one of the parties inside the legislature to undermine sitting presidents and plot their electoral defeat. "Hoover knew how to use intelligence" not only to root out potential traitors, but "as an instrument of political warfare" to help red-hunting Republicans give grief to allegedly weak-on-communism Democrats such as Truman. Hoover's back-channelling of sensitive information to Richard Nixon, serving at the time on the House Un-American Activities Committee (HUAC), is only the most famous example of the partisan-political game played by an unaccountable executive-branch bureaucracy that had no constitutional right to spend taxpayers' money to help one party depose the other.

Another of Weiner's arresting themes is the way the bureau frequently overemphasised lesser threats and underemphasised greater ones. In the immediate postwar period, the diversion of scarce resources to red-hunting meant the neglect of organised crime. In the late 60s and early 70s, similarly, the "bureau's increasingly relentless focus on American political protests" and "political warfare against the American left" drained "time and energy away from foreign counterintelligence" and counterespionage. A similarly one-sided fixation has emerged in the decade after 9/11. In response to political pressure, the FBI now routinely resorts to what seems like solicitation and entrapment to compensate for an awkward deficit of homegrown Islamic terrorists: "More than half of the major cases the FBI brought against accused terrorists from 2007 and 2009 were stings." The implication, once again, is that the bureau's irrational focus on relatively marginal threats has significant costs, diverting investigative attention from much more serious perils to the country's and the world's wellbeing. For instance: "The FBI's relentless focus on fighting terrorism had an unforeseen consequence. The investigation and prosecution of white-collar crime plummeted, a boon to the Wall Street plunderings that helped create the greatest economic crisis in America since the 1930s."

Weiner's passing reference to a "tug-of-war between security and liberty" brings us to the most pungent question raised by the history he recounts, namely: can the FBI's serial failures (for example, to prevent the Soviet theft of nuclear secrets or the 9/11 attack) really be attributed to the way the bureau's hands have been irresponsibly tied by civil liberties, Congressional oversight, freedom of the press, and the rule of law? In a few casual asides, Weiner suggests that the answer is "yes", that legality and individual rights have imposed a debilitating burden on America's intelligence agencies. He also seems to defend the same thesis from the opposite perspective: "Over the decades, the bureau has best served the cause of national security by bending and breaking the law."

Fortunately or unfortunately, this conclusion is contradicted by almost every page of Weiner's detailed narrative, where the principal obstacles to the bureau's effective use of scarce resources in the defence of national security are meticulously rehearsed. They include a lack of professionalism, cultural insularity, monolingualism, bureaucratic routine, political partisanship, ideological fixation, and a failure of agents to follow up on promising leads. Legality and liberty are barely if ever mentioned in this regard. Not surprisingly, given his last book, Weiner also emphasises the dismayingly dysfunctional consequences of the "fight between the FBI and the CIA", which he describes as "a theatre of the absurd".

The subsection that Weiner devotes to former FBI interrogator Ali Soufan also upends the facile supposition that security is invariably compromised when intelligence agents follow procedures. While Soufan quickly extracted actionable intelligence from captured al-Qaida suspect Abu Zubaydah using well-tested methods of studying the subject's background and building a personal rapport, inexperienced CIA interrogators, who excruciatingly water-boarded Zubaydah 83 times over an extended period, only motivated their helpless captive to fabricate tactically useless lies to stop the pain.

Readers of a comprehensive history such as Weiner's come away with an indelible impression that the FBI chronically misjudges and capriciously ranks the dangers facing the country. For instance, even when "the Communist party was no longer a significant force in American political life  Hoover had to continue to represent the party as a mortal threat". Such misuse of scarce resources cannot be explained by too much liberty but only by the tunnel vision of a specialised bureaucracy that needs to justify its existence to the Appropriations Committee. Hoover was convinced that the American civil rights and anti-Vietnam war movements were orchestrated from Moscow, but he consistently overestimated the danger posed to America by "the American communist underground" because only a fifth column under the control of "the international communist conspiracy" would fall squarely under the domestic jurisdiction of the FBI. Operating without reality checks, or even sanity checks, secretive intelligence agencies are unlikely to provide a rational assessment and prioritisation of the many real but not equally urgent threats to national security.

In 1943, Attorney General Francis Biddle ordered Hoover to destroy the bureau's list of American citizens who, although they had committed no crime, were deemed dangerous and worthy of military detention. (Hoover defied the order, quietly renaming and better concealing the list.) What disturbed Biddle was the impossibility of effectively challenging a conclusion reached within a secretive bureaucracy. The attempt to pick out those who were likely at a later date to break the law naturally led to a search for proxy indicators, which boiled down in practice to speech critical of government policy. As a result, during the Vietnam war, "The FBI found it hard to distinguish between the kid with a Molotov cocktail and the kid with a picket sign."

The institutional shortcomings of the FBI, Weiner is surely right, cannot be traced exclusively to Hoover's personal eccentricities and foibles. That the bureau's troubles have more enduring sources was made clear, for example, by the way Dick Cheney, after 9/11, "renewed the spirit of the red raids" and "revived the techniques of surveillance that the FBI had used in the war on communism". The root cause of the bureau's frequent ineffectiveness also lies deeper than the aversion to self-criticism characterising specialised bureaucracies or the vulnerability of secretive agencies to partisan-political capture. It is trite but true that public officials, like all of us, tend to behave irresponsibly when unwatched. This common failing is exacerbated when national security is at stake, for the simple reason that insecurity is highly emotional, subjective, variable and easy to manipulate for strategic ends. Elected and unelected officials understand that when fear levels are raised  justifiably or not  an insecure public will tend to support national security policies uncritically and will not, until many years have passed, hold its leaders responsible for misconceived actions, including mendaciously justified wars and the arbitrary snaring of confused young men in undercover stings. That such seemingly incurable ills persist well beyond Hoover's long personal shadow is perhaps the most important lesson of this carefully researched study of occult powers inside the most externally powerful modern democratic state.

Stephen Holmes's The Matador's Cape: America's Reckless Response to Terror is published by Cambridge University Press.