The "human tendency" to be seduced by coherent narratives and charismatic narrators, including ourselves, is the screw that turns in almost all of Janet Malcolm's work, from her early writing on psychoanalysis to 2007's Two Lives: Gertrude and Alice. Malcolm has made a career out of exposing our impulse towards self-deception, particularly when practised by those of us in pursuit of "pure" representations of reality. What she reveals is the reliance of factual accounts  whether produced by reporters, biographers, diarists or criminals mounting their own defences  on the conventions of fiction. The moment when we affirm the objectivity and accuracy of our vision is the moment we reveal our blind spots, to which no person or profession is immune.

In her newest book, Malcolm adds trial lawyers to the list of jobs implicated in the universal tendency to mishear, fail to see and misunderstand "so that the stories we tell ourselves will add up". A few pages into Iphigenia in Forest Hills, Malcolm's account of the 2009 murder trial of Mazoltuv Borukhova, a 35-year-old doctor accused of hiring co-defendant Mikhail Mallayev to murder her estranged husband, she quotes from prosecutor Brad Leventhal's opening statement: 28 October 2007 was "a bright, sunny, clear, brisk fall morning, and on that brisk fall morning a young man, a young orthodontist by the name of Daniel Malakov was walking down 64th Road in the Forest Hills section of Queens county . . . With him was his little girl, his four-year-old daughter, Michelle." Michelle was still with Malakov a few moments later, when he was shot and killed.

For Malcolm  who brings to language the bright, penetrating attention of the English professor or the analyst, alert to telling turns of phrase and suggestive details  the style is always the thing. The "formidable" Leventhal opens his case as if it were a bedtime story, ornamenting his facts with the kind of rhetorical flourishes (vivid adjectives, incantatory repetition) familiar from fiction, and thus imbuing them with a self-sustaining logic, of the kind the phrase "once upon a time" confers on fairy tales. There is the suggestion that every event, no matter how implausible, is held together by some invisible explanation, which the storyteller will ultimately reveal.

The "mythic underpinning" in the prosecution's case was Borukhova's alleged desire for revenge. Three weeks before the murder, a state supreme court ruling had removed Michelle from her mother's custody and sent her to live with Malakov. An essential, unspoken feature of Leventhal's portrait of Borukhova as "the archetypal avenging murderess" was her ethnicity. Like Mallayev and Malakov, Borukhova  an Uzbek immigrant  is a Bukharan Jew. Sixty thousand members of the sect, which originated in central Asia, now reside in the US, most of them in the part of Queens where Borukhova lived. Another Soviet immigrant tells Malcolm that Bukharans are stereotyped as "alien and not altogether civilised  savage, tribal people, capable of violence, even of murder". Borukhova's relatives read prayer books in the courtroom and refused to speak to journalists.

Like fiction, legal cases aim for verisimilitude, which is not quite the same as the truth. "If any profession (apart from the novelist's) is in the business of making things up," writes Malcolm, "it is the profession of the trial lawyer." The image of Borukhova as the spirit of maternal vengeance was the glue with which Leventhal pasted incoherent scraps of information  among them a garbled and possibly mistranslated recording of a conversation between the defendants, forensic evidence that failed to meet the National Research Council's standards for scientific rigour, and nearly $40,000 in deposits into Mallayev's bank account that could not be traced back to Borukhova  into a coherent dramatic arc. His creation proved convincing. In March 2009, the jury found both defendants guilty on all counts of murder in the first and second degree.

What, wonders Malcolm, had Borukhova "misunderstood about her new country that set her on her blundering course toward Strauss", the judge who removed Michelle from Borukhova's care, and ultimately towards trial judge Robert Hanophy, who has the "faux-genial manner that American petty tyrants cultivate"? The answer is perhaps the value the American legal system places on intelligibility. Ambiguity is the enemy, and anyone who does not present herself as a like- able or coherent character is doomed.

Unwilling or unable to learn the native lines, Borukhova repeatedly turned in unsatisfying performances, first in the family court and later during her trial. Both the costume  long skirt, white jacket  and the delivery were wrong. Instead of addressing the jury, "she kept her head high" and her gaze fixed on the interrogator standing before her. "She looked," writes Malcolm, "like a captive barbarian princess in a Roman triumphal procession." Borukhova's failure to follow the standard script  her implacable, inscrutable "otherness"  provoked anger and suspicion. "She didn't seem upset," said one juror. "If you were innocent and being tried for murder, you'd be upset."

During the sentencing phase of the trial, Leventhal reads to the jury a letter from Malakov's nephew, who recalls his uncle speaking fondly about Michelle while eating a pomegranate. Malcolm's interpretation is brief but astonishing: "Of course he was eating a pomegranate." Under her brilliant gaze, a seemingly incidental detail shines suddenly with meaning. Gurov's watermelon, Oblonsky's pear: "It is in the blood of Russian storytelling to take note of the fruit." And it is the nature of the American legal system  where a trial is "a contest between competing narratives"  to reward those who speak its language. Hanophy cites the New Testament in a remark addressed to Mallayev, who is Jewish. Then he sentences both defendants to life in prison.

Amazingly, Hanophy is not the most unreasonable of the people who had a hand in determining Borukhova's fate. That superlative might go to David Schnall, Michelle's legal guardian. In a long interview with Malcolm, Schnall  who despises Borukhova  reveals his belief in "ominous conspiracy theories". (The government knew about 9/11 and Katrina ahead of time, "We funded the Soviets," and so forth.) After their conversation, Malcolm  who has previously described "the 'I' character in journalism" as "almost pure invention . . . an emblematic figure, an embodiment of the idea of the dispassionate observer of life"  does something she has never done before. Notifying Borukhova's lawyer, she enters the story she is reporting as "a character who could affect its plot".

In The Journalist and the Murderer, Malcolm observed how, in comparison with fictional characters, "real people seem relatively uninteresting . . . because they are so much more complex, ambiguous, unpredictable and particular". Psychoanalysis "attempts to restore to the neurotic patient the freedom to be uninteresting . . . It proposes to undermine the novelistic structure on which he has constructed his existence." The same might be said of Malcolm's own work. The permission she grants herself in Iphigenia  the permission to step out of the role of journalist  is an example of the freedom she bestows on all her subjects. Known for its lucid ruthlessness, Malcolm's intelligence is also a generous one. By dismantling self-delusion, she liberates the self. She allows people such as Borukhova their essential human mystery, observing always the sanctity of our heart's hidden chambers, which are locked even to ourselves.

There are only two writers whose work I can say I have read in its entirety. The first is Jane Austen. The second is Janet Malcolm, who is a staff writer at the New Yorker, and whose subjects have included, over the years, psychoanalysis ("the impossible profession"), Sylvia Plath, Anton Chekhov and Gertrude Stein. Malcolm's masterpiece, though, is The Journalist and the Murderer, a brilliant and pitiless examination of journalistic ethics built around the lawsuit filed by Jeffrey MacDonald, a convicted killer, against Joe McGinniss, author of a book about his crimes. "Every journalist who is not too stupid or too full of himself to notice what is going on knows that what he does is morally indefensible," wrote Malcolm in an opening sentence that caused a sensation in the tiny, self-referential world of posh American journalism. Yet a full two decades on, she is still at it. The truth might be elusive. It might even  like psychoanalysis  be "impossible". But this doesn't make the idea of pursuing it any easier to resist; some of us are just born beady. As she puts it with characteristic understatement in a rare interview in the latest Paris Review: "Journalism, with its mandate to notice small things, was always congenial to me."

Malcolm's new book, Iphigenia in Forest Hills, tells the story of the 2009 joint murder trial of Mazoltuv Borukhova, a 34-year-old doctor, and Mikhail Mallayev, her 50-year-old cousin by marriage whom she hired to shoot her estranged husband, Daniel Malakov, in 2007. (The trial took place in Queens, where Malcolm followed its every twist for the New Yorker.) Trials, of course, are about finality: the truth is established, a gavel is banged, everyone goes home happy. But Malcolm is not so easily pleased. It's not particularly that she doubts the guilt of those in the dock. It's more that, in this courtroom, motives are so confusingly muddy. Even the judge, Robert Hanophy, behaves badly, making his bias in favour of the prosecution too clear, it is suggested, because he has a Caribbean holiday booked, and is determined to make his flight (as the trial draws to a close, Hanophy forces Borukhova's lawyer to prepare his summation overnight; the prosecutor has the whole weekend to wrestle with his).

From her reporter's seat, Malcolm observes that a trial is merely "a contest between competing narratives". The most consistent story wins, not the truth: "In life, no story is told exactly the same way twice. As the damp clay of actuality passes from hand to hand, it assumes different artful shapes. We expect it to. Only in trials is making it pretty equated with making it up." On a landing at the Queens supreme courthouse, she notices a wondrous mosaic: a complex allegory over which hovers a tipped scale of justice. Staring at it, she sees that the pan that is low on the ground is empty, while the pan that is high in the air holds a book. "Is this a comment on the weightlessness of the law? Or is it just [the artist] exercising his gravity-defying artist's imagination...?" It's in this manner, eyebrows only slightly raised, that she sets about sowing her insistent little seeds of doubt.

Malcolm has written a fascinating story, if not exactly a gripping one; indeed, she eschews dramatic tension at every turn, even when the verdict is returned (which makes me think that it was on the encouragement of Malcolm that her publisher produced such a frustratingly ascetic book; not a single photograph of the beautiful and baffling Borukhova is included). Malakov was gunned down in broad daylight in a children's playground where he stood with his four year-old-daughter, Michelle, but the bloody logistics of this interest Malcolm far less than the backstory to the case, not to mention the many smaller questions that are raised by the trial, only to hang over it, unsolved (for instance: Borukohova secretly taped even the most anodyne conversations with Mallayev, and never explained why).

The backstory has mostly to do with "otherness". Borukhova is a Bukharan Jew from Uzbekistan, and thus is a member of a particularly inward-looking sect, one viewed with suspicion even by other Jewish immigrants. Seemingly pious, she grows thinner and thinner during the trial (the correct kosher meals are not, she insists, available in prison).

Malcolm, unlike just about everyone else involved in the case, has a measure of sympathy for her. But she recognises, too, her contrary nature, her stubbornness, her unreadability (which a jury will take to be coldness). It's also her contention  or her instinct  that the die was cast when Borukhova and Malokov separated and began a custody fight for Michelle. The child's court-appointed guardian, David Schnall, took a dislike to Borukhova, with the result that she lost this battle, even though she had accused her husband of molesting Michelle. (The prosecution disbelieved these accusations, and argued that her motive for the murder was simply to get Michelle back). This startling ruling leads Malcolm uncertainly into the relatively new field of children's rights. Her tentative conclusion? The concept is a smokescreen, "a mantra invoked by adults to help them in their own fights with other adults".

Michelle, then, is the Iphigenia of the title (in the Greek myth, Iphigenia was sacrificed by her father, Agamemnon, so that his fleet might sail to the Trojan war; she is avenged by her mother, Clytemnestra, who stabs and kills her husband on his return). In the book's most unnerving scene, Malcolm passes Michelle in the street. She is with a group of her paternal relatives. Her face is "distorted by mirthless laughter".

What does all this add up to? Something of a curiosity, I think. This is not Malcolm's best book. Her story is in essence parochial, and its preoccupations are with the quotidian aspects of justice as much as the human. She has chosen to write about the trial of someone who is guilty, not to investigate some terrible miscarriage of justice (though Borukhova, represented by the celebrity lawyer Alan Dershowitz, is currently appealing against her sentence). Nevertheless, it does its work. The unease grows, like a shadow, with the result that her essay's after-effect is entirely disproportionate to its brevity. The disquiet stays with you. It's there in the pit of your stomach.