Looking back over Jeffrey Eugenides's first two novels, I wondered if a part of their enormous appeal might have been the way they brought together two apparently incompatible registers: the dank morbidness of the subject matter, and the graceful exuberance of the style. Mass self-slaughter in The Virgin Suicides, incest and hermaphroditism in Middlesex: in both cases, the elements of what might have been merely freakish narratives were transformed by a combination of witty, vigorous prose and a cinematic sense of social and historic context into something unexpectedly capacious and pleasurable.

The Marriage Plot largely (though not entirely) dispenses with the morbid element. Its cast consists mostly of bright, go-getting young Ivy Leaguers, and its storyline follows their love entanglements and spiritual crises during the early 1980s as they pursue and escape each other through a variety of colourful locations that stretch from Cape Cod to Monte Carlo to Calcutta. With one exception there is nothing seriously the matter with any of them.

It's customary to cheer when an author moves outside his comfort zone, but I'm not sure it was such a great idea in this case. In tilting the focus so emphatically towards the wholesome and ordinary, Eugenides seems to have restricted his access to his own considerable powers. The lively intelligence of the earlier books has little to grapple with in these mostly unremarkable characters as they make their intellectual and geographic grand tours, and consequently much of the writing veers between effortful smartness and a kind of half-hearted blah. "Paris was a museum displaying exactly itself"; "Calcutta felt like the first real place he'd been"; "Reading a novel after reading semiotic theory was like jogging empty-handed after jogging with hand weights"; "The mind of Thomas à Kempis, the author of The Imitation of Christ, was difficult to connect with", go some fairly typical lines. They don't, to me, suggest an imagination on fire.

The story itself, a post-modern take on 19th-century romance, is built around three characters who meet at Brown University's semiotics class in 1981. Madeleine, the ingenue, is an English major from a prosperous Wasp family with a nice home in the suitably twee-sounding town of Prettybrook, New Jersey. Mitchell and Leonard, the men she must choose between (I'm simplifying, but it boils down to this), come from humbler backgrounds and study religion and science respectively. Mitchell, well-behaved and presentable, plays the Mr Knightley role: patiently besotted with Madeleine, who flirts with him occasionally but prefers to think of him as her treasured friend. Leonard is the wild card: a 6ft 3in polymath who lives in squalor, writes brilliantly (so we're assured), is "historically hilarious", irresistible to women, and savagely self-destructive.

He also turns out to be clinically bipolar (Eugenides's one concession to his old interest in pathological conditions) but Madeleine marries him all the same, and therein lies the book's main source of tension: how is "positive, privileged, sheltered, exemplary" Madeleine, who "instinctively avoided unstable people", going to acquit herself, having chosen this loose cannon for her mate? Will she stay the course? Will the long-suffering Mitchell, trying out various forms of religion and do-gooding (including a stint with Mother Teresa in India) while he endures his rejection, begin to look more tempting when Leonard goes off his meds and starts cracking up?

Framing and shaping this story is a sustained inquiry into the great themes of love and marriage, and the question of how they might be rescued for contemporary literature. Madeleine discovers Roland Barthes's A Lover's Discourse, and her growing infatuation with Leonard is punctuated by apposite quotations from this famous deconstruction of yearning, while the novel itself squares up to the classics of marital fiction, notably Emma and The Portrait of a Lady.

"What would it matter whom Emma married if she could file for separation later?" Madeleine's professor asks. "How would Isabel Archer's marriage to Gilbert Osmond have been affected by the existence of a prenup?" This is the task the book sets itself: to see if a viable "marriage plot" might be constructed around a feminist-era heroine for whom marriage no longers means an irrevocable surrender of person and property.

It's a likably quixotic challenge; but I think it's premised on a misconception about those earlier books. What makes them great, surely, isn't just that marriage back then was a much higher-stakes game than it is now, but that their heroines are explored at such astounding emotional and moral depth, and articulated with such wit and precision, that just about anything that happens to them becomes rivetingly consequential. They would be no less interesting to us, I suspect, if the options of divorce and prenups were available to them. By the same token, I don't think the removal of those options would make Madeleine and her suitors any more interesting than they are, which is to say: moderately.

What do we ever really know about these three? We know about the books they read, the music they listen to, the things they consider cool or uncool. We know that they look like Audrey Hepburn, Jack Nicholson and Tom Waits. We're given plenty of interesting specialist information about their more arcane pursuits: Leonard's yeast experiments at the lab on Cape Cod where he gets a fellowship, for instance, or Madeleine's progress through the lit crit fads of the 80s. But information isn't the same as insight, and it seems to me that a couple of crucial things are missing.

One is the presence of some larger scale of values by which to measure these characters, than the cool/uncool scale by which they measure themselves. There's a fair amount of irony at the expense of undergraduate pretension and holier-than-thou posturing, and some of it spills over to the three protagonists. But at the same time the book wants you to take their yearnings and dilemmas seriously, and the result is a curious tonal indecisiveness. The presentation of Mitchell, especially, is so uncertainly poised between solemn and mocking that it's impossible to tell whether his gestures  hanging a cross round his neck, intoning the Jesus prayer  are intended as those of an earnestly agonised truth-seeker or of a slightly narcissistic sap. Either way, he's not a character you'd wish on any heroine.

The other missing piece, more important perhaps, is the kind of action that gives dramatic reality to the complexities of a character's inner life. This is less true of Leonard, whose volatility does create some vividly dynamic scenes (he seems to be modelled in part on the late David Foster Wallace, with his bandana and chewing tobacco). But the other two seem under-realised. There's a moment when Mitchell gets a letter from Madeleine and thinks that "she may have looked normal on the outside but once you'd seen her handwriting you knew she was deliciously complicated inside". But there's little to substantiate this; no equivalent of those precisely gauged moral interactions that make, say, Emma Woodhouse so complexly mesmerising (think of how her fatally entangled virtues and vices manifest themselves in her impact on her protégée, Harriet Smith).

What we actually see of Madeleine seems rather ruthlessly uncomplicated, in fact, and furthermore the book as a whole seems to vindicate her chilly instinct to avoid "unstable people". If she learns anything, it's that it's best not to stray too far from Prettybrook. Her tale makes a quick and sprightly read, but it's not  to borrow a word from the world of its literary predecessors  especially "edifying". And it certainly doesn't play to the idiosyncratic strengths of its gifted author.

James Lasdun's It's Beginning to Hurt is published by Vintage.

The first thing we know about Madeleine Hanna is her library. "To start with, look at all the books," Jeffrey Eugenides suggests of his heroine, and proceeds with a tracking shot of her shelves: "A lot of Dickens, a smidgen of Trollope, along with good helpings of Austen, George Eliot and the redoubtable Brontë sisters the Colette novels she read on the sly the first edition of Couples, belonging to her mother, which Madeleine had surreptitiously dipped into back in sixth grade"

Madeleine Hanna is an English major at Ivy League Brown University in 1982. Her thesis is concerned with "the marriage plot" as it existed in the 19th-century novel and the way, with marriage having lost its gravitas in her era of quickie divorces and prenups, the novel itself has been diminished. Much as Madeleine may believe this thesis as a critic, however, as a 20-year-old woman there is much about her life that seems Victorian. She is, cliche of cliches, caught in a love triangle herself, torn between two fellow undergraduates: the charismatic and depressive Leonard Bankhead on the one hand and the studious and spiritual Mitchell Grammaticus on the other. Her heart shouts Leonard (most of the time); her head and her Waspish parents murmur Mitchell.

As well as locating the style of Madeleine's dilemma, Eugenides's opening tracking shot of those library shelves is also a nudge to the reader: this is the territory we are in. And here is the challenge he sets himself: to breathe new life into the redundant marriage plot; to create a properly absorbing love triangle, not only as pastiche or irony, but as something as full of life as those books on Madeleine's shelf. In the 400-odd pages that follow he mostly succeeds in this aspiration, both knowingly and brilliantly.

This is Eugenides's third novel. It is 18 years since the precocious and perfectly formed The Virgin Suicides marked him out as a writer who would always be required reading. In between times, the fabulous family saga Middlesex, which, along the way, told of the unlikely coming of age of a hermaphrodite in Michigan, became a huge bestseller and Pulitzer prize-winner, without ever seeming entirely coherent.

The tight plotting and internalised psychology of this new novel, allied to the full sweep of ideas and social observation and quiet comedy that characterised Eugenides's earlier works, are signs of a new maturity. In the generosity and nuance of his characters and paragraphs, you are reminded of the Jonathan Franzen of The Corrections. Like that novel, this one acknowledges the brio and experimentation of American writers such as Thomas Pynchon or Don DeLillo or the late David Foster Wallace; it takes on board some of the philosophical caveats to "conventional" social realism, but does it anyway.

The book begins not only with Madeleine's love troubles, but also with the fact that those "love troubles had begun at a time when the French theory she was reading deconstructed the very notion of love". Her heart breaks even as she sits in seminars discussing Derrida on the bogus nature of romance and sentiment in life as well as literature. When she finally tells the manic-depressive polymath Leonard that she loves him, after a perfect day, he gets up from the bed in his ratty student room and starts quoting Barthes at her: "The figure [je t'aime] refers not to the declaration of love, to the avowal, but to the repeated utterance of the love cry. Once the first avowal has been made 'I love you' has no meaning whatever..." Quite understandably, she immediately chucks the book at him and chucks him. And then immediately regrets it (but not as much as Leonard, who ends up hospitalised with psychotic depression).

Eugenides inhabits the minds of each of the points of this love triangle in turn. The Greek-American Mitchell Grammaticus, who, like the author himself in his youth, volunteers as a gap-year helper with Mother Teresa in Calcutta as he searches conflictedly for spiritual enlightenment, seems very close to home for Eugenides. Madeleine is entirely believable as the ambitious, beautiful and mostly moral young woman slightly out of step with the freedoms of her time.

Leonard Bankhead, though, is both the wild card and the proper heart of his novel. Eugenides handles the difficulties of describing the manic phases of depression from within  through Leonard's eyes  with rare skill, the gradations of self-delusion measured almost incrementally in his prose as Leonard seeks to take control of his teeming thoughts and reduce his lithium dosage. The reader is asked to become just as hyper-aware of the character's mental state as those, particularly Madeleine, who try to protect him from himself. It is a highly affecting portrait that brings to mind some of those Salinger stories that walk the line between reality and mania. There were times in the book when this reader was so alarmed at the first hints of impending lapses in Leonard's behaviour that he felt like calling out his alarm to the other characters.

As he delineates these fracturing lives, Eugenides also pursues cogent inquiries into religion and philosophy and sexuality as his young trio try to make sense of things. (Leonard, who takes up postgraduate work in biology at a genius lab in Cape Cod, also brings with him some scientific insight, notably into the mating rituals of microscopic organisms.) Though the absence of email and mobile phones allows the author to explore the proper frustration and novelistic suspense of airmail letters and poste restante boxes for perhaps the last time, there is much that feels contemporary about the moment of the book: Madeleine and the others are graduating into deep recession; there seems no new idea under the sun.

For much of the novel, the suspicion is that while the disturbing pull of lust and love thrums beneath all of Mitchell's distracting quest for spiritual truth, and Leonard's unhinged normality, and Madeleine's pursuit of happiness  that the marriage plot is still the most important plot of our lives. The novel's resolution allows that possibility, but also reminds us that we are not quite Victorian after all.