There are few more awesome sights in modern, globalised intellectual life than a north American with a hot idea to sell. Milton Friedman and monetarism in the 70s; Francis Fukuyama and the end of history in the 90s; Malcolm Gladwell and his more recent pop-sociology  all have tirelessly publicised their catchphrases and key concepts until they achieved chattering-class ubiquity.

David Brooks is more like Gladwell than Fukuyama or Friedman: an upmarket journalist, a political columnist for the New York Times rather than an academic, more a synthesiser and populariser of other people's ideas than an original thinker. Like all three, he knows how to convey sometimes complex notions in primetime-ready language, and he has a good sense of timing.

The Social Animal is about social mobility, a key concern for many political people nowadays, but especially for rightwingers like Brooks, as the realisation belatedly dawns on them that the great free-market liberalisation they have unleashed over the past 30 years has not produced the fluid, meritocratic societies they hoped for. Last month, the British leg of Brooks's promotional tour for this book included widely noted meetings  very useful publicity about publicity  with both Ed Miliband and David Cameron. Yet The Social Animal will also appeal to less political readers. With its close, vivid focus on the trajectories of a few individual lives, it echoes the concerns of two fashionable recent novels, Jonathan Franzen's Freedom and Jennifer Egan's A Visit from the Goon Squad.

This is a highly ambitious volume. "Ever since college," the 49-year-old Brooks writes, "I've been interested in research about the mind and the brain . . . The people studying [them] are producing amazing insights about who we are, and yet these insights aren't having a sufficient impact on the wider culture. This book is an attempt to do that . . . to integrate science and psychology with sociology, politics, cultural commentary, and the literature of success."

To bring order to this messy-sounding fusion, he invents two characters called Harold and Erica, both contemporary Americans. Harold is "a popular, athletic high-school boy who also showed flashes of idealism". White, middle class and slightly under-motivated, he has a comfortable, loving upbringing followed by an intermittently fulfilling career as a management consultant, then a historian, then a think-tank intellectual. His wife Erica is, in predictable ways, very different: half Chinese-American and half Mexican-American, raised in poverty and relentlessly driven. She sets up her own company, moves on to reform a giant conglomerate, and ends up a big player in Washington politics.

As Brooks follows Harold and Erica from childhood to retirement, and describes the other people they interact with, he weaves in digressions on everything from IQ to the unconscious, toddler development to school discipline, management fads to modern political campaigning. The book grows into a strange hybrid: part science primer, part polemic, part self-help, part satire and part melodramatic novel. The tone shifts wildly: from wide-eyed wonder at the discoveries of science to world-weariness at the current state of western political discourse. The factual sections, which draw hungrily on scores of academic and more popular sources, are sometimes deftly integrated into the story, and sometimes not. The fictional sections are sometimes delicately drawn, even moving, and sometimes embarrassingly schematic.

Nevertheless, a central argument emerges. Brooks believes that success and happiness, and the kind of politics that make these possible for the greatest number of people, depend greatly on an understanding of "the inner mind  the unconscious realm of emotions, intuitions, biases, longings, genetic predispositions, character traits and social norms". Yet for the last three centuries, he argues, many politicians have mistakenly seen "the unconscious as [full of] primitive vestiges that need to be conquered in order to make wise decisions", and have instead concentrated on "the surface level" of human preoccupations, such as "wealth, prestige, worldly accomplishments." Thus, often "government had tried to fortify material development, but had ended up weakening the social and emotional development that underpins it."

When Brooks suggests that the state should do less, and that, left alone more by government, people become more collaborative  "social animals"  it is easy to see why Cameron and the other increasingly beleaguered "big society" enthusiasts in Westminster are said to be very keen on this book. Like Phillip Blond, author of 2010's similarly talked-up Red Tory, Brooks does also blame business behaviour and free-market economics for making the west, in their view, so unhappily individualistic and materialistic. But as with Blond, you wonder why it has taken him so long to notice that today's capitalism can be bad for your health.

In some ways, the book's most original and powerful passages are not about success but about its costs. Erica, excited to be about to go to university, is criticised by many of her relations, none of whom have degrees and who see her social mobility as a form of social abandonment. When Harold and Erica work too hard or too out of rhythm with each other, their home and their relationship begin to fall apart. Given the state of most western economies, "how failure happens" might have been an even better subject for Brooks.

But like most mainstream commentators and politicians, he avoids saying that social mobility can be downward as well as upward, or that the latter for one person often means the former for someone else. A similar rigid optimism governs his use of science: there is little sense here that its findings can be provisional, or seemingly patternless, or contradictory. Instead, the workings of "the inner mind" and its political and social implications are presented via Harold and Erica's busy lives as a challenging, but ultimately manageable and satisfying jigsaw.

The solutions offered here to reconnect politics with human nature are, like those of the coalition and many post-Reagan/Thatcher conservative thinkers, rather on the undercooked side. As well as a smaller state, Brooks wants schools to be stricter (surely an extension of state power) and individualism to be valued less, while the communal is valued more  which just sounds like old-fashioned, village-green conservatism. Ancient Tory heroes such as Adam Smith, Edmund Burke and Benjamin Disraeli are all cited.

For all its faults, though, this book has a sense of curiosity, a warmth, and a happy ending rare in political literature. You can see why it's selling and being talked about. But there's no great breakthrough here.

Andy Beckett's When the Lights Went Out: Britain in the Seventies is published by Faber.

David Brooks describes himself in this book as someone who writes about politics and policy. That is as true as far as it goes. An op-ed columnist for the New York Times, and a regular on PBS NewsHour, he can hold forth with the best of them on the weaknesses of the latest policy announcement, the character flaws of presidential wannabes, and the merits of this or that electoral strategy.

A few things, however, mark Brooks out as a bit special  things which ensure that his book will be widely read in Washington, Whitehall and Westminster. First there is his eye for developments in the sciences and their relevance to politics and policy. So far so wonkish. Then there's his still rarer ability to see American  and to some degree western  culture from a distance and in doing so cast new light on social trends and political issues.

This latter ability was put to good use in Brooks's decade-old bestseller, Bobos in Paradise, a finely observed, witty look at the values and aspirations of the "bourgeois bohemian" cultural, political and business elite that came of age in the 1990s  the generation of Madonna, Steve Jobs and Obama. Brooks, who is a member of that much depleted species the Moderate Republican, had some fun at the expense of the conceits of the US's costal elites  their obsession with working out, their love of extravagantly utilitarian kitchens, their horror of formality. But it was not an unkind portrait: Brooks understood the "bourgeois bohemians" as giving expression to contradictory but quintessentially American yearnings for freedom and community, wordly success and spiritual transcendence, sensual pleasure and self-denial.

If Bobos in Paradise had "state of the nation" ambitions, The Social Animal attempts something even grander. The idea is to draw on new scientific thinking to describe what makes for success and happiness in life. And taking its inspiration from Rousseau's novel of ideas, Emile, the book adopts a narrative form, following the life-course of a successful and ultimately "enchanted" middle-class American couple, Erica  a Chinese-Latino who rises from poverty to become a business leader and Washington bigwig  and her husband, Harold, an altogether more cerebral historian. "I use them," Brooks writes, "to describe how genes shape individual lives, how brain chemistry works in particular cases, how family structure and cultural patterns influence development."

There are already plenty of guides to success and happiness out there, but this book is defined against them in a couple of ways. First, Brooks claims that most life manuals greatly exaggerate the power of reason and will to determine our fate. In truth, our views and decisions are largely shaped through unconscious or barely conscious forces  deep impulses, emotions, and character traits. It's not, says Brooks, that we are mere victims of our unconscious selves. If we are shaped by the interplay of our genes, culture, upbringing and education, and the institutions and networks in which we live and work, we can in turn shape at least some of these. What we can't do is master them. The art of living well is to know how to steer our natures, and slowly remodel our characters.

Second, most manuals present success as a matter of material wealth and status. But in this, Brooks believes, they underestimate the vital role of family, friends and larger communities. All the evidence is that trust and reciprocity are as important as wealth and prestige to our wellbeing.

The Social Animal is an odd beast of a book with a slightly arbitrary quality. It is never quite clear on what grounds Brooks has decided to explore the implications of some new ideas and not others, other than that they confirm his own views and can be worked into his narrative. Indeed, his rather casual use of academic research sits strangely with his avowed respect for science. There are other tensions. Brooks is impressed by the evidence marshalled in Wilkinson and Pickett's The Spirit Level, and elsewhere, that "the mere fact of being low on the status totem pole brings its own deep stress and imposes its own psychic costs". Yet he remains an almost unqualified meritocrat, arguing that the great challenge for government is not to promote greater equality but to make it easier for people to rise from one class to another.

Despite such flaws, though, this is a spirited and engaging book, true in its ambition and liveliness to the spirit of Rousseau. Brooks has no delusions about his skills as a novelist and his two protagonists are deliberately thinly drawn. Yet the partly fictionalised narrative, allowing for drama and satire, does help the journey along. And there are real insights on offer. Brooks writes excellently, for instance, on the failure of a lot of traditional policy-making to work with the contours of human nature (much 20th-century urban and suburban planning being a good example) and the need for a focus on policies that strengthen "character" and life skills  especially for poorer families left behind by deindustrialisation and rising inequality.

At its best The Social Animal offers a sense of the very real limits of the assumptions that we westerners bring to life, and the possibility, generally so hard to see, of radically better, more "realistic" ways of living.