John Gray is far too forbearing to tell us that he told us so, but he did. The title of one of his key works indicates his foresight: False Dawn: Delusions of Global Capitalism. What is significant is that this closely reasoned polemic came not from the pen of some hot-eyed zealot of the left or a green-fingered son of Gaia, but from a liberal conservative thinker of a quietist cast of mind, an admirer, albeit in a qualified way, of Margaret Thatcher, a shrewd commentator on the likes of Friedrich Hayek and George Soros, and a dedicated foe of Enlightenment values. He is surely the most incisive political philosopher that we have, and one whose time has, sad to say, definitely come.

Sad, because no one wants to be around when Cassandra's prophecies come true, not even Cassandra herself. Gray excoriates the follies of our globalised world more in sorrow than in anger. He has no grand solutions to offer for the troubles of our apocalyptic age, and urges a programme that is radical only in its mutedness: "Other animals do not need a purpose in life ... the human animal cannot do without one. Can we not think of the aim of life as being simply to see?"

Gray has taught at Oxford, Harvard and Yale, was professor of European thought at the LSE and now writes full time, outside the academy. He has done as much as any thinker to wake up English philosophy from the long sleep of the postwar years, when any suggestion that professional philosophy might have a direct bearing on the conduct of life was likely to be greeted with quiet laughter or a glare of pitying scorn. His books include the wonderfully bracing Straw Dogs: Thoughts on Humans and Other Animals and, most recently, Black Mass, a timely meditation on the mischief wrought by the new Jacobins at work not only in the festering deserts of Arabia but in the corridors of western power. Gray's Anatomy is a generous and well-chosen selection of his work from the 1970s to the present, and is an ideal introduction for new readers and a useful overview for those already familiar with this most elegant, witty, incisive and quietly fierce commentator on our benighted time.

One of Gray's abiding themes is that spilt religion inevitably leads to spilt blood - "modern revolutionary movements are a continuation of religion by other means" - and he sees a more or less hidden religious impulse in all the great secular movements of the modern age, beginning with the Enlightenment. One of the most characteristic and representative essays gathered here, "The Original Modernizers" - a masterpiece of conciseness, wit, insight and lightly worn learning - traces the unbroken seam of positivism that runs from Saint-Simon and August Comte to the present-day architects of the global free market. He writes: "Without realising it - for few of them know anything of the history of thought, least of all in their own subject - the majority of economists have inherited their way of thinking from the positivists." Although Saint-Simon and Comte "envisaged a unified science in which all of human knowledge would be reduced to a single set of laws", the positivists did not aim merely to revolutionise society. "Their aim was to found a new religion. Saint-Simon believed the 'positive doctrine' would become the basis for a new 'church' when all scientists united to form a permanent 'clergy'. He envisaged an assembly of 'the twenty-one elect of humanity' to be called the Council of Newton ... In Saint-Simon's new religion, however, it was not gravity that replaced the Deity. That place was filled by humanity."

Gray has been having fun for years with that poor old crazy coot Comte - one of whose initiatives for a new world order of brotherly love was a waistcoat with the buttons down the back so that it could be put on and taken off only with the help of others - yet he points out too that, for instance, the authors of The Communist Manifesto, the proponents of "modernisation" after the second world war and the theorists of globalisation were alike animated by the positivist creed. With rueful mockery, he notes that "For Saint-Simon and Comte, technology meant railways and canals. For Lenin it meant electricity. For neoliberals it means the internet." The conviction that our own time is at last "modern" and that we are the "last men" is, for Gray, one of the most lamentable of the many delusions that humankind allows itself. We imagine ourselves original yet are mired in the past. He quotes Keynes's apposite insight: "Madmen in authority, who hear voices in the air, are distilling the frenzy of some academic scribbler of a few years back."

Keynes was writing at a time when public policy was governed by outdated economic theories. Today it is ruled by a defunct religion. To link exotic figures such as Saint-Simon and Comte with the vapid bureaucrats of the International Monetary Fund may seem fanciful, but the idea of modernisation to which the IMF adheres is a positivist inheritance. The social engineers who labour to install free markets in every last corner of the globe see themselves as scientific rationalists, but they are actually disciples of a forgotten cult. One thinks of the rascally Talleyrand, togged out as a secular bishop and marching in one of the solemn processions organised by the Jacobins to celebrate the first anniversary of the revolution, meeting Lafayette and whispering to him urgently out of the corner of his mouth: "Please, don't make me laugh!"

Yet despite Gray's disapproval of the splatters of religion that incarnadine our political, social and economic thinking, he is far from being a critic of religion itself. Indeed, he sees religion and poetry as "more realistic guides to life" than a blind faith in science and technology, politics, or "progress" (though, for him, religion is a kind of poetry).

It is not too much to say that Gray considers the Enlightenment to have been little short of a catastrophe, for it was the philosophers, unconsciously pining for the certainties of the old religion, who instituted the notion of the human adventure as an ever-ascending journey towards perfection and worldly redemption. For Gray, the Enlightenment idea of the soul progressing in tandem with technological advances is pernicious. Progress in science is real - painless dentistry and the flush lavatory, he concedes, are certain goods - but spiritual progress is a myth. "Scientific and technological advance has not, and cannot, diminish the realm of mystery and tragedy in which it is our lot to dwell."

It is impatience with and rejection of this sad fact that animates American neoconservatives and Islamist fundamentalists alike. In his book Al Qaeda and What It Means to Be Modern, Gray argues, convincingly, that Islamists are driven more by the positivist strains in western thought than by a longing to return to the simple certainties of a medieval world, and that their "closest affinity is with the illiberal theory of popular sovereignty expounded by Rousseau and applied by Robespierre in the French terror".

Gray is rightly dismissive of contemporary millenarianism which, until very recently, considered that we had arrived at the "end of history" and the dawn of a new age of endless expansion - "the project of promoting maximal economic growth is, perhaps, the most vulgar ideal ever put before suffering humankind" - and sees, in the destructive and exploitative activities of Homo sapiens, an unwilled urge towards our own destruction. He argues for an entirely reformed attitude to the world and our place in it, and above all urges that we relinquish the delusion of progress. In the long essay "An Agenda for Green Conservatism" - which, by the way, every Green politician, and voter, should read - he sets out his case most subtly and persuasively: "The idea of progress is detrimental to the life of the spirit, because it encourages us to view our lives, not under the aspect of eternity, but as moments in a universal process of betterment. We do not, therefore, accept our lives for what they are, but instead consider them always for what they might someday become."

Recognise the truth of this contention and we are on the way to the getting of wisdom.

 John Banville's novels include The Sea (Picador).

This selection from the writings of John Gray spans the last 30 or so years, which is neat, and timely, because those 30 years are starting to look like a distinct chapter in human history, now closed. It began in the late 1970s with the rise of Margaret Thatcher and it ended last September, not with the demise of Lehman Brothers, but with the nationalisation in the US of Fannie Mae and Freddie Mac, two social-welfare institutions that had turned into vehicles of capitalist democratic fantasy.

The fantasy was that anyone could afford a mortgage, regardless of income, enabling the poor to have their own homes, the politicians to have their grateful votes and the moneymen to have pots more money, just so long as everyone wanted it badly enough and no one asked any awkward questions. This was the have-your-cake-and-eat-it era.

Gray is a philosopher who has specialised in asking awkward questions about capitalist democratic fantasies, so this is his time. These essays cover a remarkable range of topics, from Isaiah Berlin to Damien Hirst, from torture to environmentalism. But their unifying theme is that our naive belief in the idea of progress has turned modern life into a constant round of shadow-boxing, as we fail to notice that what we are endlessly fighting is ourselves.

So the current fashion for militant atheism - Dawkins, Hitchens et al - is just as wishful and credulous as the thing it affects to despise, confirming that "the attempt to eradicate religion only leads to it reappearing in grotesque and degraded forms". Likewise, the war on terror is really a war on our own delusions of grandeur, as we discover that new technology and ever-increasing mobility simply give the terrorists more toys with which to play.

Above all, neoliberalism, with its faith in "technological advance that fuels economic development, and economic forces that shape society", is just a mirror of the crass Marxist determinism it thought it had vanquished, just as Marxism was itself a mirror of the Judeo-Christian theodicies it wanted to destroy. We think we are making progress, but all we are ever doing is saying boo to our own ghosts.

Gray has arranged the book thematically, but if it is read in chronological sequence then some distinct shifts in tone and mood start to emerge. The pieces from the 1980s, especially those written as the Soviet Union was creaking on its hinges, are often bluntly admonitory. Look, Gray seems to be saying as he points at the absurd mystifications of Soviet ideology and the even more absurd mystifications of its western Marxist interlocutors, this is where human faith in progress gets us.

But after the end of the cold war, when it is no longer so easy to point in the Marxist-Leninist mirror, the writing becomes more ironic and more resigned. The grand irony is that "a view of the world falsified by the communist collapse should have been adopted, in some of its most misleading aspects, by the victors in the cold war". The problem is that it is hard to get anyone to notice the irony as a new wave of hubristic nonsense about peace, globalisation and apple pie carries all before it.

But then, post-9/11, things change again. In the capable hands of George W Bush and Tony Blair, a naive liberal faith in the value of freedom, and a happy obliviousness to the difficulties of trying to spread it around the world by force of arms, makes it almost too easy to show how neoliberalism turns into a parody of itself. From this point on, Gray's style becomes pithier and more satirical (and it is no coincidence that this is when he started to reach a much wider audience). He teases the liberals with a liberal defence of torture in a manner that is both cruder but also more lighthearted than his earlier parodies of Marxism, also included here.

His 2005 demolition of Thomas L Friedman's globalisation-porn, The World Is Flat, has none of the world-weariness of a similar demolition job on Francis Fukuyama's The End of History and the Last Man from 13 years earlier. By the time we get deep into the first decade of the 21st century, with all its cartoonish absurdities, Gray seems to be really enjoying himself.

What he doesn't resolve, though, is the central tension in his answer to the question of how we can avoid the perils of progressivism. With part of himself, he still clings to his own ideal of a sceptical liberal conservatism, in the spirit of writers such as Michael Oakeshott, with its deep suspicion of rationalist fantasies and its emphasis on poetry, curiosity and a willingness to rub along together.

But with another part of himself he doesn't seem to believe any of this can make much difference. Gray rejects the idea of social progress, but not of technological progress, which he recognises has us in its grip. It's going to take more than poetry and rubbing along together to rescue the human race from the consequences of its own depredations of the environment.

Gray envisages a Malthusian future in which resource scarcity coupled with population growth and technological innovation produce nastier and more frequent wars, as all the "old evils" return and "what we see as unalterable features of civilised life vanish in the blink of an eye". At that point, it will no longer be possible to pretend that we are in control of our destiny, though it may also no longer be worth anyone's while to care.

Gray seems pretty sanguine about this prospect, particularly if it offers the chance of some respite for all the other lifeforms with whom we share the planet and whose destiny we do still have in our hands. His vision of the future is self-confessedly apocalyptic, but his tone is increasingly cheerful, almost playful. The result is a thoroughly enjoyable book, both profoundly bleak - "the coming century looks like being one of wars, massacres and forced migrations, of which the holocausts of our own century are but precursors", that from an essay on green conservatism in 1993 - and curiously bracing.

It's an awful title, though, misleading as well as gimmicky. There is nothing particularly surgical about Gray's analysis - it's much too eclectic for that - and it's not much of a practical guide. A better title might have been The Illusions of Progress, though that's been used before, most famously 100 years ago by Georges Sorel, the French philosopher and prophet of violence, who wanted western civilisation to wake up from its stupor and see itself in the mirror of its own democratic capitalist fantasies.

In truth, Sorel was slightly unhinged and he gained a reputation as the ultimate intellectual gadfly, as he flirted with most of the political fashions of his age, from syndicalism to monarchism. But unlike those who clung to their faith in Bolshevism or nationalism through thick and thin, Sorel saw something of the coming horrors of the 20th century that escaped his contemporaries.

Gray is too polite, too sceptical, too English to be another Sorel, though he has something of the same reputation as an intellectual gadfly. But like Sorel, he sees things that other people don't. That doesn't make him a prophet, however much some of his readers (and no doubt his publishers) would like to treat him as one. He doesn't have any more idea of what's coming next than anyone else does. But it does mean he is more than just a joker or an irritant. The strength of this book lies in the cumulative sense it gives of a culture out of control, unmoored from its historical foundations and untouched by any sense of its own limitations.

To read it in 2009 is to be struck by its prescience. If Gray himself sometimes seems chaotic, certainly by conventional philosophical standards, as he spreads this message, that's a small price to pay.

 David Runciman's most recent book is Political Hypocrisy: The Mask of Power From Hobbes and Orwell and Beyond (Princeton).

John Gray: A life

Born 17 April 1948 in South Shields, Tyne & Wear.

Educated Exeter College, Oxford, where he read PPE.

Career 1976: fellow in politics at Jesus College, Oxford. 1996: professor of politics at Oxford. 1998: professor of European thought at London School of Economics. 1998: False Dawn: The Delusions of Global Capitalism. 2002: Straw Dogs: Thoughts on Humans and Other Animals. 2003: Al Qaeda and What it Means to be Modern. 2007: Black Mass: Apocalyptic Religion and the Death of Utopia.

He says: "What if we give up the empty hopes of Christianity and humanism? Once we switch off the soundtrack - the babble of God and immortality, progress and humanity - what sense can we make of our lives?"

They say: "Gray's elegant anathemas on everything from the Enlightenment philosophies to, God help us, Bush and Blair make for some of the most bleakly invigorating writing around" - John Banville