When you call a thinker a "conservative communist", you sound as if you are making a weak joke. To understand the late Eric Hobsbawm's peculiar genius, however, you must see him as just that, and accept there is no contradiction. Hobsbawm stayed loyal to the Soviet disaster to the very end. But long before the Berlin Wall fell, he told the British left that socialism was dead and put his formidable authority behind the movement that led eventually to Tony Blair. In theory, he believed in the overthrow of the British state. In practice, he accepted that most refined of honours, the Order of the Companions of Honour, from no lesser personage than Her Majesty the Queen.

Fractured Times shows this revolutionary traditionalist at his best. It is an account of the collapse of the high bourgeois culture of the 19th century, and an examination of the ruins it left behind in the 20th century. He loved them both, but understood why they could not last. All the certainties of the 19th century turned out to be lies. Instead of progress there were total wars and genocides that mocked liberal optimism. Instead of a rational science, there was quantum physics, which no one, not least quantum physicists, could understand. The scientist JBS Haldane, a comrade of Hobsbawm on the Marxist left, who is celebrated in these pages, suspected in 1927 that the universe would turn out to be not only "queerer than we suppose, but queerer than we can suppose"  his suspicion has been vindicated. Above all, mass consumer culture and democracy have undermined old elite tastes and certainties.

Hobsbawm can never see them coming back. No historian was better at deploying a killer fact to make an argument stick in your mind. As he makes his case that classical music is now a museum art form, he says "of the 60 operas performed by the Vienna State Opera in 1996/7 only one was written by a composer born in the 20th century". As that statistic implies, the "revolt against tradition" of what we can loosely call the modern movement, even if it is now very old, failed. The artisan crafts of classical music, fine art, sculpture, jazz and, Hobsbawm believed, maybe rock'n'roll too, are for ageing audiences in the wired 21st century.

None of Hobsbawm's excuse-making for communism, that other revolt against all traditions, appears here. Instead there is a moving reflection on the life of Karl Kraus, the Viennese writer who saw the horrors of the 20th century before anyone else, and satirised the dying Habsburg empire with brilliant cruelty. You can satirise and protest in half-free societies like Franz Joseph's Austria or Brezhnev's Russia, says Hobsbawm. But Kraus said of Hitler "on the subject of national socialism nothing occurs to me", and the same satirical silence falls over Stalin's Soviet Union. "To this day," Stalin's old supporter accepts, "no one makes fun of it, not even in retrospect."

Yet regret at the passing of his certainties still lingers. Writing the introduction just before he died, Hobsbawm says he was, "looking forward with more troubled perplexity than I recall in a long lifetime, guideless and mapless, to an unrecognisable future." Really? The world is more troubled now than during the battles between dictators of the 1930s and 1940s? Or during the "mutually assured destruction" of the cold war? I do not think that this is just the pessimism that comes with age. The American economist Brad de Long was the first to notice that for as long as the Soviet Union survived, Hobsbawm believed that humanity was progressing in some form, as a Marxist should. Once it was gone, despair engulfed his writing. This rings true to me. My grandfather was Hobsbawm's mentor in the British Communist party. He died before the Soviet Union did, but I noticed that his friends just gave up once it had gone.

There is another problem. Fractured Times is not a fully worked book but a collection of essays from the 1960s on. I wish Hobsbawm had found the time to reconsider some of his judgments before he died. He repeats his beautifully nonchalant line, "Why brilliant fashion designers, a notoriously non-analytic breed, sometimes succeed in anticipating the shape of things to come better than professional predictors, remains one of the most obscure questions in history".

That was once true. While the old 19th-century order carried on as if nothing could change, artists, including fashion designers, somehow anticipated the chaos that was coming in their work. Can they still do that?

One of the many staggering features of the great crisis of our lifetime was how few artists and writers were interested in the roaring, rapacious City that was hurtling towards disaster in front of their eyes. So comprehensive was their indifference, the BBC after the crash had to revive Little Dorrit, Dickens's story of a banking crisis from 1855, while the National Theatre had to make do with Sir David Hare wandering round the stage trying to explain what had gone wrong.

That I would love to know what Hobsbawm thought about the silence of the artists is one of the many, many reasons why his death last year was such a loss to British culture and, for he was always an internationalist, to the world's culture too.

Eric Hobsbawm was the best-known and most celebrated historian of the 20th century, not just in Britain but all over the world. His major works, four substantial volumes covering the history of Europe in its global context from the French revolution in 1789 to the fall of communism two centuries later, have remained in print ever since their first publication. More than half a century after it appeared, The Age of Revolution is still a staple of university reading lists. The Age of Extremes has been translated into more than 50 languages, and no doubt the foreign publication record of his other books is just as impressive.

Hobsbawm was as widely known in Italy  you can view him on YouTube speaking about Antonio Gramsci, in Italian  as he was in Brazil, where President Lula's confession that he had been the biggest influence on his thinking turned The Age of Extremes into a bestseller. Hobsbawm received honorary degrees from many countries, including Uruguay and the Czech Republic. He was an honorary citizen of Vienna. He won the Balzan prize, Europe's most coveted (and richest) award in the humanities. He was awarded the Frankfurt Book Fair's Prize for European Understanding. In the UK he was made a Companion of Honour, the equivalent of a knighthood. The news of his death, on 1 October 2012, was carried by papers across the globe.

There are many reasons why Hobsbawm managed to achieve such worldwide eminence and popularity. He wrote with extraordinary wit, grace and power, qualities evident once more in this posthumously published collection of essays and lectures on European culture in the 19th and 20th centuries. Arresting phrases leap from its pages. Karl Kraus's public life was "a lifelong monologue addressed to the world". Cultures are not just "supermarkets where we provide for ourselves according to our personal tastes". "With the democratisation of politics, power increasingly became public theatre." Hobsbawm's gift for provocative obiter dicta never left him.

This suggests another powerful reason for his global appeal: the enormous, astonishing fertility of his historical imagination. Many historians have come up with one influential concept or another, but Hobsbawm came up with a whole shedload: the "General Crisis of the 17th Century"; the "dual revolution" (the French and Industrial revolutions, the formative events of modern times); the "invention of tradition"; "primitive rebels"; "social banditry"; the "long 19th century" (1789-1914); the "short 20th century" (1914-1989); these are just a few. His ability to see the big picture and devise a framing concept to sort out the diverse and unruly detail of history was breathtaking.

Clearly Hobsbawm owed much of this to his lifelong adherence to Marxism, which in his hands was a subtle and flexible tool for organising and interpreting historical material, a whole intellectual world removed from the rigid doctrinal orthodoxies of the Soviet Academy and its mindless satellites in the other countries of the Warsaw pact. No wonder they didn't really know what to make of him. Marxism did, it is true, lend much of his work a teleological flavour that is no longer to our taste in the post-Marxist world. His rural bandits and peasant millenarians, for instance, were primitive rebels precisely because history had not yet reached the stage where socialism provided the intellectual toolkit for rebels to become modern. Yet this did not prevent him from treating them with a sympathy and fascination far removed from the "enormous condescension" of a dogmatic Marxist posterity. These qualities are nowhere more apparent than in the faint but distinctly discernible nostalgia of the brilliant essays in the present volume on Jewish life in central Europe in the 19th and early 20th centuries, the milieu in which he himself grew up.

What Hobsbawm's Marxism also did, however, was to turn him from a lifelong optimist  while it was still possible for some to think, even with reservations, that it provided hope for the future  into a bewildered pessimist when it became obvious, from 1990 onwards, that it didn't. Hobsbawm's pessimism comes through in many of the essays in this book more clearly than in any other work he published after the fall of communism. The cultural experience, he says, is "disintegrating". Classical music has no future, only a past. In many parts of the world, state subsidies of the arts are being replaced by market forces, to disastrous effect. ("It is not going to happen in the UK," he says, but in this case he wasn't being pessimistic enough.) Nevertheless, his vision of culture's future is too gloomy. Modernist music may not be very popular in the concert halls, for example (as he repeatedly points out), but it goes out to millions in the form of film scores. Looking around at the visual arts or the theatre, there's not much sign of decline. As so often, his arguments invite as much dissent as agreement, the sign of a truly creative historian. As the American economic historian David Landes once remarked, you come away from a Hobsbawm book feeling like you do after a vigorous game of squash: exhausted and invigorated at the same time.

Finally, and most important of all, Hobsbawm had a global appeal because he had a staggering breadth of historical knowledge, encompassing a dizzying number of countries and cultures. I've been teaching and writing modern European history for 40 years, but in reading this book I learned an enormous amount that I didn't know before, about writers such as KE Franzos, Gregor von Rezzori, or Miroslav Krleza, about the role of the Jewish vote in elections in mid-19th-century Turin, about the reasons why Mormons are always villains in the Sherlock Holmes stories, about the appeal of the cowboy myth to the European sensibility, and much more; something new on almost every page. Hobsbawm owed this encyclopedic knowledge partly to the insatiable curiosity about everything that was surely a major factor in keeping him alive and thinking for so long; he also owed it, obviously, to his cosmopolitan upbringing, in Vienna, Berlin and London.

This cosmopolitanism was far from unique among British historians of his own and the following generation, however, even though in Hobsbawm's case it went further than anyone else's. Historians of a comparable age, such as Owen Chadwick, Denis Mack Smith, Raymond Carr and Michael Howard, all happily still with us, found it natural to research and write about the history of the European continent.

They transmitted their breadth of outlook to a younger generation  my own  producing a whole phalanx of British historians whose work is as familiar in the countries they write about, from Spain to Russia, Germany to Italy, Poland to Romania, as it is in Britain or America: Paul Preston, Ian Kershaw, Norman Davies, Dennis Deletant, Lucy Riall, Geoffrey Hosking and many others. This has made British historians the most influential and widely read in the world today. Their dominance has been the product of a broad historical education in British schools, where European and world history have been taught alongside British history for decades.

Now this grand tradition is under threat, as the secretary of state for education's new national history curriculum threatens to produce a blinkered and ignorant generation of young people who will leave school knowing nothing at all about the history of the lands beyond these shores. I can already hear Eric Hobsbawm turning in his grave.

 Richard J Evans's The Third Reich at War is published by Penguin.