There seems to be no end of books about the British empire, and the fascination it holds for historians of all descriptions is inexhaustible. But while the British empire is easy to write about, it is very difficult to summarise. This is because what we call "empire" spans 400 years in time and thousands of miles in space; every continent on Earth was directly affected by it.

The empire existed as a concept in the early 17th century and lasted, as a factor in world affairs, to the second half of the 20th century. Yet it changed so much in that time that it is difficult to trace any continuities or consistent characteristics it may have possessed.

This diversity makes it easy for historians to write what they like about empire, but that doesn't mean that certain facts can, or should, be obscured. For much of the colonial empire, for most of the time, there was no idea of promoting democracy; the slave trade, in the earlier days of empire, was an unmitigated evil; the Victorian public school was a centrally important institution of empire; sport was highly valued as an arena in which manly virtues and character could be displayed; there was a wide degree of racial arrogance on the part of the administrators.

It is to the credit of Jeremy Paxman that his sane, rather detached view of empire understands these truths. Paxman has an eye for detail, and his sharp intelligence quickly detects, and ridicules, the eccentric and plain weird. Any wider conclusions are, however, avoided in this readable introductory account.

Paxman is also very conscious of the idiosyncrasies of so many of the servants of empire, describing TE Lawrence as a figure in the tradition of Livingstone and Gordon, "the courageous, half-cracked maverick unshakeably loyal to the empire". In an equally shrewd aside, Paxman observes that a surprising number of empire-builders died childless. Such comments reveal a feel for the subject, even if the book does not give many indications of long hours spent among dusty archives.

Writers on this sprawling, often nebulous subject are often anxious to define what they mean by empire, or to set out limits to their subject area. Paxman, with characteristic insouciant cockiness, doesn't bother. So, characters as diverse as Walter Raleigh, Winston Churchill, Mahatma Gandhi, Clive of India, Cecil Rhodes, Kitchener, David Livingstone and TE Lawrence, to name but a handful, crowd the narrative of this new book. Consequently, the effect is rather rambling: dry and self-assured, yet without structure or extensive elucidation of key themes.

Geographically, Paxman covers the raj in the 19th century, India under Robert Clive and the Mughals, the Middle East, Kenya, South Africa, Sudan, west Africa, Australia, New Zealand, Canada, even Ireland. Such a widespread coverage in a book of fewer than 400 pages is impressive, but it does lead to an impression of breathless chaos  much like the empire itself, one can imagine. At the end, readers will be left scratching their heads, trying to make sense of the whole thing.

The rather rambling nature is revealed by the fact that in this slim, elegantly produced volume, Henry Newbolt's famous (or notorious) imperial anthem, "Vitaï Lampada", is reproduced in full not once but twice, on pages 12 and 1967.

Paxman's subtitle is "What Ruling the World Did to the British", and in this phrase he pays homage to the household gods of modern contemporary history. It has been fashionable, of late, to say that the British empire changed Britain as much as it did the colonies, but what fresh insight is gained from this trite observation remains obscure. Paxman nods obediently to this, but one detects a lack of engagement with the subject; he mentions that several British cities have been changed by immigration from former colonies, and that there is now more variety of cuisine to be found on the high street, but he reaches no earth-shattering conclusions on this interesting theme.

Of course, as someone who has just written a book about the British empire, I am very well aware how easy it is to pick holes in any book on this subject. The common objection, the easiest comment in the world to make, is the "What about ...?" question. Any book on empire will omit, by necessity, vast tracts of the imperial experience, and so critics can easily find facts and details to contradict an author's bold generalisations.

This hole-picking is a rather futile game; any first-year undergraduate can play it. It is, of course, much harder to try to generalise, to make some coherent story out of a phenomenon as extensive and durable as the British empire. Naturally, Paxman is too cool, sophisticated and careful a journalist to attempt anything resembling a grand theory or central argument about empire. His book is episodic, fragmentary and languid in its approach. Reading it felt rather like being led through a tropical garden by a weary retired army officer. Nothing really excites the author: though he expresses admirable outrage at the barbarities of the slave trade, the tone is one of detached, knowing indifference.

The strength and weakness of Paxman's book are therefore in its range and diversity. As a taster, a teasing amuse bouche, Empire: What Ruling the World Did to the British is effective and entertaining, yet it seems unlikely that a devotee of imperial history will learn much from this performance.

Kwasi Kwarteng is Conservative MP for Spelthorne and author of Ghosts of Empire (Bloomsbury)

Jeremy Paxman thinks we're neglecting the history of the British empire. "Perhaps in the dark recesses of a golf-club bar some harrumphing voice mutters about how much better the world seemed to turn when a great-uncle in baggy shorts ran a patch of Africa the size of Lancashire. But, by and large, no one has much to say about empire." That will come as a surprise to the authors of the dozen fat books about it that have appeared over the last few years. If Paxman thinks his Empire is filling a gap, he's mistaken.

British imperial history is highly contested territory. One reason is obvious: the number of peoples, societies and nations that still bear the marks of it, or think they do. Another is that it is so often discussed  or expected to be discussed  judgmentally. What readers generally look for first in a new history of the empire is whether it is for it or against it. More neutral books are often forced into one category or the other simply to give reviewers a handle on them; or, if they can't be, they are ignored. For those of us who burrow away at the subject academically, this last is probably the best we can hope for. It's not an option available to Paxman, however, who is too famous to be ignored. In fact it's difficult to say where he stands in the great pro- or anti-imperial debate; but that's to the credit of the book, especially if it can put over to readers that the question is not a simple one.

Judgments come thick and fast, but they are not all on one side. The worst atrocities of British imperial rule are properly highlighted and deplored. The horrors of the slave trade, for example, "should be engraved on the national conscience. It is one of the most disgraceful episodes in British history." Similarly the opium wars, the British reprisals after the Indian mutiny, "genocide" (the right word, he thinks) in Tasmania, the Amritsar massacre, Balfour's perfidy in Palestine, "offering land belonging to one people as a gift to another", the great Bengal famine, the now notorious Kenyan prison camps, and British imperial arrogance and racism generally. Few of the leading "builders" of the British empire  heroes to a previous generation of imperialists  escape Paxman's scorn. The early ones were mostly pirates and freebooters. Clive of India was mainly a fortune-seeker, "scheming, and devious in business". General Gordon was "half-cracked". Cecil Rhodes was "the most sabre-toothed of all empire-builders". Kitchener was a monster. Lawrence of Arabia was romantic, but (again) "half-cracked". Churchill may not have been "quite sane" when it came to India. Lord Meath, the creator of Empire Day, is described as looking "a bit like Father Christmas", with "a bald head, a red face and an enormous white beard"; he had some silly ideas about instilling "duty and discipline" into working-class ne'er-do-wells. (David Cameron might like to look him up.) Others were mostly hypocrites, or "self-deluders" at best. The last includes Gladstone and "more recent moralists in Downing Street".

On the other hand, the book also describes atrocities committed by the colonised in, if anything, even more gory terms: the "Black Hole of Calcutta", for example, "a horror story to rival anything among the Gothic tales which swept Britain" at the time. Paxman isn't too keen on the Mahdi, Gordon's nemesis, either, with his "usual jihadi stuff" and the regime of "slavery, institutionalised paedophilia, hand-loppings and floggings to death" instituted under him. So there's a certain balance here.

More importantly, there was "another side to colonialism". Britain's most admirable imperial agents were her ordinary administrators, who Paxman thinks have been unfairly ridiculed; they were generally good eggs, and  crucially  incorruptible. The British empire was also prone to periodic spasms of moral self-questioning, which is unusual for empires, one of which in the end got rid of slavery. The postwar transition to self-government was managed "peacefully" in the main, though "in a handful of places the British fought nasty little campaigns". (I'm not sure that many Kenyans, Malayans and Cypriots would accept that "little".) Overall, "if you had to live under a foreign government," then the British empire "was better than many of the other possibilities". That "if you had to", however, is a crucial caveat.

If you have to have a popular introduction to British imperial history, too, this book could be said to be better than many of the other possibilities, in particular it is far preferable to Michael Gove's "patriotic" approach. It's a very engaging account, built mainly around dramatic incidents in exotic places (just right for the forthcoming TV series), and the lives of fascinating people, with a good sprinkling of jokes, funny nicknames and sexual references. Paxman makes some very sharp points, and writes well. Here's an example, towards the end: "The British empire had begun with a series of pounces. Then it marched. Next it swaggered. Finally, after wandering aimlessly for a while, it slunk away." If you want the appearance of the whole process described in a few short sentences, I can't imagine it done better.

He doesn't, however, delve much beneath appearances. There's nothing here on the economic, cultural or any other roots of British imperialism. Paxman is insistent that there was no great plan behind it, and that often governments were simply bumped into taking countries without really wanting to. At times he appears to be going along with what he takes to be the 19th-century historian JR Seeley's view, that the empire was accumulated "absent-mindedly". (In fact Seeley said the opposite. Everyone gets this wrong.) To his credit, Paxman repeatedly confesses his own puzzlement: "there was something inherently nonsensical about it  How could the ultimate purpose of colonisation be freedom?" Well, there is an answer to that. He might not agree with it, but he should try to understand it. Dismissing things as "nonsense" is a way of avoiding the bother of explaining them.

In view of the book's subtitle, it's a shame he doesn't say more about what the empire did to the British. He claims it "convinced the British that they were somehow special", but is very woolly about whom he means by "the British" in this context. Later on he claims that it was "mainly a ruling-class thing", with attempts to foist it on "the people" largely unsuccessful. "The noisier the loudspeakers of officialdom, the more reverberant the empty echo." This is hotly disputed in some historical circles, even for the inter-war period, which Paxman is referring to here. If it's true, however, it goes some way to explain the situation which he alleges  and deplores  at the beginning and end of his book: of widespread public amnesia about the old empire, while our prime minister pompously lectures the "lesser breeds" as if he were still wearing those baggy shorts. Hopefully he'll be watching the TV series, too.

Bernard Porter's The Lion's Share: A Short History of British Imperialism is published by Longman.