Opening this book, I wondered with a premonitory shudder whether I'd find an anatomy of my own collecting habits inside it: would I turn out to be a thieving magpie, or a squirrel frantically storing its foraged nuts? Other people move house to accommodate an enlarging family; my own moves have been determined by the need to domicile what Americans would call my "stuff"  a couple of hundred Hitchcock posters, even more old photographs of Manhattan, a morbid hoard of Wagneriana (including a death mask with a crown of gilded laurel), along with a gathering of eggless birds' nests from my garden and a thicket of Australian Aboriginal totem poles.

My heart sank when I discovered, late in Yallop's group biography of Victorian scavengers, that in Freud's opinion the obsession with curios "suggested abnormality, sexual impotence and personal failure".

Gilbert Osmond in Henry James's The Portrait of a Lady acquires a wife to complement his array of rare porcelain: collecting is a chilly substitute for love. It's the same sad logic that led Nick Hornby to conclude that collectors of records  and yes, I have a room full of those  are all nerdy, reclusively solitary males.

Luckily Yallop dispels the slur by offering the example of an intrepid and indomitable female antiquarian called Charlotte Schreiber, who with a compliant husband in tow crisscrossed Europe during the 1870s driving hard bargains for ceramics. Another of Yallop's characters, Michael Mayer  whose indiscriminate acquisitions ranged from the detritus of Anglo-Saxon burial sites to Napoleonic memorabilia, illegible Burmese manuscripts and a taxidermically treated crocodile  does appear to have been disappointed in love, and Yallop briefly speculates about the internal emptiness his litter or lumber was meant to fill; she pardons him, however, because he opened a Liverpool museum to make his accumulation of oddments available to all-comers. Philanthropy converts a private vice into a public benefit.

For the most part, this is a breezily anecdotal social history, often almost novelettish in its determination to bring the musty, mouldy past back to life: discussing the foundation of the Victoria and Albert Museum, to which so many of her collectors entrusted their finds, Yallop feels obliged to assure us that when spring arrived in South Kensington in 1854 "the leaves split green on the trees" and that before Christmas in 1867 the windows of Harrods "were bright with gifts and treats and lights".

The glittery allure, of course, is window-dressing. Yallop's book is about the nasty collusion between wealth and aesthetics, culture and social climbing. The first collector she examines is John Charles Robinson, who "bought and sold furiously" and in the process helped stock the new galleries in South Kensington. Robinson was denounced as "bumptious and odious" by his rival Austen Henry Layard, a parliamentarian, archaeologist and trustee of the National Gallery.

"He is nothing but a dealer," sniffed Layard: collectors were genteel amateurs, but dealers, dirtily contriving profits for themselves, counted as tradesmen. I rolled my eyes at this and then, remembering the double-breasted toffs who have galleries in Mayfair, wondered whether Layard may not have had a point.

Yallop usefully notes the new imperial scope of collecting during Victoria's reign. The aristocratic tourists of the 18th century aimed to bring home paintings or sculptures from Italy, but Yallop's collectors and the agents they employed could plunder a wider world.

Murray dug up Egyptian sarcophagi, and Stephen Bushell, an enterprising diplomat posted to Peking in the 1880s, supplied the V&A with Chinese bronzes and Ming porcelain, as well as exploring the Qing tombs to trap some rare monkeys that he shipped back to the Zoological Society in London. The countries that were being despoiled looked on helplessly, relying on art to act out their fanciful revenge. Crowds at the Colonial and Indian exhibition in South Kensington in 1886 admired Tippoo's Tiger, "an animated mechanical organ, made of painted wood and carved to look like a tiger mauling an Englishman"; the beast had been commissioned by the Sultan of Mysore as a satire on the predatory tactics of the British East India Company. Our grand museums, it is worth remembering, are warehouses stuffed with stolen goods.

Collecting, like the empire, was a by-product of capitalism, and Yallop has some good tales to tell about attempts to sabotage this busy, greedy market. Travellers in India naively paid for indigenous treasures that had been mass-produced in Birmingham factories, while the forger Louis Marcy churned out fakes as an anarchic subversion of the system.

The most endearingly cranky character in the book is the painter Dante Gabriel Rossetti, whose elegant riverside house in Chelsea was a junkyard crammed with bric-a-brac  "brass and pewter, oriental rugs, velvets and chintzes, Sheraton furniture, Spanish cabinets, and banks upon banks of mirrors". But Rossetti was more than an aesthete fussing over inanimate objects of desire: the same premises doubled as a zoo, overrun at various times by dogs, owls, rabbits, dormice, hedgehogs, wombats, kangaroos, racoons, parrots and lizards. (There were squirrels in Rossetti's menagerie, though the inventory does not mention any magpies.) Pride of place in this squabbling, rooting, insanitary scrum was claimed by a lordly, strutting peacock, which screeched at visitors and then bedazzled them by displaying the blue-green fan of its iridescent tail. The animals and birds were Rossetti's reproof to his own acquisitive mania. The collector makes do with corpses like Mayer's stuffed crocodile, or with mechanical simulations of life like the Sultan of Mysore's tiger; art can never compete with the rowdy plenitude and bizarre beauty of nature.

Human collections of objects can be instruments of research about the world or objects of aesthetic delight. Linnaeus, Darwin and Alfred Russel Wallace made systematic collections of creatures and things. Linnaeus in the 18th century travelled widely and sent students all over the world. At his death in 1778 Joseph Banks tried to buy his collection, which was finally sold to James Edward Smith in 1783 and is now housed in the Linnaean Society in Burlington House in London. There were 14,000 plants, 3,198 insects and 1,564 shells. His pioneering geological collection appears to have been thrown away by the carriers as just bags of old stones. Collecting has its terrors and tragedies. Wallace spent four years in the Amazon and collected and mounted thousands of specimens of plants and animals, only to lose them all in a fire at sea in the Atlantic.

Humans also collect artefacts made by humans. Balzac, in the introduction to The Human Comedy, says that the world consists of "men, women and things  that is to say, persons and the material representation they make of their thoughts  that is to say, man and life." He seems to conceive the making of objects as a kind of evolution which differentiates one man from another, one society from another. Almost a kind of DNA. In The Human Comedy battles rage for the acquisition of objects and collections. Lives are destroyed by rapacious aesthetes and dealers. And Balzac is a great describer of sculptures and clocks, plates and paintings.

Freud too, as Jacqueline Yallop reminds us in her interesting book, was both a collector and interested in collecting. He collected antiquities and described intense collecting as a form of fetishism.

In our own time The Antiques Roadshow and Flog It! have fed our interest in artefacts and added a new word, "collectible", to our vocabulary. The surrealists collected random objects (a collection of which is now on display in Houston in the Menil Collection). Modern artists collect  Tom Phillips's amazing collection of African gold weights was recently on show, and Peter Blake apparently has to be restrained from excluding himself from his studio by filling it with found objects.

Yallop is interested in collecting as an expression of the Victorian age  "a compulsive urge to do things bigger and better, to bring as many elements as possible under one protective umbrella, to control, to regulate and to extend." Her book tells the stories of five very different collectors, against a background of the Great Exhibition of 1851 and the birth of the Victoria & Albert Museum. This was a time when aristocratic good taste was giving way to strong middle-class curiosity about crafts and domestic arts. Her collectors are interested in china and fans, in silver and fabrics, not only in great paintings and sculptures.

She begins with John Charles Robinson, an aesthete who collected, in France and Italy, objects displaced or disturbed by the chaos of French politics after the revolution. Robinson was the superintendent of art collections at the South Kensington Museum, which became the V&A, and was constantly at odds with the director of the museum, Henry Cole. They didn't like each other, but their battle was part of the larger battle between beauty and utility that went on in the world of museums and art education at the time. Cole, and Prince Albert, believed museums were educational and should show ordered examples of good and bad craft work for the instruction of British artists and craftsmen. Cole had instituted a Chamber of Horrors  a collection of bad taste, "a gloomy chamber, hung with frightful objects in curtains, carpets, clothes, lamps and whatnots". Robinson collected for art's sake, and admired the continental museums where objects were displayed in recreated rooms of their periods. There was a school at the museum who believed in orderly rows of all possible knives and all possible bowls. Yallop's account of Murray Marks, a collector who was a friend of DG Rossetti, is about the aesthetic movement, about peacocks and blue and white china  sumptuously displayed in the current V&A exhibition of The Age of Beauty.

Her most vivid section is about the amazing and resourceful Charlotte Schreiber  an aristocrat who made two happy misalliances, and seems to have been indefatigable. As a young woman she learned "many languages, including Arabic, Hebrew and Persian", read Chaucer and Ariosto and "played a mean game of billiards". She had a flirtation with Disraeli but married a Welsh businessman and threw herself into workers' welfare, education and the furnaces and forges at her husband's ironworks. She was active in work for the abolition of slavery, free trade and church reform. She found time to translate the Mabinogion from Welsh to English. When her first husband died she married her son's tutor, Charles Schreiber, and became a serious collector, and patron of Layard the archaeologist. Yallop's book describes the Schreibers, with their big red collecting bag, trundling across rural France to collect something, before it was snapped up by Duveen, and then proceeding to Paris, only to find that the city was destroyed by the Commune, the barricades and their aftermath. They nevertheless made some bargains with a dealer whose shop had not been ransacked or burned.

I was also fascinated by Yallop's account of Stephen Wootton Bushell, who went to Shanghai in 1868 as a young doctor and, as he is described here, took to educating himself about, and collecting, Chinese arts and crafts almost out of boredom, or loneliness. He ended up a great scholar, and went to study the Chinese porcelain collection of William Thomas Walters in Baltimore. In 1896 he published Oriental Ceramic Art, 10 volumes "lavishly bound in yellow cloth, backed with yellow silk" and "full of colour plates and black and white illustrations".

The brief biography of Jacqueline Yallop on the book jacket ends "She does not collect anything." And as I read the book I thought to myself that I could have guessed that. She collects collectors but seems not to be essentially interested in objects; very few are described, and even the big red bag of Mrs Schreiber can be imagined only as an area of redness  we don't know if it is cloth or leather, what its handles are like, how it fastens. She is interested in the psychology of individual collectors  what tips a man or woman over into ferocious acquisitiveness, into seeking particular things across continents. She is interested in the competitiveness of collectors, their desire to have the first or the best example of something. She is interested in collaboration  Bushell and Walters  and in stealth and warfare and stealing marches.

She is interested beyond that in dealers  whose social standing is much lower than that of the great collectors  and in fakes, and the dealers who deal in them. She has a splendid section on a fake bust of Flora "by Leonardo da Vinci" which took in the art world, and on the professional forgers  often highly skilled craftsmen  whose work sometimes stayed undetected in museums for decades.

She is also very good on the relations between collections and the larger world around them. Several of her subjects instituted schools or libraries to educate the world. Joseph Mayer founded a public library and a museum in Liverpool and believed, Yallop says, "that his memorial would be secure". The desire to leave a lasting memorial drove many of her collectors. But in more cases than not, she suggests, the collections are broken up by museum administrators and display artists. Labels that once carried the names of the collectors are replaced by educational ones. Mayer's library was too expensive to keep up, and the Egyptian collection in Liverpool was bombed.

Collections in fiction tend to be sinister. Browning's last duchess is frozen into a smiling painting. In Balzac's Le Cousin Pons the skirmish over a collection destroys the collector. My favourite collector is the innocently smiling Adam Verver in Henry James's The Golden Bowl. He is an American who collects the rarities of old Europe and displays them in his museum in American City. He also collects, and marries, Charlotte Stant, his daughter's beautiful friend, who is having an affair with his daughter's Italian husband. Charlotte is "shipped" to American City. She is seen gracefully showing off the garlands looped round a specimen of vieux Saxe porcelain. Maggie, the daughter, perceives her on an imaginary "long silken halter", describing the collection in a "high coerced quaver before the cabinets in the hushed gallery". She has become an exquisite specimen.

AS Byatt's Ragnarok: The End of the Gods will be published in September by Canongate.