The porcelain of the potter Edmund de Waal doesn't usually stand solitary in a glass case or alone on a table. His pieces are often grouped as families on shelves or in niches, as in the new ceramics galleries at the V&A, where they talk among themselves and hold converse with the space they're living in: its past, its associations, the qualities of its light. Sometimes they look like the stacked contents of a cooled kiln, waiting for the selection to reject the misfires; or survivors retrieved from a cargo long sunk on its voyage back from the Far East. De Waal barely mentions his pots in this unique memoir of his family, though. They're present as an absence, the vocation he evaded for a couple of years while he researched how he came to inherit a collection of 264 netsuke from his great-uncle Iggie.

Netsuke seem the opposite of frangible porcelain. They were carved in finegrained wood or ivory to reward touch and endure wear while doing their job, as the toggle on a cord from which a container was slung, most often for medicine or tobacco. A man in traditional Japan tucked the cord behind his obi-sash, and the netsuke prevented it from slipping out. Private satisfaction and public display contended in them, as it does in many small, personal objects. Some were carved by great craftsmen, some by gifted amateurs; all held forever a moment of time  the twist of a tiger's shoulders, the twitch of a hare's head.

De Waal's intent in studying the collection was to use their acquisition by Charles Ephrussi, a cousin of his great-grandfather, to understand the first wave of Japonisme as it surged through Paris in the 1870s. (De Waal has the credentials, years studying Japanese aesthetics.) But that proved too narrow a subject. He came to want to hear the dialogue between all the possessions of Charles, wealthy son of a pan-European Jewish dynasty of grain brokers and bankers who had migrated from Odessa on the Black Sea. He needed to know how Charles had educated himself in art  Charles first wrote for, then became proprietor of the Gazette des Beaux-Arts magazine; how Charles evolved his taste from tourist souvenirs, albeit the grandest ones, to the purchase of lacquer, and bold commissions of his friends, the Impressionists. Charles paid Manet so amply for a painting of a bundle of asparagus that, a week later, Manet delivered an extra canvas with a single stalk, and the note: "This seems to have slipped from the bundle." Recognise the story? Proust borrowed it. Charles became one of the models for Swann in In Search of Lost Time, although by the year Proust began to write that, Charles had moved on from Japan, his passion for which had been intertwined with his relationship with his netsuke-collecting mistress. They had exhibited their treasures together  there's Parisian sexual daring.

De Waal moderates the exchanges between art and life (the back of Charles's top hat, so out of place, in Renoir's informal canvas, The Boating Party Lunch; the anger of Renoir when Charles bought an auric Moreau painting, which Renoir regarded as a Jewish lapse of taste), and explores as close as he can get to Charles's apartment, its contents and their meaning. A sure way to retrieve lost time, or at least to feel that retrieval is possible, is to make contact with the dead fingers that left their impressions on what they created, and with the eyes and hands that appreciated that creation. De Waal has a mystical ability to so inhabit the long-gone moment as to seem to suspend inexorable history, personal and impersonal.

A lot happens next  that's what stories are for  but none of it seems inevitable. Charles on a whim gave the netsuke, and the vitrine that was their glass carapace, as a wedding present to his cousin Viktor, who married the Baroness Emmy Schey von Koromla in 1899. They were shipped to the marble burg of the Palais Ephrussi in Vienna, and being too intimate for its salons, were stashed in Emmy's dressing room. She made up stories about the netsuke as her children took them out for play, but otherwise they remained undisturbed for almost 40 years, while the Austro-Hungarian emperor died and his empire evaporated. Viktor and Emmy, confident in assimilation  his Russianness was vestigial, their Jewishness marginal  had invested in that empire and lost much of their fortune, but they lived on diminished in the palais. The paintings and miscellanea, no longer supplemented with fresh-bought excitements, barring Viktor's choice books, solidified into a family heritage.

The central passage of this narration is overwhelming. Emmy's daughter Elisabeth fought her way through university to become a lawyer, then escaped Vienna to study further before a marriage to a Dutch businessman; son Iggie, in pursuit of a minor talent for couture, and a major discovery about his sexuality, slipped away to Paris, then America. They travelled light to their future lives, assured that all the familiar things remained in their remembered places back in the old home, the ivory rats set out in the vitrine; while their parents were immobilised by those same things, withdrawn from the thickening, darkening public world of Austria after Hitler's rise into what seemed safe domestic space, the silver calm in its room, the rare books wise in their locked case. We all know the dates, and events on those dates, yet De Waal's description of the Anschluss and after comes as an absolute shock: the palais breached by night, the initial smash and grabs. Then, by many methodical days, the inventories, as the Property Transactions Office sent in its appraisal valuation official to divest Viktor and Emmy of everything from the ownership of the Ephrussi bank to a bundle of umbrellas. Viktor and Emmy left with two suitcases for the limbo of a country estate in a Slovakia that no longer existed, and half their little luggage was confiscated on the way. Emmy committed discreet suicide; Elisabeth, by now a bicycling, observant Christian bringing up her own children in Tunbridge Wells, extricated her father. He arrived with the key to his case of books  all Aryanised by diktat into Austrian libraries  on his watchchain, and not much more.

And the netsuke? That's De Waal's revelation, the "hidden inheritance". Anna, who for decades was Emmy's personal maid (no other details specified), was suborned by the Nazi asset strippers to crate up the Ephrussi household goods. Allowed to dwell on in her servant's chamber, she pilfered the netsuke  so portable they could be slipped in an apron pocket one at a time  until she had stolen them all away, and bedded them down in her straw palliasse. Postwar, Elisabeth reached a wrecked Vienna to retrieve any morsels of what had been home, to find the palais had been a Nazi, then a US occupation, office. The Americans reunited her with Anna, who returned all 264 netsuke in an attaché case, and dematerialised from the story. Elisabeth, who was De Waal's grandmother, gave them to Iggie, thereby making up his mind to take up a business appointment in despoiled Japan. Iggie made his life there, installed the netsuke in a wall case in Tokyo, and found his true partner, a young Japanese man who inherited the netsuke at Iggie's death, and passed them back to De Waal at his own.

That had to be told because this story compels telling; but it is such a bald summary of a work that succeeds in several known genres: as family memoir, travel literature (De Waal's Japan is the nearest thing to being there, and over decades), essays on migration and exile, on cultural misperceptions, and on De Waal's attempt to define his relationship with his own kaolin creations. His book is also a new genre, unnamed and maybe unnameable. A thing-book, perhaps, or a Wunderkammer  cabinet of marvels  except they're both such German concepts.

The netsuke, and other works of art and craft gained and lost along the way, never serve merely as accessories, trophies or substitutes for their owners and looters, but retain their own secret identities, and hold within them, as they always will, the time of their creation; both the moment they show  that hare with his front feet so briefly lofted off the ground  and the many, many hours of their manufacture. If you have ever cleared a house after a death you will recognise this feeling, that each handmade thing matters of itself, even when mortality casts it loose sequentially from maker and owners; the sense that responsibility for the present of an object is also a duty to its past, and an obligation to its onward transmission.

What happened to the hare with amber eyes, and the carved medlar that almost felt as if it might squish when handled, after their return from Japan? De Waal bought them a secondhand vitrine from the V&A and set it up in his London house, its door unlocked so his own children could play with its contents. "Objects have always been . . . stolen, retrieved and lost. It is how you tell their stories that matters." He has told their story wonderfully. Oh, and this is a beautiful and unusual book, as a physical object. Somebody really cared.

Edmund de Waal is a potter, perhaps the most famous potter working in Britain today. His bowls and beakers, thrown in porcelain and glazed in celadon, are domestic,  in theory, you could fill them with hot tea  but they also exist in a more contemplative realm; arranged in pale lines and marked by various dents and asymmetries, they whisper a story of limitless but rather fragile possibility. This is what they say: that the potter may throw any shape he likes; that no two of his pots will ever be precisely the same; and that a pot may disappear  crash!  in an instant. I find them exquisite, but I'm not sure that I would ever want to own a row. As an ever-present metaphor for human endeavour, I fear they would slowly drive me mad.

In his memoir, de Waal alludes early on to the existential hum some objects emit. Things do "retain the pulse of their making" and this intrigues him: "There is a breath of hesitancy before touching or not touching, a strange moment. If I choose to pick up this small white cup with its single chip near the handle, will it figure in my life?" De Waal believes the way objects are handed on has as much to do with storytelling as happenstance. You know the drill: this belonged to your aunt, who looked just like you. But such anecdotes, prettified over time, obscure as well as reveal and this worries him (he's always worrying).

De Waal has inherited 264 Japanese netsuke  wood and ivory carvings of animals, plants and people, none larger than the palm of his hand  from his beloved great uncle Iggie, and though they're a relatively recent arrival at his London home, already he fears their story is growing too "poised". A netsuke is a "small, tough explosion of exactitude". It deserves exactitude in return. "I want to know what the relationship has been between this wooden object that I am rolling between my fingers and where it has been." So, leaving his studio in the care of others, off he went. He would tell their story.

Where does it begin? Paris. The netsuke were bought from a dealer there in the 1870s by Charles Ephrussi, a relative of his great grandfather, Viktor. Charles, scion of the fabulously rich Jewish banking family and one of the models for Proust's aesthete Charles Swann, is a collector who once bought a still life of asparagus from Manet at a price so generous the artist sent him a canvas of a further, single stalk in gratitude. Charles bought the netsuke during the craze for Japonisme. They were kept in a black lacquer vitrine until, one day, Charles sent them to Vienna as a wedding present for his cousin Viktor. Why send these rather than, say, a vase? De Waal speculates that they must have been lost among all the tapestries and the Renoirs; probably, Charles had outgrown them.

But at Viktor's home, they were equally out of place. "It looks like the foyer of the opera," said his bride, Emmy, on being shown her new apartment. The vitrine and its homely curiosities  netsuke were originally designed as toggles  were banished to her dressing room, where, in due course, her children would play with them while she chose her jewellery. And there they stayed, a cuckoo in the nest, as the first world war began, and ended, and then, as Austria, unable to feed its people, allowed antisemitism to take hold. In March 1938, the Ephrussi home was invaded by men in swastika armbands. Some things were stolen, others destroyed, but the netsuke remained mysteriously intact.

After the Anschluss, the family fled. Emmy took her own life in the Ephrussi country house in Czechoslovakia. Viktor and his children escaped elsewhere: his daughter, Elisabeth (de Waal's grandmother), took her father to Tunbridge Wells. After the war, she travelled to Vienna to discover what remained of the family's possessions. Not much was the answer, but a maid, Anna, saved the netsuke from the Nazis, hiding them in her mattress.

In 1947, Elisabeth's brother, Ignace (Iggie), visited Tunbridge Wells between postings for an international grain exporter. Should he go to the Congo or to Japan? They looked at the netsuke together and his decision was made for him. And it was in Japan, in 1991, that de Waal first set eyes on his future inheritance, now repatriated by Iggie. The young potter was studying in Japan and every week he lunched with his great uncle. Afterwards, they examined the netsuke, one by one. The hare with the amber eyes. A tiger. A tumble of tortoises.

De Waal has researched his story with obsessive diligence and he tells it with an imaginative commitment  searching, yet wide-eyed  sadly lacking in some of our more wizened biographers. He is wonderful on place, forever turning doorknobs, real and imaginary, and inviting the reader in. But I could not understand, and became annoyed by, his conviction that he is not in the business of memorialising the diaspora. There is something precious about this, as though such territory is beneath him. "I don't really want to get into the sepia saga business, writing up some elegiac Mitteleuropa narrative of loss," he says.

The question is: do the netsuke enable him to resist such a tale? No. Their survival is wondrous, but I don't think their presence turns The Hare With Amber Eyes from memoir into book of ideas, as de Waal seems to believe. Sometimes, they are more distraction than narrative thread and the need to return to them often bogs the author down; there are, after all, only so many ways to describe the feel of carved wood and only so many times such an image can be made to work as a symbol of patinated memory without the reader feeling that a point is being laboured. I loved the story of the Ephrussis, but I am mystified by de Waal's insistence on gilding it with his own flimsy abstractions. There is no shame in telling people what happened to Jewish families in the last century. Such elegies, sepia or otherwise, grow every day more vital.