"Nothing changes more than the notion of what is shocking," wrote the novelist Elizabeth Bowen in 1959, on the eve of two decades of major disruption in the public notion of what should be kept under wraps. Deborah Cohen's excellent and illuminating book explores, in painstaking but never tedious detail, what society from the Victorians onwards kept secret, the relationship between secrecy and shame and the subtle interdependence of the secret and the private.

The central body of the book is structured around its topics: interracial intercourse and its progeny, adultery, illegitimacy, mental disability and homosexuality (in fact, the subjects that novelists since the 19th century have depended upon). The final section considers the family and the culture of confession that challenged, and continues to challenge, the very British traditions of reserve.

Cohen quotes Thackeray, "Every house has its skeleton in it somewhere" (The History of Pendennis), and nowhere was this apparently more the case than in the houses of those who, during the heyday of the British empire, repaired, for what ever reason, to India. "The nabob was "a controversial new species of grandee who made his fortune in India and returned home to enjoy his treasure". But many, if not most, also enjoyed the treasures of the country they had come to exploit, not least the sexual availability and erotic allure of its women. It was, Cohen assures us, commonplace to have a native mistress who was frequently given the Hindu title of Beebee (Mrs), and these mixed-race households were "unapologetically open to view".

The open secrets only became problematic when the men returned home, often bringing with them the children they had fathered in a very different social climate. Cohen gives us some poignant examples of the fates of these children who not infrequently, in the case of "Margaret Stewart", for example, prompted genuine affection and loyalty in their British families. Margaret was the illegitimate daughter of Robert Bruce, who, ruthlessly separating her from her mother (there's an interesting correspondence between the nabob's possessiveness of his offspring and brutality towards his mistress), brought her back to his native Edinburgh under cover of being her guardian. Apart from this act of oblique acknowledgment, Bruce seems to have been an indifferent parent, but his two siblings took the blood connection to heart and even ensured that Margaret was acknowledged as the rightful heir to her father's estate, thus cutting themselves out of a considerable fortune.

Margaret eventually married a much younger man who, against expectations, apparently loved and cherished her. Together they lived a life of luxury unlike Susan Cochrane, another Eurasian, who was brought to England as a small child, raised as a "lady" and who spent her later years fruitlessly fighting to prove that her English father's liaison with her Indian mother had been a legal marriage.

The fate of the empire's mixed-race offspring brings to mind Marlowe's "That was in another country, and besides, the wench is dead". Morality is shown as distinctly relative. Here the book is not so much a study of secrecy as a study of pragmatism  "When in Rome" becomes the moral compass. What is fascinating is the tenacity with which the English nabobs maintained their blood ties with their Eurasian offspring. Even if these "brownies" were accorded lesser fates than their legitimate, white brothers and sisters, they were by and large cared for and frequently acknowledged, albeit tacitly.

Much the same can be said of those children in the Victorian era who were born with some version of mental inadequacy. Cohen reveals that far from eschewing any potentially embarrassing connection with their "simpleton" offspring, the Victorians tended rather to dote on them. It is not until the 20th century that attitudes sharply alter. Again, it is in the fastidious detail that her account comes alive.

"Lucy Gardner and Elizabeth Scott-Sanderson arrived at the Normansfield Training Institution with the same diagnosis: both were deemed 'imbeciles' from birth. Five-year-old Lucy brought with her trunks full of pretty clothes On her visits home, Lucy attended garden parties and teas. When she was away, neighbours and acquaintances inquired about her progress. After four years of training, Lucy Gardner returned to a family that delightedly pronounced her much improved.

"Elizabeth Scott-Sanderson never came home, not even for holidays"

By 1920, the date of Elizabeth's birth, Victorian optimism that "idiots could be educated, even cured" and a certain largesse that tolerated difference as part of God's plan had succumbed to a dread of a hereditary taint. With God dislodged, afflicted children became not His most vulnerable lambs but manifestations of bad blood. The new science of eugenics promulgated the notion that "feeble-mindedness" was the prerogative of the lower social orders. A feeble-minded child reflected dangerously on the apparent normality of other family members. Gradually, originally progressive institutions, like Normansfield, "founded to promote the integration of the mentally disabled became the means by which they could be segregated for a lifetime".

Cohen is equally engaging on adoption and homosexuality, both of which involved the very British art of turning a blind eye. In the final sections of the book, she considers our post-Freudian culture, in which shame and secrecy are not so much social arbitrators as bars to personal authenticity. Cohen bravely raises the paradox of a culture in which "transparency" and public confession seemingly go hand in hand with an idealising of privacy. If this is the least compelling section of the book it is because there are fewer examples of what in the past it was felt needful to conceal. And, as Freud recognised, the concealed inevitably becomes alluring. Cohen does not directly tackle the phone hacking scandals but she might agree that our current censoring of such behaviour  as with the recent Jimmy Savile revelations  may well be in proportion to our fascination with what we like to suppose we deplore.

Salley Vickers's latest novel, The Cleaner of Chartres, is published by Viking

The biggest thing that Deborah Cohen achieves in this book of marvels is to dislodge, once and for all, the whiskery idea that the Victorians were a generation of secret keepers. The image of all those attics stuffed with mad wives, strange sons and idiot siblings waiting for modernity to shine its cleansing light on festering family shame will no longer do.

Cohen argues that for the past 200 years Secrecy has been engaged in a frisky gavotte with its first cousin Privacy. First one leads, then the other takes a turn. And in the 19th century Privacy, that classic constituent of political liberalism, mostly had its best foot forward. What went on in an Englishman's [sic] home remained a family's own business, which meant, paradoxically, that there was nothing much to hide. It was only once the 20th century started to breath down everyone's neck that a retreat into lock-down seemed necessary. Now any family that wanted to shield its sadnesses and embarrassments from public view was obliged to take refuge in a thick wadding of camouflage, misdirection and pointed silence.

This sounds confusing, but Cohen's great strength is the way she puts flesh on theory, making it come to life before our eyes. Take her chapter on mental disability. In the mid-Victorian period, she argues, well-resourced families found themselves able to cherish mentally handicapped children. God had chosen you to care for one of His less finished creatures, and that was that. Such children joined in family life, appearing at the dinner table, splashing in the sea, becoming Mama's special pet. If they were sent away to a school, such as the progressive Normansfield Training Institution in South London, they went wrapped in love and with the promise that everyone was counting the days until the holidays.

By the time the apparently permissive Edward VII was on the throne, it was all quite different. "Imbeciles" were increasingly left at Normansfield all year round and for decades at a time. Communication between school and parents, once suffused with loving chatter, had dwindled to the occasional chilly letter from the family solicitor or doctor. Children who did occasionally come home, such as clergyman's son Percie Weldon, were told to keep out of sight when visitors arrived. Cohen finds tragic evidence of handicapped children systematically expunged from the record, their names left off parents' obituaries, ending their lives in unmarked graves.

All this had happened because heredity had replaced God as the driving force in human history. The turn to eugenics in the early years of the 20th century made families with a mentally handicapped child, gay uncle, or criminal cousin frantic with worry about a possible "taint" in their bloodline. Would anyone want to marry into a family that might be deemed "contagious"? In this new age of curtain twitching the only solution was to behave as if your mentally handicapped child, alcoholic brother or adulterous mother simply did not exist. If it was too late for that, you could simply kill them off in an accident and pray they never turned up to embarrass you.

Cohen finds a similar pattern when it comes to those permanent bachelors without whom no Victorian home was complete. A flamboyant uncle who always arrived with a "best friend" in tow was easily accommodated at a large family gathering where a variegated palette of appearances and behaviours was on show. But by the mid-20th century, and with families smaller in size, a homosexual son or brother was harder to hide in plain sight. And even when heredity was replaced by theories that favoured "nurture" over "nature", there was no let-up in the potential for shame. As ideas derived from psychoanalysis began to circulate in educated families, mothers of gay sons and shoplifting daughters blushed at the thought that their clued-up friends must surely be wondering about their parenting skills. Cohen is too subtle an historian to suggest that every kind of family issue fitted one cultural pattern. Or, as she succinctly puts it, "secrecy tends to run in circles rather than straight lines." In the case of adopted children, the Edwardians' insistence on total discretion seems to have worked a treat. Before the Adoption Act of 1926, childless couples who used the services of a private agency such as The Haven of Hope for Homeless Little Ones regularly passed off the new arrival as their own.

Adoptive mothers resorted to large winter coats or long holidays by the sea in order to disguise the fact that they had not given birth. The decision never to tell the child about her origins served to graft her seamlessly on to her new family tree. It also dealt with any neighbourhood gossip about her moral make-up and the worry that, once she hit puberty, the little stranger might go the same way as her mother.

While the new act of 1926 certainly shored up the legality of the adoption process, its formalised paperwork also blew away the chance to keep things private. No matter how discreet the new parents were, someone usually found out and, from there, it was a short step to whispers over the coffee cups and taunts in the playground. Legislation designed to eradicate the earlier nightmare in which the birth mother might reclaim her infant had the unintended effect of exposing the adopted child to the trauma of knowing that someone had once given her away.

What marks out Family Secrets as an important book is not so much its breadth  there are also chapters on race, divorce and family therapy  as its depth. Each chapter has a painstaking architecture of original sources, Cohen having spent years working through the archives of institutions including the Tavistock Clinic and the Edinburgh Marriage Guidance Centre. The result is a clear-sighted investigation into what our forebears felt was private, and what they kept secret  and, most importantly, the difference between the two.

 Kathryn Hughes's biography of Mrs Beeton is published by Harper Perennial.