This byzantine study of the home life and milieu of a fictional Edwardian children's author, Olive Wellwood, is a beguiling offshoot from the growing popular interest in the personal lives of the creators of 20th-century children's literature, from JM Barrie and Lewis Carroll to Enid Blyton. Curiosity about such types has perhaps been fuelled by the amazing grip on modern culture still held by fantasy-jocks such as JRR Tolkien and JK Rowling. Like Charles Elton, whose 2009 debut novel, Mr Toppit, explored a similar theme, Byatt has set herself the task of persuading readers to jump into the world of a make-believe writer, with no existing body of work to draw us in.

But The Children's Book is more than the anatomy of a bohemian writer and her family; it's a critique of a whole era. Introduced to Wellwood's unconventional set through the eyes of a poor runaway, Philip, we see how liberal philosophy and Fabian politics can be equated with the fairy tales she weaves. Phyllis, her favoured daughter, shows young Philip a tree in the Wellwood garden that bears silver pears and golden apples. But "you have to believe" to spot them. In the same way, through a thicket of plot, the idealistic social theories current in the run-up to the first world war are revealed as dubious, dangling articles of faith. A sense of encroaching disaster gives the story pace, and at points it has the raw appeal of good children's literature, with dirty Philip washed and put to bed like Oliver Twist at the Brownlows. (Though, unlike Oliver, Phillip proceeds to masturbate carefully inside the pristine sheets.)

Byatt's first book for seven years takes on the artistic themes of the day in exhilarating detail. She takes us to a premiere of Barrie's Peter Pan and to Paris to see the astonishing art in the Exposition Universelle. Sometimes, sadly, there is so much historical interplay that there is no space left for purchase on the modern age. The Children's Book has been called "ambitious" not just for its scope, but because it is "a novel of ideas". The trouble is, as a period piece its philosophical forays sometimes have the flavour of parody and so are less involving than they might be standing on their own feet.

The moral seriousness of AS Byatt's fiction derives much from her concept of responsibility; and responsibility, for her, is most importantly the business of marshalling and applying one's intellect to every area of one's life. Her new novel, a staggeringly detailed and charged re-creation of the period between the end of the 19th century and the first world war, overflows with people attempting to define their responsibilities, whether to fulfil them or to evade them; with those in pursuit of enlightenment or seeking to manipulate it; and with some simply attempting to unearth who they are and what they should do to survive.

For Philip Warren, the boy whose discovery in the store-rooms of the South Kensington Museum (now the V&A) opens the book, survival is a matter of making good his escape from life in the Potteries and finding, somehow, a way of unravelling and realising his semi-articulated desire to "make something". His hideaway uncovered, he is brought before Major Prosper Cain, the special keeper of precious metals, and Olive Wellwood, a "successful authoress of magical tales" who is visiting Cain for research purposes. Enquiring about the starved boy's background, Olive suggests, "You wanted to make something of your life, of yourself ... That's natural."

We are clear, immediately, that Philip's sense of what "a life" might constitute is different from Olive's, though their backgrounds are similar: Olive, together with her sister Violet, has also fled an upbringing of unrelieved poverty and traumatic loss. By marrying Humphry Wellwood, a banker by trade though not by inclination, she has created a life of plenitude, productivity and no little luxury; by harnessing her memories of the mining community in which she grew up and linking them with her instinctive feel for fairy stories and folk-tales, she has also created an independent life for herself. But over the course of several hundred pages, Byatt demonstrates that creativity means entirely different things to Philip and to Olive, and has entirely different effects on those around them.

Humphry and Olive's rambling farmhouse on the Kentish Weald, named Todefright, is a wonderfully achieved emblem of the particular slice of late 19th-century society Byatt wishes to show in all its precariously utopian varieties: swarming with children who are allowed to speak not only when spoken to, with rebels ranging from politely insistent Fabians to fugitive Russian anarchists, with unstable artists, with ideas and projects and determination. Its inhabitants pride themselves on their ability to speak their minds freely and to arrange their affairs according to progressive and humane values rather than in the service of unblinking Victorian stricture; Byatt is slyly comic on the extent to which these ideals are allied to the profusion of decorative earthenware plates and bountiful mid-summer parties. Beneath the house's surface, though, secrets multiply: infidelities that have made a mess of lines of paternity and maternity; expedient accommodations of truth and finance; lapses of thought or care for the consequences of one's actions. For Todefright's children, given terrifyingly partial glimpses into the adults' muddied affairs, the family home shifts from idyll to prison and back, their parents - or who they think their parents are - from beneficent protectors to child-like incompetents.

Olive's work stands at the centre of the novel, and extracts of it run throughout - the baby prince whose shadow is stolen from his crib, the girl who imprisons a group of miniature human beings in her dolls' house only to be imprisoned herself by another, larger child - thickening not only our sense of her subjectivity but also providing Byatt's imitative commentary on the children's literature of the time. JM Barrie, Edith Nesbit and Kenneth Grahame all appear on these pages, their attitudes towards children both as consumers of their fictions and as conduits for their interpretations of the world nimbly highlighted; and, in Olive, Byatt probes the motives and contours of the process of writing for children further. When Olive is pregnant once again, she seals herself away in her stories, partly out of financial necessity but also to shore up her individuality and to insulate herself from her unborn child, reflecting on her pleasure thus: "Blood flowed from heart to head, and into the happy fingertips, bypassing the greedy inner sleeper."

For each of the household's children, Olive has created a book, revealing her allegiances and, indeed, her actual biological connection by the differing amounts of time and creative energy she bestows. "The stories in the books were, in their nature, endless," we are told. "They were like segmented worms, with hooks and eyes to fit on to the next moving and coiling section. Every closure of plot had to contain a new beginning." Outstripping the others by far is Tom, Olive's eldest son, a boy who loathes the pretence of Peter Pan ("It's make-believe make-believe make-believe") and who decides to mimic an animal's non-attachment to the material world in order to protect himself. Eventually brought to grief by his mother's inability to separate him from his fictional alter ego, he is perhaps the novel's saddest and most significant casualty.

But casualties abound among the dizzyingly extensive cast of this novel, one that is powered by unexpected doublings, sudden appearances, disappearances and couplings, individual histories - and, for that matter, history itself - rapidly and uncompromisingly deployed. Mirroring the superficially orderly and harmonious Todefright is the ghastly Purchase House, the dilapidated home of Benedict Fludd, the master potter to whom Philip is apprenticed, a crazy makeshift household in which kilns explode and sexually explicit statues of children are kept in a locked cupboard. Here, the horrors of genius are writ large on the lives of Fludd's drugged-up wife, his traumatised daughters and angry son, his beleaguered and abused friends. Byatt's cleverness, in the dual figures of Olive and Fludd, is to keep the reader in thrall to their talents while showing the sinuous stealth of their neglect; and to underline that neither artist is left undamaged themselves.

It will probably never be said of Byatt's writing that she wears her learning lightly, and her lengthy disquisitions on the building blocks of her narrative both support and bloat the novel; her briskly delivered but expansive accounts of, among other things, the development of London's museums, of late Victorian banking crises, of pottery and puppetry and of the Arts and Crafts, Fabian and suffrage movements are never less than informative but sometimes a little less than compelling. A visit to the Exposition Universelle de Paris, in which electricity and machine guns are displayed alongside a Tyrolean Castle, and a later section in which the wonderfully drawn child-woman Dorothy travels to Germany to establish her parentage ("things got out of hand at a carnival", Humphry tells her), stay far longer and more suggestively in the mind because they are animated by characters with so much at stake.

But Byatt is brilliant on the gathering forces of England and Germany at the beginning of the 20th century, their contrasting attitudes towards the part that the land plays in the collective unconscious, their differing forms of nostalgia. That her novel closes with the first world war, with its arbitrary culling of so many of the characters whose "book" the novel purports to be, feels entirely appropriate. The war never feels like an inevitability, nor the numerous characters artificially inflated in order to provide soldiers for it; instead, it feels like the vast, traumatising shock that it was, its victims randomly alighted on, its effects making nothing, and yet everything, of what has gone before.

One could say, in the survivors' parade that forms the very final pages of The Children's Book, that Byatt provides us with glimmers of hope; connections are reforged, tiny instances of justice done, gestures towards continuity sketched. But this is a very dark novel, driven by an unsparing view of human nature and a clear-eyed analysis of the idea of human perfectibility. Despite some of its structural similarities to Byatt's earlier novel Possession, and its thematic links to the tetralogy that featured Frederica Potter, it reminded me most of Byatt's Little Black Book of Stories. In that collection and in this novel, Byatt reminds us in chilling fashion of the perils of artistic creation, and the duties of its exponents to find out the difference between what is real and what is not.

 Alex Clark is the editor of Granta

There's a moment in AS Byatt's immense new novel when a young woman gets some advice about the tentative designs she has drawn: "Define their limits." Her patterns acquire force and coherence almost magically when surrounded by bold lines. Defining limits is a principle that Byatt has rejected in this book, though she produces not shy shapes "blushing mildly to be present at all" but proliferating themes and characters, which spread out steadily past any possible frame until the centre is pulled apart.

Like Byatt's 1990 Booker Prize winner Possession, The Children's Book is a historical novel about writers and artists and their entanglements. Though the period here is somewhat later (the book runs from 1895 to 1919), the two books have formal resemblances. The new novel starts in a museum, as the earlier one did in a library. Byatt has again produced pastiche texts  children's stories and poetry  for her characters.

Byatt is an admirer of Iris Murdoch's fiction and has taken on a version of the typical Murdoch double plot. At the centre of one family group is Olive Wellwood, a successful children's writer. Benedict Fludd, a brilliant but unstable potter, is the head of another household within visiting distance. There's a flow of characters between the groups, so that, for instance, a boy found hiding in the Kensington Museum is taken in by Olive but then moves in with Fludd as his apprentice.

Benedict Fludd may owe something to Eric Gill; Olive Wellwood certainly owes much to E Nesbit, not just the writing but the Fabian viewpoint and the unorthodox married life. Art, politics and sex  most writers would consider that enough to be getting on with, but Byatt adds class by making Olive a runaway from a deprived childhood in the north.

If Byatt is a novelist of ideas, then they are often overlapping and contradictory ideas, lumped together rather than dramatised in isolation. An author of children's stories who is also a mother is someone who shuts herself away from her own children in order to weave spells for other ones.

Do Olive's stories start as stories for her children? Not really. She would make up tales for Tom, her first born, about inch-high warriors marching over the counterpane, but later she got into the habit of writing stories down in a special book for each child. Each book explores a different vein of fantasy  Tom's involves his search in the underworld for his shadow, stolen from him in his cradle. The stories are modelled on them (Olive feels free to raid the book for material to be worked up and published) rather than created for their pleasure. One of her daughters, Dorothy, doesn't actually like "her" story.

There is a suggestion that in some way Olive has decanted the essence of her children into the stories. When she turns Tom's story into a stage spectacular, perhaps she does something symbolically similar to separating him from his shadow (if she knew he had been hideously abused at his public school she might feel less guilty, or guilty about different things).

What is the part of Olive that produces the stories? Speaking to a friend, she identifies it as what we would call her "inner child", but that isn't the whole truth. Her stories, sharp and sinister as they are, are evasions of her real terrors, which are of the mines that killed her father and a brother. Both the prosperous household and the fantasy world are constructed "against and despite the pitched life of ash pits, cinders, rumbling subterranean horrors and black dust settling everywhere".

It's hard to imagine a writer less willing to identify with her inner child than AS Byatt. That's not to say she can't write brilliantly from a child's point of view: "One day she stood up, and staggered from grass to border, crashing down amongst the delphiniums and knowing them now as an acrid smell as well as a blue series of towers." But her need to elaborate the elements of her fictional world independently hampers their ability to combine. It takes even this astonishingly accomplished writer a long time to bring one vast sub set of characters to a simmer, and by the time we go back to the last lot they've gone cold again.

It's a strange choice not to conclude the novel's action before the Great War. Every year after 1900 makes the characters' preoccupations a whistling in the dark they don't yet see. Chronicling such a wide tranche of time has required more research than the characters can plausibly hold and it spills over into the narrative's own voice. One of the advantages of Possession being built around the yawning, yearning gap between past and present was that modern-day academics could provide some background unobtrusively.

A teacherly element is undoubtedly part of Byatt's literary personality and gradually it becomes dominant in The Children's Book. There is a potentially fatal unwillingness to trust the reader to get the point or the full range of reference. How does it help, for instance, to be told that one of Olive's children, Hedda, "was the traitor in all tales of chivalry and myths. She was Vivien, she was Morgan Le Fay, she was Loki"? Hedda is not yet 10. Joyce used to boast that his books would keep professors and critics tied up for generations  Byatt would rather do their work for them, to make sure they get things right.

The feverish desire to have nothing be missed can only be self-defeating. A gravely eloquent sentence about a bereaved mother's perception of her son's image everywhere, at every age  "They were all equally present because they were all gone"  loses some of its force when four of the 10 words are italicised.

It's important to get details right, but sometimes even an exact detail produces clutter rather than clarity. At the first night of Peter Pan, Olive looks at how family members are reacting to the plea that they show their belief in fairies by clapping. The only important point is that Tom, the closest to being an actual Peter Pan, doesn't clap at all, but Byatt describes each character's style of applause: "Dorothy and Griselda, somewhere between enthusiasm and good manners. Phyllis wholeheartedly, eyes bright. Humphry, ironically. Violet, snappishly. She herself, irritated and moved. Hedda, intently." It's like a word game that has got out of control.

Benedict Fludd describes how any one of the four elements can betray a potter and reduce months of his work to dust and ashes and spitting steam  "failure with clay was more complete and spectacular than with other forms of art". This is perfectly true and The Children's Book is not a failure on that scale, or anything like. It contains magnificent things, but readers are entitled to feel short-changed when a family drama slowly turns into a history lesson.