The title of James Kelman's new novel is more welcoming than his past choices, which have tended towards the jagged (Not Not While the Giro) or the melancholy-dour (How Late It Was, How Late, which won the Booker prize in 1994). Mo Said She Was Quirky is almost a Roddy Doyle title, but the book carries on Kelman's campaign for truthfulness in writing, as he sees it. Unfortunately this is also a campaign against vitality. There's no set formula for vitality, of course  Dickens's version very different from Beckett's  but without it no writer can maintain a hold on a readership.

"Mo said she was quirky but it was more than quirky." "She" is Helen, 27, a Glaswegian working in a London casino, Mo her live-in boyfriend. Mo is a restaurant worker from a Pakistani family, the name she uses for him presumably the abbreviation for a full form that doesn't get a mention. The two of them met in Glasgow and moved south, partly to get out of range of Helen's nasty ex, father of her young daughter Sophie. The relationship is strong and solid, though Mo has no plans to introduce her to his family.

On the opening pages Helen sees, from the taxi that is taking her home from work in the early hours, a shambling down-and-out who may be her brother Brian. The rest of the book covers the next 24 hours or so of her interactions with Mo and Sophie, but mainly her thoughts. There's a little flurry of incident at the end of the book, by which time the reader no longer expects any pandering to the dramatic and doesn't quite know how to react.

One woman's life in a day  this is Mrs Dalloway territory, but drastically defoliated. The method is "virtual first person", so that the character is represented as "she", but with the bare minimum of context or commentary from outside. Kelman's technique is a little awkward in those opening pages. There must be less bald ways of explaining your set-up than saying "Mo was her boyfriend. She and her six-year-old daughter lived with him", or of indicating which side of the family a grandmother belongs to than explaining "She was Mum's mother", something of which Helen presumably doesn't need to be reminded.

Kelman's background is hardly more privileged than his central character's, and it's obviously important to him politically to give oral culture priority over the stale and literary. This sounds admirable but isn't straightforward. Kelman everywhere purges the apostrophe from abbreviated forms like "wasnt" and "didnt". Sometimes he drops the core conventions of the sentence (initial capital, final stop), so as to convey rhetorically the urgency and shapelessness of Helen's thought.

When you try to customise the toolbox of punctuation in this way you can end up multiplying the contradictions you want to eliminate. Kelman isn't the only writer to dislike "wasn't" and "didn't", Cormac McCarthy being another, but if the apostrophe is a class traitor, why retain it elsewhere, in possessive forms? If informality is the aim, what about the semicolon? It's a mark of which Kelman is fond, but this virtually ancien régime piece of punctuation must have pulled strings to survive the typographical Terror. And what's that double dot in the middle of the word naive, looking suspiciously like a tiara?

Real speech is repetitive and goes in zigzags, and the same goes for Helen's refracted monologue, but authenticity is an irrelevant notion in this context. Nothing remains oral once it's written down. Accent, intonation, eye contact, body language, social setting, all these fall away, and after that the choices are a matter of literary convention. When Kelman writes of Helen: "She had experienced a thing similar in the past to do with confidence, and her ex, it was him and whatever, she didnt know what to do just like standing still, that was all, like a panic but just so quietly and that cold sweat, just so  not able to move," the incoherence is artificial, making uphill work for the reader though a friend saying something similar would be perfectly easy to understand. We forgive friends their conversational tics, but it's harder to forgive Kelman for Helen's unrelenting use of "so so" ("so so wrong, just so so wrong" and "so sad, so so sad, really" on the same page), the constant overemphasis that achieves with so much labour no emphasis at all.

The strangest authorial decision governing the book is the virtual abolition of detail. Perhaps the idea is to reach general applicability without passing through specifics, as poetry occasionally can, but the trick can't be worked in prose. Even a novel as mightily indeterminate as Finnegans Wake is made up of melted particulars.

Helen worked at a casino in Glasgow before the one in London, but there's no portrait of either establishment, no account of atmosphere, rituals or tricks of the trade. Only towards the end of the book is it possible to believe that the author has even visited such a place. It's as if abstention from detail is a piece of righteous self-denial, mortification of the writer's spirit, but mortifying your reader into the bargain can't be a good idea. Kelman withholds detail even when it's unnatural to do so. Helen (for instance) obviously doesn't think of her ex as her "ex" but by his name, which doesn't appear. She refers to "the scary exhibition" near London Bridge station, as if for Kelman identifying the London Dungeon would be culpable indulgence, the beginning of a recovering addict's relapse into binge naming.

The few details of the Glasgow casino that do get through the puritan filter are bizarre. The only drink reported as being taken is tea. Old Chinese women apparently drop in to drink tea and to chat with their friends, and the management tolerates this, even when their voices are loud and carrying, because they are the place's "bread and butter". I'm not saying that there are no Glasgow casinos where old Chinese women drink tea and gossip at the top of their lungs, but when these are the only details given they seem utterly unreal. They stand out stark and stunted on the skyline of the novel.

Detail is the rain that makes the soil of a novel imaginatively fertile. Without it nothing can grow. (The semi-arid ecology of Beckett's novels can sustain life but attracts few visitors.) There are widely separated bits of succulent Scots vernacular in Mo Said She Was Quirky ("shoogly" and "fankled together" early on, then a long wait for "crabbit") which resemble occasional drops of dew let fall, either mercifully or sadistically, on to the reader's cracked lips. It's a sign of a truly desperate thirst to be pouncing on a specified place name for its illusion of moisture  Charing Cross station, mentioned on page 190, where Helen has to change on her way from her generic south London home to her generic West End workplace  like someone sucking stones in the delirium of dehydration, or a castaway cut loose in an open boat reduced to drinking seawater.

Helen's life is about confinement, the most poignant emblem of which, perhaps, is the cupboard in which her six-year-old daughter sleeps. Converted by Helen's boyfriend, the Mo of the title, from a walk-in storage area into a makeshift bedroom, it represents various and conflicting aspects of the trio's life together. In one sense, it is hugely hopeful: Mo, a Pakistani Muslim and not Sophie's biological father, has through ingenuity and determination hived off a private space for her in their cramped flat. But it is also scarily provisional: both faintly transgressive (their landlord might object to the alterations, if they are discovered) and potentially dangerous (what, wonders Helen, would happen if the makeshift shelves above were to crash down on to her daughter?).

Negotiating the contrary emotions produced by the ambiguous, equivocal contours of her life is what occupies Helen throughout much of this curious and curiously compelling novel. The vast majority of it takes place inside her head, with only the occasional foray to the family breakfast table or to the casino in which Helen works as a croupier threatening to interrupt the internal monologue; and its "action", such as it is, unfolds over the course of 24 hours.

During most of that day, Helen is to be found in the tiny south London flat she shares with Mo and Sophie. We first encounter her, though, in another constricted place, the minicab that she is sharing with her two workmates, on the way back from the West End. It is the dead of night, edging towards dawn; the streets are empty, apart from, suddenly, "a pair of homeless guys" shuffling across the road in front of the cab as it's stopped at lights. Helen, captivated by a tiny human drama  will they make it to the other side before the lights change, and why is no one else as gripped as she?  is suddenly utterly wrong-footed. One of the men, she realises, is her brother, Brian, whom she hasn't seen for 12 years and didn't even realise was in the same city (both of them are far away from their native Glasgow). Unsure of what she has seen and unable, in any case, to do anything about it, Helen allows the cab to trundle on and deliver her back to her sleeping household.

What follows is a day at once relentless in its grinding routine and peculiarly liberated from it; it's as though Helen's maybe-glimpse of her maybe-brother, a definition of estrangement, has made everything else strange as well, and as though she herself has been tipped out from her warm, cloistered flat into the dark and empty city. As she goes about her daily business  the night-worker's frequently futile attempts to get some sleep, the services she must perform for her daughter and for Mo, the all-too-quick preparations for another night at work  her thoughts loop and swirl, running round in circles, repeating themselves, petering out.

There is little that doesn't cross Helen's mind, from the imperfections of domestic life  the lack of money, the clanking boiler, the dripping kitchen tap  to the unfathomable unknowns of personal history. At the centre of her thoughts is Brian, which leads her to contemplate, obliquely, what might have fractured their family, from her cold, undemonstrative mother to the father who couldn't let her brother be ("It isnt strange to find in families a parent doesnt like a child. You had it in stories and films. It was an old subject; even in the bible.") At other moments, she remembers her ex-husband, Sophie's father, a macho man from whom she had to escape; and swings between a whole-hearted gratitude to Mo, for the life he offers her now, and an impatience with what she sees as his frivolity and lack of gravitas.

But these summaries, although accurate enough, suggest a shape to Helen's thoughts and, consequently, Kelman's prose, that simply isn't there from paragraph to paragraph  and isn't intended to be. Rather, its stream-of-consciousness style  from the (somewhat affectedly) dropped apostrophes and the repeated words ("stupid stupid", "tired tired tired"), to the sudden expostulations against the world's cruelty ("Cages for children my God that was so evil") and its fragmentation as unconsciousness beckons ("But she needed bed, eyelids / A thickness too / But Brian")  presents us with a succession of fugitive, unreliable thoughts in which one must struggle to discern a pattern.

It makes for a narrative that would be hard to describe as attractive, but is also hard to draw away from; it's more like trying to read through a thicket. Its least successful moments, funnily enough, come when it is at its most concrete. In these cases the book seems to be aiming for some real-world significance  in Helen's anxieties about her daughter's future as a woman ("Screaming was so important. Men could laugh but it was. It was men had to learn. Some did and some didnt") or in her preoccupation with the way people react to her relationship with Mo. Here, Kelman veers close to becoming didactic and sententious; one wonders if it is as a result of writing, for the first time in his career, from the point of view of a woman. What works best in this peculiar but powerful book is its more fabular moments; that tiny bedroom-in-a-cupboard, for example, or the image of a woman peering through the windscreen at the figure of a man she thinks she once knew.