There are a lot of log cabins in the work of the Alaskan author David Vann. His much-admired debut, Legend of a Suicide, was an interlinked set of semi-autobiographical stories in which the disastrous outcome of a father and son's attempt to subsist in a cabin in the woods reworked Vann's own experience of his father's suicide. The following novel, Caribou Island, featured the same set of characters and reached a similarly bloody outcome, though this time the ill-considered attempt to build a wooden retreat on a remote Alaskan island became a metaphor for a disintegrating marriage.

In Dirt, the setting has shifted from the wastes of Alaska to the suburbs of California, yet in all other respects it is business as usual. The subject is a violent and abusive family, the prose remains uncompromisingly direct, and the principal action takes place in a cabin in the woods.

Galen is a 22-year-old virgin, who has never met his father and lives with an overbearing mother on a walnut orchard enclosed by the urban sprawl of Sacramento. He's a Buddhist, a bulimic, and a voracious reader whose key texts are Khalil Gibran's The Prophet and a stack of Hustler magazines under the bed. Galen believes himself to be "an old soul  nearing transcendence, learning his last and most difficult lessons, his final disengagements from family". Yet his determination to resist the temptation of the flesh is tested to the limit during a family holiday to  where else?  a cabin, where his aggressively flirtatious younger cousin Jennifer torments him with flashes of her underwear.

The young protagonist of Legend of a Suicide conceived of himself as "part of a large despair that went everywhere his father went". Dirt revolves around a frenzied fantasy in which Galen seeks to dispose of his mother (to reveal how would destroy the dramatic thrust of the novel). The sense that Vann's fiction is all part of an ongoing attempt to purge himself of inherited trauma is reinforced by graphic descriptions of Galen's eating disorder: "If only there were some way he could throw up his family and not have them inside him any more."

The difficulty for the reader is deciding whether to take Galen quite as seriously as he takes himself. His reading of Gibran convinces him that he is a prophet "living in a time that was preparing to recognise him". Yet the marathon masturbation sessions (Vann seems to give his protagonist a "boner" on practically every other page) and his sense of "the entire planet conspiring against him" suggest nothing more than an emotionally arrested teenager.

At times one can be reasonably confident that Galen's moods are intended to have comic intent: "He lay on his bed, thinking that this was perhaps the prophet he was meant to be, the prophet who would free everyone from religion and send them back to bed for more sex." Yet his karmic musings on samsara  the Buddhist concept of "continuous flow"  become an excuse for some extremely loose and self-indulgent writing. There's a lax infiltration of unnecessary adverbs ("Galen's mind was just empty"; "It just seemed hopeless") and some bizarrely redundant phrases: "He lifted the lid of the piano, a large flat polished piece of wood on a hinge." We're even told at one point that Galen "used his opposable thumbs" to grip an axe, as if to clear up any ambiguity over whether the protagonist possesses hooves.

It may be that Vann, having become closely associated with the barren frontier conditions of Alaska, but now living a relatively comfortable life as a professor in San Francisco, has deliberately chosen to change tack by limiting himself to a couple of locations and a course of action that unfolds in under a fortnight. Yet Galen's epic bouts of introspection frequently threaten to bring everything to a halt. To give an example, here is his thought process on the subject of carpentry: "Each nail individual, metal worked by machine but not perfect  Lines cut on the shaft, also, and in this light, there was no shadow. Light as a presence, without source or direction or heat, a cold illumination that was general, and it was only in this light that you could see the true shape of the thing, the fullness of a nail." No doubt this ungainly pile-up of clauses is meant to convey the intensity of meditation: yet Vann's seemingly random dispersal of commas and conjunctions is altogether symptomatic of a novel that takes an extraordinarily long-winded approach to hitting the nail on the head.

There are plenty of novels out there that leave you feeling safe, that make it clear from start to finish what you're supposed to feel and think. And they clearly have a readership. Yet I often wish that unease sold more copies and won more prizes; don't you want to be shaken up by fiction, to have your complacencies hassled and frazzled, to be forced to take a peek into the abyss?

I don't know quite when I first got a whiff of real, vicious discomfort in David Vann's new novel. Was it just a few pages in, when Galen, its apparently normal 22-year-old protagonist, getting into his mother's car to visit his grandmother, decides  for no apparent reason  to attempt the manoeuvre without the use of his arms and falls flat on the gravel? Or was it some pages later when, eating dinner with his relatives, he crams an entire plateful of hot dogs into his mouth before heading upstairs to spew them into the lavatory, acknowledging to himself that he would also have dipped his head in there for a drink "if his mother had been watching"?

Certainly, it was long before the moment when, having been discovered having sex with his 17-year-old cousin, Galen realises his mother's plan to shop him to the police for statutory rape is for real. If he is to escape a life sentence, it is time for desperate action.

People often say of thrilling books that they read them in a single sitting; with this extraordinarily alarming and convincing piece of work, I had no choice. I would actually have preferred to eke it out but, whizzing through it in the small hours, there seemed to be no point where it felt safe to break off and turn out the light. Better  surely?  to know the worst than to keep on worrying and wondering?

And Vann's novel does get worse. Possibly his biggest achievement is never to allow you to guess quite how far it  or he  will go. No author is better at making you lose your literary balance and a large part of his brilliance is that he knows how to adjust the level of derangement to just short of most disturbing. And yet, though I believed every word, I'm still not certain I really understand this novel. Conviction without comprehension  a queasy but compelling mix. The last time I had this feeling was while reading Vann's hellish 2009 debut Legend of a Suicide.

At first sight, Dirt is a fairly straightforward family money saga. Galen lives with his mother in an isolated corner of Central Valley, California. His grandmother  in the early stages of dementia  is in a rest home and his aunt and cousin are coming to visit. For some reason never entirely explained, Galen's mother seems to have control of the not inconsiderable family fortune and is hoarding the money rather than allowing either Galen or his cousin, Jennifer, the opportunity to go to college.

The tensions  inevitable bickering between the two sisters and a less inevitable but shockingly precocious seduction of Galen by his underage cousin  are chillingly realised. But it's Galen himself  a superbly uneasy and memorable creation  who rocks this novel from start to finish.

Intelligent, imaginative, unruly, possibly unhinged and borderline repellent, Galen's responses to his world are intense. He runs around with hardly any clothes on, rolls in the dirt outside and, arguably worst of all, spouts Kahlil Gibran and Carlos Castaneda. It's hard to know whether to applaud or recoil from his intuitive lack of inhibition, his thirst for self-mortification. But when we learn that he has recently been banned from a new-age bookshop for insisting on "aligning his aura with a young woman who worked there", alarm bells start to ring.

And yet the brilliance of Galen is that so many of his ideas and actions, at least in the first half of this novel, make a certain kind of comic sense: when he eats a meal with his bare hands because "he didn't want to distance himself with a fork"; when he observes his family as "two-dimensional, flattened"; and feels that "the bulb and its harsh light made it seem that if you removed his grandmother, you'd have to cut her from the fabric of the world and there'd be a hole left". Here, you feel, is a young person's psychological and emotional anomie conveyed in all its solipsistic splendour. And then, just when you've allowed yourself to relax and enjoy it a little, the novel coils up tight and pounces.

Vann's gift  his quest, almost  is a willingness to explore the unimaginable, the unthinkable, on the page. He is the real thing  a mature, risk-taking and fantastically adept fiction writer who dares go to the darkest places, explore their most appalling corners. I haven't read a novel as rough and shocking or, importantly, as wise and warm as this one in a long time. It's not safe and it doesn't seek our approval  and I've certainly no idea what Vann wants us to think or feel about it. But isn't that a plausible definition of truly great writing: a piece of work that leaves our heads and hearts in flux  rolling, churning and, if we're lucky, changing?