The title of the second volume of Susan Sontag's diaries comes from a note in the margin next to an entry from 1965: "spiritual project  but tied to making an object (as consciousness is harnessed to flesh)". It is a curious phrase, suggesting the paradox that is art: a real, tangible thing resulting from a long, indefinable process. It evokes, too, the duality of Sontag herself: the public figure, whose provocative essays can seem to readers intimidatingly confident, and the mind that made them  which, as her diaries reveal, was unusually full of pain.

The title was presumably chosen by Sontag's son, David Rieff, who will edit three volumes in total; the first, Reborn, which came out in 2009, started in 1947, when the precocious 15-year-old Sontag took herself off to the University of California, where she discovered her own bisexuality and the novels of Thomas Mann. Having transferred to Chicago, she married her professor, Philip Rieff, with whom she had a son, before leaving him to go to Oxford and then Paris. Critics responded to the self-aggrandising tone and the recounting of heartaches in that book with mixed feelings  partly perhaps because Sontag didn't approve the publication of the notebooks before her death, and partly because the angst-ridden diaries made her seem almost ordinary. This second volume covers the period in which she produced the main body of her essays and fiction. Rieff edits his mother's innermost thoughts only lightly, adding in names and correcting factual errors, but leaving it up to the reader to establish which of the notes might refer to which of the essays.

Having abandoned academia, Sontag spent her 30s writing and consuming New York's culture: watching films, attending "happenings", visiting the studios of her artist friends Robert Rauschenberg, Paul Thek and Jasper Johns. Her diary entries are often short and elliptical, consisting of observations ("Jasper Johns = Duchamp painted by Monet"), quotations from her reading, and, most of all, lists. But the majority of the volume is devoted to Sontag's consciousness, her notes clearly coloured by the "deconditioning" that took place on the couch of her therapist, Diana Kemeny. Still wounded by her breakup with the Cuban playwright María Irene Fornés ("I am frozen, paralysed, the gears are jammed  "), she embarks on a romance with "Carlotta" or "C", a woman who likes Sontag for her independence but who makes her feel needy ("I must not offer her my suffering  as proof of my love"). Unlike her essays, which warned against looking for hidden depth, her personal prose champions Freudian conjecture: on her dislike of her body (particularly her legs), her desire to please others, her "insatiable" appetite for culture.

Sontag's essays are arch, intransigent  so it is a rare pleasure to read, in her diary, discoveries being made in real time. She applies her mind to itself with enthusiasm, in one gripping sequence unravelling the unconscious inspiration behind her second novel, Death Kit, in which the protagonist, Diddy, plans his suicide. "Diddy. Daddy," she writes. "That's the source of the meditation on death I've carried in my heart all my life. Diddy is 33 years old. So was Daddy when he died." Heartbreak is also useful material; having been rejected by C in Paris in 1970, she returns to New York apparently broken, before using the experience to enlighten her "psychobiography". "I feel once again, and I rejoice that I'm not busy dying," she writes. "I'm still busy being born."

Her self-revision was necessarily helped by world politics. While Sontag's stylish essays can seem designed to shock  "Fascinating Fascism" is more about the lure of leather than the reality of Nazi Germany  the 60s forced her to face the realities of the politics she had ignored or aestheticised. In 1968, she participated in a trip to Hanoi for antiwar activists, and her longer, more polished notes on that time betray her bewilderment with a communist Asian culture in which "everyone talks in the same style, and has the same things to say". "Trip to Hanoi", written after her return to New York, is one of the few essays in which Sontag allowed the first-person to peek out through her public prose.

"My various selves," she writes in 1967, "woman, mother, teacher, lover, etc  how do they all come together?" The overall portrait gained from these journals seems to be of an impossibly fractured author  but the diaries also remind us that Sontag the writer and Sontag the woman, inevitably, occupy the same territory, so that even when she is writing about culture, she is, in a sense, exploring herself. Her interest in Greta Garbo, an actor with a talent for always being Greta Garbo, and in Ingmar Bergman's Persona, a film in which two women's identities merge, suggest a mind searching to reconcile interior motive with public image.

In the 70s, she realises her tolerance of communism is no longer tenable. The complications of real life impinge on her neat theses  as when she understands that, having advocated absolute free speech, a lack of censorship might inure people to sadism, or that, having championed only radical new art, classical literature could be forgotten. Being an "adversary writer" has a downside: "I can't help being dismayed when my minority taste  becomes majority taste  I can't help but be in an adversary relation to my own work."

Sontag appears, toward the end of this difficult, fascinating volume, diminished  by poor reviews of her films and, one might suppose, by the surgeries on her breast cancer (which, revealingly, she barely mentions, making only a few remarks about the book it inspired, Illness as a Metaphor). Her personal consolations  her partner, the poet Joseph Brodsky, and books by Elizabeth Hardwick and Nietzsche  do little to stave off writer's block.

The experience of reading the diaries as her depression sets in again is strangely mesmerising. When Sontag does find a way through, it is in a kind of creative destruction  a proposal for Notes on Notes, a book on the aphorism, that punchy, provocative mode she had adopted in her youth, and which she had since rejected for the pluralistic vision of On Photography. "To write aphorisms is to assume a mask  a mask of scorn, of superiority," she writes in her diary, adding that, in the end, "the aphorist's amoral, light point of view self-destructs". The book, like so many of Sontag's proposals, was never written. As her diaries reveal with such intensity, she harnessed only a fraction of her mind to produce the writing we have seen until now; the rest is consciousness.