Critics were hard on Stephen Spender, but not as hard as he was on himself. He felt a failure, a man famous only for having famous friends. Each time he published something new, it was like throwing himself to the wolves. "People I meet sometimes say 'I've read your books.' Secretly, I do not believe that anyone has read anything (apart from a few anthologised poems)." Even achievements he might have been proud of brought him little satisfaction: "My lecture an enormous success (by this I do not mean it was a good lecture)." Accompanying everything there was a voice that said "You are wrong."

In a long career, he took up many noble causes  for peace, democracy, freedom of expression, intellectual exchange. But despite, or because of, his missionary goodwill, he tended to be seen as a duffer  shy, awkward, naive. "You will always be a poet because you will always be humiliated," Auden said soon after they met, and humiliation came naturally to him. The scandal over the CIA funding of the magazine he co-edited, Encounter, was his most public humiliation. His journals also include more comical examples, such as his account of walking down a street in Covent Garden and unwittingly, noisily farting, to the delight of some nearby boys and girls. He's more amused than embarrassed by this, until a "self-important" thought strikes him: what if they'd realised it was Stephen Spender farting? What would they think then?

Such agonised self-consciousness is an endearing feature of these journals, spanning more than 50 years and including personal material that hasn't appeared before. This isn't to say that they're confessional: on the contrary, when asked whether he was totally candid in his journals, Spender replied that he "did not feel impelled to be  or rather, I felt impelled not to be". Short on bitchiness, and with half an eye on publication, he's too nice to be a great diarist. But he's unfailingly curious and liberal-minded  and so down on himself that you can't help but take his side.

A close friend once put it to him that the reason he'd lost his way as a poet was that "you haven't been able to deal with the problem of your homosexuality in your writing." The friend, Reynolds Price, was also a lover, though you would never know that from the journals, which rather proves the point. Spender defends himself on the grounds that writing about his homosexuality would have resulted in sentimentality and "a lack of contact with the ordinary life of family marriage". More to the point, he wanted to spare his wife Natasha, whose sensitivity about his public image was greater than his own.

Were Natasha alive (she died two years ago), she would certainly have censored the most intimate passage in this book, which describes Spender's relationship with an American biology student almost 50 years his junior, Bryan Obst. Nothing explicit is recorded, in the manner of Edmund White or Alan Hollinghurst (neither of whom Spender seems to have read). But the pleasure he takes in his young lover is palpable. And there is one tempestuous scene, when Natasha overhears or intuits a furtive phone call he's making to Obst from the next room and he comes through to find her crying. The relationship continued, over odd snatched weeks, till Obst's death from Aids. But Spender denied or underplayed it, as he did his affairs  past and present  with other young men.

Natasha's preoccupation with her (and his) reputation is a recurrent theme in the journals, notably when a biography appears alleging that she had an affair with Raymond Chandler  distressed and indignant, she takes to her bed. Spender's response is sweetly supportive: even if true, it's irrelevant, he tells her; whatever she did for Chandler was done from compassion. A "bad conscience" about his own affairs made it easier for him to be understanding, perhaps. But his concern for her was genuine. He couldn't imagine life without her. The marriage wasn't just a front.

His absorption in "family marriage" is also evident in the many entries here about his two children, Matthew and Lizzie. He and Matthew regularly go off on "honeymoons"  bonding trips abroad  and he's equally close to Lizzie: they watch Some Like It Hot together on TV, which "really cheered us up after a dreary day". Even when the children have grown up and are doing well, he still worries about them: "One doesn't have reserves of philosophical  or cynical  indifference to one's children. Their unhappiness seems unbearable. It is worse than one's own unhappiness."

If Spender the devoted father comes as a surprise, Spender the dinner guest and lunch host is more familiar  but mightily impressive nonetheless. Can any other 20th-century writer claim to have hobnobbed with all of the following: Igor Stravinsky, Charlie Chaplin, Princess Margaret, Truman Capote, Ted Heath, Yehudi Menuhin, Guy Burgess, Sarah Ferguson, Hugh Gaitskell, Michel Foucault, Jackie Onassis, Margaret Thatcher, Susan Sontag, Beth Chatto, Vivien Leigh, Francis Bacon, Tony Benn? It's typical that while visiting the Guggenheim Museum in Venice, Spender is spotted among the tourists by Peggy Guggenheim and invited to cocktails. When he analyses the work of the artists and musicians he knew, he's not insightful or deep. But they're charmed by him and he by them. Whatever his other talents, Spender was good at making (and keeping) friends.

The endless lunches and lecture tours took their toll on his poetry, which came slowly or not at all. The journals were a fallback  a record of interesting times and interesting people that would guarantee him a place in posterity  but, as Spender was the first to admit, they were no substitute for the masterpiece he'd hoped to write before he died: "Under it all, there is the feeling I have never done my best." The later journal entries, some written during spells in hospital, are much possessed by death, without being terrified of it: with so many friends having predeceased him, he feels more at home among the dead than the living. His chief worry is how his death will inconvenience those left behind.

Among those who died before Spender was Auden, the dominant presence in the journals and a constant point of comparison. Spender envied his poetic gift but noted that Auden likewise envied him, first for having a large penis ("He was certainly affected by this and mentioned it on many occasions") and second for being a father. He admired Auden's devotion to work but was also pulled in the opposite direction, towards sociability, hedonism, "travelling first-class, giving people delicious meals, etc". When the choice came  perfection of the life or the art?  he went for life. And on balance, for all his doubts and sense of failure, and despite the voice telling him "You are wrong", he believed he'd chosen right.

In a famous photograph from 1931 (reproduced in this book), Auden, Spender and Isherwood face the camera, though Spender's eyes are looking off at an angle. Auden looks like an overgrown schoolboy, Spender like a cricket captain, Isherwood like a pocket film star or glamorous jockey. Spender is the central figure, but only as a requirement of photographic composition, thanks to his height. His arms are behind his friends, though it's not clear if he actually has his hands round their shoulders, as he does when the grouping was repeated in front of another camera on Fire Island in 1947. Spender's eyes are closed on the later occasion (he's in mid-smile and the day is sunny), while Auden and Isherwood grin warmly at each other.

All three writers tried to be true to literature without ignoring politics, and also to balance the claims of desire, commitment and public image. In 1935 Isherwood rejected the idea of marrying Erika Mann, to give her citizenship and safety, because he hated the idea of seeming to want a respectable facade. Auden stepped in without hesitation, as if marriage held no sacredness for him, yet he committed himself completely to his partner Chester Kallman in what seemed to his friends an arbitrary martyrdom (the relationship was open, but only at Chester's end). As a young man Spender was relatively frank about his interest in his own sex, but encouraged the idea that this was some sort of phase after he married Natasha Litvin in 1941, by whom he had children, Matthew and Lizzie.

It's a bit of a jolt to read in this new selection from Spender's journals (5 August 1980) that Tony Hyndman, with whom he had a difficult affair, "was the visible manifestation of something which was the deepest thing in my nature  my loyalty to the 'queer' world, the gay". And later in the same entry: "In the long run, I did, of course, ditch Tony  but I never lost my loyalty to a commitment which he represented. I know what Christopher Isherwood means when he writes unforgivingly of his 'queer' friends who get married." This passage was understandably omitted from the 1985 selection of journals, and appears here (along with a number of unambiguous entries) almost literally over Natasha Spender's dead body. Her name is given in the new book as having edited it alongside Lara Feigel and John Sutherland, but Feigel's introduction records that she was "reluctant" to include such entries. If you're reluctant you can be won over in time, but given that the decision to include the material wasn't made while she was alive (she died in 2010) the word "opposed" might be more exact.

No one reading the journals could reasonably think that the marriage was a fiction, but if there was an element of wishful thinking on both sides, Natasha paid most of the price for it. It would be odd for her to want the world to read about Spender's ecstatic involvement in the 1970s with a man 40-plus years younger than himself, and the humiliation it caused her at the time, when her husband made declarations over the phone without realising that an acoustical peculiarity of the building made them perfectly audible to her, two rooms away.

You could make the case that Spender wouldn't have kept those journal entries if he hadn't wanted them published, and that Natasha could have destroyed them herself if that had been her deepest need. It may be that the children have given their blessing  but it's still a decision that needed to be justified by the editors, not smoothed over by describing the passages as essential and fascinating.

It has to be said that Spender's journals aren't as entertaining as Isherwood's diaries. The word "journal" itself has a whiff of pretension, even before it gets a capital letter, as it often does here: "My life is getting absurdly social, and now it is worse because I am stimulated by curiosity about experiences to put in my Journal" (July 1955). Sometimes the style of referring to intimates is oddly stilted, seeming more appropriate to a public speech than any sort of private utterance: "Matthew (aged 9), who was sleeping in the twin bed during my wife's absence" Referring to "Sundrin Dutta, the great Bengali writer" may be well-meant, but repeating the phrase exactly in a later entry makes it look as if you haven't actually read a word.

Isherwood had the advantage of prose being his primary product, so that a diary could double as a workshop. When Spender reports a conversation about the candidness of his journals he refers to "one or two things in my life I would not write about because I did not understand them myself". This category includes "experiences of falling in love which seemed almost hallucinatory". Isherwood would have been baffled by this impulse to retreat rather than examine, and to ban the richest samples from the laboratory.

Still, keeping a diary is a sort of yoga, a stretching exercise almost guaranteed to promote suppleness of mind, and Spender's sensibility opens up unpredictably. He becomes better company as the book goes on. No one ever accused the later Auden of suppleness, though as Spender puts it with rather desperate gallantry, "if Wystan [] seems a bit fixed, it is in a fixed direction, not that he is stuck."

Right to the end, Auden remains a mystery to him, almost on a par with sex and death. As late as 1979 Spender is troubled by both the character and the working methods: "I did not think of him as having human feelings and I felt about his early poetry a lack of a personal 'I' at the centre of it." This could be rewritten in Auden's favour by saying that he didn't make a fetish of subjectivity in those poems, and this is part of what made them durable  the sense of their being full of electrical activity but not charged in the conventional ways.

Spender returns to the argument a few months later, suggesting that in Auden's case "the poet at once knows his lovers and friends more completely than they know him, because of his very intelligent powers of analysis, and less well because he never lapses into that mutuality which is shared knowledge of each other by the other". The lapse into mutuality as something Auden instinctively opposed is a strong and rewarding idea.

As for Spender's feelings towards someone who simultaneously pushed him forward and hampered him, they could only be a tissue of gratitude and suppressed resentment. In March 1995, only months before his death, he tells a story as if it was discreditable to Cyril Connolly when actually it is Auden who is shown in a bad light (it's to do with the appropriation of a valuable book). He can only express a grievance against Auden with a cover story.

In an earlier pair of entries, he garbles something Auden said on a visit, so that "he surprised me by saying he thinks endlessly about what form would best suit his subjects" (this recorded at the time) soon becomes "Auden said 'What obsesses me is form. So I put poems into them arbitrarily and make them as abstruse as possible'". Passing on both a true copy and a corrupted file, he can be both the faithful disciple and the betrayer.

In 1979 Stephen Spender spent a sleepless night asking himself "did I really like Wystan?" Part of his answer is to discuss Auden's jealousy of his endowment (not the poetic one). "To be totally honest now," he writes, "I should ask whether Auden was not a bit envious of me because I had a large penis. He was certainly affected by this and mentioned it mockingly on many occasions." There's a certain mutuality of abasement here, with one poet's littleness being put on record, while the other is diminished by having needed to mention it.