Has there been a pop singer more flagrantly reticent than Scott Walker? Summon the YouTube spectre of him in his 1960s pomp and you'll discover a performer seemingly cold-stored at his very moment in the sun. He mimes his part in the Walker Brothers' "The Sun Ain't Gonna Shine (Anymore)" (1966) as if from some debilitating distance, and seems just shy of a fastidious shudder when co-singer John Maus goes all snake-hipped for "Land of a Thousand Dances". A couple of years later Scott pouts, solo now, through Jacques Brel's "Jackie" in a mocked-up bordello on, of all things, The Frankie Howerd Show. The voice soars, but the boy shivers. As he sang later, in a line much quoted in this collection of essays on his darkling oeuvre: "This is how you disappear."

The exaltation and Icarus fall of the man born Scott Engel have been sketched before: from rock'n'roll prodigy to reluctant object of Walkermania, Europhile crooner to avant-garde enigma, via alcoholic MOR lostness. The chronology emerges piecemeal in No Regrets, which is apt given the acute angles of Walker's career graph, commercially and (so the myth has it) artistically. Here he is in his teens being touted as "the gonest gasser of all gassers", gleaning production smarts as a studio hand for Phil Spector, recording the first Walker Brothers tracks with Spector's legendary arranger, Jack Nitzsche. The apprenticeship is important; by the time he arrived in London in 1965, Walker was well apprised of the lure of well-groomed professionalism versus the scarier potential of artistic self-fashioning.

Maybe he was already stranded. The rest, anyway, is well known: brief manic stardom, diminishing sales for four astonishing solo albums, the long slow trek to a place where he was in control. What is there to add to that narrative arc? Rob Young, author of Electric Eden, a remarkable history of visionary music in Britain, is well placed to fill the analytic gaps. A former editor of the Wire magazine, he has corralled a book that at first glance seems to depend on Walker's current experimental credentials. His last two albums have essayed a music of extraordinary intimacy and strangeness, schooled in the last century's avant gardes but with baritone phantoms of the familiar Scott still in attendance. Young mounts a compelling case for Walker as belated modernist: an American remaking himself according to English and French models.

Brian Morton pursues that comparison in an essay on Tilt, from 1995: Walker's song had evolved to a state of extreme lyrical abstraction, the voice grown higher and panicked among blocks and wires of indeterminate sound. By the time of The Drift in 2006, the now 60-something singer had vanished inside soundscapes that were constantly, as David Toop puts it, "flexing, sagging, cracking, breathing, stretched over bloody fluidity". There's a persuasive argument here that the recent Scott is the true Scott.

The longest and best essay in No Regrets, however, is Ian Penman's "A Dandy in Aspic", on the "lost" years when Walker, facing commercial failure, was huddled inside the smooth carapace of the consummate pro: taking care of business, grinding out standards after a Transit ride to the sticks, boozily committing to "the album of the TV show of the solo act". Penman is acute on the adult pleasures of 1960s MOR, and on the sheer craft and work ethic demanded of a pop drone at the time. But he is also alone among the writers here in truly, or at least imaginatively, divining the state of mind of such a figure, lost to internal exile and "so withdrawn into a hinterland of edgeless, glassy introspection and landscaped musical anonymity he almost doesn't seem sexually human. Almost, at times, an angel."

This is heady stuff, but it gets to the heart of Walker's 60s persona, and perhaps also explains something of the coldness of the later work. (Consider Scott's four songs on the reformed Walker Brothers' last album Nite Flights, in 1978, where he knowingly trumps the dystopic abstractions of recent Bowie.) In her essay on Walker's first solo album, Nina Power makes a similar point about the singer's sexual ambiguity, verging as it does on froideur, even (or especially) while he effortlessly emotes. "The bigger the songs get, the more he disappears." The vanishing act surely had something to do with his amazing looks, so unlike the cheeky or hairy male pop stars of his day: the young Scott was Bacall-beautiful, hard to credit, never mind touch. Toop cannily aligns him with Dirk Bogarde, another pretty studio pawn turned serious artist and commercial suicide.

But the more apt comparisons may be with women artists of his ilk: artists barred from, or allergic to, rock credibility, for whom the path from teatime TV to left-field studio fulfilment was equally arduous, figures as unalike as Marianne Faithfull and Kate Bush. Walker has seemed to some (as Penman puts it) "provocatively and unforgivably inauthentic". You just have to return to his first single from 1958, when he was 15, for an inadvertent hint of how he'd flummox expectations about surface and depth. "This must be," sings little Scott Engel brightly, "the living end."