"A printed collection [of lyrics]," says Stephen Sondheim at the beginning of Finishing the Hat, "is a dubious proposition." Indeed: like making a musical out of the invasion of Japan by the west, or the tortuous painting of a canvas, or the intimate relationships of a group of New Yorkers, or the assassination of American presidents. All dubious propositions, all triumphantly carried off, as is this book, one of the greatest books ever written about craft in the theatre, which also happens to be a self-portrait of one of the most striking and original artists of our time. It is entirely typical of Sondheim that in writing a book of such apparently narrow focus, he should have produced a work of vast breadth and scope. The surprise is how moving, how deeply romantic the book is  surprising, that is, to those who persist in thinking of Sondheim as merely ingenious, a deviser of musical crossword puzzles, instead of the passionate explorer that he is, irrepressibly searching for the new forms that will keep alive the art to which he has devoted his life.

We student actors and directors who piled into Her Majesty's Theatre in 1972 to see Company had no doubt that we were present at a revolution in musical theatre: all other musical shows we'd ever seen suddenly seemed childish. Urban loneliness, self-deceptions within relationships, the fear of commitment  no one had ever expected to hear about those sorts of things in a musical. "It's the little things you share together / Swear together / Wear together / That make perfect relationships. / The concerts you enjoy together, / Neighbors you annoy together / Children you destroy together, / That make perfect relationships." The telephonic pulse of the opening bars  a switchboard in music  and the great underlying melodies surging up through the orchestral texture with thrilling metropolitan energy were to us the very sound of now, the characters' emotions identical to our own. We laughed, we cried, but most of all we hugged ourselves for having, as we felt, discovered this genius, our contemporary. In fact he was 42 at the time; I don't believe we even knew then that he was the lyricist of West Side Story. But when, the year after Company, Gypsy finally arrived in London and he stood awkwardly on the stage with the composer Jule Styne as the applause went on and on at the triumphant first night, we realised that he was no guerrilla waging war on the musical, but someone deeply plugged in to its great tradition.

Finishing the Hat is to a large extent an appraisal of that tradition and his place within it. Thanks to the happy chance of being the best friend of the son of one of the giants of American musical theatre, Oscar Hammerstein II, who all but adopted him, Sondheim was superbly well placed to learn not only his basic craft but the lofty principles of the musical play from the horse's mouth. Now, of course, Sondheim is himself the horse's mouth, and he imparts what he has learned with playful rigour. The book starts off bracingly, with a statement of the three principles of lyric writing (Content Dictates Form, Less is More and God is in the Details) and a brisk outline of the ground rules, in the sixth of which we learn that "Oscar Hammerstein II is not my idol". We have already learned that Sondheim "cordially but intensely" dislikes the lyrics of Noël Coward (a statement which earned him hisses at the Cheltenham literary festival a couple of weeks ago), and will soon discover that he has equally rebarbative things to say about the work of some of the deities of lyric writing  Alan Jay Lerner ("a chameleon of one colour"), Lorenz Hart ("sloppy"), Ira Gershwin (who, he believes, showed off shamelessly because he knew that he was merely talented, whereas his brother was a genius), even Hammerstein, whose much-loved lyric for the great Jerome Kern number "All the Things You Are" is thrashed to within an inch of its life. He comes back to Coward, who really gets it in the neck second time around: "overstated, sentimental . . . 'written' rather than experienced".

There is another lyric writer, though, to whom he gives the hardest time of all: Stephen Sondheim, many of whose most admired efforts are found to be obtuse, inelegant, exhibitionistic. The sin of sins, in his view, is when a lyric forces you to hear the composer's voice rather than the character's, a self-confessed failing of his. Reading his deconstruction and damnation of the lyric of "The Little Things You Do Together", the very words we kids so adored in Company back in '72, is like overhearing a devastating tutorial in which the tutor and the tutee are the same person. In every instance, one is forced to agree  if, that is, you believe, as he does, that logic is the sine qua non of a lyric, and that tortured syntax, strained rhymes, "poetic" writing, are simply poor craftsmanship. The severity of his judgments is underpinned by his personal philosophy: "if you think of a theatre lyric as a short story, as I do, then every line has the weight of a paragraph." Very occasionally, he is quietly boastful about his own lines  "The Ben I'll never be, / Who remembers him?" from Follies makes him glow with forgivable pride.

But despite his brilliant facility, nothing is ever easy, because on each new show he has set himself a specific challenge  a challenge that always finally resolves itself as technical: the doing of it is all. Scattered throughout the book are observations about his craft which are riveting, and rivetingly expressed: of one of his songs, "Getting Married Today", for example, he notes that "the patter sections may seem difficult to sing in one breath as they ought to be sung, but in fact they're calculated to alternate vowel and consonant sounds in such a way as to make them easy for the tongue, teeth and breath to articulate . . . in the best patter songs, the faster you sing the easier it is." If Finishing the Hat is not the Bible of lyric writing, it is certainly its New Testament, expounding an approach to the work which provides the best hope for the future of musical theatre, whose possibilities Sondheim has done more than any other writer to open up. Picasso and Stravinsky endlessly reinvented themselves, but he has constantly reinvented the form, and  true to his own principle  he has done that by constantly seeking out new content.

He claims to believe that the musical theatre is now, finally, in its death throes, but nothing in his practice reflects this. He has been famously generous to hopeful young composers, and, shy though he is, has made himself easily accessible. Early on he says that to him teaching is a sacred profession: he pays his debt to it over and over in these pages  by some rather brilliant teaching, but also by celebrating his own teachers, who have been many and various. This is, ultimately, what makes the book so moving: the creative breakthroughs, the sudden understanding of what one is up to, of what the craft requires, the things that transform a show and make it live. It is crammed full of brilliantly etched accounts of such moments, and trenchant character sketches of some of the great monstres sacrés of the theatre, but above all, of course, it is a cornucopia of some of the greatest lyrics of our  of all  time, some familiar, some rescued from oblivion, all given context and history. (Is there another writer who could have rhymed Gielgud with feel good?) The very best thing of all about the book (which takes us up to Merrily We Roll Along) is its last word: INTERMISSION. Act II is already on the stocks.

Simon Callow's My Life in Pieces is published by Nick Hern.

 This is a preview from tomorrow's Guardian Review. Also in tomorrow's Guardian Review, a very personal piece by Jonathan Franzen on writing The Corrections, Sarah Churchwell on Mark Twain's autobiography, Ian Jack on the Glasgow Boys painters, an interview with Günter Grass and much more ...

 Find out how to subscribe to the Guardian at a special rate