David Hume, an Edinburgh man, discovered, when he was just over 20 years old, that he could not believe the Gospel stories of those miracles upon which Christianity was founded, their improbability far outweighing the credibility of their authors. He was cautious enough, however, to suppress his arguments for more than a decade and when he did publish them, as part of An Enquiry Concerning Human Understanding, he did so in a spirit of irony: "Upon the whole we may conclude that the Christian religion was not only at first attended with miracles, but even at this day cannot be believed by any reasonable person without one."

Richard Holloway, in contrast, did resign as Bishop of Edinburgh in 2000 when he was 66. He did so with sorrow and his memoir is the account of his growing breach with the Episcopal church, until he sadly walked away from it and, perhaps less certainly, from Christianity.

The difference between Hume and Holloway is that Holloway is a romantic. Before he was 14, when he went to board at Kelham, the Anglican seminary in the Midlands that prepared boys for the priesthood, he had experienced the "latency" of landscape. He used to walk miles in the hills above the Leven Valley, usually alone, feeling the yearning for that something more, something nameless of which the landscape spoke but which eluded him.

In a less Wordsworthian spirit, he sought this "something more" in the American movies he loved, many of them westerns. It was his romantic imagination that "propelled" him to embrace the monastic rigours of Kelham and to love and understand its ethos. The Christian faith, indeed, might well have absorbed and satisfied him if its institutions had not so often demanded not mere faith but dogmatic certainty.

Holloway's apparently bottomless candour reveals a touch of bad faith, even at Kelham. He felt that while others were really pious, he played the pious part, swinging the incense just a little too enthusiastically. As children say, he was "being" a novitiate, he was not really one, just as Sartre's waiter was "being" a waiter, as, duty-bound, he attended almost too assiduously to his clients' needs. Later, as the priest in charge of Old Saint Paul's, a poor parish in Edinburgh, Holloway seemed to reach a plateau of near-peace, by becoming one of those active Christians who find God only in the hopeless and dispossessed, the less said and the more done the better.

But here, too, he watches himself "being" a sort of social-worker/priest, and slightly overplaying the part; at one time, he decided that this involved following the early Christians in holding all things in common. Money, time, food were all to be shared not only with his curates, but with every vagrant, mad or sane, drunk or sober who knocked on the door. This, fortunately, was short-lived. Holloway, the real imaginative solitary, could not stand it. But it may have increased his sense of being a whited sepulchre, a disappointment to himself and others, which had haunted him since he left Kelham without becoming a celibate priest.

Richard Holloway's developing thoughts about the nature and purpose of religion, and especially about the status of the Christian narrative, slot seamlessly into the story of his own life; in fact, they form its principal drama. There were two things wrong with the work of the Christian fathers who shaped the Bible and established the church. The first was their ignorance of the origins of the universe. We cannot blame them for this, but we should not pretend to share it when we know better.

The second was culpable, even then. They did not understand the nature of myth. This failure has had a profound effect on religion, producing the finally intolerable tension between pretending to believe a narrative to be factually true and understanding the meaning of that narrative, the truth that it contains, without denying that it is the product of imagination. It is Holloway's insistence that Christianity is a great work of the human imagination that makes his memoir so compelling and so intense. What he loves about the narrative is its central figure, who possesses endless pity for human beings and is endlessly subversive, in preferring compassion to rules. What he came to hate about the church is its insistence on rules, which turns it to cruelty, not pity. The attitude of the church towards women and homosexuals, which Holloway in the end could stand no more, illustrates the way the supposed rules drive out love.

Nobody, whether interested in religion or not, could fail to be intensely moved by the last chapters of the memoir. Holloway, unlike Hume, is not a cautious man. He does not tell us what he said to George Carey, then Archbishop of Canterbury, when he, visiting Holloway's patch, publicly uttered a fatwa on the recently published Godless Morality. He tells us only that he was "not pleased". But another time, his unruly tongue got him into deep trouble. Hearing of the absurd arrangements of "flying bishops" to look after those who could not agree to women priests, he exclaimed: "Oh the miserable buggers! The mean-minded wee sods!" What a deeply lovable man; and what a wonderful book he has written.

When Richard Holloway resigned as Bishop of Edinburgh in 2000, the tabloids crowed over what they saw as his final confusion: for years he'd been their favourite "barmy bishop". He must have been sorely tempted to justify himself with some straightforward autobiography. In fact, he produced a series of books on doubt, forgiveness, meaning and the human condition  and has only now turned to himself. It's been worth the wait. Leaving Alexandria gives a profound sense of the benefits, as well as the difficulties, that accrue from taking a zigzag path through life.

Holloway was the child of working-class parents who lived "in a small town on the west coast of Scotland called Alexandria". At the age of 14 (in 1948), partly to save the expense of another mouth at home, and largely because he was already in thrall to the notion of the "given-away life", he entered Kelham Hall, the mother house of the Society of the Sacred Mission; he describes this as "an Anglican religious order that trained uneducated boys for the priesthood in a monastic setting that was its own world, self-sufficient, entire unto itself". For the next four years he devoted himself to laying the foundations for a training in the priesthood.

Kelham was tough. Because it was "a complete culture that appeared to be unconcerned with what went on outside its gates", it encouraged an almost slavish adherence to particular codes and practices; at the same time, any reasonably questioning student was bound to jib at its strictures. Time and again, in later parts of the book, Holloway refers to his "divided mind", to his sense that he is "one of life's leavers", and to the gulf between his intentions and his capabilities.

It was Holloway's intention to return to Kelham after two years of National Service, but first came a holiday in the West Country  and the mute recognition that he was in love with his colleague-companion. Holloway makes this admission  and all subsequent references to the same man  very quietly, and for this reason alone it feels significant. Especially since it is closely juxtaposed with much more candid references to the frustration he felt around women. When he was appointed as the priest and personal assistant to the Bishop of Accra, he says the daily sight of scantily-clad females drove him almost crazy. Precisely what we are meant to make of this contrast is not spelled out. But it is striking that one of Holloway's most distinguished contributions to the church in later life was his support for gay priests.

Back in Scotland, and now ordained, Holloway found the emphasis of his faith had shifted. It was not so much good thoughts that preoccupied him, as the need to perform good deeds. A spell working in the Gorbals only strengthened his attraction to the idea of "the rebel Jesus", and although this meant an increase in certain kinds of doubt, the same emphasis survived when he was appointed rector of Old St Paul's in Edinburgh.

Holloway remained in post at Old St Paul's for 12 years; he was married by the time he arrived, and had a young family from whom he evidently drew great comfort, but the tensions in his life continued to grow. Caring for parishioners, giving hospitality to the needy, living briefly in a mini-commune, comforting the dying and burying the dead: these all proved his commitment to the given-away life, and brought a sense of settledness despite their exhaustions. But they reminded him of qualities and convictions that eluded him. St Paul's lament haunted him more and more audibly: "Demas hath forsaken me, having loved this present world."

Not surprisingly, the years after leaving Old St Paul's were peripatetic and filled with self-arguing: he accused himself more or less continually of lacking faith and obedience. At the time he became Bishop of Edinburgh in 1986, the churchman Holloway understood that he had a highly conspicuous role to play before God and in society, but the sceptical Holloway felt the force of his doubts was irresistible. Although he did some wonderful work as bishop, especially in the cause of women priests, as well as gay priests, by his own admission, he was "deficient in the carefulness gene".

Holloway has good reason to feel angry about his resignation, even though his dwindling faith had made it inevitable. Angry with certain elements of the church, and angry in particular with George Carey, who was Archbishop of Canterbury at the time, and might with more tact and cleverness have prevented the débâcle. But he governs his tongue. The memoir ends with certain statements of plain good sense ("The problem for the church in the 20th century was that many of the pressures for human emancipation were coming from the secular rather than the religious sphere"; "The church can never just do the right thing because it is the right thing to do; it has to find religious reasons for doing it").

Even more valuably, it meditates on the ways in which a doubt-filled life can still be filled with grace. "The mistake," he says, "was to think religion was more than human. I was less sure whether God was also just a human invention, but I was sure religion was." This is simply put, but with the whole weight of a very thoughtful and courageous book behind it, it summarises an argument that a lot of people will find sympathetic, as well as compelling.

 Andrew Motion's The Cinder Path is published by Faber.

 This article was amended on 23 February 2012 because it referred to people's favourite "balmy bishop" instead of "barmy bishop".