Joachim Fest was the editor of Albert Speer's correspondence, as well as the author of an early biography of Adolf Hitler and the definitive account of the last days of the Third Reich. But if his subject matter was challenging, then he had a personality to match. Not only a noted historian, he was a public intellectual of rigour, unrelenting in his pursuit of the issue at hand.

During the bitter historians' dispute in the 80s about the Holocaust, referred to as the Historikerstreit, Fest argued tenaciously against the "singularity" of the Holocaust; an iconoclastic position at the time, and still controversial  and painful  to many now. Co-publisher of the Frankfurter Allgemeine Zeitung, one of Germany's most respected broadsheets, Fest was a thorn in the side of the West German leftist intellectual orthodoxy before the fall of the Wall. His great value as a journalist was that he could cut through to the nub of a question, often with a cool, if cruel, wit. Take his response to Günter Grass's Waffen SS membership, revealed 50 years after the fact. "I can't understand how someone who for decades set himself up as a moral authority, and rather a smug one, could pull this off."

So the uncompromising "Not Me" of the title seems entirely in keeping with Fest's character. Actually, it was his father's maxim, taken from the Matthew's gospel: "Others betray you, not I." Fest had an early grounding in intransigence, and this is what makes this memoir so intriguing. It covers not his career, but rather his passage into adulthood during the Third Reich.

The family milieu is fascinating, and will be unfamiliar to most British readers. Fest's parents were devout Catholics, proudly Prussian, and had been ardent supporters of the Weimar Republic. Regarding both communists and Nazis with deep distrust, they kept company with like-minded political activists and intellectual Catholic clergy, and had many close Jewish friends.

Great weight was given to learning in this circle, and Fest marks the stages of his formative years by the books he was given to read. At his boarding school in Freiburg, southern Germany, he buried himself in Schiller's Wallenstein "throughout the autumn and early winter of the Russian campaign, so that for me, Freiburg is forever associated with Schiller and as an absurd contrast to the ice storms and blizzards outside Moscow".

Immersed in both high culture and current events, the young Fest was also taken seriously by the adults who passed through his household. This was evidently stimulating for a quick and developing mind, and perhaps also established the intellectual certainty that so marked him out in adulthood. After Fest made clear to his headmaster how parochial he thought him, his fellow pupils asked: "Have you no manners?" "I certainly did, I retorted, but only where there was mutual respect."

The author's bone-dry humour is evident throughout the memoir. It emerges as just one of the many defences the family had to cultivate to endure the Nazi era. They lived with poverty, social exclusion, and under constant threat of visits from the Gestapo, but Fest doesn't take the moral high ground. If ever a book demonstrated the slow but seemingly unstoppable reach of tyranny, even into the private realm, it is this one. He shows how tempting compromise becomes, even while it remains loathsome, and bears witness to the cost and limits of individual courage.

His memories of his father are particularly affecting in this regard. Johannes Fest was a school teacher, whose repeated refusal to join the Nazi party led to him being banned from employment. From 1933 until the end of the war, this intelligent and energetic man was reduced to an observer, watching the gradual erosion of civil society around him, first with incredulity and then with horror. He concentrated his efforts on his immediate circle, urging Jewish friends to leave, and seeing off the Hitler Youth who came calling to forcibly recruit his sons.

Fest absorbed his father's scathing view of the regime, scratching caricatures of Hitler on to his school desk. At the meeting that confirmed his subsequent expulsion, Fest's father put his arms about him  a rare and bravely public show of approbation. His blessing was not easily won. Of Fest's chosen field, his father said the Third Reich was "no more than a gutter subject. Hitler's supporters came from the gutter and that's where they belonged. Historians like myself were giving them a historical dignity to which they were not entitled".

Fest senior learnt of the developing Holocaust through friends, and his sense of impotence must have been intolerable; it certainly left its scars. When his sons entreated him to write about his experiences after the war, he responded: "I did not want to talk about it then, and I do not want to now. It reminds me that there was absolutely nothing I could do with my knowledge." While others accepted requisitioned villas in leafy suburbs from the Allies, Fest senior refused. "For compensation, my father had only the knowledge of meeting his own rigorous principles."

Not Me is a quiet, proud, often painful, always clear-eyed memoir. That it was a bestseller in Germany is perhaps not surprising, but it surely deserves wide attention in the English-speaking world. It is illuminating of the man, of the times he lived through, and also of a rare kind of moral resolve, both sobering and inspiring.

 Rachel Seiffert's Afterwards is published by Heinemann.