Why are architects so mercurial? Why is architect-speak so impenetrable? Rowan Moore, the Observer's architecture critic, answers question one and could not be accused of accusation number two. One of the UK's most accomplished writers on the profession, he critiques the most important buildings and the people who masterminded them with a style that is both entertaining and cuts through the crap.

Why We Build, his new book, does not disappoint. It does not set out to be a definitive study, more a set of musings. As he tours the world, Moore dips into themes. He guides us through the relationship between architecture and eroticism, power and finance in a journey from São Paulo to New York, Beijing and beyond.

The more eccentric the destination, the more amusing is the language. The Atlantis hotel in the ridiculous bling fest of Dubai is "a Blofeltian phantasmagoria of giant aquaria and rooms with views of sharks, which suffered the rare ignominy of being accused of bad taste by the British tabloid the Sun". He stops off at the super-rich folly of self-made software mogul Larry Dean in Atlanta, Georgia  a mansion the colour of salmon mousse. The marital bed is "engulfed by writhing turquoise vegetables  and by gooey blossomings the colour of vulvas".

Architecture is perhaps the ultimate form of human self-expression. A less charitable term would be self-indulgence. Sex plays a major part in this narrative, from a farmhouse in Sussex designed for swingers, to the compulsiveness of American architect Stanford White, a "man with fire between his legs and lead between his eyes". Frank Lloyd Wright apparently built his career "servicing the wives of rich men while designing their houses".

From the late 1960s housing estate outside Amsterdam called Bijlmermeer, the author takes us to Moscow, to the Park of National Economic Achievements and the marbled metro stations. These testaments to Stalin personified "force with charm, the twinkling eyes with the mailed fist".

He describes Kijong-dong, a prosperous-looking North Korean town in sight of its enemy to the south, "which turns out to have no glass in its windows nor rooms behind its doors". Architecture as tool of propaganda is a long-established motif, but instead of the bombast, Moore chooses to focus on the relationship with power, not from the perspective of client, but of architect. Thus in his view Mies van der Rohe "lingered longer in Nazi Germany than was decent, apparently in the hope that the regime might adopt his architectural style". He also tries to explain the thinking of Rem Koolhas when he agreed to build a new HQ for Chinese state television.

Moore has conjured a rare feat in producing a work that will be appreciated by professionals and punters alike. My only quibble was the use of small, dusky black-and-white photographs and illustrations. I would have preferred more boldness and colour.

His overarching message is that buildings are inextricably linked to time and space. They are defined by their context. I agree with this assertion, with a little personal experience. As chair of Turner Contemporary, I watched as David Chipperfield created a gallery by the sea in Margate that has taken the UK art world by storm. Chipperfield is doggedly proud of his achievement; he is also self-deprecating. His "shed", as he calls it, has transformed not just the built environment but the mood in a town that has seen so little success in recent times.

"Buildings will never exist independently of the stuff around them, and the events and thoughts that occur inside them and out," Moore concludes. "One description of bad architecture is that it ignores this inescapable circumstance."

John Kampfner is the author of Blair's Wars and Freedom For Sale

Imagine a speculative development, on the edges of London. The designers were anonymous; the style was out of fashion; the materials were cheap; the build quality low. Soon after it was built, and left partly unfinished, local people started calling the half-occupied units "the Stumps". One of the streets was nicknamed "Coffin Row". "The whole estate," wrote one contemporary, "was a graveyard of buried hopes." Later, this depopulated failure became a byword for overcrowding, with a notoriously exploitative slum landlord. It was known for rioting and drug dealing. Imagine, finally, that this place had been praised by an architectural writer as "a wonder of the world, a feat of construction the equal of the Pyramids".

You get zero points if you suspect we're talking about something modernist, gigantic and built of concrete, or even somewhere made of steel, Trespa and glass, dropped on a former dockside. In fact, the description is of an area now so expensive and desirable that Peter Mandelson needed a personal loan to be able to afford to live there, somewhere now so safely establishment that many in the Conservative front bench are based there. It is, of course, Notting Hill, and its potted history is one of many that make up Rowan Moore's Why We Build. The point is not to stress how foolish were those who denounced the area in the 1860s, nor to hail it as an entirely successful form of urbanism and architecture that could be easily repeated  but to chart how structures can change their meaning and function, can shift radically in public estimation, without anything much changing in their outward appearance. Throughout Why We Build, Moore, former architect, former director of the Architecture Foundation and current Observer architecture critic, has no axe to grind, unusually in an art form that lends itself to polemic. There is advocacy here, but not on behalf of any particular architectural style, ideology or period; the prospect of architecture as a fight between Quinlan Terry and Richard Rogers clearly depresses him. Yet after charting the failure and eventual success of somewhere like Notting Hill, a reason has to eventually be found. Class? Gentrification? Nostalgia? Centrality? Moore opts for adaptability, a certain laissez-faire within buildings outwardly defined by upright classical rectitude.

This extends to most of the places described in the book, and Why We Build is nothing if not broad. Subjects include: John Soane's unhappy family life, American classicist Stanford White's ferocious sexual appetite, Adolf Loos's curious ideas about tattoos, Ken Livingstone and the failure of Blairite urbanism, the difficulty of getting contractors to build a Zaha Hadid building, the surprisingly delicate architecture of Italian fascism, love hotels and, inevitably, Dubai. These are vivid and witty sketches, often elegantly flowing into each other. The scope does, however, make it hard to work out exactly what Why We Build really is. Partly it's a treatise on architecture's often uncomfortable links with finance and power, and an admirably unmoralistic one  Moore finds that only the Third Reich among history's various vile regimes was wholly incapable of creating interesting architecture. He doesn't try to extricate the buildings of, say, card-carrying fascist Giuseppe Terragni from their politics, but finds that they can be ambiguous and fascinating as well as fascistic. It's a rare and difficult line to follow, although in many ways quite obvious  a medieval cathedral, Moore points out, might have looked humane and beautiful to Victorian neo-Gothicists, but less so to a burning heretic.

Mostly it's a book about what happens when other non-architectural matter  capital, sex, family life, the caprices of function  barges into a discipline that sometimes likes to think of itself as pure and self-referential. Persistently, Moore calls for a happy but atypical medium between purpose and aesthetics, and outlines the dangers of mistaking an image of something for its actuality. A pungent example is the image of sensuality and public participation in London's City Hall, as compared with its functional inability to accommodate these things. The architecture of the Italian-Brazilian modernist Lina Bo Bardi is the nearest Moore comes to a model. Her informal but often massive and powerful structures in São Paulo are, like the stucco clerks' houses of Notting Hill, designed  here deliberately  in such a manner as to encourage use, transformation, rough treatment, while remaining the same buildings. He finds the work of the Buckminster Fuller or Norman Foster, which trumpets its adaptability, a failure by comparison. Stansted Airport's beautiful faceted canopy appears to look embarrassed by the chaos that goes on underneath it.

Moore's even-handedness and intelligence make this fine as a meditation on and history of architecture and its tortured relationship with everything else  yet it's hard not to wonder what an architectural philosophy based on it might look like. So many of architecture's great leaps and transformations have been driven by unreasonable, unfair people  Alberti, Ruskin, Loos, Mies  with ideologies to match. But they are capable of breaking with the profession's ingrained tendency to quietude and conformity; only after them can the work of assessment and adaptation begin. When their schemes creak, collapse and need to be redesigned, a reading of Why We Build will come in handy.

 Owen Hatherley's A New Kind of Bleak is published by Verso.