No one has ever written quite like Iain Sinclair. He will, without doubt, prove the indelible diarist of our age - our post-punk Pepys - though his diaries take the form of walking tours. After his celebrated circumnavigation of the capital in London Orbital, Sinclair here returns to his centre: the streets around Hackney that he has walked almost every day for 40 years from his house in Albion Drive (how he must love the Blakean resonance of that name). His quest  for nothing less than the grist and meaning of E8 before it is flattened by the Olympic dream  is enlivened by his fellow travellers, all of whom become, in his hands, remarkable eccentrics. He makes you work to stay with him, but if you persevere there are many moments in the journey when a paragraph will make the hairs on the back of your neck stand up, so acute is your guide's comic observation.

No one has ever written quite like Iain Sinclair. He will, without doubt, prove the indelible diarist of our age - our post-punk Pepys - though his diaries take the form of walking tours, striding in particular that margin between obsessive note-making, internal dialogue and scabrous prose. He makes you work to stay with him and if you try to keep pace without a knowledge of underground London phrase and fable, of American poetry and French cinema, you may sometimes fall behind. If you persevere, though, there are many moments in the journey when a paragraph will make the hairs on the back of your neck stand up, so acute is your guide's comic observation.

This is a book in search of home. Having done his celebrated circumnavigation of the capital in London Orbital, that rancorous perambulation of the M25, Sinclair here returns to the centre, his centre, to the streets around Hackney that he has walked almost every day for 40 years, from his house in Albion Drive (how he must love the Blakean resonance of that name). Home is in a constant rearguard action against change. Sinclair, or the brilliant construction of himself that narrates these books, is the developers' worst nightmare, the scourge of "progress".

He needs a monumental focus for his rage. In the book in which he found his compulsive voice, Downriver, it was the Big Bang hubris of Canary Wharf; in London Orbital it was the empty folly of the Millennium Dome. Now, a decade on, it is the £9bn hole in the ground, the blue-fenced Olympic site, the crater of dreams. So scathing has Sinclair been of the 2012 project, in particular in an essay in the London Review of Books, that he has become an unlikely pariah among the authorities in his borough. A reading he was due to give at a Hackney library was cancelled at the request of the council. Hackney is therefore "a confidential report", the book Tessa Jowell would love to ban - sort of.

Sinclair is sceptical about many things, among them structure. His film-making career never got off the ground, he suggests, because he never believed in the editing process; there was nothing he ever wanted to leave out. The wanderer was overwhelmed with footage. He adopts the same principle in his writing: nothing and no one that crosses his path appears unworthy of deconstruction. He begins, tellingly, with a digression about Hackney's recycling habit which serves as a handy metaphor for his own method. "We are the rubbish," he declares. "Outmoded and unrequired. Dumped on wet pavings and left there for weeks, in the expectation of becoming art objects." The "we" here is, I guess, everything with a soul and for Sinclair, a soul is the preserve of the derelict and the discarded; it is found in the market where no one ever goes. "It is my own choice," he writes, "to identify with detritus in a place that has declared war on recyclers while erecting expensive memorials to the absence of memory. This is a borough that has dedicated itself to obliterating the meaning of shame."

If this sounds a bleak prospectus, it doesn't stay so for long; there is redemption at every comic turn. Sinclair's quest - for nothing less than the grist and meaning of E8 before it is flattened by the Olympic dream - is enlivened by his fellow travellers, all of whom become, in his hands, remarkable eccentrics. Some are familiar: the indefatigable pamphleteer Stewart Home; the feminist pioneer Sheila Rowbotham; the ever-stylish cultural mortician Will Self. Others Sinclair has made famous before: film-maker Chris Petit, the only man who can make the author seem optimistic; Renchi, his former housemate and alter ego. And there is the expected cabal of unlikely discoveries: the Owl Man of Albion Drive (a long-term squatter with a fetish for stray strigiformes); the Mole Man of Mortimer Road (a resident tunneller who began with a wine cellar and ended with a direct subterranean route to the underground).

You can never work out quite how Sinclair holds this all together, keeps it moving forward. Quests come and go - for the spirit of Julie Christie in Bethnal Green, for the heart of Joseph Conrad's Hackney darkness, for Godard's East End masterpiece. Occasionally an argument surfaces. There's an interesting dialogue at one point with photographer Stephen Gill, who has also been recording the erasure of memory brought about by the Olympic vision. Having gone on a walk with Sinclair, he balks a bit at the relentless apocalypse of the writer's take on things. "Hackney gets so much stick and of course we should present the truth as we see it," Gill writes, "but I would also love to help Hackney get back on its feet." Sinclair includes this, but he does not respond to it, quite. There's no place in his world for regeneration, only regret and the endless spur of curiosity.

It's this latter tension that makes his books, despite all their arcana, so human. Sinclair thought about leaving Hackney before he wrote this. He bought a bolt hole in Hastings, which reminded him of how Hackney had been before the estate agents exaggerated its iniquities. He could have gone, but now he knows he never will; if he discovers anything in these pages, it is that there is too much to stay for. His wife Anna, who has been not much more than the psychogeographer's version of "her indoors" in previous books, is here more of a muse, working as a teacher, raising their three children while her husband's mind wanders.

They first came to Hackney half a lifetime ago, full of the Sixties, and they have stuck with it, for better and for worse. Sinclair may have set out wanting to make this book another gazetteer of subversion, but along the way a different tone emerges, one that he perhaps least expected: that of an old-fashioned love story dedicated to a life lived in one place.

This is a book of approaching 600 pages about a single London borough. It has been produced by a big publisher, in the middle of a recession, with specially-commissioned illustrations and a lovely lavish cover. It is densely, sometimes opaquely written and has been obsessively researched for more than a decade. It is full of digressions, forgotten east London characters, and details about local bus routes. If you have never been to Hackney, more of a metropolitan vanity project may be hard to imagine.

And yet Iain Sinclair, as ever in his long and singular career as an explorer of the capital, is on to something here. Early on, he describes discovering a volume of photographs taken in one of the most untamed corners of this notoriously untamed inner-city borough. The photographs are fuzzy, hastily composed shots of traders and their customers at a car-boot sale held at a disused dog track: not much more than hungry clusters of people from many nations standing over scatterings of junk, as litter eddies around their ankles. The pictures were taken on a secondhand camera bought from the same market for 50p. Sinclair, always on the lookout for other urban chroniclers and London subcultures to write about, contacts the photographer. When they meet, the photographer tells Sinclair the book of Hackney car-boot sale pictures can "be found on eBay at £400".

In Hackney, perhaps more than anywhere else in Britain, or even Europe, the messy, unruly inner city has become a desirable commodity. Artists and bankers, squatters and politicians, punks and estate agents, in the last five decades all have moved to Hackney in large numbers, in search of excitement, kindred spirits, affordable property. London's centre of gravity has shifted accordingly. When I first moved to Hackney 15 years ago, to Stoke Newington, by then a relatively well-known and gentrified bit of the borough, most people I knew in richer parts of London had still never heard of my area. Now most of them live there.

Sinclair has lived in Hackney since 1968. His previous books, like this one vast intoxicating mazes of fact and fiction and memoir, have played a part in the rise of the borough and of the east of the capital in general. Yet this book is the first he has devoted solely to Hackney. With the marshy east of the borough being transformed for the Olympics, and the tightly-packed south and west increasingly occupied by wealthy incomers and property developers, he wants to capture the Hackney he loves, its history and deeprooted bohemia and intricate social mix, before they become absorbed into a more orthodox Britain. Sinclair, you suspect, also wants to get Hackney out of his system. "To construct a Hackney book, after all these years," he writes, "was to say goodbye."

This is a much more emotional, less oblique book than Sinclair has written before. A central element is simply the story of his life in Hackney: from a communal house in the late 60s, all smoke and hippie home cooking and fantasy film-projects, through a slightly more conventional 70s and 80s, as he takes labouring jobs and deals in secondhand books and has children, to his present-day existence as a cult author with a large following and a large home without a mortgage. Sinclair has off ered glimpses of all this before, but never so directly, and the results are surprisingly aff ecting. The great relisher of London's dark crevices now can't stop mentioning his wife Anna, his beloved companion through decades of burglaries and rat infestations and other pestilences. We even hear about the birth of their first grandchild  although Sinclair does cut straight from that to an interview with the former Baader-Meinhof member Astrid Proll. In the late 70s, when that terrorist group disintegrated and she had to flee Germany, Hackney proved a perfect refuge. "Here, all of a sudden, I met the whole world," Proll remembers. "I had friends in Hackney who ... had connections with the revolutionary struggle ... They did communal work, housing work. They did strikes. I worked for this drug-rehab programme ... Hackney council paid for my training."

Such reminiscences, going on for pages at a time, are another major feature of the book. Sinclair listens respectfully to local radicals and gentrifiers, to elderly Jews and ex-ravers, to ex-Hackney residents in bored exile in other parts of London and Britain. This, too, is a break from the usual Sinclair tone. He is a considerable stylist, with a quick and imaginative eye (huge Hackney street trees have "hawser roots") and endless wiry sentences. But his tough-guy, domineering prose can sometimes get wearing. Here, this writer of compelling monologues lets in other voices, and the book is warmer and more powerful as a result.

Out of these testimonies and his own memories Sinclair assembles a modern history of the borough: from slowly decaying backwater in the postwar decades, as factories closed and much of the original working class left for Essex, to new frontier for radical Londoners in the late 60s, to profit pportunity and staging post for the ambitious from the 80s onwards  a new Islington. Between 1980 and 1986 Tony and Cherie Blair lived in Hackney, just up the road from Sinclair, and without presenting many hard facts about the Blairs' life in the borough  here as elsewhere he prefers polemic and speculation and tantalising rumour  Sinclair makes much of their presence as a sign of things to come. By the 21st century, he sees bohemian Hackney disappearing fast: artists cannot afford to rent studios, his fellow 60s adventurers are dying off.

The role Sinclair and his writings have played in the buffing-up of the borough, in making Hackney seem attractive to potential incomers, is not something the book deals with directly. Instead it rails entertainingly against the Olympic Delivery Authority and the council and other agents of the borough's current "regeneration". His view of politicians and of government remains resolutely that of the 60s counterculture: no good can come of them. But Sinclair does concede that ageing rebels such as he, who bought their Hackney homes decades back, have done rather well out of the longterm increase in property values. And he perceptively suggests that the borough may attract a certain kind of middle-class incomer: "intelligent, focused, aggrieved ... conforming in nonconformity".

Sinclair writes less about Hackney's other classes. The borough is still one of the poorest in the country, with miles of council estates alongside its enclaves of period properties. He talks to a few members of the area's old working class, respectable and otherwise. There is a memorably gothic interview with Tony Lambrianou, an associate of the Kray twins, beside one of the soupy local canals. "As a boy I can remember coming down here to kill rats," Lambrianou tells him. The canal is also a favoured spot for disposing of weapons: "It's like an armoury down there, in the mud, goes back generations."

Since the second world war, immigrants from Africa and the West Indies, from Turkey and Vietnam and eastern Europe, have changed the borough as much as the waves of British incomers. Sinclair finds their worlds harder to penetrate. He spends pages loitering outside the many Kurdish barbers in Dalston, probably the busiest, most volatile, most fashionable quarter of Hackney. "Guns, knives, populist murals, bands, firecrackers, barrios, Dalston was Mexico City," writes Sinclair drolly, "without the justification of a successful revolution." But the owners are reluctant to let him take photographs or to answer his inquiries. Instead, Sinclair shows how multiculturalism has altered Hackney with a trademark piece of historical juxtaposition. A local school where 50 languages are currently spoken, he mentions almost in passing, was used in 1948 by Oswald Mosley to give a speech to supporters about "the threat of devastation from Oriental forces".

This book, like all of Sinclair's, is as much about loose ends, about subjects earmarked for future research, by himself or others, as it is about conclusive revelations. "I had no ambition," he writes, "to produce a definitive account." Part of the point of the book is that Hackney, like London, like any city, will never be completely knowable. There is not even an index here for readers who want to take a short cut to their favourite Hackney topic. Instead they must be patient and open to surprises, like drivers inching their way through the road works and reckless pedestrians of Dalston Junction. It may take a while, but along the way almost anything can happen.

 Andy Beckett's When the Lights Went Out: Britain in the Seventies will be published by Faber in May.