When I first visited Moscow, in what now seems a far-distant era, a giant red neon sign beamed Lenin's famous phrase "Socialism plus electrification equals communism" across the Moskva river. The slogan encapsulated several central aspects of communist thought. First, optimism for the future. Second, that science and technology were both by definition progressive forces but were impeded by capitalism; only under communism could they enable the building of a society of abundance for the many, not just the few.

The roots of this respect for science lie deep in Marxist thought, not least because Marxism was seen as itself a science. When Friedrich Engels's fragmentary text, The Dialectics of Nature, dating from 1883, was rediscovered in the 1930s, it was regarded as demonstrating that nature itself conformed to the philosophical principles that he and Marx had formulated in their decades of collaboration. Then in 1948 the American mathematician Norbert Wiener published his path-breaking book (and invented the word itself) Cybernetics, a way of thinking about how self-regulating systems interact dynamically with their environment, both changing it and being changed by it. Cybernetics ("circular causation") was a way of understanding how systems could show apparent goal-directed behaviour without consciousness. Soviet philosophers seized on the concept, elevating it to the status of a "fourth law of dialectics".

Following Stalin's death and the slow thaw, initiated by Khrushchev, that lasted from the mid-1950s to the mid-60s, Soviet planners, economists, physicists and mathematicians flourished. They persuaded the Soviet leadership that, using cybernetic principles and the newly developed computers, the centralised, planned Soviet economy could at last be made efficient. By 1980, Khrushchev claimed, the Soviet Union would overtake America; communism would have defeated capitalism. For a while, in the aftermath of Sputnik and Gagarin's space flight, it looked as if he might be just be right.

So what went wrong? It is this question that Francis Spufford explores in Red Plenty. Not as history, nor yet exactly as a novel, but in a series of loosely linked chapters, each a vignette in which fictional characters rub shoulders with real ones. The cast list at the front distinguishes real from invented, although two of the latter, as Spufford makes clear, are "stand-ins" for the real economist Abel Aganbegyan and the molecular geneticist Raissa Berg (the latter still alive at last count and now resident in the US). The key real figures are the mathematical prodigy and later Nobelist Leonid Kantorovich, whose initial calculations on how to improve plywood production blossomed into a comprehensive plan for the economy, and Sergei Lebedev, designer of the first generations of Soviet computers.

Spufford has long had a somewhat eclectic interest in the interactions of science, technology and society, as evidenced by Backroom Boys, his tales of post-1945 British "boffins", and he has certainly done his homework here. It isn't every work of historical faction that is backed up by 70 pages of footnotes, references and sources. He speaks no Russian and has built heavily on the work of English-language historians of the Soviet Union such as Sheila Fitzpatrick, but his learning sits lightly. Admittedly, his fictional characters are two-dimensional types without inner life, appearing often only for a single chapter. They are chosen to illustrate daily details of Soviet life, from the primitive conditions of collective farms and the career moves of young, upwardly mobile party apparatchiks to the semi-criminal underworld of the fixers who helped to circumvent the idiosyncratic inefficiencies of central planning, and the brutally authoritarian medicalised childbirth of 1960s Moscow.

It is through their voices that the impossibilities of Khrushchev's dreams are revealed. The planners were, above all, rationalists, committed to a cognitive view of the world that allows no space for human frailties, greed, corruption, or mere inadequacy. The fictional, but all too realistic, Maksim Maksimovich Mokhov is deputy director for chemical and rubber goods at Gosplan, the USSR's central planning agency. He is faced with a problem at a viscose plant. It shouldn't have happened; the plant is fairly new and the process simple  new machinery, simple inputs of chemicals and woodpulp, and "Trees into sweaters! Brute matter uplifted to serve human purposes". But one machine is unaccountably wrecked. The plant managers request a new machine  but that requires revising the targets for the factory that produces the machine. And that requires replanning the inputs into the machine tool factory. Everything connects in a nightmare combinatorial explosion. Down the line, only the semi-criminal fixer, so characteristic of Soviet life, can sort out the problem.

Human error and human corruption are inevitable features of being human and not a machine, and need to be built into planning assumptions. What cybernetics should have taught the mathematicians and planners, tucked away in the relative comfort of the Siberian science city of Akademgorodok, and the Moscow offices of Gosplan, is that systems work best when self-organised from below, not centrally planned from above in a command economy. Although this is now widely recognised, it has been an expensive learning process in terms of both political and individual human tragedy.

Self-organisation is a fundamental feature of living systems, and was indeed well understood by Soviet biologists of the 1930s (and some in the west), before Stalin's dogmatic destruction of scientific creativity  and of the creators themselves. Spufford is weaker on his biology than on technology. He also pays far too little attention to what fatally weakened the Soviet economy  the escalating arms race with the US. It wasn't just that there was no trickle-down from military innovation into the desperately inadequate production of consumer goods. Even civil science was denied the computers, centrifuges and relative freedom from rigid planning constraints that so privileged the military.

But could it have worked? Can we ever once more believe that we, the people, could create a just, equal and abundant society? Red Plenty ends with the question that must carry all our hopes and fears, as Khrushchev, the deposed pensioner, broods: "Years pass. The Soviet Union falls. The dance of commodities resumes. And the wind in the trees of Akademgorodok says: can it be otherwise? Can it be, can it be, can it ever be otherwise?"

 Steven Rose's The 21st Century Brain is published by Vintage.

 This article appeared in the corrections and clarifications column on 27 August 2010. In the article above, we incorrectly rendered a famous pronouncement by Lenin as "Socialism plus electrification equals communism". According to the Oxford Dictionary of Political Quotations, the phrase Lenin uttered at the All-Russia Congress of Soviets in 1920, where he was reporting on the work of the Council of People's Commissars, was: "Communism equals Soviet power plus the electrification of the whole country".

One day in 1992, just after the USSR collapsed, I met a mid-ranking officer from the old Soviet interior ministry on a train heading to southern Ukraine. He'd lived in the Soviet system since he was born half a century earlier and knew no other way of doing things. We got talking. He was astounded when I told him that in Britain we had no ID cards or system of residence permits to keep track of who lived where. I saw a look of panicked incomprehension forming on his face and waited for a question such as, "How do you keep tabs on people, then?" What he actually asked was, "How do you know how much bread to make?"

I'd known the Soviet Union had a planned economy, where bureaucrats, rather than the market, decided what goods and services would be supplied to whom and how much they'd cost. But that system died at the end of 1991, replacing a world of shortages (full pockets, empty shops) with a world of poverty and hyperinflation (empty pockets, full shops) overnight, and it was only in occasional startling remarks like the officer's that I glimpsed the deep otherness of the communist system from that with which I grew up.

Insofar as there's a general, popular sense of the past, the first thing we forget is the way people used to do business. When we think of the Soviet Union we think of Stalin, the labour camps, parades on Red Square and propaganda posters, not men in suits and ties in a Moscow skyscraper working out how many summer dresses the ladies of Vladivostok should be allowed to buy.

Daily life in the Soviet Russia, Ukraine or central Asia is generally considered to be contained in one word: queues. In this strange, risky and compelling book, effectively a collection of short stories with the Soviet economy of the 1950s and 60s as its theme, Francis Spufford points out that westerners didn't always see it that way. In the 1950s, the Soviet Union was growing faster than any other country except Japan.

"For a while, in the late 1950s and early 1960s, people in the west felt the same mesmerised disquiet over Soviet growth that they were going to feel for Japanese growth in the 1970s and 1980s, and for Chinese and Indian growth from the 1990s on," writes Spufford. "Beneath several layers of varnish, the phenomenon was real."

Not everyone in the non-communist world felt disquiet, of course  socialists in the west took inspiration from what seemed to be a working alternative to capitalism. In the middle years of the cold war, the Khrushchev-Eisenhower-Kennedy era, competition on the economic front between the superpowers was as intense as on the military. As Spufford puts it in the voice of Khrushchev, one of his characters, it was "a race to see who could do the best job at supplying the ordinary fellow on the beach with his cold drink".

Disappointingly for romantic socialists, the book highlights the materialism of Khrushchev's project and, indeed, the Russian admiration for American consumerism that predated his rule. "The Americans got it," muses Spufford's Khrushchev on his first visit to the States. "They understood that if ordinary people were to live the way the kings and merchants of old had lived, what would be required was a new kind of luxury, an ordinary luxury built up from goods turned out by the million so that everybody could have one."

Moscow failed; Moscow lost. For all the idealism, brilliance and ingenuity of the minds that tried to fashion a Marxist utopia on the ruins of post-second world war Russia and Ukraine, the greater part of Red Plenty is given over to why and how the Soviet communist project failed.

There were plenty of Soviet economists in the 60s who wanted to bring computers into the system, who wanted to put prices on things that bore some relationship to the labour required to make them, who knew that rules like the tonne-kilometre target  whereby a factory that did its job by moving 100 tonnes of materials over 1,000 kilometres was thought more successful than a factory that achieved the same result by moving half the goods over half the distance  made no sense.

But with the death of Stalin, the director class had lost the main incentive it had to be more efficient  fear. The replacement of Khrushchev by the Brezhnev clique in 1964, and the discovery over the next few years of vast reserves of oil in Siberia, made it easy for the people running the country's economy to brush the reformers aside.

Even Karl Marx, I suspect, would have found the Soviet economy of the 1960s an inside-out sort of place. Each spring, factories would guess the quantities of goods and materials they were going to need the following year, and order them; only in summer would the state planning committee Gosplan tell them what they were supposed to produce, and how much.

In a piquant series of linked short stories in the middle of the book, Spufford shows how it worked in practice. We see Maksim Mokhov, a kindly bureaucrat in Gosplan, helping out a viscose factory that has had one of its machines destroyed in an accident by commanding a plant in the Urals to supply them with an updated version of the machine.

We move to the bosses of the viscose factory, who secretly got their engineer to arrange the accident, as the original machine couldn't make enough viscose to fulfil the plan they'd been given.

Next, we cut to Chekuskin, the shady middleman working outside official channels to help desperate factory bosses deal directly with one another. Chekuskin is called in because the Urals plant is refusing to give the viscose factory the updated machine. It's the wrong price, apparently  it's not too expensive, it's too cheap. The reason it's too cheap is because it weighs less than the machine it replaces, and since it's a machine from the chemical industry, it's priced according to how much it weighs, as if it were an amount of coal.

It should be pointed out that this isn't an actual, recorded event. Spufford's method is to create fictional characters and fabricate incidents closely based on real anecdotes and contemporary observations. His book comes without an index, but it does have 53 pages of notes explaining the convergence and divergence of fact and fiction.

It's a method that would normally repel me, but the audacity of the subject and the superb craftsmanship of the writing won me over. This is not a dry book, even though at times the economic reasoning can be hard to follow. Spufford invests his characters with loves, joys, whimsy and weakness and puts them in believable worlds  sometimes eerily so, given that he doesn't speak Russian.

He can get carried away with his own virtuosity  the detailed descriptions of how a computer works and how cancer begins are as superfluous as they are brilliant  but more often his stories cut richly, subtly to the point. The interlude where Chekuskin the middleman enters the parallel Soviet universe of life criminals, the Lawful Thieves, foreshadows the criminalisation of the Soviet economy, and a gruesome tale of childbirth, with pain forcing a woman to exploit her husband's party connections to get some painkillers, suggests the birth of corruption in a grimly literal way. Apparently Soviet obstetricians used to tell expectant mothers that labour pain was a myth invented by capitalist doctors.

Red Plenty is not merely a series of quaint historical vignettes. By presenting a society in which business and finance were ordered so differently from our own, it provokes us into comparisons and connections; to consider, for instance, the extent to which ingenious software, marketing and the consolidation of big corporations has put us at the mercy of a kind of ad hoc capitalist Gosplan.

In the end, although there were moments in the 20th century when the Soviet Union ran the US and western Europe pretty close when it came to the war on want, it was never a contender in the war of people getting what they want. But the materialism of both sides, the idea that plenty is the ultimate goal of society, is a mean-minded sort of dream.

Soviet planning of the kind Spufford writes about lives on now only in North Korea and Cuba. But in the contrast between the two kinds of capitalism today  authoritarian capitalism like modern China's versus democratic capitalism like America's  we see the same race to be the better provider of consumer plenty. It's become commonplace to hear Britons returning from Moscow, Dubai or Singapore ridiculing democracy as so much "faffing about", getting in the way of business. I'd like to think that Britain and America would prefer democracy even if we lost Khrushchev's race to get the best cold drink to the guy on the beach, not because we've won so far. But I'm not sure.