Banaz Mahmod was aged 20 when she was murdered in Mitcham, south London, by her father Mahmod Mahmod and her uncle Ari Mahmod. The pair, who were jailed for life in 2007 for the crime, believed she had brought shame on their family by taking up with a man they considered unworthy. In fact, Banaz had warned police she was being targeted by her family but was ignored. Her death, like other "honour killings", raises real concerns about the kinds of societies humans create for themselves. Indeed, for scientists such as Mark Pagel, a biologist who heads the University of Reading's evolution laboratory, murders like these are genuinely vexing  for it is not easy to account for their existence using standard evolutionary explanations.

"Killing an offspring is the most costly and direct thing parents can do to harm their own reproductive success," Pagel points out. "Parents rarely do it unless affected by temporary insanity brought on by divorce, separation or depression." As he notes, however, there is one exception to this rule: honour killings  "one of humanity's most extreme and repugnant behaviours"  which are tolerated in several societies.

But why do some cultures excuse such horrific acts? For a father or mother to extinguish the most precious input they have into future generations makes no obvious biological sense. Yet the United Nations estimates there are around 5,000 honour killings of young women every year.

This statistic is startling. Humans have created cultures that bring out the best of our species in terms of innovation, self-sacrifice and co-operation but at the same time can produce horrors which include parental murders that are excused by neighbours and friends. If we want to explain our social behaviour in scientific terms, we must not only highlight the advantages of close co-operation with others, but also account for the perverse actions that can emerge simultaneously  hence the efforts taken by Pagel to account for honour killings.

Central to his thesis is the fact that humans do not co-operate mindlessly, unlike other creatures that establish elaborate societies, such as ants and termites. In these cases, the role of the individual is suborned totally to the greater good of the nest or hive. Humans are still capable of expressing great individuality within a society. So think of our role in society as more like that of a venture capitalist who is trying to invest money, says Pagel. We seek out individuals with whom we can form the best alliances needed to set up friendships and businesses. The rewards are bountiful and can be seen in all the glories of modern civilisation, though we have to take care. This process only works if we select good candidates for co-operation and are selected, in turn, by others. To make sure this happens, says Pagel, we need to have good reputations. "Reputations act as the currency we use to buy trust and co-operation," he states. Thus we hold open doors, stand aside for others, help the elderly, give to charity and even risk our lives to save animals. It is all done to build up our own reputations so that others will seek us out and co-operate with us.

But sometimes, says Pagel, it all goes a little bit too far and reputations are elevated to an almost religious status. They are considered to be heritable and are reckoned to run in families. As a result, those who are thought to be endangering a family's reputation are attacked by their close relatives. The result is an honour killing. Seen from this perspective, the act is a co-operative one taken to a grotesque, overzealous level. "A reputation acquires the worth of a human life," as Pagel puts it.

By a similar token, we honour our war dead with elaborate ritual, he adds. "They deserve our highest respect for their sacrifice but somehow in doing so we acknowledge that some form of payback  reputation enhancement  is needed to keep families willing to send their sons off to battle. No ant, bee, or wasp would make such a request."

Pagel's arguments are complex but skilfully assembled, creating a convincing thesis that accounts for the rise of human culture, a process that began to flourish around 70,000 years ago. "We had to acquire the social and psychological systems that could somehow overcome and tame selfish instincts born of millions of years of evolution by natural selection to cheat, exploit, dupe and even murder one's rivals," he states. The fact that we succeeded in this task can be seen in the art, spacecraft, computer games and particle accelerators of modern humanity.

Crucially, Pagel's arguments steer away from reliance on biological determinism. There is nothing pre-ordained in our genes to account for the societies we have created. They are not the product of hard-wiring in our brains, but of careful, considered rules of behaviour, a point that is also stressed by Jesse J Prinz, a professor of philosophy at the City University of New York. Just as we have built societies through the use of rules of behaviour, so these cultures determine how we think. Genes and biology have nothing to do with it, says Prinz. The vast majority of human behaviours  from pub fights to mental illness  vary in form and frequency from culture to culture. "Our actions are not ingrained," he states.

A slightly different tack is taken by Richard Sennett. He emphasises co-operation as a personal craft. We need to listen well and discuss common concern with others. Skilful practitioners are rewarded by being allowed to share the company of others, a process that we have enhanced over the centuries through co-operative rituals, from churches to guilds. Such institutions are fading from public prominence, however, and our capacity for co-operative behaviour is weakening, he warns.

All three books present compelling arguments that cover a vast range of human behaviours. Pagel's is perhaps the most thorough in its analysis, Prinz the easiest to read, while Sennett provides the best historical perspective. All are emphatic on one point, however. We are not prisoners of our genes. The societies we have created by following careful rules of engagement largely leave us free to act as we see fit, for good  and bad.

The search for the sharp dividing line between us and other species may be a wild goose chase. But as Mark Pagel's comprehensive history of human co-operation shows, there are capacities we have in abundance that appear as mere traces in the animal kingdom.

Chief among these is our remarkable sociability, which gives rise to what has the best claim to be the distinctive mark of the human: culture.

Culture is made possible by social learning. We can imitate and copy others to an extraordinary degree  no other creature comes close. Once homo sapiens learned this trick, all sorts of innovations could be passed on from person to person, group to group, meaning culture could change and diversify at a rate that far exceeds the glacial progress of biological evolution. Interestingly, there are more languages (around 7,000 in total) spoken by just one species of mammal than there are species of mammal.

Social learning depends on co-operation, which allows individuals to specialise to a degree which is unique in the natural world. Look at any group of animals and every member of each sex will be pretty much doing the same thing. Look at even a small human settlement and you will see different people doing a wide variety of tasks. This division of labour enables a thousand cultural flowers to bloom.

But how did evolution (which Pagel not uncontroversially takes as a survival competition for genes) permit co-operation to flourish? Simply because "together your outcomes are better than if you acted alone".

Pagel borrows a stark illustration of the icy, selfish logic of social bonding from Sebastian Junger's book War. While following a small platoon of US soldiers in Afghanistan's Korangal Valley, Junger was struck by their willingness to die for each other. But their bond was not love of country or even their fellow-man. Rather, says Pagel, they simply realised that they "were individually more likely to survive when they were all prepared to die for each other". Like the starving slime-mould amoebae who form a tower so that a few can be blown away to survive in more fertile land, the fact that this is near-suicide for most is better than certain death for all.

Co-operation can also make sense when the goal is selfishly to keep as much of a resource for yourself as possible. In a timely example, Pagel suggests that the reason the other EU countries are willing to cough up to bail out Greece is simply because the money they are giving away is worth less than the cost to their own economies if Greece defaults. In the end, richer Eurozone countries keep more of their wealth by giving quite a lot of it away.

Culture gives rise to a number of such paradoxes, such as the fact that it is precisely our exceptional ability to co-operate that makes us the most divided species on the planet. Put a Neanderthal in a time machine and take her to another Neanderthal culture 1.5m years later, and she would not notice the difference. Take a gorilla and put it in another troop, and it would know exactly what to do. On the other hand, if you took an inhabitant of Milan and put him in a mountain village in next-door Francophone Switzerland, the poor devil would be at quite a loss.

Social learning enables small groups of otherwise identical humans to create distinctive cultures. One purpose of the diversity this creates is to provide cues that enable group members to recognise people as "one of us"  someone they can trust.

This creation of an in-group, however, entails a clear differentiation from outsiders, and this is sometimes the explicit goal of cultural change. A community of Buian language speakers on New Guinea, for instance, once decided at a meeting to switch all of its masculine and feminine gender agreements at a stroke in order to distinguish its dialect from that of nearby villages. This need to differentiate becomes more important the more closely packed people are: 15% of the world's languages are spoken in the 312,000 square miles of the island of New Guinea.

The same drive that pulls people together can also make them turn on anyone different they perceive as a threat. Hence the alarming rapidity and viciousness with which it was possible to make Hutus massacre Tutsis in Rwanda, or for Jews in Germany to be identified as the enemy under the Third Reich. So too can the value of reputation. This is so important for securing trust, but can quickly damage us. Massacres and dictatorships are chilling examples of when culture comes to "exercise a form of mind control over us".

In these and other ways, co-operation's base roots in selfish survival do not always grow into more nourishing fruits. Culture has its thorns and toxins too. And because it is tied to self-interest, our willingness to co-operate is vulnerable. Authoritarian regimes, for instance, keep a very tight grip on power by spreading suspicion within social groups, undermining trust, and thus disabling co-operation. In Pagel's view, that is precisely why such regimes cannot sustain themselves indefinitely.

It is with a hopeful political message that he concludes this compendious account of the interaction between biological and social adaptation. What we need to do in a changing world is to work with our evolved capacities to create the kind of trust, common values and shared purposes that the crude markers of language, ethnicity and cultural differences cannot provide. And he sees evidence that this is already happening in the large cosmopolitan cities where people of all shapes and sizes rub along more or less contentedly together.

"Nothing in our evolutionary history specifically prepared us to live in large societies," he says. But in a refreshing rejoinder to the argument that creatures who evolved to hunt and gather in small groups on the savannah can never flourish in the contemporary world, Pagel adds "almost everything about the way culture works does."

The clarity of Pagel's absorbing account is enhanced by the fact that he looks at everything through the one lens: evolution. No doubt other histories of cooperation from other perspectives would have different, perhaps conflicting, things to say. But partial though his view may be, he paints a broad picture, impressive for its detail, accuracy and vivacity.

 Julian Baggini's The Ego Trick is published by Granta.