Published on December 15th, 2023 | by Hady George


Modern Vs Archaic Mammals, a match-up decided by brains

For over 150 million years, mammals scurried beneath the feet of dinosaurs. Despite being confined to small sizes, they diversified into numerous forms alongside their dinosaurian contemporaries. Although some Cretaceous mammals were about the size of modern badgers, most were not much larger than mice. This all changed 66 million years ago, when a giant meteor collided with Earth, wiped out the terrible reptiles, and set the stage for mammals to reach a new levels of diversity. Some of these post-apocalyptic mammals evolved remarkably high degrees of intelligence and opposable thumbs, others took to the skies. Many of these new mammals got much, much larger than badgers, and a few even got so large they surpassed even the largest of the dinosaurs.

The rise of mammals in the Cenozoic has been a thoroughly investigated topic for many decades now, however, much is still unknown about most of the early forms that lived during the first half of the Palaeogene (the first geologic Period of the Cenozoic). These archaic species would have been very bizarre to us, as they had all sorts of unusual body plans (morphology/anatomy) and unique lifestyles (ecology). Some of these extraordinary early Cenozoic mammals were the pantodonts, which include some of the largest animals of the Palaeogene that reached the size of some modern cows. The pantodonts Barylambda, Coryphodon, and Titanoides were especially large and specialised for herbivory, the latter even had exceptionally large canines that may have aided in slicing vegetation. There were many other herbivorous mammals in the early Palaeogene such as the tillodonts, which had rodent-like incisor teeth, as well as numerous primitive hoofed mammals such as Hyrachyus. There were even peculiar omnivores such as the arctocyonids. The namesake of the group, Arctocyon, was formidably large and had both large canines specialised for eating flesh and molars dedicated to grinding vegetation.

The large bear-like Arctocyon was a ferocious archaic mammal that would have had little to fear in the Palaeogene. Archaic placental mammals such as Arctocyon were very common in the post-dinosaurian world.

By the time of the final epoch of the Palaeogene, the Oligocene, began about 33 million years ago, most of these bizarre archaic mammals had gone extinct. This is when many modern mammal groups began to take over. The Oligocene saw the rise of modern hoofed mammals, carnivorans (dogs, cats, hyenas, bears, seals, etc.), elephants, and many other mammals we would recognise in zoos and documentaries such as primates and whales. This dramatic turnover in species has fascinated and bewildered palaeontologists for centuries. Despite how much we know about the success story of mammals over the last 66 million years, there’s still a great deal of uncertainty surrounding this drastic change in the late Palaeogene.

The tiny Leptictidium was a fascinating ancient relative of placental mammals. Mammals such as this would be outcompeted by larger-brained species better adapted for the Neogene.

Recently, research led by Dr Ornella Bertrand at the University of Edinburgh (where she was previously based), has attempted to decipher this ancient mystery. By tackling the problem from a completely novel angle, they made groundbreaking discoveries about mammalian evolution in the Palaeogene. Instead of examining the fossils with just the naked eye, they produced digital 3D models of fossil skulls using CT scans, which allowed them to virtually examine the brain cavities (endocasts) within the skulls. While the brain cavities within fossil skulls do not perfectly represent actual brains, they can give us an idea about the size of the brain relative to body mass and what parts of the brain are exceptionally large or small. The study of fossil brains has gained considerable traction in recent years, with palaeontologists all over the world discovering what sort of brains dinosaurs, Cretaceous marine reptiles, and even Permian protomammals may have had. From this information, we can determine aspects of the behaviour of extinct creatures. For example, we can tell Tyrannosaurus rex had an incredible sense of smell since the olfactory part of its brain is exceptionally large, even relative to the predator’s gargantuan body size. Additionally, we can infer the intelligence of many fossil mammals based on their very large brain to body size ratios, and also from their remarkably large neocortices, which is the part of the brain dedicated to problem solving, communication, and other cognitive functions.

The research by the University of Edinburgh-based team compiled digital brain cavities from a wide assortment of Palaeogene mammals and even some from end-Cretaceous mammals. They then compared body to brain size ratios between these ancient mammals and found an astonishing evolutionary trend. Right after the asteroid wiped Earth clean of the non-avian dinosaurs, mammals sky-rocketed in body size, but their brains did not keep up leading to a decrease in their brain to body size ratio. As a result of having proportionally smaller brains, many of these Palaeogene mammals effectively became less intelligent. Interestingly, it was discovered that the early members of modern groups were an exception to this phenomenon. While the archaic species kept relatively small brains for their body size, the early modern species evolved proportionally large brains. The proportional size of their neocortices expanded as well. These adaptations likely granted them increased intelligence compared to their archaic contemporaries. This discovery helps explain the success and longevity of modern mammals whilst the archaic mammals went the way of the dinosaur.

Once thought to closely resemble a tapir (as shown in the outdated silhouette), we now know that Palaeotherium was more closely related to horses. Relatives of animals similar to Palaeotherium including other hoofed mammals (ungulates) became the dominant land herbivores after taking over from archaic placental herbivores such as the pantodonts.

Despite this incredible discovery, there is still a lot we don’t know about the extinction of archaic Palaeogene mammals and the rise of modern groups. As the Palaeogene transitioned into the Neogene about 23 million years ago, global temperatures generally dropped. Perhaps modern mammal groups were better adapted to a cooler planet than archaic species, and this may not have been solely related to their brains. Many archaic mammals are not thoroughly researched and remain surrounded by obscurity. The genealogical relationships between them are infamously enigmatic and many well-preserved near complete specimens have not been properly studied for decades, if at all.

Finally, it’s always important to get the thoughts and opinions of the scientists behind these awe-inspiring discoveries. Below is an interview with the lead researcher behind this ground-breaking discovery, Dr Ornella Bertrand, that reveals some of the behind-the-scenes of being a palaeontologist interested in mammal brain evolution.

What was it like generating so many digital brain cavities? Were any species particularly challenging to work with?

Relaxing… for the modern species. Once you know what you are doing, you can put on a good podcast and do the work. Because you have to fill in sometimes more than 2000 slices of CT data, and even if you can interpolate between every 4 slices, it takes some time. Patience, learning the anatomy, and keeping in mind why you are processing the CT data was key.

The fossils can definitely be challenging because in contrast to modern species, many of them are filled with matrix (sediment), so you have to do a bit more manual work when processing the CT data. But honestly, I was always so excited to see what I would get that it was okay. When you are done, you can finally see what the brain or the inner ear of this fossil from millions of years ago looked like and you are the first person to see it. It is quite special and particularly exciting work to do.

What are some limitations of using brain cavities instead of actual brains (which don’t fossilise)?

Yes, they are definitely some limitations, the big one is the fact that we cannot see inside the brain, do thin sections, and really study the different regions of the neocortex that deal with high cognition. This is the part of the brain that is really expanded in humans, cats, dogs, whales, cows, horses… (actually in quite a bit of mammals!).

BUT of course, there are lots of advantages in working with endocasts. First, we can study the neurosensory system of fossils. You can also integrate a lot of modern species that are already in museum collections. Working with actual brains is tricky because you need to use preserved remains of living species and the sample availability becomes limited very quickly e.g. when considering endangered species.

What are some big unanswered questions in brain evolution? Could we try to answer them by studying brain cavities in extinct species?

They are so many! From understanding when different senses such as vision and olfaction emerged in animals to how much ecology and the environment shaped the brain throughout evolutionary times.

I think that fossils can provide answers to these questions that modern species alone cannot. Regarding the emergence of senses, fossils are crucial because they represent intermediate morphologies between the modern species and their common ancestor. Without them, your result might be misleading since homoplasy (convergent evolution) is everywhere in the animal Kingdom.

If others want to break into the world of palaeoneurology, what sort of research opportunities do you recommend they pursue?

First, get some experience with handling and processing 3D CT data. You can start from home. Download a skull from Morphosource (some are open access) and use a free software to start doing some segmentation. The best way is to do it as a research project with someone familiar with the technique. I have trained many students throughout the years, and it was an amazing experience each time. I continue doing so at my current institution at the Institut Català de Paleontologia in Spain. It is better to start with a modern specimen because fossils might be too challenging. At the same time, read papers to learn about neuroanatomy so you know what to segment.

Do you think other factors helped modern mammals replace archaic mammals in the end of the Palaeogene?

Probably. There are still a lot of things that we don’t know about the emergence of modern placental mammals specifically. The environment changed with a global cooling starting during the early Oligocene. Beyond the fact that modern clades appear to have bigger neocortices than archaic taxa, they may have had other morphological adaptations that gave them an edge over the archaic groups.

Do you have any follow up research projects you are interested in, if so, what are they?

Yes, for sure! Long term, I want to understand how the brain, but also other sensory organs such as the inner ear, evolved through deep-time in mammals and in other vertebrates. I want to study how animals behaved in the past from different perspectives, that means using other aspects of the skeleton like craniodental and postcranial elements. Collaborations are key because you cannot be an expert in everything! For instance, I would like to incorporate genetics, ethology, and AI into my research to improve our understanding of the emergence and diversification of senses in animals.

Scientific article this is based on:

Bertrand et al. 2022. Brawn before brains in placental mammals after the end-Cretaceous extinction. Science, 376(6588): 80-85.

Hady George is a palaeontology PhD student at the University of Bristol researching the jaws of the earliest tetrapods among other things, and seemingly always has a pop science book somewhere in his bag

Tags: , , , , , , , , , , , , ,

Back to Top ↑