The Art of Perfection: Avoiding Apophenia

Date: April 20, 2017

SP - The Art of Perfection - Avoiding ApopheniaIn The Art of Perfection series we are exploring what asset managers can learn from the worlds of art, science and studies into the human body. 

In this fourth issue: finding “meaning” when it’s not really there.

The experience of seeing patterns or connections in random or meaningless data was coined apophenia by the German neurologist, Klaus Conrad.

He originally described this phenomenon as a kind of psychotic thought process. Conrad focused on the finding of abnormal meaning or significance in random experiences by psychotic people; though this is now viewed as being a ubiquitous feature of human nature.

Science historian Michael Shermer has called the same phenomenon patternicity. He defined it as ‘the tendency to find meaningful patterns in meaningless noise’. Shermer pointed out that our brains do not include a ‘baloney-detection network’ that would allow us to distinguish between true and false patterns. He cites the gambler’s fallacy – the belief that past random events can influence the probability of future ones, such as the increased likelihood of a flipped coin landing heads after a run of tails. He also cites the clustering illusion – the belief that the clusters that are always found in random data actually indicate something meaningful, such as a run of wins in a game of dice indicating that you are on a winning streak.

Apophenia in asset management

In the same way for asset managers, the market and underlying variables can seem to collude against or for the manager, performance wise. The tendency to find meaningful patterns in meaningless noise is useful in a world where the meaningless noise comes in the form of so much data. But the trick is to know whether logically the fact that xyz happened did actually cause abc performance, or whether it was just a random series of events. Evidence is crucial.

A piece in Psychology today gives the following everyday life example: The alarm clock does not wake you up and you are now late for work. The cat has peed on the couch. The coffeemaker is making strange noises and doesn’t seem to be working. The kids are fighting with each other. It’s raining. And on top of everything else, the car won’t start. What do you conclude? One or two of these minor irritants would seem insignificant and unmemorable. Once the list grows, however, it begins to take the shape of a plot—a plot of unseen forces perhaps conspiring in a meaningful way against you.

In science, apophenia is related to what’s known as a Type I error, in which a test seems to show that two variables are related when, in fact, they are not. It can also contribute to confirmation bias, in which an investigator deliberately looks for evidence which supports a favoured model and avoids evidence which refutes it. In other words the brain actively looks to find patterns and causal effects even when logically none are there and the events are indeed complete chance.

Seeing patterns is important in all aspects of life

But while it’s tempting to view apophenia as merely a defect in cognitive processing capacities, there is cognitive efficiency built into this equation: quick reactions depend on a cost-benefit ratio that favours safety and survival. On balance then it is better to err on the side of caution and identify every possible pattern. What the brain then does is evaluate whether the pattern is true – looking at logic and evidence in the face of a seemingly obvious pattern.

Without such assessment, we humans would be unable to make predictions about survival and reproduction. The natural and interpersonal world around us would be too chaotic. In the asset management world, not being able to make predictions about performance and future positioning would also have the same chaotic effect.

Seeing false patterns is clearly something that asset managers need to avoid. They need their analytics to provide true visibility and clarity and have the capacity to back up any patterns with robust underlying source data. They need to look again at the data and prove the relationship between the causal events and the outcome.

The power of the middle office

What any decent middle office analytics system should do then, is give the asset manager the underlying data needed to prove this causal effect – thus allowing the manager to distinguish between a true pattern and a spate of unrelated events that had the same effect.

In this sense the middle office needs to replicate the pattern detection machines that we all have in our brains. The system, like the brain, needs to connect the dots making it possible to uncover and prove meaningful relationships among the barrage of input – be that sensory or back-end accounting data.

In practice this works in a middle office context by having great analytical capability and data management controls in place.  A clear pathway and workflow is also key so that any pattern that emerges does so as the result of its facets having been treated in exactly the same way time after time.

Analytical sophistication also comes to the fore now that there are so many complexities in the data and it is more granular. There are more facets to today’s data, and accurate interpretation of data is a difficult task. And with the ever increasing volumes of data available, the opportunity to discover these non-existent patterns is only becoming more pronounced.

Clean data is thus key and the system must also be able to identify and verify data instantly so as to eliminate anything that might cause or contribute to a false pattern. Data controls and workflow processes in place mean that the quality of the data is more reliable and that there is greater overall visibility over the entire performance and attribution process.

Evidence is crucial and any system needs to provide that. Finding meaningful patterns in meaningless noise is only useful if those patterns can be proven to be true.

The end game is that once the data has been analysed effectively and true patterns have been identified, then the manager can use that intelligence and know how to position the portfolio going forward – whether that be to anticipate a combination of events and to steer clear of any negative impact, or in the case of a positive impact, how to position the portfolio for maximum uplift.


  • Finding logical, meaningful patterns can be difficult when there is so much data available
  • An efficient middle office will supply enough data to allow the asset manager to determine whether a pattern is true or not
  • Identifying and dismissing false patterns is crucial for an asset manager’s performance
  • Supplying clean and clear data is the key to helping an asset manager make sense of patterns
  • Once a true pattern is identified, an asset manager can act accordingly to optimize the situation

Find new ways to make the complex simple


To learn more about 13f-2 watch our webinar replay Part 1: Unpacking the SEC's New Disclosure Rules for Shareholders
Join us for Part 2: Operationalizing the SEC's New Disclosure Rules, for Shareholders on December 12.