Algorithms are now acknowledged as having a serious edge over human expertise in forecasting the outcomes of a whole range of activities.
An algorithm is a formula that consists of a set of instructions for calculating or defining a sequence of operations. In mathematics and computer science, algorithms are used to analyse huge amounts of data way beyond the capability of the human mind.
The word ‘algorithm’ was first derived from the name of an Arabic mathematician, Mohammed ibn-Musa al-Khwarizmi, known in the Bagdad royal court from 780 to 850. So, they have been around for a long time.
If you think there is no relevance to meaningful living then you would be mistaken. For example, one dating website (not that I’m looking) claims that by asking a handful of inane questions that it can predict whether or not a couple will stay together. One question is Wouldn’t it be fun to chuck it all in and go live on a sailboat? That beats John Gottman, the American researcher and couples therapist, who only needs 7 minutes with a couple to “know” if they will stay together.
We just love to predict the future, we want to know the weather, we seek info from the stars as to what the day has in store for us and of course, the stock market is everyone’s favourite for tips, spruiking the future and crystal-ball gazing. There have been some spectacular successes, too.
The whole industry of gambling is based on guessing what will happen, and when we’re right, we feel so special. If only we could do it with Lotto!
We use surveys to forecast elections though recent experiences in the UK and US have cast some doubts. For shares, we have trendlines, stochastics, moving averages, other charting techniques, Bollinger bands, Elliot Wave Theory, Fundamental Analysis, Value Analysis, Cash Flow Analysis, ROI and throwing darts at the shares page in the newspapers. There are still tea leaves, palm reading, various types of cards like Tarot, and my personal favourite, the I Ching. Let’s not forget psychics and clairvoyants who are in demand and compete in the same space as counsellors, psychologists and psychotherapists.
The 2003 Complete Idiots Guide to Psychology states that the best predictor of future behaviour is past behaviour which it claims is “a psychological fact of life”! – Well, it is an idiot’s guide. However more sophisticated psychologists know that it is not quite this simple and would modify this “fact” with certain qualifications. The more frequent the behaviour, the shorter the time frame, the more consistent the context, the absence of anything to change that behaviour, the person concerned must be much the same and typically consistent in their habits – all make this fact more likely to be borne out in practice.
The bottom line, however, is that for all our methods and methodologies, we are incredibly bad at forecasting the future with any modicum of reliability.
The psychologist Daniel Kahneman, in his book Thinking Fast and Slow discusses a study of counsellors who believed they could predict the high school grades of their students with all the information available to them about their students’ past performances. They accessed multiple aptitude test scores, a 4-page personal statement and a 45-minute interview. Their success was then compared with an algorithm that only had access to one aptitude test and school grades. Despite ‘knowing’ the students personally to some degree, the algorithm was consistently more accurate in predicting future grades.
But there is good news now, algorithms are being developed that cut-through the self-deceptions, the prejudices, and the gut-feel and intuitions of silly humans who think they know something. Wishy-washiness is dead. More importantly, this outperformance of algorithms over humans is occurring in virtually every context where human judgements are compared with a formula. All kinds of complex and formerly uncertain activities are now increasingly predictable with higher and higher probabilities such as how long a cancer patient will live, the credit risk of applicants, recidivism, and of course, the holy grail of forecasting – the share market. And let’s not forget footy tipping!
Yet, some algorithms are better than others and some are not reliable at all. I don’t believe the sailboat question above has any meaningful predictive value and dating websites are hardly the best source for scientific rigour. But it is the dating websites that have increasingly become the way that we meet each other and couple-up, or sometimes just ‘hook-up’. First launched in 2000, these sites typically claim to have developed complex algorithms to match you with your perfect mate.
Such algorithms are based upon our more than a hundred years of studying couples who are compatible and stay together, then matching profiles and Hey! Presto! – you have found THE ONE. You are given on a platter your soul-mate, who ticks all the boxes, who will remain faithful and true, and give you beautiful orgasms until Le Grande Mort do you part.
All you have to do now is supply the magic of inter-personal chemistry that no algorithm can predict. Forgive me for scepticism but algorithms for all their computational power cannot substitute for love, and never will.
You have to be very careful with algorithms. We know them best from Google, a company that has transformed itself through the use of algorithms into a multi-billion dollar corporate behemoth.
One American marketing company appeared to get it badly wrong when it began sending a teenage girl coupons for nappies. Her irate father lodged an official complaint but later he had to phone and apologize admitting that he was, in fact, going to be a grandfather. The algorithm analysed her Google searches and knew she was pregnant before he did.
But another company had the clever idea to take the modern meme Keep Calm and Carry On (from the British war-time poster) and use algorithms to create slogan variations to be printed on T-shirts. It advertised, marketed and sold the printed shirts automatically without even checking what the computer came up with. Some eyebrows were more than a little raised when Amazon offered T-shirts emblazoned with Keep Calm and Grope a Lot and best (or worst) of all Keep Calm and Rape.
Ultimately, we need to be discerning about how we use algorithms. Romantic love and the durability of our primary relationships is fair game for the cold analysis of data. The statistician Robyn Dawes came up with the simplest algorithm that does indeed seem to have amazing predictive powers as to whether or not a marriage will work and last. Subtract the frequency of quarrels from the frequency of love-making. A negative number suggests you should consult a lawyer – or a couples therapist!