Machine Learning and Artificial Intelligence have been around for a long time. Artificial Neural Networks emerged in the 1950s as a promising direction for 'thinking machines', but their statistical foundations go back further. Least Squares, discovered around 1800, was first demonstrated by predicting where the asteroid Ceres would reappear after it disappeared behind the sun's glare. The more modern story of how statistics, machine learning and artificial intelligence became business essentials is more recent.
The Golden Age of Statistics (Pre-2010)
Prior to 2010, the most commonly used techniques were members of a family called Generalised Linear Models, or GLMs. The beauty of GLMs is that they are all, at their core, the same thing: a straight line with a trick applied to it.
Linear Regression is something most of us started learning in our teens: finding the line of best fit and describing the trend.
Linear regression: the line of best fit
We can get the equation of that line and use it to make predictions. With more than one variable we enter n-dimensional space, which is impossible to visualise, but the maths still holds.
So perhaps we are predicting the value of a property. We could have a formula like:
price = β₀ + β₁ × bedrooms + β₂ × bathrooms + β₃ × squarefootage + β₄ × desirable_location
Each β coefficient tells us how much that variable contributes to the price. The model is interpretable, auditable, and you can explain it to a stakeholder in five minutes. This mattered enormously in regulated industries like insurance and banking, where "the model said so" was never an acceptable answer.
But a straight line can only take you so far. What if the thing you are predicting is not a continuous number?
Logistic Regression takes the same linear equation and passes it through a sigmoid function, bending the straight line into an S-curve that squashes the output between 0 and 1. Instead of predicting a value, it predicts a probability: Will this customer churn? Will this claim be fraudulent? Will this applicant default?
Logistic regression: same line, bent through a sigmoid
Same line underneath. Different trick on top.
Poisson Regression applies yet another trick: exponentiation. You fit a straight line, then raise e to the power of it. The result is always positive and can only produce whole numbers in expectation, which makes it perfect for modelling counts. How many claims will this policyholder make? How many calls will hit the support desk this hour? How many defects per batch?
Poisson regression: exponentiation for count data
Gamma Regression handles the awkward cases where the outcome is continuous and positive but heavily skewed. Think claim amounts, time-to-event, or repair costs, where most values cluster low but a long tail stretches out to the right. The line is still there, but the trick this time reshapes how the variance behaves: bigger predicted values come with bigger uncertainty, which matches how these real-world quantities actually work.
Gamma regression: handling skewed, positive-only outcomes
These four models and their various relatives (negative binomial, ordinal, Tweedie) are all the same fundamental idea: a linear combination of inputs, transformed through a link function to match the shape you are trying to predict.
But GLMs were not the only game in town. The pre-2010 toolkit was broader and not everything in it was a line in disguise. ARIMA handled time series. Survival analysis estimated how risk changes over time. Decision trees split data into regions with no linearity assumption at all. k-nearest neighbours, kernel methods, and naive Bayes each brought their own geometry to the problem.
What these techniques shared was not a common mathematical form but a common culture. They were built to be understood. A regulator could pick up a GLM, read the coefficients, and ask "why does this variable have that sign?" An actuary could interrogate a survival curve. An economist could test whether an ARIMA model's residuals were white noise.
They were not glamorous. They did not have venture capital. But they worked, they were understood, and they could be challenged. And they have not gone away. A Poisson GLM will predict insurance claim counts more accurately than any GPT or Claude, not because LLMs are bad, but because they were built to do something else entirely. When the task is "predict a number from structured inputs," the old guard wins. It is not close.
The people who built these models were called statisticians, actuaries, analysts, or econometricians. Nobody called them data scientists yet.
I intend to write more about these types of techniques in the future, how they work and where they still win.
The Hadoop Era and the Birth of "Data Science" (2010–2015)
Around 2010, two things happened at once: data got big, and someone coined a job title.
Hadoop and its ecosystem (MapReduce, Hive, Pig, HDFS) promised to let you process terabytes across clusters of commodity hardware. At the same time, "Data Scientist" started appearing in job listings. Harvard Business Review famously called it the "sexiest job of the 21st century" in 2012, and the gold rush began. In practice, most data scientists spent 80% of their time cleaning data and 20% explaining to stakeholders why the data wasn't ready yet.
Remember those decision trees? This is where they came into their own.
A single decision tree: intuitive but fragile
A single decision tree is intuitive but fragile. Random Forests fixed this by training hundreds of them in parallel on bootstrapped samples and averaging the results. Gradient Boosting Machines took a different approach: train small trees sequentially, each one correcting the errors of the last. XGBoost, released in 2014, became so dominant on Kaggle that competitions essentially became hyperparameter tuning contests.
Random Forest: many trees, averaged together
The real craft of this era was feature engineering: encoding domain expertise as model inputs. Ratio of claim amount to policy limit. Days since last customer interaction. Rolling averages over carefully chosen windows. A data scientist who understood both the algorithm and the business could build features that made a simple model perform like a complex one.
Feature importance: the craft of encoding domain expertise
This is the era that everyone forgot about. Most of the companies that stood on conference stages claiming they were "doing Machine Learning" were not. Many still are not. But this era produced enormous business value. Recommendation engines, fraud detection, demand forecasting, churn prediction, pricing optimisation. These were revenue-generating production systems, and most of them were built on gradient boosting, not neural networks. If you have ever had a fraudulent transaction flagged before you noticed it, you probably have this era to thank.
GBMs are still an absolute powerhouse and I intend to write more about them in the future.
Deep Learning Changes the Conversation (2015–2020)
Deep learning had been simmering in academic labs for years, but three catalysts brought it into the mainstream: GPUs became cheap enough to train large networks, ImageNet proved that deep convolutional networks could classify images better than humans, and TensorFlow (2015) followed by PyTorch (2016) made the tooling accessible.
ImageNet error rates: the cliff that changed everything
Look at that cliff. In 2011, the best image classifiers got the wrong answer 26% of the time. By 2015, deep networks had pushed that below human-level performance. Nothing else in the history of machine learning looks like this. It was not incremental progress. It was a phase change.
For computer vision and natural language processing, the impact was genuine and profound. Problems that had resisted decades of hand-engineered feature extraction suddenly yielded to architectures that could learn their own representations. Image classification, object detection, machine translation, speech recognition: all took dramatic leaps forward.
Something that tends to be forgotten though is that NLP was already a rich and productive field before deep learning arrived. Bag-of-words models, TF-IDF, and topic modelling (LDA) had been extracting structure from text for years. spaCy could do named entity recognition, dependency parsing, and text classification on a laptop in milliseconds. Sentiment analysis was a solved problem for most practical purposes. Word2Vec and GloVe gave us dense word embeddings that captured meaning without a GPU cluster. By the time BERT arrived in 2018, it was building on a deep foundation, not starting from scratch. These tools were fast, cheap, deterministic, and good enough for the vast majority of text-based business problems. Remember that, because it matters when we get to LLMs.
Word embeddings: meaning encoded as geometry (2013)
Words that mean similar things end up near each other. Words that share a relationship (king is to queen as prince is to princess) form parallel lines. Nobody told the model any of this. It learned it from patterns in text. This was 2013.
For tabular business data? A well-tuned XGBoost model on structured data still beat a neural network more often than not, and it trained in seconds rather than hours. But the narrative had shifted. Executives who had never heard of gradient boosting had definitely heard of deep learning, and "we use AI" started appearing in pitch decks regardless of what was actually running in production.
Data scientists found themselves spending increasing amounts of time managing expectations: explaining that not every problem needed a neural network, that data quality mattered more than model architecture, and that a dashboard was not AI.
Neural networks and "deep learning" absolutely have their place and I intend to write more about when and where they should be used.
The LLM Earthquake (2020–2024)
GPT-3 arrived in 2020 and the ground started to shift. By the time ChatGPT launched in November 2022, the earthquake was in full swing.
There was a sense of magic with these things, and fun too. DALL-E, OpenAI's text-to-image generator, sucked many an hour of thinking up absurd images to generate. But the magic is worth examining, because understanding what is actually happening under the hood is the difference between using these tools well and using them badly.
A large language model is, at its core, a next-token prediction engine. Given a sequence of words, it predicts what word comes next. That is it. It does this extraordinarily well because it has been trained on a staggering amount of text with a staggering number of parameters, and it turns out that learning to predict the next word requires learning a lot about the structure of language, reasoning, facts, and style along the way. But it is not thinking. It is not reasoning from first principles. It is pattern-matching at a scale that produces behaviour which looks like reasoning.
This matters because it explains both why LLMs are so impressive and why they fail in ways that surprise people. They can write fluent legal prose because they have seen millions of examples of legal prose. They can summarise a document because summarisation is a pattern in their training data. But ask one to predict next quarter's revenue from a spreadsheet and it will give you a confident, fluent, wrong answer, because that is not a language task. It is a statistics task, and the tools from the first two sections of this article will do it better.
Large Language Models did something that no previous ML technology had managed: they made AI tangible to non-technical people. Your CEO could type a question and get a coherent answer. Your marketing team could generate copy. Your legal department could summarise contracts. The abstraction barrier between "AI" and "useful tool" collapsed almost overnight and it was incredible.
But it also created a gravitational pull that distorted everything around it.
The thing is, LLMs are non-deterministic. Same prompt, different answer. Run it again, different answer. Every model in the earlier sections of this article had clear evaluation metrics. You could measure RMSE, log loss, AUC, deviance. You could validate a model against holdout data and get a number that told you how good it was. You could put that number in a report and defend it.
How do you evaluate whether an LLM's output is "good"? There are useful methods but most people are using vibes. They are using "it looked right to me." They are showing two examples to a stakeholder and calling it tested. This is the exact opposite of the scientific discipline that made the earlier eras productive, and it is happening at scale in production systems right now. This is the problem I want to solve.
Data science teams that had spent years building careful, validated, domain-specific models suddenly found themselves fielding requests to "just use ChatGPT for it." The nuance of when a language model is appropriate and when it is expensive, hallucination-prone, and wildly overkill for the task became the new frontline of technical leadership.
The NLP case makes this concrete. Sentiment analysis that spaCy could run on a laptop for free now gets routed through an API that charges per token and returns a probabilistic answer that might be different next time you ask. Named entity recognition that ran deterministically in microseconds gets replaced by a prompt that costs money, adds latency, and occasionally hallucinates entities that do not exist. These are not edge cases. These are real decisions being made in production systems right now, and in many of them the older, cheaper, faster tool is also the more reliable one.
NLP cost comparison: local tools vs LLM APIs (2026 pricing)
Even with budget LLM APIs like DeepSeek V4 Flash at $0.14 per million input tokens, local NLP tools are still 50-100x cheaper for tasks like sentiment analysis and NER. The cost gap has narrowed since 2024, but it has not closed. The one place LLMs genuinely earn their cost is summarisation, where condensing a long document into something coherent was always beyond traditional NLP. For everything else on that chart, you should be asking hard questions about why you are paying for an API call.
Job titles shifted too. "Data Scientist" started to sound quaint. "ML Engineer" had a brief moment. "AI Engineer" emerged as the new magnet for recruiters. The underlying skills had not changed as dramatically as the branding suggested, but perception matters in hiring markets.
I love using AI and I particularly enjoy building agentic AI networks. I intend to write more about how to do this and the real power of mixing agentic AI networks with traditional techniques.
Where We Are Now (2025–2026)
The dust has not settled. Agentic AI, multimodal models, and the race to deploy AI into every workflow have created an environment where the gap between what is promised and what is delivered has never been wider.
Here is what I see from the inside:
The fundamentals still matter. The organisations getting real value from AI are the ones that already had clean data pipelines, well-defined business problems, and teams that understood the difference between a demo and a production system.
Most business problems are still tabular.For all the excitement about LLMs, the majority of business decisions are driven by structured data. A well-built XGBoost model with solid feature engineering still outperforms a fine-tuned LLM on most of these problems, at a fraction of the cost.
Governance is the bottleneck. Deploying AI without governance frameworks and human oversight is not innovation. It is negligence. The organisations that survive the inevitable regulatory tightening are the ones building these structures now.
The economics do not always work. Running LLM inference at scale is expensive. For many use cases, the ROI is thinner than the sales pitch suggests. Measuring this honestly is one of the most valuable things a data leader can do.
The Timeline We Should Remember
The AI narrative wants us to believe that everything before 2022 was a prelude. That the real story started with large language models, and everything else was just warming up.
That is wrong.
The golden age of statistics gave us interpretable, auditable models that still underpin critical systems today. The data science boom built the teams, the tooling, and the culture of evidence-based decision making. Deep learning solved genuinely hard problems in perception and language. Each era produced real, lasting value.
LLMs are a powerful addition to this lineage. They are not a replacement for it.
The most dangerous person in a boardroom today is someone who thinks AI is new. The most valuable is someone who knows what came before, what still works, and when the new thing is actually the right tool for the job.
This article is me setting out my stall and I want to follow up with more in-depth articles that try and show the best tools for the job and how to hang on to a real rich vein of talent built over the last decade.
