Why do so many AI projects fail before hitting the production stage?
Julia Rose
Marketing & Design
21. Oktober 2022
Stop treating AI projects like a standard IT project
According to well-known research from Gartner in 2018 , around 85% of AI projects were expected to fail. Even though the companies have gained much more hands-on knowledge in applying AI to their own projects and products over these four years, unfortunately still many of the projects never see the production phase, or they run into problems after some time after the deployment to production.
In this blog post, we would like to focus on a few chosen (but not only) reasons of this issue – which are treating the AI projects as standard IT projects, and treating the project as finished in the moment of deployment.
Why are AI projects different from standard IT projects?
In short – unlike standard IT projects, where you can exactly predict the results of every step of the program, the situation with AI projects is different. AI projects depend heavily on the data on which the models are trained and how well they mimic the real-life environment, in which they will be applied. Machine learning models find patterns in data and the closer the data set is to reality, the better the result will be afterwards. However, there can always be borderline cases. There are situations that do not occur in the data set, new conditions that have not been expected on the planning and development phases or are so specific that the applied algorithms’ architecture might not be able to generalize that well to cover them.
The common misunderstanding in this area is the expectation (sometimes even by skilled IT professionals) of achieving very high metrics, without any prior research of the problem, current state of technology and availability of reliable training datasets.
Imagine a company using data to train their AI model. However, this data is only collected under controlled and pre-determined conditions. Unfortunately, this control means that their model often performs very poorly under real conditions – which might be very different from the training data.
An important aspect of overcoming this challenge is good communication and expectations management involving the most important stakeholders of the project – and of course the thorough analysis and understanding of the underlying business problem by the whole development team.
The Data Drift Problem – Changing Conditions
Another important aspect, partially connected with the previous one, is treating the AI projects as closed after the deployment to production and finished hyper care period. Unfortunately, in case of AI project things work differently, because of the so-called data drift. Over time, the data distribution changes, and this means that the AI model must also adapt to this new distribution.
What does this mean exactly?
Data changes over time. That’s why AI models need to be taught about new situations regularly, so they can adapt to the new patterns in the data and respond to new situations accordingly. This requires from the development team constant control over the model performance. An important area of ML projects is therefore the application of Machine Learning Operations (MLOps).
And that’s also a big advantage: AI provides companies with a regular inflow of new insights. That’s important for adapting quickly in a world of constant change, driving innovation and gaining competitive advantage. So long-term support pays off and should always be an option for companies.
The growing impact of Machine Learning Operations (ML Ops)
MLOps is a critical process that enables organizations to gain valuable insights from machine learning models as those models continuously access and analyze the organization’s data.
Machine Learning Operations (MLOps) is a technology that helps organizations implement overall management, continuous monitoring and maintenance, and re-training for production use of Machine Learning models, when the data changes or deviates over time.
Example Natural Language Processing (NLP)
Natural Language Processing (NLP) is a branch of Artificial Intelligence that enables computers to understand and interpret human language. Natural language processing is based on text and speech-based data and is extremely versatile. Since language changes over time (so-called language change or language dynamics), these models must also be regularly adapted.
Another interesting example here are the Large Language Models – such as the GPT family – which help us solve many NLP tasks without specific retraining for the task. An example of a data drift here is, for example, that GPT-2 doesn’t know anything about the coronavirus pandemics as it has been trained before that point in time.
Example AI-based CRM systems
Artificial intelligence can be used to optimize customer relationship management (CRM) processes to understand customers more precisely, identify and produce revenue potential, save costs, and increase customer loyalty. But even here, unforeseen problems can arise. Customer behavior changes, data must be adapted, and AI must be continuously retrained to achieve the best possible results.
AI as a Service – Work with an experienced partner
The above mentioned aspects, are one of the reasons why it’s important to make sure that we either build a strong AI focused team in-house (instead of giving it as additional task to other departments) or to work with an experienced partner, who will not only implement the requested models, but understand the real underlying business problem and the possible business value of the solution, iteratively implement and evaluate the models and take care of the models maintenance over time. This is what we call an AI as a Service approach.
We will help you to implement and maintain your AI-based solution or project
If you would like to learn more about our AI-as-a-Service offering, please do not hesitate to contact us. We will help you to implement and maintain your AI-based solutions sustainably.
At theBlue.ai, we offer extensive expertise in Artificial Intelligence, Machine Learning and Deep Learning and are committed to the success of your projects with our proven and world-class team.
―
1 https://www.gartner.com/en/newsroom/press-releases/2018-02-13-gartner-says-nearly-half-of-cios-are-planning-to-deploy-artificial-intelligence
