A new phenomenon termed Data-Centric AI has gotten a lot of attention in the AI world in the last 6 to 9 months. As we all know, data and algorithms are essential for constructing an AI system. Dealing with data accounts for 80% of the data scientists’ time on a machine learning project despite this, a survey of current research efforts found that 99% of efforts are focused only on improving the model.
With only 1% of research focused on data, there is a pressing need to bring the AI community's attention to this massive gap. Instead of being a costly afterthought, data should be treated as a valuable and adaptable asset in the Data-Centric Architecture. Security, integration, portability, and analysis are all made much easier with data-centricity, and quicker insights are delivered throughout the whole data value chain. Most industries are adopting AI solutions and while AI models have improved over time, a fundamental shift is required to fully realize AI's promise.
“Instead of focusing on the code, companies should focus on developing systematic engineering practices for improving data in ways that are reliable, efficient, and systematic. In other words, companies need to move from a model-centric approach to a data-centric approach.”
— Andrew Ng, CEO and Founder of LandingAI
Companies from a variety of industries (including automotive, electronics, and medical device manufacturing) have seen improvements in deploying AI and deep learning-based solutions by adopting a data-centric AI approach. Data Scientists have seen that it's easier to get an algorithm to do what you want it to do if you feed it high-quality data rather than mixed-quality data. The following are some of the benefits we've experienced as a result of adopting a data-centric approach, compared to the traditional model-centric approach as mentioned here:
A model-centric approach focuses on improving the algorithm, code, and model of the AI System. Andrew Ng, widely regarded as the foremost pioneer of modern artificial intelligence, has a very clear opinion on the matter: he believes AI ecosystems need to move from a model-centric to a data-centric approach.
While working on the model has shown phenomenal progress in AI, we are slowly reaching a point where, for many practical applications, the code (or model) is a solved problem and can be simply repurposed from open-source sources. Working on further optimizing these models will not show worthwhile improvements in performance, instead, systematically working on engineering the data can help enhance performance notably. So, rather than asking “How can you change the model to improve performance?” you should ask “How can you systematically change your data (inputs or labels) to improve performance?”
Due to popular belief model-centric approach focuses on continuously improving the algorithm, similarly, data should also be iteratively engineered and improved through the AI systems lifecycle. AI practitioners must train the model on the data, perform error analysis to decide on what segment of data needs to be further engineered, and improve that data by working on both the labels and the input data itself. This process needs to be repeated throughout the AI system’s lifecycle. As mentioned above, if one produces clean high-quality data, AI is capable of learning faster and gaining higher accuracy. The key is consistent labeling.
In his seminar at TransformX 2021, Andrew Ng showcases the differences between model-centric and data-centric approaches:
A steel plant had developed a computer-vision solution to inspect steel sheets for defects. Their solution had a baseline accuracy of 76.2%. Their goal was to increase the accuracy to over 90%. Different approaches were used to improve accuracy, model-centric and data-centric. The team working on improving the model achieved 0% improvement after months of trying and the team that engineered the data was able to improve the model by 16.9% to achieve 93.1% accuracy in just a few days.
A machine learning model can only be effective if the training data is of high quality. The consistency and accuracy of labeled data are the determining factors of quality. Several industry standards exist to confirm the quality of data, such as benchmarks, consensus, and review.
When labels are applied to datasets, the labels must be consistent across labelers and batches. This means that, ideally, the instructions should define a deterministic function mapping the inputs (x) to the labels (y). Having consistent labels, that follow a deterministic function mapping, not only increases the accuracy of the model but also exponentially increases the rate at which the error during training decreases.
Use multiple labelers to find inconsistencies. Define a standard for your labelers to reduce inconsistencies. Repeatedly, find examples where the label is inconsistent between multiple labelers. Define a standard for how each point of inconsistency should be labeled and document this decision in the labeling instructions.
An annotation's accuracy is determined by how closely it matches the Data Ground Truth. A subset of the training data is labeled by a knowledge expert or data scientist to measure the annotation's precision. In order to assess accuracy, benchmark datasets are used. Benchmark datasets give data scientists insight into the labeler's accuracy, letting them assess the quality of their data over time.
As a second way of ensuring that the data is accurate, an expert assesses the annotations after the labels have been completed. Typically, the review is done by visually spot-checking labels; however, some projects can review all labels. Reviews are used to identify inconsistencies and inaccuracies in labeling while benchmarks are often used to judge labeler quality.
Data that is noisy can have a variety of impacts depending on the quantity and use case. For example, if the dataset size is small, the number of noisy labels will be more, but on larger datasets, the label noise will be averaged. Further, noisy data can influence model training in critical use cases like medical image analysis.
At Biz-Tech Analytics we believe a data-centric approach to data annotation and labeling makes your AI system easier to train and more capable of producing accurate results. We take steps to ensure high-quality labels by:
Please get in touch to learn more about how Biz-Tech Analytics can bring a data-centric workflow to your AI project.