By Kamlesh K
Artificial intelligence or AI refers to the simulation of human intelligence in machines. (Refer the article Are AI and machine learning the same if you wish to understand more on what is AI) Artificial intelligence-based system learns from data (also referred as “training data”). These systems find relationships hidden in the training data, develop an understanding of the data, make decisions and then evaluate their performance from the data. And better the training data, better the system performs. Or in other words, the training data (both the quality and quantity) dictates how well the AI system will perform. In this article I will cover some hacks one can use if there is not enough training data available. So read on.
But first let’s understand how the training process works in the AI world.
Let’s say you want to make an AI-based system capable to do visual search i.e. it should be able to recognize a particular object in a given image (example: is the person in the photo wearing a jacket or not?). The process of training this AI is as follows (refer the picture below)
Businesses often struggle to find enough relevant training data to train their AI and this is because most often the data they need resides in multiple sources, are usually available in an unstructured format and therefore not useful for machines to learn from. So in such situations, how can you still make sure that AI works for you?
Here are 5 possible hacks you can try out and it is very likely that one or more will work in your situation.
Transfer learning is about using the knowledge gained while solving one problem for which a lot of labeled data is available and applying it to a different but related problem for which only little labeled data is available.
For example, the knowledge gained while learning to recognize a person in an image could apply to recognize the types of clothes that the person is wearing and finding dominant colors in his clothing.
Data augmentation is a process that increases the diversity of data available for training models, without actually collecting new or more data. As an example, for computer vision-related tasks we have the following options:
For text (NLP) related task you can try the following options:
Problem reduction approach refers to modifying the new data or unknown problem to a known problem such that it can be easily solved using existing techniques. Here is an example:
Neural networks are good at learning from high dimensional data like images or spoken language, but only when they have huge amounts of labeled examples to train on. Humans, on the other hand, are capable of one-shot learning – if you take a human who has never seen a fork in his entire life and show him one, next time if you ask him to bring one from that messy drawer of your kitchen, he might be able to fetch that with high precision. For this type of approach, there are advanced algorithms that we will try to cover in later blogs.
Various data sets are available in the open-source community or on the internet, which can be easily downloaded and used for training purposes. Some examples like the amazon dataset and google dataset are also accessible easily. Apart from these, search engines like google and bing also provide APIs to download datasets and pre-trained models, which can be used in many ways to improve the performance of an AI-based system.
While availability of relevant training data decides whether an AI system will perform well or not, it is certainly not a hopeless case if one does not have enough data. There are multiple techniques and hacks that one can still try out and I have listed the top5 above.