Machine Learning Testing

Sometimes people argue that machine learning is complicated because we can't know what it is doing under the hood. The truth is that many of our actual systems are so big that not a single person know what the system is doing all the time, we can't hold all this information in the head. So to bypass this problem we need to find a way to trust in our systems. This way is to have a comprehensive test suite for the software at hand. We do this already for many types of complex systems, why not do it for ML projects? Below I list the types of tests that I find useful working with.

Unity tests

Mainly use unit tests to not let the model regress on a given and fixed problem. Example:

• We have a image sample that isn't working, we want to make sure that we will fix the problem and it will not happen again.

Integration tests

Mainly used to evaluate the model in the field, we must setup a comprehensive dataset that we can test all the possible difficulties that will happen to our model and then calculate something like precision and recall. So we can keep track of the evolution and data drift.