{"id":54675,"date":"2024-05-23T12:01:15","date_gmt":"2024-05-23T11:01:15","guid":{"rendered":"https:\/\/www.n-able.com\/?p=54675"},"modified":"2024-06-10T18:13:54","modified_gmt":"2024-06-10T17:13:54","slug":"how-and-why-we-applied-machine-learning-to-cove-continuity-part-2","status":"publish","type":"post","link":"https:\/\/www.n-able.com\/fr\/blog\/how-and-why-we-applied-machine-learning-to-cove-continuity-part-2","title":{"rendered":"How, and Why, We Applied Machine Learning to Cove Continuity, Part 2"},"content":{"rendered":"

If you haven\u2019t already read part 1, click here<\/span><\/a> to do that first.<\/span><\/p>\n

If you look at the screenshots, they\u2019re actually quite simple to understand. Anyone can easily identify whether the OS booted successfully at first glance. Look at the following examples and you\u2019ll see what I mean<\/span>\u00a0<\/span>:<\/span><\/p>\n

\"\"<\/p>\n

So, rather than the existing deterministic method, which relied on indirect evidence, we opted to use machine learning and neural networks to analyze and classify screenshots like a human being.\u00a0<\/span><\/p>\n

Only a few years ago, this might have sounded like science fiction. Today, there are a variety of mature AI\/ML tools and technologies ready for practical application. Better still, many of them are open-source and publicly available. So, our job was to look for proper tooling and use it accordingly.<\/span><\/p>\n

We knew what to do, however we had little experience in the area. So, we decided to start with a POC. We evaluated a number of neural networks that can be used to classify images, and opted to use SqueezeNet<\/span><\/a>\u2014a well-known open-source model that proved its efficiency in numerous contests.<\/span><\/p>\n

Preparing the dataset and training the model<\/span><\/h2>\n

Roughly speaking, the SqueezeNet model is a kind of an architecture or algorithm that can solve a general problem. In this case classifying images. However, with some training we can make it solve our specific problem\u2014classify VM screenshots.<\/span><\/p>\n

To train the model, we first must provide it with a proper dataset. This allows the model to \u201cknow\u201d what we are trying to classify and what we expect as a result. Since Recovery Testing has been in production for several years, we had tons of VM screenshots to use for training. To prepare the dataset, we removed all PII and manually labeled 2,500 screenshots with a specific class, e.g., failed to boot, successfully booted, or loading.<\/span><\/p>\n

We used\u00a0PyTorch<\/span><\/a> to conduct the training. It took us around\u00a060\u00a0minutes on the above dataset. And as a result of training, we got a 5MB file with parameters. Pretty dense and will not eat a lot of RAM. Nice!<\/span><\/p>\n

As an output, we received model weights, or parameters. If we apply these parameters to the model, it should be able to solve the problem on any set of screenshots, even if they were not part of the initial training set.<\/span><\/p>\n

Evaluate results and tune parameters<\/span><\/h2>\n

Theoretically, we could have applied that model in production, but before doing that we had to validate the results and, if needed, tune the model parameters. In the first stage, we checked the results of the model training on 1,000 screenshots that were not part of the training set. In the second stage, we checked it on another 36k screens.<\/span><\/p>\n

The results were tremendous. It takes 1 second (max) on a decent developer’s workstation with Intel i7 to classify a screenshot. More importantly, we now have a mechanism to classify screenshots with an accuracy of 99%<\/b>. Not bad!<\/span><\/p>\n

A summary in numbers:<\/span>\u00a0<\/span><\/p>\n