Evaluating a trained network

Once the training has finished you can access the evaluation tab. Here you can evaluate the performance of the network on the validation set.

To the left, a list of the images currently selected for evaluation is displayed. By default the 10 hardest images of each class are selected: the 10 Bad images with the lowest scores (meaning the network is less certain they are actually Bad) and the 10 Good images with the highest scores (meaning the network is more certain they are actually Bad).

Clicking "Select images" at the top of the list expands it to let you change which images to include. In the list are all images that were used for evaluation during the training. If you need to look at certain images, or if you want to see the evaluation of all of these images, you can check the images you want to include or use the buttons at the top to select all. Usually however, the default "Hardest" selection is enough to give a good picture of whether the network successfully solves the problem.

The validation set

The data you provide is automatically split into a training (~80%) and validation (~20%) set. The validation set is used to evaluate the performance of the network during training and in the evaluation tab. In the first stage, the network is only trained on the training set. The evaluation displayed in dStudio is done with a version of the network that has never seen the validation data. After the evaluation is generated however, a final training stage is run where the network is trained on all data in your snapshot. This leads to a final network that performs slightly better, but might mean that the score for a given image when running on device might not be exactly the same as in the evaluation in dStudio, which can be relevant to keep in mind if evaluating the network manually later on.

When you are done selecting images, you can collapse the list again for a more compact view. Since the anomaly detection assigns a score to each image the classification accuracy depends on the threshold you choose.

You can see how the threshold affects the performance on the validation set by dragging the dark green threshold line in the graph. Doing so will update the list of images to the left, indicating with icons which images are correctly or incorrectly classified. The numerical value for the threshold is shown in the top left corner of the graph. The orange line in the graph shows the total number of Good images classified as Bad at any given threshold, while the red line likewise shows the number of Bad images classified as Good for any given threshold. The optimal threshold, calculated on the dataset during training, is marked in light green.

As the threshold changes, the images in the evaluation shift between the different cells of the confusion matrix to the left of the graph. The numbers in the cells indicate how many images are classified a certain way, and the colors help indicate the classification status: green for images correctly classified as they were labeled (both true positives and true negatives), orange for Good images classified as Bad (false positives), and red for Bad images classified as Good (false negatives). Clicking a cell in the matrix toggles displaying its images in the image grid below the graph. The images in the grid have borders with colors matching those of the graph and the matrix, to indicate which cell they belong to.