Pa
ge
1


ECE1512 2019 FALL
ECE1512 – Homework 3: Monday, October 28, 2019

Background
Machine learning (ML) has played a significant role in data analysis and decision making.
Though promising, there is little insight in how and why a decision is made. The lack of
interpretability hurdles the practical usage of ML (especially neural network) in applications that
are highly regulated and may have serious consequences (for example, medical diagnosis,
autonomous cars, financial decision, etc.). To understand a rationale of a ML solution and thus to
build trust on the outcomes, explainable AI (XAI) that attempts to “opening the black box of AI
algorithms” is highly desirable.
Introduction
In this assignment, you will be exploring a simple approach to detecting anomalies in a dataset of
industrial inspection texture images, MVTec Anomaly Detection Dataset. Finding defective
components is an important problem in manufacturing and black-box machine learning
algorithms is unacceptable. You will be asked to apply a texture classification CNN pre-trained
on normal images, infer pixel-level activation maps with Grad-CAM to detect defects, and assess
the quality of the binarized defect activation maps to explain the defect decision.
Dataset
MVTec Anomaly Detection (AD) Dataset:
https://www.mvtec.com/company/research/datasets/mvtec-ad/
 5000 images (700-1024 pixels) = 3629 normal training images + 467 normal test images
+ 1258 defective test images
 15 classes in total: 5 texture and 10 object classes
Instructions
Setup
1. Prerequisites:
a. python
b. keras (TensorFlow backend)
c. sklearn (suggested)
d. numpy (suggested)
e. matplotlib (suggested)
2. Download a3_release.zip from Quercus
3. data is the re-formatted MVTec AD dataset for texture classes only
a. 1781 images = 1266 normal training images + 133 normal test images + 382
defective test images
b. Only includes images from the 5 texture classes (carpet, grid, leather, tile, and wood),
re-formatted into a folder structure compatible with Keras
c. data/train includes normal training images
Pa
ge
2


ECE1512 2019 FALL
d. data/test includes normal and defective test images
1. Normal images: _test_good_<###>.png
2. Defective images: _test__<###>.png
e. data/ground_truth includes binary defect masks for defective test images
4. save/texturenet.h5 is a pre-trained classification CNN for Keras trained on texture
classification (alternatively, you may train your own CNN for texture classification)
a. Architecture customized to be lightweight and ideal for applying Grad-CAM (first
three blocks of VGG16, GAP, and one FC layer)
b. The CNN was trained on classifying the 1266 normal training images to the five
texture classes
5. gradcam_utils.py is a Python file including utility functions needed for Grad-CAM
in this assignment (alternatively, you may use other code for Grad-CAM, such as that
provided on GitHub: https://github.com/jacobgil/keras-grad-cam)
6. train_cnn.ipynb is a Python notebook explaining how the CNN was trained, included
as a reference (train_cnn.pdf is the PDF version)
7. demo_gradcam.ipynb is a Python notebook explaining how to use the Grad-CAM
utility functions, included as a reference (demo_gradcam.pdf is the PDF version)

Task #1: Texture classification [4 marks in total]
1. Load the pre-trained classification CNN save/texturenet.h5 and the test set images
from “data/test”, then classify the texture classes of the test set images, assuming that the
most-confident class is predicted for each image
2. Report the following [2 marks]:
a. Overall classification accuracy on the test set
b. Plot the classification ROC curves (and find the AUC) for each class
c. Plot the normalized confusion matrix
3. Analyze the success cases and failure modes: for which classes does the texture classifier
succeed? For which classes does it fail? Can you explain why? [2 marks]

Task #2: Defect detection [6 marks in total]
1. Apply the pre-trained CNN to each test image (both normal and defective) and
implement the Grad-CAM method using gradcam_utils.py to infer pixel-level
activation maps for each predicted texture class
2. Without training on the defective test images, design your own approaches to the
following – report and justify your methodology [1 mark]:
a. Create a “defect activation map” to quantify the probability that a pixel is “defective”
in each image
b. Classify the test image as normal or defective using your “defect activation map”
3. Report the following [2 marks]:
a. Overall defect accuracy on the test set
b. Plot the defect ROC curves (and find the AUC) for each class
4. Analyze the success cases and failure modes: for which classes does the defect detector
succeed? For which classes does it fail? Can you explain why? [3 marks]
Pa
ge
3


ECE1512 2019 FALL

Task #3: Defect detection explanation [5 marks in total]
1. Load the ground-truth binary defect masks of the defective test images from
data/ground_truth (assume a blank mask for normal test images)
2. Obtain your predicted binary defect masks for all test images (normal and defective) by
applying a threshold to your “defect activation map” – this is your defect detection
explanation
3. Report the following [1 mark]:
a. Overall mean Intersection-over-Union (mIoU) between the ground-truth and
predicted binary defect masks
b. Intersection-over-Union (IoU) for each ground-truth class between the ground-truth
and predicted binary defect masks (regardless of the predicted texture class)
4. Analyze the success cases and failure modes: for which classes does the defect detection
explainer succeed? For which classes does it fail? Can you explain why? [2 marks]
5. Analyze the general psychology of the defect detection explanation – how well does it
explain the defect decision to a human? Qualitatively, how is it successful or lacking? [2
marks]
Notes
1. Coding is expected for this assignment, and your code must be included in your
submitted report – use the Python programming language
2. No machine learning models need to be trained for this assignment
3. External code may be used only if properly cited and intended for other problems – you
cannot use code intended for the MVTec dataset!
4. Include as many data visualizations as possible – a good visual is worth more than a
thousand words
5. It is recommended that you write your Python code as a Jupyter notebook and export as a
PDF report for submission – Jupyter notebooks enable easy insertion of figures with
Markdown text
Resources
Certain concepts and methods used in this assignment may be unfamiliar to you. Refer to these
online resources for more details (cite if code is used):
 Keras Installation: https://keras.io/#installation
 Training and test sets: https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets
 Evaluating a pre-trained CNN in Keras: https://medium.com/@vijayabhaskar96/tutorial-
image-classification-with-keras-flow-from-directory-and-generators-95f75ebe5720
 Summary of Grad-CAM: https://medium.com/@ninads79shukla/gradcam-73a752d368be
 ROC curves: https://machinelearningmastery.com/roc-curves-and-precision-recall-
curves-for-classification-in-python/
 Confusion matrix: https://www.geeksforgeeks.org/confusion-matrix-machine-learning/
 Intersection-over-Union / mean Intersection-over-Union:
https://www.jeremyjordan.me/evaluating-image-segmentation-models/
Pa
ge
4


ECE1512 2019 FALL
Format and Grading

Your report should be approximately 10 pages and list any additional references you may have used for
your answers. The page limit is a general guideline only, and includes any figures and code that you might
include in the report. Links to external web pages and code repositories, listing your results and code
implementations, such as “GitHub” are permissible.

Kindly provide answers to the above questions in complete sentences and in paragraph form. You will be
marked on the correctness, preciseness and comprehensiveness of your answers and quality of your
presentation. You are not required to observe the “Required Format for Submitting Assignment Reports”
guidelines included in your course syllabus for this assignment.

Turnitin scores will be visible upon submission, so make sure that you are not penalized for
plagiarism in uncited text and code portions.

The report should be submitted as PDF email attachment, using the following subject line:

ECE1512_HOMEWORK3-by-STUDENTNAME