1


Lab 4 – Dynamic Convolutional Filters for CIFAR-10 Classification
ENGN8536, 2019
20th September 2019
1. Introduction
The goal of this lab is to introduce you a state-of-the-art idea called the Dynamic Filters, which is
closely related to the concept of Conditional Probability and the Attention Mechanism, it has been
widely applied in many machine learning networks.
You might never come across any of the terms above, but with all the knowledge that you have learnt
in this course so far, you should be able to complete this lab. However, if you want to have a deeper
understanding of the knowledge behind the lab tasks, please refer to the papers in the reference list.
Let’s first take a glance at the dynamic filter with a simple example: Consider a system which takes
two signals as input, one is fixed (and assume it is the main signal) while the other one changes with
respect to time, the system predicts an output based on the two input signals. Clearly, the output will
be different if the second signal varies. An easy way to address this problem is to simply concatenate
the two signals and pass to the network for prediction, however, this requires the network to have a
strong capability to learn about the variation of the second signal and its relationship to the first signal
at the same time, which could be very difficult, especially when the variation is large. What could
happen is either the network only learns about the first signal because it is fixed and therefore easy
for learning features, or the network only cares about the second signal since it varies all the time and
contains more information. Of course, you can initialize numbers of submodules each dealing with a
specific type of the second signal. However, in the case that if the signal type is unknown, you will
need to do clustering which means lots more work, or you can deal with it implicitly (e.g. Multihead
Attention [4]) which makes your network much larger.
This is where the idea of Dynamic Filters comes in – what if we can train a network that can learn to
generate different filters each suitable for an/some input itself?
For this lab, we are going to apply dynamic filters on one of the convolutional layers to achieve ‘label-
guided feature extraction’, please read and follow the instruction carefully, and don’t forget to answer
the questions. Make good use of the PyTorch official documentation and you may find some tricks
for coding in the PyTorch discussion forum.







Figure 1. Network schematic.
The network that you are going to build in this lab is shown in Figure 1. In brief, the network takes
an image from the CIFAR-10 Dataset as input, encode by the first several layers of a pre-trained
ResNet-18 (backbone) and a 1-by-1 dynamic convolutional layer (DC). The encoded image features
will pass to a fully-connected layer (FC) to produce a single score for a class at a time. The weights
for the dynamic convolutional layer are generated by a multi-layer perceptron (MLP), which takes
Class
Embeddings
Input RGB
Image
R
esN
et-1
8
B
ackb
o
n
e
v
D
yn
am
ic
C
o
n
vo
lu
tio
n vො
Class Score
Predictor -
MLP
Output one classification score
for a single class at a time
Score
Weights
Generator -
MLP
Generate different weights for
each class separately

Class 1
w
2

one class embedding at a time as input. By repeating this process for 10 times, each one for a single
class and the same input image, you will get 10 prediction scores and now you can use argmax() to
assign a label to the input image.

2. Task 1 – Implement the Dynamic Convolutional Network
2.1 Build the network by filling in the code for the BaseModel() class in model.py
For the BaseModel initialization: (2 marks)
• Initialize an Dim128 representation for each class label randomly using nn.Embedding().
• Initialize the Weights Generator MLP as FC1 – ReLU – FC2 – Tanh, where FC1 maps input
Dim128 to hidden Dim256, FC2 maps hidden Dim256 to output Dim64.
• Initialize a pretrained ResNet-18 model from torchvision.models, however, only use the
model up to (layer1) as your Backbone. You can print out the ResNet-18 model from
torchvision.models to see how each layer is named. Be careful, do not freeze the weights for
the Backbone (i.e. do NOT set params.requires_grad=False).
• Initialize the Dynamic Convolutional (DC) layer as a 1-by-1 2D convolutional layer which
maps 64 input channels to 1 output channels with stride 1 and no padding.
• Initialize the Class Score Predictor MLP as FC1 – ReLU – FC2, where FC1 maps input Dim64
to hidden Dim64, FC2 maps hidden Dim64 to output Dim1.
For the forward path: (2 marks)
• Pass the input image to the backbone convolutional network to get feature map . Just for
sanity check, should be dimension of batch_size × 64 × 8 × 8.
• Take out one vector representation for a class (e.g. Car) from the Class Embeddings, pass it
to the Weights Generator to get a tensor of Dim64, which is the weight for the DC layer.
• Apply L2-normalization on .
• Set the weights of the DC layer as the normalized .
• Pass the feature map to the DC layer to get the response map ො. For numerical stability
reasons, rescale ො by √64, where 64 is the size of the dynamic filter.
• Reshape the scaled response map ො and pass it to the Class Score Predictor to get a
classification score for a single class (e.g. Car), corresponding to the class-specific DC filter.
• Repeat the above for 10 times for each class to obtain the classification score for each class.

2.2 Fill in the missing code in train.py as indicated by the comments
• Save the checkpoints. Basically, for each evaluation(), you need to save the states of the model
and the optimizer whenever you trained a better network (e.g. whenever the new testing
accuracy is higher than the previous best testing accuracy). (0.5 mark)
• Fill in the code for resume() to load the saved model and optimizer, so that you can resume
your network to a state with the highest performance anytime and continue training or run
testing. (0.5 mark)


3


2.3 Train the network and compare to the network without dynamic filter
• Train the network which you have built in 2.1, record the training and testing statistics (any
information that you think can best reflect the training process and the result). (1 mark)
• Compare the training stats to the same network without the dynamic mechanism, i.e. stop
using the Weights Generator MLP and make the 1-by-1 convolutional layer trainable. (1 mark)

3. Task 2 – Understand the Dynamic Convolutional Network
3.1 Visualize the response map for an arbitrary input
As shown in Figure 2, the prediction score for class Plane is the highest among all for the input image.
We can see that (although it is very coarse), the shape of the object in the input image is most similar
to the shape of region with higher response in the map for Plane.
• Choose a correctly-labelled image from the train or the test set (and different from the image
in Figure 2) and visualize its response maps, one for each class. You need to rescale each
response map to the same size as the input image. (optional task, you will 1 extra mark if you
do the visualization nicely)
• Briefly talk about what do you think the possible usage of the response map could be. (1 mark)

Figure 2. Response map visualization.

3.2 Answer the following questions
• What will happen if you freeze the weights for the Backbone and why is that? If you have to
freeze the weights, what could be a possible way to address the issue? (1 mark)
• List at least two advantages and two possible downsides of using the Dynamic Filter. (1 mark)

Notes
The lab report will be due on Sunday, 13 October 2019, 6:00pm. You have two weeks to complete
the lab. You are required to complete all tasks and answer all questions. Please submit a single zip
file containing a report in pdf format and all the code. Name your pdf as Lab_4_uxxxxxxx.pdf,
replacing uxxxxxxx with your uni-ID.
This Lab is worth 5% of the total course assessment.
4

References
[1] Jia, X., De Brabandere, B., Tuytelaars, T. and Gool, L.V., 2016. Dynamic filter networks. In Advances in
Neural Information Processing Systems (pp. 667-675).
[2] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R. and Bengio, Y., 2015, June.
Show, attend and tell: Neural image caption generation with visual attention. In International conference on
machine learning (pp. 2048-2057).
[3] Landi, F., Baraldi, L., Corsini, M. and Cucchiara, R., 2019. Embodied Vision-and-Language Navigation
with Dynamic Convolutional Filters. arXiv preprint arXiv:1907.02985.
[4] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, and Polosukhin, I.,
2017. Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008).
[5] Gavrilyuk, K., Ghodrati, A., Li, Z. and Snoek, C.G., 2018. Actor and action video segmentation from a
sentence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5958-
5966).