COMP5623 Coursework on Image Caption Generation
Deadline
10am 28th April 2020.
Submission
Submit work electronically via Minerva. This coursework is worth 20% of the credit on the
module.
Time allocation
An indicative duration for this coursework is a total of around 17 hours.
Motivation
Through this coursework, you will:
 Understand the principles of text pre-processing and text embeddings
 Gain experience working with an image to text model
 Compare the performance and usage of RNNs versus LSTMs as sequence generators

Setup and resources
Please implement this coursework using Python and PyTorch. This coursework will use the
Flickr8k image caption dataset [1].
Sample image and ground truth captions


A child in a pink dress is climbing up a set of stairs in an entry
way
A girl going into a wooden building
A little girl climbing into a wooden playhouse
A little girl climbing the stairs to her playhouse
A little girl in a pink dress going into a wooden cabin
You are provided with a Jupyter Notebook file “COMP5623 CW2 Starter” with starter code.
Dataset download
The zip files containing the image data and text data can be downloaded here:
https://github.com/jbrownlee/Datasets/releases/tag/Flickr8k
This dataset is larger than the ones we have used previously, so if you would like to work on
Colab, it is recommended to download the zip files, unzip them, and then upload all the files to
your Google Drive. This initial upload may take a while, but from then on you will only need to
mount your Drive every time you start a new Colab session and the data will be immediately
accessible. Mounting only takes a few seconds.
Text preparation
The first step is to build a vocabulary. The vocabulary consists of all the possible words which
can be used - both as input into the model, and as an output prediction. To build the vocabulary:
a. Parse the lines variable in the starter code, splitting the image ID from the caption
text.
b. Split the caption text into words and trim any trailing whitespaces.
c. Convert all words into lowercase by calling word.lower().
d. Remove any punctuation (periods, commas, etc.).
e. Since the vocabulary size affects the embedding layer dimensions, it is better to remove
the very infrequently used words from the vocabulary. Remove any words which appear
3 times or less; this should leave you with a vocabulary size of roughly 3440 (plus or
minus some is fine).
f. Add all the remaining words to an instance of the Vocabulary() object provided.
Note that vocab.add_word() deals with duplicates already.
Complete the encoder network
Carefully read through the __init__() function of the EncoderCNN and complete the class
by writing the forward() function. Remember to put the part involving the ResNet in a with
torch.no_grad(): block.
BLEU for evaluation
One common way of comparing a generated text to a reference text is using BLEU, or the
Binlingual Evaluation Understudy. This article gives a good intuition to how the BLEU score is
computed: https://machinelearningmastery.com/calculate-bleu-score-for-text-python/
The Python ntlk package for natural language processing provides a function
sentence_bleu() which computes this score given one or more reference texts and a
hypothesis. For the following two sections, generate this score by using all five reference
captions compared against each generated caption. You can import it on Colab like this:
from nltk.translate.bleu_score import sentence_bleu
If working on your local machine you may need to install the nltk package using pip or conda
first.
Train using an RNN for decoder
Add an RNN layer to the decoder where indicated, using the API as reference:
https://pytorch.org/docs/stable/nn.html#rnn
Keep all the default parameters except for batch_first, which you may set to True.
Add code to the training loop provided which generates a sample caption from two test set
images before any training, and after each epoch. Display both the sample images, the reference
captions, generated caption, and its BLEU score after every epoch. Train for 5 epochs.
Displaying an image

To generate a prediction from your trained model, you will need to first pass the image tensor
into the encoder to get the image features. Remember to first process the image with the
data_transform. Because the EncoderCNN has a batch norm layer, and you are feeding
in single images (plus we do not want the shift the mean and std deviation), run this line before
passing in your images:

encoder.eval()

Pass the features from the encoder into the decoder sample() function.

The decoder output is a one-dimensional vector of word IDs, and length max_seq_length.
This means that if the caption is shorter than max_seq_length you can ignore any padding
after the token.

Finally, use the vocab to convert the IDs into words and string the caption together.

Train for exactly five epochs. No need to use a validation set.
Using the fully trained model, generate captions for at least five additional test images and
display along with BLEU scores and reference captions.
Train using an LSTM for decoder
Now replace the RNN layer with an LSTM (leave the RNN layer commented out):
https://pytorch.org/docs/stable/nn.html#lstm
Again, keep all the default parameters except for batch_first, which you may set to True.

As with the RNN, display the same two sample images, generated captions and BLEU scores
before training and after each epoch. Train for five epochs. Using the fully trained model,
generate captions for at least five additional test images and display along with BLEU scores and
reference captions.

Deliverable and Assessment

You should submit a report of not more than 8 pages (excluding Python code):
1. A common practice in natural language processing is to lemmatize words before creating
tokens. While we did not do this for our vocabulary, take a look at
Flickr8k.lemma.token.txt and briefly explain what the advantages/disadvantages might be
of using lemmatized vs regular tokens. (max. 5 marks)
2. Present the sample images and generated caption for each epoch of training for both the
RNN and LSTM version of the decoder, including the BLEU scores. (max. 30 marks)
3. Compare training using an RNN vs LSTM for the decoder network (loss, BLEU scores
over test set, quality of generated captions, performance on long captions vs. short
captions, etc.). (max. 30 marks)
4. Among the text annotations files downloaded with the Flickr8k dataset are two files we
did not use: ExpertAnnotations.txt and CrowdFlowerAnnotations.txt. Read the readme.txt
to understand their contents, then consider and discuss how these might be incorporated
into evaluating your models. (max. 10 marks)
5. Python code for all your work (max. 25 marks)

Your submission should include one PDF file for your report, and one Python or .pynb file,
zipped together into a .zip file.

References
[1] M. Hodosh, P. Young and J. Hockenmaier (2013) "Framing Image Description as a Ranking
Task: Data, Models and Evaluation Metrics", Journal of Artifical Intellegence Research, Volume
47, pages 853-899 (http://www.jair.org/papers/paper3994.html)