Assignment 1: First Steps

Deadline: Sunday October 21, 6 PM

In this assignment, you will implement and train your first deep model on a well-known image classification task. You will also familiarize yourself with the Tensorflow library we will be using for this course.

General Assignment Notes

• You will need to do some extra reading for these assignments. Sorry, but there is no way around this.
• Assignments are posed in a very open-ended manner. Often you only “need” to complete a rather basic task. However, you will get far more out of this class by going beyond these basics. Some suggestions for further explorations are usually contained in the assignment description. Ideally though, you should really see what interests you and explore those directions further. Share and discuss any interesting findings on Mattermost!
• Please don’t stop reading at “Bonus”; see above. Don’t be intimidated by all the text; pick something that interests you/lies within your capabilities and spend some time on it.

Setting Up

Install Python (3.x is recommended) if you haven’t done so, and install Tensorflow. This should be as simple as writing

pip install tensorflow

in your console. This will install the CPU version; for now, there is no need to bother with the GPU version since you will usually use your own machine only for development and small tests.

Download the raw MNIST files from Yann LeCun’s website. MNIST is a collection of handwritten digits and a popular (albeit by now trivialized) benchmark for image classification models. Download our conversion script. Unpack the data, put the script in the same folder and run it as

python conversions.py -c -n

This will create both csv tables and numpy arrays of the data (you don’t need the csv’s but the arrays are created from them). If you want to, you can also append the flag -p to create folders with the actual pictures to get an impression of the data (this will take a bit longer).

Note: For some reason, the MNIST file names seem to differ slightly between operating systems. You might need to adjust conversions.py accordingly if you get some “file not found” error.

Tensorflow Basics

NOTE: The Tensorflow docs went through significant changes recently. In particular, most introductory articles were changed from using low-level interfaces to high-level ones. We believe it’s better to start with low-level interfaces that force you to program every step of building/training a model yourself. This way, you actually need to understand what is happening in the code. High-level interfaces do a lot of “magic” under the hood. We will proceed to these interfaces after you learn the basics. This is why some of the links below lead to old versions of the TF docs that still have the low-level tutorials. Unfortunately, it seems that the ability to access old tutorials straightforwardly over the website has been removed, forcing us to link you to Github instead. Note that these tutorials sometimes have broken formatting (formulas especially). Sorry!

Get started with Tensorflow. There are many tutorials on diverse topics on the website, as well as an API documentation.

• Tensorflow Concepts: You need to understand tensors and variables, what a computational graph is, and how to train a model. You may ignore the section on tf.estimator for now.
• Basic MNIST Tutorial: A logistic regression “walkthrough” both in terms of concepts and code. You will of course be tempted to just copy this code; make sure you understand what each line does. Note that Tensorflow comes with an MNIST dataset already, but we recommend that you download and process the data yourself (see above) and use this simple dataset class. This way, you get used to processing/reading your own datasets.
• The Programmer’s Guide has more in-depth articles on many Tensorflow concepts. Right now you could read the ones on Tensors and Variables and maybe the one on Graphs and Sessions.

Play around with the example code snippets. Change them around and see if you can predict what’s going to happen. Make sure you understand what you’re doing.

Building A Deep Model

If you followed the tutorial linked above, you have already built a linear classification model (softmax regression). Next, turn this into a deep model by adding a hidden layer between inputs and outputs. There you go! You have created a Multilayer Perceptron. Hint: Initializing variables to 0 will not work for multilayer peceptrons. You need to initialize values randomly instead (e.g. random_uniform between -0.1 and 0.1). Why do you think this is the case?

Next, you should explore this model: Experiment with different hidden layer sizes, activation functions or weight initializations. See if you can make any observations on how changing these parameters affects the model’s performance. Going to extremes can be very instructive here.

Also, reflect on the Tensorflow interface: If you followed the tutorials you were asked to, you have been using a very low-level approach to defining models as well as their training and evaluation. Which of these parts do you think should be wrapped in higher-level interfaces? Do you feel like you are forced to provide any redundant information when defining your model? Any features you are missing so far?

Tensorflow Execution Model

It is extremely important that you understand the graph-based execution model of Tensorflow. As a rule of thumb, tf.'anything' builds the graph on a symbolic level, which should only be done once, after which the graph is run repeatedly to produce results. To make sure you understand this, below you will find some “problematic” code snippets. Analyze what is going wrong with these snippets and propose ways to fix them.

• Snippet 1
• Snippet 2

Later in the class, we will look at eager execution, which is more akin to how things would work e.g. in numpy or Pytorch.

Bonus

Feel free to explore Tensorflow and MNIST more. For example, this tutorial gives a more complete coverage of topics such as saving a trained model and using it to make predictions.

There are also numerous ways to explore your model some more. For one, you could add more hidden layers and see how this affects the model. You could also try your hand at some basic visualization and model inspection: For example, visualize some of the images your model classifies incorrectly. Can you find out why your model has trouble with these?

Finally, think about the semantics of your model(s). Can you describe what a specific activation value (in the output as well as in the hidden layer) “means”? Can you do this for the weights that were learned during training? You should start thinking about this for the logistic regression model (having no hidden layers) and then proceed to your MLP. Take note of how much more difficult it becomes to reason about your network as it gets deeper. Visualization is extremely useful here!

You may also have noticed that MNIST isn’t a particularly interesting dataset – even very simple models can reach very high accuracy and there isn’t much “going on” in the images. Luckily, Zalando Research has developed Fashion MNIST. This is a more interesting dataset with the exact same structure as MNIST, meaning you can use it without changing anything about your code. You can download the data and use conversions.py with it, or you can follow this to use it instead of the built-in Tensorflow MNIST. You can attempt pretty much all of the above suggestions for this dataset as well.