Assignment 4: tf.Estimator & Assorted Programming Puzzles

Deadline: November 11th, 8pm

Same submission options as last time.

High-level Modeling in Tensorflow

You have hopefully seen that working with neural networks becomes more comfortable as you move from low- to higher-level interfaces. However, if you have been following the tutorials so far, you have been defining things such as training loops, evaluation procedures etc. yourself. This is rather annoying – compare this to libraries such as scikit-learn where such things can usually be done in a single line of code.

Luckily, Tensorflow also comes with similar high-level interfaces. In this assignment, we will be having a look at Tensorflow’s own Estimator class. Unfortunately, you will need to do quite a bit of extra reading again to get an overview. You can have a look at some or all of the following docs:

• An extremely high-level overview
• A tutorial using pre-built models for the well-known Iris classification task.
• The tutorial on checkpoints gives a quick overview over saving and restoring trained models.
• There is another tf.data overview.
• The last introductory tutorial shows how to build your own models for the Estimator API.
• Finally, this tutorial walks you through building a CNN for the MNIST task, allowing you to place the Estimator API in context with the methods you used in previous assignments. This is likely the most interesting/complete of these tutorials.

Note that the above tutorials mention “feature columns” quite a lot; feel free to ignore these beyond what is needed to follow along since we won’t be needing them anytime soon (they are not needed for “custom” estimators and are most interesting for categorial input data).

Build a functioning CNN using the Estimator interface, supporting all of training, evaluation and prediction on new inputs. You should have a grasp on these core components of building models with tf.Estimator:

• Build an input function, preferably using tf.data.
• Build a model function that can operate in train, predict and evaluate modes, usually based on tf.layers functions.
• Set up hooks as desired and run the model in the appropriate mode.

Once again, if you haven’t done so, use the Fashion-MNIST dataset instead for more of a challenge (or feel free to train models on other datasets). With the more convenient Estimator interface, experimenting with different models/ hyperparameters is hopefully more comfortable. Try to achieve the best results you can!

Some Estimator fun facts:

• A tf.summary.merge_all() op is automatically set up. No need to include this in your model functions when using Tensorboard.
• Summaries are saved every 100 steps by default, but you can change this if you want. They are saved in the same directory as the checkpoints are stored (the model_dir).
• A tf.summary.scalar is automatically set up for the loss in training mode.
• Similarly, a logging hook is set up for the loss when training, and a streaming metric when evaluating. No need to include these yourself.
• Never forget: tf.logging.set_verbosity(tf.logging.INFO) or else the estimator won’t talk to you.

Exploring Tensorflow

Following all those tutorials can be boring, so we will now be focusing on getting to know the Tensorflow core a little more. In the long run, knowing what tools you have available will allow you to write better/shorter/faster programs when you go beyond the straightforward models we’ve been looking at. Below you will find several (mostly disconnected) small programming tasks to be solved using Tensorflow functions. Most of these can be solved in just a few lines of code, but you will need to find the right tools first. The API docs will be indispensable here. Note: Below we are sometimes linking to the 1.10 API because for some reason the “API Guides” grouping functions by their usage etc. have been removed in the most recent API versions. If this leads to any issues, you might want to look at the most recent API version of the function you are interested in after you found it using the API Guides in the old version. Since the API can be a bit overwhelming at first, hints are included for each task.

• Given a 2D tensor of shape (?, n), extract the k (k <= n) highest values for each row into a tensor of shape (?, k). Hint: There might be a function to get the “top k” values of a tensor.
• As above, but instead of “extracting” the top k values, create a new tensor with shape (?, n) where all but the top k values for each row are zero. Try doing this with a 1D tensor of shape (n,) (i.e. one row) first. Getting it right for a 2D tensor is more tricky; consider this a bonus. Hint: You should look for a way to “scatter” a tensor of values into a different tensor.
• Implement an exponential moving average. That is, given a decay rate a and an input tensor of length T, create a new length T tensor where new[0] = input[0] and new[t] = a * new[t-1] + (1-a) * input[t] otherwise. Do not use tf.train.ExponentialMovingAverage. Hint: You might want to have a look at higher order functions to simulate a loop over the input. Alternatively, with the full input already being available you might be able to find a way to do this without looping. Do not use Python loops!
• Given three integer tensors x, y, z all of the same (arbitrary) shape, create a new tensor that takes values from y where x is even and from z where x is odd. Hint: An op from Sequence Comparison and Indexing could help.
• Given a tensor of arbitrary and unknown shape (but at least one dimension), return 100 if the last dimension has size > 100, 12 if the last dimension has size <= 100 and > 44, and return 0 otherwise. Hint: You will need some Control Flow Operation for this.
• Given two 1D tensors of equal length n, create a tensor of shape (n, n) where element i,j is the ith element of the first tensor plus the jth element of the second tensor. No loops! Hint: Tensorflow supports broadcasting much like numpy.