Reading Assignment: Revision (selected topics)

We will discuss about the topics you liked to revisit (Mainly RNNs/LSTMs and Attention). Please:

• find and share material (blogs, videos etc.) to understand the topics better
• post questions on MM which you’d like to discuss in-class next week
• also post something to MM if you like to discuss other topics

Here are the Reading Assignments of RNNs/LSTMs and about Attention

Reading Assignment: RNNs, LSTMs

Deep Learning Book - Chapter 10: Sequence Modeling: Recurrent and Recursive Nets, Sections 10.6, 10.7 and 10.10 to 10.12

Reading Assignment: Attention and Memory

Overview

For an overview on the topic, start with the blog post “Attention and Memory in Deep Learning and NLP” by Denny Britz and the distill.pub article “Attention and Augmented Recurrent Neural Networks” by Olah & Carter.

Main Reading

Continue by reading the following articles that provide details on two very recently proposed approaches,

• Graves et al. “Hybrid computing using a neural network with dynamic external memory” in Nature 538.7626 (2016): pp. 471-476.

• Ashish et al. “Attention Is All You Need” in NIPS (2017) (The version on arxiv seems to be most recent.)

Further Reading (Optional)

Follow the links from the blog post to the referenced articles for further reading.