CS 6320.002: Natural Language Processing
Fall 2020
Homework 5 — 45 points
Issued 11 Nov. 2020
Due 11:59 pm 25 Nov. 2020
Deliverables: Answers can be typed directly into Gradescope. LaTeX can be handtyped or generate
...
CS 6320.002: Natural Language Processing
Fall 2020
Homework 5 — 45 points
Issued 11 Nov. 2020
Due 11:59 pm 25 Nov. 2020
Deliverables: Answers can be typed directly into Gradescope. LaTeX can be handtyped or generated using Mathpix Snip. See the assignment guide for more details.
What does it mean to show your work? Write out the math step-by-step; we
should be able to clearly follow your reasoning from one step to another. (You can
combine obvious steps like simplifying fractions or doing basic arithmetic.) The point
of showing your work is twofold: to get partial credit if your answer is incorrect, and to
show us that you worked the problem yourself and understand it. We will deduct points
if steps are missing.
1 Recurrent Neural Networks — 30 points
The questions in this section are based on material covered in Week 13.
W = (n × m), U = (n × n), and b = (n × 1). V = (3 × n).
1.1 Backpropagation Through Time
Suppose we have the following small recurrent neural network:
h
(t) = RELU(w1x
(t)
1 + w2x
(t)
2 + uh(t−1) + bh)
y
(t) = σ(vh(t) + by)
(The superscript notation (t) is used to avoid confusion when we also want to use subscripts for our parameters; h
(t) means the same thing as ht
in the slides.)
Suppose we have the following loss function:
L(y) = −ln(y)
(For the problems in this section, it is not required to submit a picture of the computation
graph, but it is highly recommended to draw one for yourself to aid in answering the
questions.)
1.1.1
What is the partial derivative of the loss L(y
(1)) with respect to the parameter w1,
ie. ∂L(y
(1))
∂w1
, in this network? Show your work and simplify the expression as much as
possible. You can assume that all values are positive for R
[Show More]