Prezent od córki: burząca cisza starości

Ludwika Nowak krzątała się w kuchni od świtu: kroiła sałatki, gotowała rosół, wkładała do piekarnika kurczaka z czosnkiem, czyściła kryształowy wazon na kwiaty. Dwukrotnie zbiegła do sklepu – wróciła z tortem i wypchanymi siatkami, wpadając po drod# CS 224n Assignment #2: word2vec (43 Points)

## 1. Understanding word2vec (12 points)

### (a) (3 points)
* Repeating for completion:
\begin{align}
\mathbf{J}_{\text {naive-softmax }}(\mathbf{v}_{c}, o, \mathbf{U}) &=-\log P(O=o \mid C=c) \\
&=-\log \left( \frac{\exp \left(\mathbf{u}_{o}^{\top} \mathbf{v}_{c}\right)}{\sum_{w \in \text{Vocab}} \exp \left(\mathbf{u}_{w}^{\top} \mathbf{v}_{c}\right)} \right) \\
&=-\mathbf{u}_{o}^{\top} \mathbf{v}_{c}+\log \left(\sum_{w \in \text{Vocab}} \exp \left(\mathbf{u}_{w}^{\top} \mathbf{v}_{c}\right)\right)
\end{align}
* Therefore:
\begin{align}
\frac{\partial \mathbf{J}_{\text {naive-softmax }}}{\partial \mathbf{v}_{c}} &= \frac{\partial}{\partial \mathbf{v}_{c}} \left ( -\mathbf{u}_{o}^{\top} \mathbf{v}_{c} + \log \left( \sum_{w \in \text{Vocab}} \exp \left(\mathbf{u}_{w}^{\top} \mathbf{v}_{c}\right) \right ) \right ) \\
&= -\mathbf{u}_{o} + \sum_{w \in \text{Vocab}} \frac{\exp \left(\mathbf{u}_{w}^{\top} \mathbf{v}_{c}\right) \mathbf{u}_{w}}{\sum_{w \in \text{Vocab}} \exp \left(\mathbf{u}_{w}^{\top} \mathbf{v}_{c}\right)} \\
&= -\mathbf{u}_{o} + \sum_{w \in \text{Vocab}} P(O=w | C=c) \mathbf{u}_{w} \\
\end{align}

* Interpretation: the derivative of the loss with respect to $v_c$ is the difference between the vector of actual outside word $u_o$ and the expectation of all outside word vectors under the current model. This makes sense as we are trying to maximize the likelihood of the actual outside word $o$ appearing around the center word $c$ while minimizing the likelihoods of all other words.

### (b) (3 points)
* Repeating for completion:
\begin{align}
\frac{\partial \mathbf{J}_{\text {naive-softmax }}}{\partial \mathbf{v}_{c}} &= -\mathbf{u}_{o} + \sum_{w \in \text{Vocab}} P(O=w | C=c) \mathbf{u}_{w}
\end{align}
* Therefore:
\begin{align}
\frac{\partial \mathbf{J}_{\text {naive-softmax }}}{\partial \mathbf{u}_{w}} &= \frac{\partial}{\partial \mathbf{u}_{w}} \left ( -\mathbf{u}_{o}^{\top} \mathbf{v}_{c} + \log \left( \sum_{x \in \text{Vocab}} \exp \left(\mathbf{u}_{x}^{\top} \mathbf{v}_{c}\right) \right ) \right ) \\
\end{align}
* When $w=o$:
\begin{align}
\frac{\partial \mathbf{J}_{\text {naive-softmax }}}{\partial \mathbf{u}_{w}} &= -\mathbf{v}_{c} + \frac{\exp \left(\mathbf{u}_{o}^{\top} \mathbf{v}_{c}\right) \mathbf{v}_{c}}{\sum_{x \in \text{Vocab}} \exp \left(\mathbf{u}_{x}^{\top} \mathbf{v}_{c}\right)} \\
&= – \mathbf{v}_{c} + P(O=o | C=c) \mathbf{v}_{c} \\
&= \mathbf{v}_{c} (P(O=o | C=c) – 1)
\end{align}
* When $w \neq o$:
\begin{align}
\frac{\partial \mathbf{J}_{\text {naive-softmax }}}{\partial \mathbf{u}_{w}} &= \frac{\exp \left(\mathbf{u}_{w}^{\top} \mathbf{v}_{c}\right) \mathbf{v}_{c}}{\sum_{x \in \text{Vocab}} \exp \left(\mathbf{u}_{x}^{\top} \mathbf{v}_{c}\right)} \\
&= P(O=w | C=c) \mathbf{v}_{c}
\end{align}
* Combined:
\begin{align}
\frac{\partial \mathbf{J}_{\text {naive-softmax }}}{\partial \mathbf{u}_{w}} = \mathbf{v}_{c} (P(O=w | C=c) – \mathbb{I}_{w=o})
\end{align}

### (c) (2 points)
* The derivative we computed above is extremely expensive to compute because we have to loop through the entire vocabulary in order to calculate $\sum_{w \in \text{Vocab}} P(O=w | C=c) \mathbf{u}_{w}$. Even though we have to loop through the entire vocabulary for each training example, we may be able to use matrix algebra to parallelize the calculations (such as the calculation of the dot products and $P(O=w | C=c)$). We also have to store the entire outside embedding matrix in memory in order to perform the computations, which may be difficult for very large vocabularies (unless we use sparse matrix representations).

### (d) (4 points)
* **Sigmoid function**:
\begin{align}
\sigma(x) &= \frac{1}{1 + e^{-x}} \\
\frac{\partial \sigma(x)}{\partial x} &= \frac{e^{-x}}{(1 + e^{-x})^2} = \sigma(x) (1 – \sigma(x))
\end{align}
* **Negative sampling loss**:
\begin{align}
\mathbf{J}_{\text{neg-sample}}(\mathbf{v}_{c}, o, \mathbf{U}) &= – \log(\sigma(\mathbf{u}_{o}^{\top} \mathbf{v}_{c})) – \sum_{k=1}^{K} \log(\sigma(-\mathbf{u}_{k}^{\top} \mathbf{v}_{c})) \\
\frac{\partial \mathbf{J}_{\text{neg-sample}}}{\partial \mathbf{v}_{c}} &= – (1 – \sigma(\mathbf{u}_{o}^{\top} \mathbf{v}_{c})) \mathbf{u}_{o} + \sum_{k=1}^{K} (1 – \sigma(-\mathbf{u}_{k}^{\top} \mathbf{v}_{c})) \mathbf{u}_{k} \\
&= – \sigma(- \mathbf{u}_{o}^{\top} \mathbf{v}_{c}) \mathbf{u}_{o} + \sum_{k=1}^{K} \sigma(\mathbf{u}_{k}^{\top} \mathbf{v}_{c}) \mathbf{u}_{k} \\
\end{align}
* **Derivatives for outside word vectors**:
\begin{align}
\frac{\partial \mathbf{J}_{\text{neg-sample}}}{\partial \mathbf{u}_{o}} &= – (1 – \sigma(\mathbf{u}_{o}^{\top} \mathbf{v}_{c})) \mathbf{v}_{c} = – \sigma(-\mathbf{u}_{o}^{\top} \mathbf{v}_{c}) \mathbf{v}_{c} \\
\frac{\partial \mathbf{J}_{\text{neg-sample}}}{\partial \mathbf{u}_{k}} &= (1 – \sigma(-\mathbf{u}_{k}^{\top} \mathbf{v}_{c})) \mathbf{v}_{c} = \sigma(\mathbf{u}_{k}^{\top} \mathbf{v}_{c}) \mathbf{v}_{c} \quad \forall k \in \{1, \ldots, K \}
\end{align}

* Negative sampling is much more computationally efficient than naive softmax because we do not need to compute derivatives with respect to all outside word vectors in the vocabulary. Instead, we only need to compute derivatives with respect to the one actual outside word and $K$ negative samples. This makes the cost of each training step independent of the vocabulary size, which is a major advantage when the vocabulary is large.

## 2. Implementing word2vec (21 points)

### (a) (6 points)
* See the `word2vec.py` file for the implementation of `sigmoid` and `naiveSoftmaxLossAndGradient`.
* Loss and gradient calculations were verified numerically using `gradcheck_naive` for a toy model and vocabulary.
* The softmax probabilities were also compared against a manual calculation on the toy model for the same test case and found to be exactly identical.

### (b) (3 points)
* See the `word2vec.py` file for the implementation of `negSamplingLossAndGradient`.
* Loss and gradient calculations were verified numerically using `gradcheck_naive` for a toy model and vocabulary.

### (c) (4 points)
* See the `word2vec.py`Liść spadł na grób Ludwiki, a staruszka westchnęła ciężko i odwróciła się plecami do bloku, w którym nigdy już nie rozlegnie się radosny głos jej przyjaciółki.