SpanBERT: Exploring Architecture

In SpanBERT, we mask a contiguous span of tokens in the sentence. Let $x_s$ and $x_e$ be the start and end position of the masked tokens, respectively. We feed the tokens to SpanBERT and it returns the representation of all the tokens. The representation of token $i$ is represented as $R_i$. The representation of the tokens in the span boundary is denoted as $R_{s-1}$ and $R_{e+1}$.

The span boundary objective

Let's first look at the SBO. To predict the masked token, $x_i$, we use three values, which are the representation of the tokens in the span boundary ($R_{s-1}$ and $R_{e+1}$), and the position embedding of the masked token ($p_{i-s+1}$). Okay, how exactly do we predict the masked token with these three values? First, we create a new representation called $z_i$ using a function $f(⋅)$, with these three values as shown: