Solving Jigsaw Puzzles

Jigsaw puzzles

Similar to predicting the relative position of patches, this pretext task involves asking a neural network to solve jigsaw puzzles to develop a visuospatial representation of objects in the image. As shown in the figure below, the input image $X_i$ is first split into a $3\times3$ grid, and all nine patches, $X_i = [X_i^{p_1}, X_i^{p_2}, ... X_i^{p_9}]$, are shuffled based on a random permutation ,$p$, selected from a set of predefined permutations, $P$. This shuffled image $\text{permute}(X_i, \ p)$ is passed through the neural network $f(.)$ to predict which permutation from set $P$ was applied to the image (i.e., $f(\text{permute}(X_i, \ p)) = p$). Mathematically, the neural network performs a$\vert P \vert$- way classification problem and minimizes the following loss function: