Convolutional Neural Networks (CNNs) are now a well-established tool for solving computer vision and imaging problems, and modern CNNs often obtain state-of-the-art performance. Perhaps surprisingly, it has been recently shown that, despite being highly overparameterized, such networks can be trained with a single corrupted image and still perform as well as fully trained networks - a phenomenon encapsulated in the deep image prior (DIP). Here we attempt to explain what might be going on in terms of recent advances of Neural Tangent Kernel (NTK) theory, which characterizes the large system limit of neural networks. We identify strong links between CNN architectures and well-known signal processing techniques such as non-local means, showing that the function associated with a CNN to a given image can be obtained in closed form without need to train the network. Although our analysis shows that the NTK still does not fully explain the DIP phenomenon, we argue it suggests that CNN's inductive bias is better characterized by images with non-local self-similar structure.