Document Details

Clip: Every Model Learned by Gradient Descent Is Approximately a Kernel Machine Pedro Domingos pedrod@cs.washington.edu Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA 98195-2350, USA Abstract Deep learning's successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods. We show, however, that deep networks learned by the standard gradient de- scent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are eectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel.

Filename: 2012.00152.pdf

Notes: