To add on: Zeyuan Allen-Zhu and Yuanzhi Li had a great theoretical paper earlier last year proving a fundamental discrepancy between the concept classes learnable by kernel methods and deep learning models.

https://arxiv.org/abs/2001.04413