Have you checked out Pachyderm (disclaimer: my company) for data pipelining on Kubernetes?

Our product has been available since the first Kubernetes release and is specifically designed for data analysis and pipelining tasks on Kubernetes.

github.com/pachyderm/pachyderm

pachyderm.io