No. It is a mathematical fact that matrices fulfill the conditions of forming a vector space (subject to reasonable conditions like having elements drawn from a field). Matrices are just as differentiable as (other) vectors!

I think you're arguing that a matrix with constant real entries isn't differentiable with the standard derivative. This is true of course, but hasn't anything to do with vectors.