* Container images are generally huge. Layer caching helps to the extent that layers are reusable. If you're lucky, you can use something like alpine for smaller huge images. If you write in a language that supports statically linked executables (e.g., Go) then you can get by with a scratch in many cases, but then you could more easily ship a naked executable
* Container images take a long time to build and build caching only works well if your dependency tree is linear (it's not). (Caveat: there are niche alternatives to docker build for which this may not hold, but then you're subject to the other issues with niche tools--support, docs, compat, etc)
* Your deployment target has to have a container runtime compatible with your image format. This is a PITA for end users in many cases.
* Most popular container runtimes make container processes children of the daemon process, not children of the client process. This means killing the parent process will not kill the child process, which means CI environments need to handle all sorts of edge cases. For example, a CI job is waiting on a stuck container to terminate, but eventually times out or the job is killed or similar; each time this happens a container is leaked--this means that CI tools have to support the container runtime explicitly and handle all of these edge cases because it's prohibitively complex for users to manage.
IMO the only advantage (and this is a huge advantage where applicable) for containers is that they can be orchestrated by things like Kubernetes or AWS Fargate.
