From my personal experience using homebrew it is two things.
1) The lack of packages combined with lack of quality of the packages (e.g. they do not build or have a poor defaults).
2) That homebrew installs stuff under /usr/local as owned by your user. That does not meet the expectations of UNIX applications and seems to me like it could be unsafe due to messing with expectations. Stuff under /usr/local/bin is supposed to be owned by root:root.
The good part of homebrew is that writing homebrew scripts seems easier than writing dpkg or rpm scripts.
1. Installing everything from source (normally I don't mind compiling, but large packages can take an hour or two. Ironically, OS X doesn't need to support many different architectures, so there's no excuse for not having binary compatibility like Linux does).
2. It doesn't manage the entire system. The process for installing from a .dmg or from the app store is completely different from homebrew, and the two don't talk to each other well (or really at all). I want a single source to manage all of my executable files and their configurations, not a small handful that each manage part of it.
In OSX, /usr/local is really user-space. It comes empty on an install, and the system never touches it (that means no mess with /opt). The FHS also doesn't mandate /usr/local/bin to be owned by root. Don't root-owned utilities go in /usr/sbin?
Because the ability to write to a directory in a user's PATH makes it easy to trick this user into running arbitrary code. And /usr/local is a particularly poor choice for package management in a system like OS X without a "universal" package manager, because lots of prepackaged, precompiled third-party software already uses it. It's also not a particularly good idea to put your package manager's paths ahead of the system paths in PATH unless you want shell scripts to mysteriously break when some package pulls in an unanticipated dependency that includes newer, older, or different tools by the same name as system utilities (two examples that come to mind are HEAD from libwww-perl on a case-insensitive filesystem and xattr from the Python module of the same name; also GNU foo where the OS ships some GNU-incompatible version of foo). Instead, create another directory to hold symlinks for the few utilities you want to override system defaults (on my own systems, these include python, emacs, and git).
1) The lack of packages combined with lack of quality of the packages (e.g. they do not build or have a poor defaults).
2) That homebrew installs stuff under /usr/local as owned by your user. That does not meet the expectations of UNIX applications and seems to me like it could be unsafe due to messing with expectations. Stuff under /usr/local/bin is supposed to be owned by root:root.
The good part of homebrew is that writing homebrew scripts seems easier than writing dpkg or rpm scripts.