Malware detection relies on signatures and/or heuristics. Signatures won't work with a brand new malware that hasn't been seen before. Heuristics can also be defeated.

Antimalware is great for the low hanging fruit but don't expect it to detect something where the author has put effort into it.