I think org-drill is just better overall since it's all stored as plain-text at the end of the day and each card is really editable and accessible. Anki in my experience has a clunky, slow GUI. It can't beat the creation and editing speed from vim keybindings and macros.
The thing I miss with org-drill is being able to do it on the phone. I have considered trying to integrate it into Orgzly but it does seem like a big project, unfortunately.
This sounds like an issue with something else you have installed in Emacs. I suggest running the profiler to see what is actually taking time. Org-drill itself on an otherwise vanilla config is fast, so it could be that you have hooks that are slow or something.
