The only mechanism I can come up with is to detect illegal instruction traps on small cores, then flag the thread as big-core-only and re-start the execution at the bad instruction. That's not ideal but maybe workable.
(If it traps on the big core, too, it's just a bad instruction and SIGILL is raised to userspace like usual.)
If a binary does that, the OS can just not migrate that binary across cores.