What if it's a scheduler daemon running whenever-it-happens-to on one core, making policy decisions about when to preempt processes running on other cores (and which sleeping processes to schedule in their place onto those cores), and then queuing those decisions up into a "dumb" kernel-side scheduler using e.g. an io_uring?
Then the actual kernel-side scheduling process would just involve 1. the kernel popping things off the "to preempt" queue and setting interrupts on those cores; and 2. the preemption interrupt handler popping a process-block ref off the "next workload on this core" queue and switching to it. With no actual decision-making happening at preempt time.
(This is irrelevant to the greater point, though, because eBPF isn't a userspace process in the first place.)
You also need to handle the synchronous IPC case, where a thread wakes up another thread and goes to sleep until some event happens (e.g. the other thread finishes running). You still need to intelligently choose what thread to wake up in that case, and ideally you don't have to wait for another core to make that decision.
That's true. But I guess it could be handled by addressed by either the IPC invoker also placing the info on which process to wake into a BPF map before doing the blocking call, and the kernel code using the info to make the scheduling decision. That would however require an additional syscall per IPC. But maybe that informmation could be either cached or derived based on some other BPF visible state, so that it gets cheaper?
Then the actual kernel-side scheduling process would just involve 1. the kernel popping things off the "to preempt" queue and setting interrupts on those cores; and 2. the preemption interrupt handler popping a process-block ref off the "next workload on this core" queue and switching to it. With no actual decision-making happening at preempt time.
(This is irrelevant to the greater point, though, because eBPF isn't a userspace process in the first place.)