You could get a slight speedup by matching letter frequency with easiest/fastest semaphore, similar as to morse code and dvorak typing do it. You loose compatibility with the rest of the world though, but if you are only using it for personal typing it's not a problem. You could also do a braille mapping instead of using semaphor so braille readers can get a workout.

A good way to play interactive fiction games: X, I, U, D, N, S, W, E, Z, OPEN, CLOSE, GIVE, ASK/TELL ABOUT https://www.microheaven.com/ifguide/step3.html