Perhaps, like me, your serial port was driven by an older UART chip like an 8250 [0] which could be a bottleneck for modems faster than 9,600 baud.
I remember in teen years phoning the local computer shop every week asking if they had the 16550 UART [1] serial cards in stock yet to upgrade my 486DX4-100 to get full 28.8kbps from my modem. Can't remember if they ever did.
I think the 16550 was faster largely because it had a larger buffer, so it wouldn't wait or drop data as often when the CPU was slow/busy.
I remember in teen years phoning the local computer shop every week asking if they had the 16550 UART [1] serial cards in stock yet to upgrade my 486DX4-100 to get full 28.8kbps from my modem. Can't remember if they ever did.
I think the 16550 was faster largely because it had a larger buffer, so it wouldn't wait or drop data as often when the CPU was slow/busy.
[0]: https://en.m.wikipedia.org/wiki/8250_UART
[1]: https://en.m.wikipedia.org/wiki/16550_UART