These things take time. A lot of people at Apple are heavily invested in Swift on the server, including the Swift core team: https://swift.org/server/
Swift itself still needs time to mature, too. Things like the memory model, async/await, generics, etc. The future is bright, it will just require some patience to get there.
There seems to be very little promotional stuff about the runtime for server-side. What's the story with that for Swift?
I don't think many people are going to start learning Swift just to build servers with it unless the platform/runtime provides something special. Otherwise it'll stay isolated just to the current iOS devs.
Swift itself still needs time to mature, too. Things like the memory model, async/await, generics, etc. The future is bright, it will just require some patience to get there.