Absolutely. However often you reseed the generator you need it to be unguessable.
Server activity would be a good source of entropy with the caveat, as you point out, that you need to be sure that there is more activity on the server than just the person receiving the random number. If server activity is the only source you use though you're vulnerable to some attacks. If an attacker can force your server to reboot or can utilise all available connections, locking anyone else out, then they could control all the server activity you're relying on.
The solution is simply to use multiple sources. Server activity plus any other sources of entropy you have available on the server.
