"Do they still store public/private keys on the same server?"
As phrased, this is not a problem - there's never any worry to including your public key wherever you have your private key; your attacker can be assumed to have your public key anyway if it'll do them any good.
The problem was private keys (encrypting important things!) on a web-accessible server, was my understanding.
Well - it is not likely you will need you public key on the non-web-accessible server. In this type of application the public key is needed in the place that encryption happens and the private key is needed where decryption happens. If the two are on the same machine it likely means you messed up.
You're assuming a single piece of information has a single key-pair. E.g.:
1. Obtain sensitive information
2. Generate a new key-pair
3. Encrypt with public key
4. Store encrypted info
5. Delete public key
6. Use private key to decrypt when reading the data
It's also likely they they were using one key-pair to encrypt all of their data (or all of a specific type, e.g. one key-pair to encrypt all passwords). In this case, the public key would be needed to encrypt new data coming in.
As phrased, this is not a problem - there's never any worry to including your public key wherever you have your private key; your attacker can be assumed to have your public key anyway if it'll do them any good.
The problem was private keys (encrypting important things!) on a web-accessible server, was my understanding.