Perhaps my use of 'falsely' is confusing, but what I mean is if what you are refusing to say is not actually a crime, and it is discovered, you could be charged with a crime (i.e. perjury, giving false statements under oath).
It doesn't matter that adultery isn't a crime: you're being questioned by the police, and any answer has the possibility of including details which might seem incriminating for whatever they're actually investigating, regardless of your innocence.
It seems like you're approaching the 5th amendment as a direct point/counterpoint arrangement: the police accuse you of robbing a bank, you know you didn't because you were cheating on your spouse, you can't take the 5th because cheating on your spouse isn't a crime, so you must answer "I couldn't have robbed the bank because I was cheating on my spouse at another location".
But the amendment is much broader than that. Since you don't know what other information the police are attempting to validate or even what crimes they're aware of, you have no way to know if "I was at $hotel with $other_person at $time" may incriminate you in this case or some other case.
You are right that the police can't force you to talk and that a judge can't if you are on trial.
However a judge could force you to testify in someone else's case. You can't broadly plead the 5th in that case unless the testimony is incriminating. Let's say you saw a murder happen at a hotel you were at while cheating with your wife. A judge is allowed to force you to answer the question.
Yes.Infact, if you conceal the information about the crime and the crime happens to be felony, you will be charged for "misprision of felony."
The federal definition of misprision requires that, “(1) the principal committed and completed the felony alleged; (2) the defendant had knowledge of the fact; (3) the defendant failed to notify the authorities; and (4) the defendant took affirmative steps to conceal the crime of the principal.” See United States v. Baumgartner (6th Cir. Sept. 24, 2014)
Disclaimer: IANAL.
EDIT: I'm getting some downvotes, so I will link to this that seems to support what I'm saying:
http://criminal-law.freeadvice.com/criminal-law/white_collar...
Perhaps my use of 'falsely' is confusing, but what I mean is if what you are refusing to say is not actually a crime, and it is discovered, you could be charged with a crime (i.e. perjury, giving false statements under oath).