There's an option in the Preferences to "Show a badge on Slack's icon to indicate new activity" which, when unchecked, may stop the behaviour you're describing.