I would say it's one thing if people perform their duties adequately and in the name of public service and then go on to use the skills they've gained in the private sector.

This is different, his actions/inactions in his public service directly affected his future employer in a beneficial way. Is it so weird that people would be distrustful in a situation like this? If there was no issue do you really think anybody would doubt he doesn't have a conflict of interest?