Yes. Adapter latency is guaranteed to be 0.5-5 usec depending on generation (later revisions are faster+lower latency). Switches add about 170 nsec per hop, plus whatever speed-of-light delays your cables add. 1 usec end-to-end is realistic for modern gear.
Of course with IP-over-IB you also need to factor in whatever latency is added from your network stack, but Infiniband also supports RDMA so IB-aware applications can essentially bypass this for high-performance or hard-real-time usecases.
Basically, Infiniband takes the network out of the equation here. The bottleneck then becomes how precisely you can time the software.
