Sounds like you got caught in a rip current, it's why you should always swim back to shore at a 45 degree angle, you'll eventually make enough horizontal progress to escape the current. https://en.wikipedia.org/wiki/Rip_current