When you visit a website that downloads Javascript to your computer, you are giving the site an implied license to execute code reasonably related to the website. If they go off and use that capability to do something else, you've exceeded the scope of that implied license.
It's like going to a kitchen remodeling store and hosting a dinner party there. Yes, they invited you in, but the scope of that invitation was looking at new counters, not hosting a party. That's trespassing. In the computer realm, it may well be a CFAA violation.