This is a good example of the difference between an excuse and a reason. The excuse people use to oppose roaming is "risk of lawsuits" but the actual reason is the "what's mine is mine and you can't use it" mentality.
If you own the land I don’t see any why it’s a bad thing to not want others trespassing on it. No diff than I don’t want people randomly deciding to sit in my car for whatever reason while I’m at work.
Keep in mind that 'right to roam' applies to unimproved and (some) agricultural land, and roamers can't disturb the right of enjoyment of the owner. This isn't your normal urban backyard.
Its more like someone walked through the parking spot that you paid to park in on their way to somewhere else.