I don't know if this is a good faith question or not. But of course the concern isn't people getting diseases from plants.

Crop yields go down and in some cases the resulting grain becomes poisonous to people or animals which can in some cases cause disease (fungal most often).

good point, as i didn't word it correctly at all: how tricky is it to control diseases _among_ plant cultivars?

You're correct. Pigs are very similar to humans anatomically, and biochemically mammals share a LOT of genetics, so its much easier for mutations to lead to species hopping pathogens. This is one reason why Eurasian cultures dominated historically speaking: they'd lived with livestock for much longer and evolution crafted those human populations to have numerous immunities that isolated, non livestock holding communities didnt have.

Put in another way or with another interpretation: the arms race between disease and immune system in the animal-human-disease relationship led to very powerful diseases which the humans carried in their populations but were not largely affected by.

When the Europeans started crossing oceans to contact very different civilizations they brought the diseases which themselves committed genocide because the new populations hadn't had the same difficulty or interactions with animals and diseases.