One alternate reason to not let your dog on other people's grass is pesticides and other lawn treatments. Many people do chemical warfare on their lawns to keep them nice, and it is not good for dogs or people to have contact with that.
Personally I always kept my dog off other peoples property. Grass or dirt...it doesn't matter, because it isn't mine.
And yes, just your dog walking on a lawn isn't going to damage it much, but the next 100 might. It's like walking in the woods. One person doesn't do much damage, but hiking trails get worn in by many people doing the same thing over and over.
That guy might have just finished replanting the edge of his lawn from the last time people walkied on the edges, and was a little touchy about it. Yes, he was rude, but technically he was right.
--Mia