Yeah, someone else’s stealth would be a vs roll, but the other stuff like a foggy night would normally just be modifiers. There’s actually big difference statistically between doing a versus roll against 4 (which would cancel out 1-2 successes on average no matter what your skill was) and a -2 modifier (representing a “significant” challenge under the rules), which highly skilled people would just shrug off.
That’s tangential to your actual point, but just FYI 