Germ Theory 2.0

So, I realize how important germ theory was to the evolution of our society. Doctors stopped starting surgeries without wearing gloves directly after completing autopsies, and finally realized why all of their patients were dying.

But, this weekend, I was working outside in our yard and got visibly dirty hands. So, I went inside and washed them twice, because I could SEE that they weren't yet clean after my traditional 10-second washing.

Isn't this the same bad logic? Aren't I probably just wrongly assuming that when I can't visibly see the evidence, that the germs are probably gone?