Absolutely. What you're feeling is pretty normal. RSV concerns are minimal until about October, when you'll want to be hypervigilant about germs again, but you can relax a bit in summer. Obviously major germs like grocery carts are a no-no, but every day house germs are good for Isaac, so that he builds more immunity before winter germ season.
You have a mother's intuition and you know Isaac better than anyonr, so you can relax a bit, and know that if something is off, you'll know it. I don't think I relaxed until my daughter started crawling, and even then, I only relaxed a little. It's just natural for us to protect our precious children. It really only becomes a problem when it prevents us or our babies from enjoying life, going places, and having fun.
Hang in there. It's tough.