I'm of the opinion that cadence DOES matter, but then again, maybe I'm just old school. A steady rhythmic cadence is something you want to perfect.
Being in the right gear means your cadence is roughly 85/110, and, consequently, keeping your cadence roughly about 100 means you're in the right gear, but if you're making progress in your personal distance, and, most importantly, making progress in your personal enjoyment, then fine. Go for it.
Building up your distance and your stamina involves slow progress. If you feel frustrated because you feel you're in too low a gear the temptation is to up the gear and maybe burn yourself out.
Let yourself find the right cadence over the summer and marvel at the progress you'll make. You'll soon find that gears you could only use while going downhill come naturally while you're on the level.