Should I take my dog to the vet at this point?

As long as their isn't any bald spots then I wouldn't worry about it yet. Try giving him omega fatty acid vitamin supplements or washing him in shampoo especially for shedding. Omega 3 fatty acids keep coats and skin healthy which significantly reduces shedding. I'd wait it out a little longer. It doesn't sound like anything life threatening. If it comes time that you have to take him to the vet call around and ask for prices first. There is a vets by my house that only charges 27 dollars for a consultation. Good Luck.