The damage on hair is cumulative: once hair is damaged, it won't repair itself over time.
This being said, if you blowdry only seldom, though it may damage hair, the damage cumulates more slowly.
There are also things you can do to protect the hair when you blowdry. The heat is what makes blowdrying damaging, so if you avoid too much heat, you can eliminate damage. Use a cool setting if possible (never use hottest setting), use a diffuser if possible, move the dryer around when you use it so that it doesn't heat same spot too much, don't bring the dryer too close to your hair.
HTH
