The taxe foncière is a tax on the owner of the property, and yes, you have to pay that even if you aren't there very often. (Frankly, even if you are never there.)

The taxe d'habitation is the tax on whoever is living in the property. It applies to furnished residential property. There are some reductions possible in the taxe d'habitation for people of modest means and other special conditions, but these are normally only available if the property is your primary residence.

If you were renting the property out to someone, they would be responsible for paying the taxe d'habitation. But in your case, because you have the right to live there any time you like, you're considered to be living there - even if you only visit every now and then.
Cheers,
Bev