If you stay away from hotels and shopping malls you should be fine :) But it is good advice to first go there on holidays! I'm moving to Hat Yai 15 of june to stay there for 3 months with my gf. When are you planning to go?