Can you go to Budapest baths in winter?

Yes, the thermal baths are open all year round.