Why does my cat hate the bathrooms and toilets in my house?

My cat is a 2 year old male. We keep him and his brother downstairs, in the conservatory, kitchen and dining room area. They do not have access to any part of the house. I sometimes take him upstairs to my room so he can explore and get used to other parts. However when I take him past the toilet or bathroom when the door is open, he goes absolutely crazy. His body stiffens up and trembles and he claws himself to me and tries to hide in my hair. Why is this? It never happens in any other room.