the idea that restrooms, locker rooms, etc need to be single-sex spaces in order for women to be safe is patriarchy's way of signalling to men & boys that society doesn't expect them to behave themselves around women. it is directly antifeminist. it would be antifeminist even if trans people did not exist. a feminist society would demand that women should be safe in all spaces even when there are men there.