It's easier for someone with a TalkTalk affliction to get a new ISP than it is for someone with Tritanomaly to get new retinal cones.
If improving a user's experience is foremost on your priority list, usability and accessability should be the most important things you are concerned about. Creating a "plug-in" style system to allow seamless blending of accessable colour schemes, I feel is worth an extra stylesheet, even if it reduces the load time by 100ms.
