I don't see why. It depends on your purposes.
A slow server would be perfectly acceptable for personal testing purposes, or for sharing files/content with friends but not in a wide public website.
Or it would be fine if you don't need a fast website-- fast responses, but slow loading, if it is, for example, only text (no media).
But, sure, for an average website, a fast server is useful.