Google thinks the web should be fast, and that’s kinda the final word. Here’s some (older but still useful) info on site speed as a ranking factor.
I recently found a useful website for testing your website’s speed – it checks the website by accessing it from multiple locations (in multiple countries) and thus gives a better test than just checking the speed in your own browser.