Hi Daniel,
Sorry you’re facing such slow load times.
Properly configured, caching can make an incredible difference (and not just page caching but object, bytecode and query caching … though depending on how much control you have over your hosting environment not all of these may be possible). Beyond that – though we’re always trying to improve performance – it often comes down to a matter of numbers and having sufficient resources.
If you have a very busy site or a site with lots of events / a very high proportion of recurring events, sometimes some extra “grunt” is required – which might entail scaling up or even moving to a quite new configuration.
It’s hard to offer definitive solutions for questions like this I’m afraid, simply because there are so many variables to consider.
Are there are any other specific questions you have on this one?