Hi there @wheretogokiddo,
Thanks for getting in touch with us! Interesting question, I’m mostly sure that won’t create a huge impact on the calendar performance but the only way to really confirm this is to give it an actual try. If you have a staging site you can import a copy of the live database and perform some test deleting past events and measuring load times. Does this makes sense?
You might also take a look at other ways of improving the site performance: The Events Calendar Performance considerations.
Please let me know if there’s anything else I can help you with,
Best,
Nico