Hey Wouter,
Thanks for reaching out to us!
I’m limited on the support I can give for customizations such as this but hopefully I can help to get you pointed in a good direction.
This is definitely an interesting problem. Would it be acceptable in this scenario for the view all events link to be a toggle that shows the additional events? If so, you could accomplish this by modifying the archive template for the event categories to have two queries. The first for the first 3 events and the second, with an offset, for the remaining events. This way you could create the functionality for all the event categories by modifying just a single template file.
Let me know if this helps.
Thanks!