Howdy there, Lars!
Unfortunately, there is no simple way to do this right out of the box. The intention of the meta box was to display all Categories associated with an event so that’s how it functions.
One possibility you might consider is using a filter to find instances where the text for certain category names occurs and then replace the text with blank instead. Here’s an example snippet that might help get you started in that direction.
Sorry I don’t have more of a concrete solution for you here, but I hope this helps.
Cheers!
Geoff