Howdy Jacob,
That is definitely possible as a small customization. We already have a really good example code that could be slightly modified to fit your exact needs:
https://gist.github.com/elimn/c47fb3e65d437c2479bd
Basically it allows you to hide all events within specific categories from the calendar. Further on line 15 it shows an example of making these events still viewable to specific WP Roles, such as the administrator. This will apply equally to the teh event details and the event itself, so this one snippet would allow you to hide the events in their entirety except for certain user levels.
Does that all make sense? Does that answer your question? If you need help implementing this, would you mind logging in so I can verify your support access? Or, if you don’t have/want a license (which includes a year of support) you should checkout the volunteer community on WordPress.org: The Events Calendar. It’s not the same level of support, but it’s free! We even check those forums once each week and help to the extent we can.
Cheers!
– Brook