Hello Jesse (or is it really Rambo? or some other action figure…?)
I think you’re right that there has to be an end time, unless it’s an all day event. However, if you check out the Themer’s guide to customizing the calendar, you may be able to get rid of the “end time” from showing up on the front end of your site: https://theeventscalendar.com/support/documentation/events-calendar-themers-guide/
Another idea I just thought of is, you could mark the event as an all day event, then add an additional field (you can find this under the WordPress Settings Menu > The Events Calendar and click on the tab that says “Additional Fields”), and call that “Start time” – so that can be where you fill in that info. It’s sort of a short-cut work around compromise, but it might work for you.
I hope this helps!