Hi Nathan!
Thanks for getting in touch with us. Let me help you with this topic.
By default, the week calendar should scroll down to the first event visible so that you don’t see the blank times.
In your site, this might not be working do to a conflict with your theme or another plugin styles. You can verify this by testing for conflicts to see if any difference > https://theeventscalendar.com/knowledgebase/testing-for-conflicts/
Unfortunately, there is no default option to only display hours in some times, but you could try to accomplish this by customizing the week view, for which I recommend you use one of the methods described in our themer’s guide > https://theeventscalendar.com/knowledgebase/themers-guide/
I hope that helps. Let me know if any follow up questions.
Best,
Victor