How to handle my shady boyfriend without being rude?

Take a break from each other for a few weeks, then see how both of you feel then. If he continues this behavior, part company and find someone more available and less tired and distant.