Limit AzureAD app access to certain users / groups

I have an Azure app that uses the Graph API to read calendar & mail data on behalf of users in the domain. The app is usually installed by an admin, and gains access to all available users.

Can an admin restrict access to certain users / groups, so that the app will only be able to access them? I looked into user assignments but it didn't seem to do anything.

Is there another way?