TFS 2018 Server Groups Active Directory Setup

I am trying to setup a layer of security for our on prem TFS 2018 server.

We have created AD groups for each project collection that acts as an “authorized list of users” for the collection. Unfortunately, as a project collection admin, I can add a user that is not part of that valid user group, but is a valid network account. This is to not be confused with a TFS ValidUser account. For Example, account first.last1 is in a TFS ValidUser group, account first.last2 is not, but is a valid network (non-TFS) account, and I am still able to add that user to a project.

What we are trying to achieve:

  1. TFS Server ValidUsers (server level) -- we would like for this to just be the custom created AD groups per project – Ex: Project1-ValidUsers, Project2-ValidUsers, etc.

  2. Project Collection ValidUsers – would just be the specific project user group – Ex. Project1-ValidUsers – Not allow other Project Collection’s ValidUsers to be listed in the Add Member option.

  3. Project Level - All People/Group fields would only pull users from that Project Collections ValidUsers, not the entire TFS Systems.

I assume we have missed something on #1 above and it’s pulling in all network users as a Built In group or something similar but I am not familiar with that granular level of permissions. The current setup is default TFS groups and permissions.