Gradle cannot resolve package for java 11 module when in one of two directories

I am trying to compile a java 11 module which contains mixed java and groovy code, and the compilation keeps failing with the following message:

package is empty or does not exist: my.sample.map

I changed the GroovyCompile task to use the java 11 compilation flags:

tasks.withType(GroovyCompile) {
    doFirst {
        options.compilerArgs += [
                '--module-path', classpath.asPath,
                '-verbose',
                '--module-source-path', ["$rootProject.projectDir/*/src/main/{java,groovy}/",
                                         "$rootProject.projectDir/*/src/main/"].join(File.pathSeparator)
        ]
    }
}

I have the following folder structure for the module that I am trying to compile:

my.sample.module
└── src
    ├── main
    │   ├── groovy
    │   │   └── my
    │   │       └── sample
    │   │           └── map
    │   │               ├── MapComponent.groovy
    │   │               └── MapViewPerspective.groovy
    │   ├── java
    │   │   └── my
    │   │       └── sample
    │   │           └── map
    │   │               └── config
    │   │                   └── MapViewConfig.java
    │   ├── module-info.java
    │   └── resources
    │       └── MapView.fxml
    └── test

The contents of module-info.java is as follows:

module my.sample.module {
    exports my.sample.map.config;
    exports my.sample.map;

    requires javafx.fxml;
    requires javafx.controls;

    requires org.codehaus.groovy;

    ...
}

After hours of trying to fix this, the only problem I can see is that it is skipping the groovy folder and only looking for stuff in the java folder.

Then I tried to add an empty class to the package my.sample.map in the java folder, and the compilation proceeds past that point but fails when compiling classes in the groovy folder which need MapViewConfig from the java folder.

What am I doing wrong? I think I have to update the classpaths used for compilation, but I'm not sure which classpaths to add.