Service Worker takes long to download content

I have a service worker and am caching both css and js files.

While doing performance testing I noticed that the css and js files take a substantially longer time in the Content downloading stage with service worker enabled as compared to my build with no service worker and content being served from disk cache.

Could anyone guide me on why this is happening?

To provide more info about the times

SW enabled style.css - 15ms main.js - 120ms
No SW styles.css - 8ms main.js - 75ms