Executing shell commands in parallel but limiting jobs (Windows without Cygwin)

Here is what I am trying to do. Suppose I have a program called myprogram.exe, which I have to execute 1000 times.

Under Windows, I could usually do something as simple as:

for /L %n in (1,1,1000) do start /myfolder/myprogram.exe

However, suppose I only have 5 CPU threads I can devote to running the 1000 instances of myprogram.exe, such that I launch only 5, then when one of these finishes another one is launched, etc until the whole 1000 end.

Under Linux and using GNU Parallel, I could simply do:

seq 1000 | parallel -N0 -j5 "nohup myprogram.exe"

How could I achieve something like that in Windows command line? Notice that in my case using Cygwin is not an option, so resorting to xargs and GNU Parallel under Windows are not options either.

2 answers

  • answered 2018-08-14 07:29 Stephan

    This will run five processes in parallel. Each time one of them finishes, the next process will be started (so there are always 5 of them until they all are done)

    @ECHO off
    setlocal enabledelayedexpansion
    set bunch=5
    
    for /l %%a in (1,1,1000) do (
      call :loop 
      echo processing: %%a
      start "MyCommand" cmd /c timeout !random:~-1!
    )
    call :loop
    goto :eof
    
    :loop  REM waits for available slot
    for /f %%x in ('tasklist /fi "windowtitle eq MyCommand"  ^| find /c "cmd.exe"') do set x=%%x
    if %x% geq %bunch% goto :loop
    goto :eof
    

    Add the /min switch to start to minimize the started processes.
    Give them a unique windowtitle (MyCommand here) to be able to count them.
    Replace cmd /c timeout !random:~-1! with your actual command.

    EDIT a slightly modified script, which may work better, if myprogram is a GUI (script above will work better with CLI applications):

    @ECHO off
    setlocal enabledelayedexpansion
    set bunch=5
    
    for /l %%a in (1,1,100) do (
      call :loop 
      echo processing: %%a
      start notepad.exe
    )
    call :loop
    goto :eof
    
    :loop  REM waits for available slot
    for /f %%x in ('tasklist /fi "imagename eq Notepad.exe"^|find /c "."') do set x=%%x
    if %x% geq %bunch% goto :loop
    goto :eof
    

  • answered 2018-08-14 08:20 Gerhard Barnard

    Here is a way, by using powershell to do the process count. and using a simply set /a as counter.

    @echo off
    setlocal enabledelayedexpansion
    set /a cnt=0
    :counter
    if !cnt! lss 1000 (
    for /F "tokens=*" %%i in ('powershell ^(Get-Process -Name 'myprogram'^).count') do set proc=%%i
                 if !proc! lss 5 (
                              start "C:\myfolder\myprogram.exe"
                              set /a cnt+=1
                  )
        goto :counter
       )
    

    You could add echo !cnt! in the line before goto :counter if you want to see it count.

    It can be done without using delayedexpansion but I prefer to use it here.