Executing shell commands in parallel but limiting jobs (Windows without Cygwin)
Here is what I am trying to do. Suppose I have a program called
myprogram.exe, which I have to execute 1000 times.
Under Windows, I could usually do something as simple as:
for /L %n in (1,1,1000) do start /myfolder/myprogram.exe
However, suppose I only have 5 CPU threads I can devote to running the 1000 instances of
myprogram.exe, such that I launch only 5, then when one of these finishes another one is launched, etc until the whole 1000 end.
Under Linux and using GNU Parallel, I could simply do:
seq 1000 | parallel -N0 -j5 "nohup myprogram.exe"
How could I achieve something like that in Windows command line? Notice that in my case using Cygwin is not an option, so resorting to xargs and GNU Parallel under Windows are not options either.
This will run five processes in parallel. Each time one of them finishes, the next process will be started (so there are always 5 of them until they all are done)
@ECHO off setlocal enabledelayedexpansion set bunch=5 for /l %%a in (1,1,1000) do ( call :loop echo processing: %%a start "MyCommand" cmd /c timeout !random:~-1! ) call :loop goto :eof :loop REM waits for available slot for /f %%x in ('tasklist /fi "windowtitle eq MyCommand" ^| find /c "cmd.exe"') do set x=%%x if %x% geq %bunch% goto :loop goto :eof
startto minimize the started processes.
Give them a unique windowtitle (
MyCommandhere) to be able to count them.
cmd /c timeout !random:~-1!with your actual command.
EDIT a slightly modified script, which may work better, if
myprogramis a GUI (script above will work better with CLI applications):
@ECHO off setlocal enabledelayedexpansion set bunch=5 for /l %%a in (1,1,100) do ( call :loop echo processing: %%a start notepad.exe ) call :loop goto :eof :loop REM waits for available slot for /f %%x in ('tasklist /fi "imagename eq Notepad.exe"^|find /c "."') do set x=%%x if %x% geq %bunch% goto :loop goto :eof
Here is a way, by using powershell to do the process count. and using a simply
set /aas counter.
@echo off setlocal enabledelayedexpansion set /a cnt=0 :counter if !cnt! lss 1000 ( for /F "tokens=*" %%i in ('powershell ^(Get-Process -Name 'myprogram'^).count') do set proc=%%i if !proc! lss 5 ( start "C:\myfolder\myprogram.exe" set /a cnt+=1 ) goto :counter )
You could add
echo !cnt!in the line before
goto :counterif you want to see it count.
It can be done without using
delayedexpansionbut I prefer to use it here.