Using forfiles, how do you search for date created rather than date modified?

See also questions close to this topic

• How to submit all R scripts in a directory using with a single .sh file with qsub?

  I have a directory diffexp with 50 R scripts, which are called diffexpRcommands_neurons_nn.R (where nn is a number from 00 TO 50.

  I would like to run as many of them as possible in parallel on a cluster.

  Is there a way in which I can submit all the scripts using qsub, but without having to create an individuaal .sh script for each .R script?

  Essentially, I am running differential expression tests between pairs of cell clusters (using the Seurat function FindMarkers().

  Since there are 52 cell clusters, there are 1,326 comparisons to make. I've generated a command for each of the combinations, and then split those commands into 50 different files, with ~25 comparisons in each one. The idea is to try to run every set of 25 commands at once, or at least as many as possible.

  Each diffexpRcommands_neurons_nn.Rscript contains the same header:

  source("/path/to/code/setupCSE.R")
  library(gmodels)
  library(Matrix)
  library(RANN)
  library(igraph)
  
  load(file="/path/to/seuratobjects/alltogether_neurons.RData")
  

  Followed by 26 comparisons between pairs of cell clusters:

  # 1 vs 2
  cluster1vs2.markers <- FindMarkers(alltogether_neurons, "1", "2")
  write.table (cluster1vs2.markers,
   "/path/to/alltogether_neurons_1vs2_markers.txt",
    sep="\t", quote=F, col.names=NA)
  
  # 1 vs 3
      cluster1vs3.markers <- FindMarkers(alltogether_neurons, "1", "3")
  write.table (cluster1vs3.markers,
   "/path/to/alltogether_neurons_1vs3_markers.txt",
    sep="\t", quote=F, col.names=NA)
  
  .
  .
  .
  

  and so on, until the end of the last file diffexpRcommands_neurons_50.R:

  # 51 vs 53
      cluster51vs52.markers <- FindMarkers(alltogether_neurons, "51", "52")
      write.table (cluster1vs3.markers,
        "/path/to/alltogether_neurons_51vs52_markers.txt",
        sep="\t", quote=F, col.names=NA)
  

  I started this thinking I could somehow run a job array, but I don't know if this is possible since each job is a different set of commands (different set of combinations).

  Thank you.
• SLURM releasing resources using scontrol update results in unknown endtime

  I have a program that will dynamically release resources during job execution, using the command:

  scontrol update JobId=$SLURM_JOB_ID NodeList=${remaininghosts}

  However, this results in some very weird behavior sometimes. Where the job is re-queued. Below is the output of sacct

  sacct -j 1448590

  JobID NNodes State Start End NodeList

  ---

  1448590 4 RESIZING 20:47:28 01:04:22 [0812,0827],[0663-0664]

  1448590.0 4 COMPLETED 20:47:30 20:47:30 [0812,0827],[0663-0664]

  1448590.1 4 RESIZING 20:47:30 01:04:22 [0812,0827],[0663-0664]

  1448590 3 RESIZING 01:04:22 01:06:42 [0812,0827],0663

  1448590 2 RESIZING 01:06:42 1:12:42 0827,tnxt-0663

  1448590 4 COMPLETED 05:33:15 Unknown 0805-0807,0809]

  The first lines show everything works fine, nodes are getting released but in the last line, it shows a completely different set of nodes with an unknown end time. The slurm logs show the job got requeued:

  requeue JobID=1448590 State=0x8000 NodeCnt=1 due to node failure.

  I suspect this might happen because the head node is killed, but the slurm documentation doesn't say anything about that.

  Does anybody had an idea or suggestion?

  Thanks
• How to schedule AWS Batch parallel?

  I am trying to understand AWS Batch concept and i want to know how does aws batch schedules the jobs.

  I want to perform : 1. reading 1000 csv files from s3 2. write them into dynamodb

  I want to read 500 files from one ec2 and another 500 files from another ec2, by using single job.

  Here is my all codes:

  Dockerfile:

  FROM python:2.7
  RUN python --version
  RUN pip install boto3
  RUN mkdir /src
  COPY . /src 
  CMD [“python”, “/src/readfile.py”]
  

  readfile.py

  import boto3 
  
  client = boto3.client('s3') #low-level functional API
  resource = boto3.resource('s3') #high-level object-oriented API my_bucket = resource.Bucket('my-bucket') #subsitute this for your s3 bucket name.
  dynamodb = boto3.resource("dynamodb",region_name="us-east-2")
  
  def read_files(keyid):
      obj = client.get_object(Bucket='csv-lambda-files', Key=keyid)
      rows=obj['Body'].read().split('\n')
      rows=rows[:-1]
      table = dynamodb.Table('new_data')
  
      for row in rows:
        table.put_item(
          Item={
              'first_number':row.split(',')[0],
              'second_number':row.split(',')[1]
          }
      )
  
  def get_s3_keys(bucket='csv-lambda-files',suffix='.csv'):
      """Get a list of keys in an S3 bucket."""
      keys = []
      resp = client.list_objects(Bucket=bucket)
      for obj in resp['Contents']:
          key = obj['Key']
          if key.endswith(suffix):
              read_files(obj['Key'])
      return keys
  get_s3_keys()