How can I make a setup command only run the first time a Jupyter notebook is ran?

Im doing a machine learning project in google colab. Each time an instance is started, I want to run these commands:

  ! mkdir ~/.kaggle # make directory ".kaggle"
  ! cp kaggle.json ~/.kaggle/ # copy the json file into the directory
  ! chmod 600 ~/.kaggle/kaggle.json # allocate required permission for the file
  ! kaggle datasets download -d alessiocorrado99/animals10 # download animal set
  ! unzip animals10.zip

These commands download and extract a dataset I need. However, it only needs to be ran the first run through only. When clicking "Run All" after the initial download of the dataset, it requires user input to decide whether to replace the files or not. I also don't want to keep downloading from kaggle and use resources unnecesarily.

My current approach is to run the script once then comment out the initialization script, but this takes time and effort.

How can I automate this process so a certain cell only runs on the first run of the runtime?

2 answers

  • answered 2022-05-06 23:43 Anton Frolov

    1. After initial execution of your commands create a dummy variable.

    2. If you happen to re-execute that code cell an IF will check if that variable is in memory:

      import os
      
      INIT_SHELL_COMMAND_LIST = [
        'mkdir ~/.kaggle # make directory ".kaggle"',
        'next_cmd',
        ]
      if not 'INIT_HAPPENED' in locals().keys():
          for command in INIT_SHELL_COMMAND_LIST:
              os.sys(command)
          INIT_HAPPENED = True
      

  • answered 2022-05-06 23:45 Jacob Waters

    This answer is specific to my problem but I hope this helps others.

    Essentially the code that I ran had some measureable effect, in my case it creates a directory called 'raw-img'. In order to detect if the code ran, I used a conditional testing for the existance of the file path.

    import os
    
    if not os.path.isdir('raw-img'):
      print("firstRun")
      ! pip install kaggle # install kaggle library
      ! mkdir ~/.kaggle # make directory ".kaggle"
      ! cp kaggle.json ~/.kaggle/ # copy the json file into the directory
      ! chmod 600 ~/.kaggle/kaggle.json # allocate required permission for the file
      ! kaggle datasets download -d alessiocorrado99/animals10 # download animal set
      ! unzip animals10.zip
    

    However, in any case, whether importing libraries, or really anything else, there should be some code you can write to test if that operation has occured. So simple use that test to determine whether the code should run again.

    For example, an answer here shows how to test if a library is installed. https://stackoverflow.com/a/55194886/9053474 If the string is empty, then you know your library has not been installed and so you should run the script.

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum