Google colab, RAM memory full when converting CSV pixel data to images

See also questions close to this topic

• Python File Tagging System does not retrieve nested dictionaries in dictionary

  I am building a file tagging system using Python. The idea is simple. Given a directory of files (and files within subdirectories), I want to filter them out using a filter input and tag those files with a word or a phrase.

  If I got the following contents in my current directory:

  data/
     budget.xls
     world_building_budget.txt
  a.txt
  b.exe
  hello_world.dat
  world_builder.spec
  

  and I execute the following command in the shell:

  py -3 tag_tool.py -filter=world -tag="World-Building Tool"
  

  My output will be:

  These files were tagged with "World-Building Tool":
  data/ world_building_budget.txt
  hello_world.dat
  world_builder.spec
  

  My current output isn't exactly like this but basically, I am converting all files and files within subdirectories into a single dictionary like this:

  def fs_tree_to_dict(path_):
      file_token = ''
      for root, dirs, files in os.walk(path_):
          tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
          tree.update({f: file_token for f in files})
          return tree
  

  Right now, my dictionary looks like this: key:''.

  In the following function, I am turning the empty values '' into empty lists (to hold my tags):

  def empty_str_to_list(d):
      for k,v in d.items():
          if v == '':
              d[k] = []
          elif isinstance(v, dict):
              empty_str_to_list(v)
  

  When I run my entire code, this is my output:

  hello_world.dat ['World-Building Tool']
  world_builder.spec ['World-Building Tool']
  

  But it does not see data/world_building_budget.txt. This is the full dictionary:

  {'data': {'world_building_budget.txt': []},  
  'a.txt': [],
  'hello_world.dat': [], 
  'b.exe': [], 
  'world_builder.spec': []}
  
  

  This is my full code:

  import os, argparse
  
  
  def fs_tree_to_dict(path_):
      file_token = ''
      for root, dirs, files in os.walk(path_):
          tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
          tree.update({f: file_token for f in files})
          return tree
  
  
  def empty_str_to_list(d):
      for k, v in d.items():
          if v == '':
              d[k] = []
          elif isinstance(v, dict):
              empty_str_to_list(v)
  
  
  parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter)
  
  parser.add_argument("--filter", action="store", help="keyword to filter files")
  parser.add_argument("--tag", action="store", help="a tag phrase to attach to a file")
  parser.add_argument("--get_tagged", action="store", help="retrieve files matching an existing tag")
  
  args = parser.parse_args()
  
  filter = args.filter
  tag = args.tag
  get_tagged = args.get_tagged
  current_dir = os.getcwd()
  
  files_dict = fs_tree_to_dict(current_dir)
  empty_str_to_list(files_dict)
  
  for k, v in files_dict.items():
      if filter in k:
          if v == []:
              v.append(tag)
              print(k, v)
  
          elif isinstance(v, dict):
              empty_str_to_list(v)
  
      if get_tagged in v:
          print(k, v)
  
• Actaully i am working on a project and in it, it is showing no module name pip_internal plz help me for the same. I am using pycharm(conda interpreter

   File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\Scripts\pip.exe\__main__.py", line 4, in <module>
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\__init__.py", line 4, in <module>
      from pip_internal.utils import _log
  

  I am using pycharm with conda interpreter.

  
• Looping the function if the input is not string

  I'm new to python (first of all) I have a homework to do a function about checking if an item exists in a dictionary or not.

  inventory = {"apple" : 50, "orange" : 50, "pineapple" : 70, "strawberry" : 30}
  
  def check_item():
      x = input("Enter the fruit's name: ")
      if not x.isalpha():
          print("Error! You need to type the name of the fruit")
      elif x in inventory:
          print("Fruit found:", x)
          print("Inventory available:", inventory[x],"KG")
      else:
          print("Fruit not found")
  check_item()
  

  I want the function to loop again only if the input written is not string. I've tried to type return Under print("Error! You need to type the name of the fruit") but didn't work. Help
• how to print all parameters of a keras model

  I am trying to print all the 1290 parameters in dense_1 layer, but model.get_weights()[7] only show 10 parameters. How could I print all the 1290 parameters of dense_1 layer? What is the difference between model.get_weights() and model.layer.get_weights()

  >model.get_weights()[7]
  array([-2.8552295e-04, -4.3254648e-03, -1.8752701e-04,  2.3482188e-03,
         -3.4848123e-04,  7.6121779e-04, -2.7494309e-06, -1.9068648e-03,
          6.0777756e-04,  1.9550985e-03], dtype=float32)
  
  >model.summary()
  Model: "sequential"
  _________________________________________________________________
  Layer (type)                Output Shape              Param #   
  =================================================================
   conv2d (Conv2D)             (None, 26, 26, 32)        320       
                                                                   
   conv2d_1 (Conv2D)           (None, 24, 24, 64)        18496     
                                                                   
   max_pooling2d (MaxPooling2D  (None, 12, 12, 64)       0         
   )                                                               
                                                                   
   dropout (Dropout)           (None, 12, 12, 64)        0         
                                                                   
   flatten (Flatten)           (None, 9216)              0         
                                                                   
   dense (Dense)               (None, 128)               1179776   
                                                                   
   dropout_1 (Dropout)         (None, 128)               0         
                                                                   
   dense_1 (Dense)             (None, 10)                1290      
  _________________________________________________________________
                                                                   
  =================================================================
  Total params: 1,199,882
  Trainable params: 1,199,882
  Non-trainable params: 0
  _________________________________________________________________
  
• Can't use Keras MeanIoU to train semantic segmentation model

  I'm working on a binary semantic segmentation problem. I built an UNet model with MobileNetV2 backbone. Here is my model code:

  def upsample(filters, size, apply_dropout=False):
      initializer = tf.random_normal_initializer(0., 0.02)
      layer = Sequential()
      layer.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same', kernel_initializer=initializer,
                                       use_bias=False))
      layer.add(layers.BatchNormalization())
      if apply_dropout:
          layer.add(layers.Dropout(0.5))
      layer.add(layers.ReLU())
      return layer
  
  
  def UNet(image_size, num_classes):
      inputs = Input(shape=image_size + (3,))
  
      base_model = applications.MobileNetV2(input_shape=image_size + (3,), include_top=False)
      layer_names = [
          'block_1_expand_relu',
          'block_3_expand_relu',
          'block_6_expand_relu',
          'block_13_expand_relu',
          'block_16_project',
      ]
      base_model_outputs = [base_model.get_layer(name).output for name in layer_names]
      down_stack = Model(inputs=base_model.input, outputs=base_model_outputs)
      down_stack.trainable = False
  
      up_stack = [
          upsample(512, 3),
          upsample(256, 3),
          upsample(128, 3),
          upsample(64, 3)
      ]
  
      skips = down_stack(inputs)
      x = skips[-1]
      skips = reversed(skips[:-1])
  
      for up, skip in zip(up_stack, skips):
          x = up(x)
          x = layers.Concatenate()([x, skip])
  
      outputs = layers.Conv2DTranspose(filters=num_classes, kernel_size=3, strides=2, padding='same')(x)
  
      return Model(inputs, outputs)
  

  To load the images and masks for training, I built an image loader inherits from keras.Sequnce.

  class ImageLoader(utils.Sequence):
  
      def __init__(self, batch_size, img_size, img_paths, mask_paths):
          self.batch_size = batch_size
          self.img_size = img_size
          self.img_paths = img_paths
          self.mask_paths = mask_paths
  
      def __len__(self):
          return len(self.mask_paths) // self.batch_size
  
      def __getitem__(self, idx):
          i = idx * self.batch_size
          batch_img_paths = self.img_paths[i:i + self.batch_size]
          batch_mask_paths = self.mask_paths[i:i + self.batch_size]
  
          x = np.zeros((self.batch_size,) + self.img_size + (3,), dtype='float32')
          for j, path in enumerate(batch_img_paths):
              img = utils.load_img(path, target_size=self.img_size)
              img = utils.img_to_array(img)
              x[j] = img
  
          y = np.zeros((self.batch_size,) + self.img_size + (1,), dtype='uint8')
          for j, path in enumerate(batch_mask_paths):
              img = utils.load_img(path, target_size=self.img_size, color_mode='grayscale')
              img = utils.img_to_array(img)
              # [0, 255] -> [0, 1]
              img //= 255
              y[j] = img
  
          return x, y
  

  In my segmentation problem, all the labels are in the range [0, 1]. However, when I try to compile and then fit the model using Adam optimizer, Sparse categorical cross entropy loss and metric tf.keras.metrics.MeanIoU, I encountered with the following problem:

  Node: 'confusion_matrix/assert_non_negative_1/assert_less_equal/Assert/AssertGuard/Assert'
  2 root error(s) found.
    (0) INVALID_ARGUMENT:  assertion failed: [`predictions` contains negative values.  ] [Condition x >= 0 did not hold element-wise:] [x (confusion_matrix/Cast:0) = ] [-1 -1 -1...]
       [[{{node confusion_matrix/assert_non_negative_1/assert_less_equal/Assert/AssertGuard/Assert}}]]
       [[confusion_matrix/assert_less_1/Assert/AssertGuard/pivot_f/_31/_67]]
    (1) INVALID_ARGUMENT:  assertion failed: [`predictions` contains negative values.  ] [Condition x >= 0 did not hold element-wise:] [x (confusion_matrix/Cast:0) = ] [-1 -1 -1...]
       [[{{node confusion_matrix/assert_non_negative_1/assert_less_equal/Assert/AssertGuard/Assert}}]]
  

  At first, I used accuracy as a metrics for training and I didn't encounter this problem, however when I changed to MeanIoU, this problem happened. Does anyone know how to fix this problem? Thank you very much!

  UPDATE: I've searched on StackOverflow and found this question about a similar error, however the fix mentioned in that link (reduce learning rate) doesn't work in my case.
• Training plot is not appearing properly for keras model

  I have data where I need to train it with X and Y. Traning part is done but when I want to plot the prediction and actual data, it is appearing with so many lines instead of showing just non-linear regression line.

  model= Sequential()
  model.add(Dense(7,input_dim=1, activation="tanh"))
  model.add(Dense(1))
  
  model.compile(loss="mse", 
  optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), metrics= 
  ["mae"])
  history=model.fit(X,Y,epochs=1000)
  
  predict=model.predict(X)
  plt.scatter(X, Y,edgecolors='g')
  plt.plot(X, predict,'r')
  plt.legend([ 'Predictated Y' ,'Actual Y'])
  plt.show()
  

  Please see the attached imageplotting image
• How can I make a setup command only run the first time a Jupyter notebook is ran?

  Im doing a machine learning project in google colab. Each time an instance is started, I want to run these commands:

    ! mkdir ~/.kaggle # make directory ".kaggle"
    ! cp kaggle.json ~/.kaggle/ # copy the json file into the directory
    ! chmod 600 ~/.kaggle/kaggle.json # allocate required permission for the file
    ! kaggle datasets download -d alessiocorrado99/animals10 # download animal set
    ! unzip animals10.zip
  

  These commands download and extract a dataset I need. However, it only needs to be ran the first run through only. When clicking "Run All" after the initial download of the dataset, it requires user input to decide whether to replace the files or not. I also don't want to keep downloading from kaggle and use resources unnecesarily.

  My current approach is to run the script once then comment out the initialization script, but this takes time and effort.

  How can I automate this process so a certain cell only runs on the first run of the runtime?
• Where is the official manifest / definition of google colab's environment?

  Is there a website or git repository where there is a clear listing of all the python packages and other dependencies installed on Google CoLaboratory? Is there a docker image that corresponds to a Google CoLab environment?
• Jupyter progress bar empty

  I'm running the following line of code from a Huggingface tokenizer on a Jupyter notebook:

  my_data.map(tokenize_function, batched=True, num_proc=4, remove_columns=['text'])
  

  I know it has been executed because there are 6 download bars, but there are all empty and show no progress. The numbers are not moving either. I'm wondering if there is an issue, or if I should add tqdm although it does not seem like the solution.