Python: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

See also questions close to this topic

• Python File Tagging System does not retrieve nested dictionaries in dictionary

  I am building a file tagging system using Python. The idea is simple. Given a directory of files (and files within subdirectories), I want to filter them out using a filter input and tag those files with a word or a phrase.

  If I got the following contents in my current directory:

  data/
     budget.xls
     world_building_budget.txt
  a.txt
  b.exe
  hello_world.dat
  world_builder.spec
  

  and I execute the following command in the shell:

  py -3 tag_tool.py -filter=world -tag="World-Building Tool"
  

  My output will be:

  These files were tagged with "World-Building Tool":
  data/ world_building_budget.txt
  hello_world.dat
  world_builder.spec
  

  My current output isn't exactly like this but basically, I am converting all files and files within subdirectories into a single dictionary like this:

  def fs_tree_to_dict(path_):
      file_token = ''
      for root, dirs, files in os.walk(path_):
          tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
          tree.update({f: file_token for f in files})
          return tree
  

  Right now, my dictionary looks like this: key:''.

  In the following function, I am turning the empty values '' into empty lists (to hold my tags):

  def empty_str_to_list(d):
      for k,v in d.items():
          if v == '':
              d[k] = []
          elif isinstance(v, dict):
              empty_str_to_list(v)
  

  When I run my entire code, this is my output:

  hello_world.dat ['World-Building Tool']
  world_builder.spec ['World-Building Tool']
  

  But it does not see data/world_building_budget.txt. This is the full dictionary:

  {'data': {'world_building_budget.txt': []},  
  'a.txt': [],
  'hello_world.dat': [], 
  'b.exe': [], 
  'world_builder.spec': []}
  
  

  This is my full code:

  import os, argparse
  
  
  def fs_tree_to_dict(path_):
      file_token = ''
      for root, dirs, files in os.walk(path_):
          tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
          tree.update({f: file_token for f in files})
          return tree
  
  
  def empty_str_to_list(d):
      for k, v in d.items():
          if v == '':
              d[k] = []
          elif isinstance(v, dict):
              empty_str_to_list(v)
  
  
  parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter)
  
  parser.add_argument("--filter", action="store", help="keyword to filter files")
  parser.add_argument("--tag", action="store", help="a tag phrase to attach to a file")
  parser.add_argument("--get_tagged", action="store", help="retrieve files matching an existing tag")
  
  args = parser.parse_args()
  
  filter = args.filter
  tag = args.tag
  get_tagged = args.get_tagged
  current_dir = os.getcwd()
  
  files_dict = fs_tree_to_dict(current_dir)
  empty_str_to_list(files_dict)
  
  for k, v in files_dict.items():
      if filter in k:
          if v == []:
              v.append(tag)
              print(k, v)
  
          elif isinstance(v, dict):
              empty_str_to_list(v)
  
      if get_tagged in v:
          print(k, v)
  
• Actaully i am working on a project and in it, it is showing no module name pip_internal plz help me for the same. I am using pycharm(conda interpreter

   File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\Scripts\pip.exe\__main__.py", line 4, in <module>
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\__init__.py", line 4, in <module>
      from pip_internal.utils import _log
  

  I am using pycharm with conda interpreter.

  
• Looping the function if the input is not string

  I'm new to python (first of all) I have a homework to do a function about checking if an item exists in a dictionary or not.

  inventory = {"apple" : 50, "orange" : 50, "pineapple" : 70, "strawberry" : 30}
  
  def check_item():
      x = input("Enter the fruit's name: ")
      if not x.isalpha():
          print("Error! You need to type the name of the fruit")
      elif x in inventory:
          print("Fruit found:", x)
          print("Inventory available:", inventory[x],"KG")
      else:
          print("Fruit not found")
  check_item()
  

  I want the function to loop again only if the input written is not string. I've tried to type return Under print("Error! You need to type the name of the fruit") but didn't work. Help
• Only download certain label tf dataset

  Looking to do some fine tuning. The dataset (found here: https://knowyourdata-tfds.withgoogle.com/#dataset=sun397&filters=kyd%2Fsun397%2Flabel:%2Fh%2Fhouse&tab=ITEM&select=kyd%2Fsun397%2Flabel&item=%2Fh%2Fhouse%2Fsun_blpzjomvikwtulrq.jpg&expanded_groups=sun397) that Im trying to finetune w is pretty large and i just want to use/download the images with label /h/house. Any tips on how I can best accomplish this? Thanks!

  import tensorflow as tf
  import tensorflow_hub as hub
  import tensorflow_datasets as tfds
  import numpy as np
  import matplotlib.pyplot as plt
  import functools
  import pandas
  
  (train_ds, valid_ds), info = tfds.load("sun397", split=["train", "validation"], as_supervised=True, with_info=True, label = "/h/house")
  int_to_class_label = info.features['label'].int2str
  
  
• TFF: How can I train any model using a server running tff-runtime and a client running tff-client?

  I read all the tensorflow-federated tutorials, including this one https://www.tensorflow.org/federated/gcp_setup, but I couldn't understand how to use this for training a model.

  I'm doing a graduation project, to start I need to do this POC using tensorflow-federated to train a model with one server and one client in order to apply cross-silo setup for recognition of organs affected by covid in the future. If anyone can point me a direction, I'd be very grateful.
• Can't use Keras MeanIoU to train semantic segmentation model

  I'm working on a binary semantic segmentation problem. I built an UNet model with MobileNetV2 backbone. Here is my model code:

  def upsample(filters, size, apply_dropout=False):
      initializer = tf.random_normal_initializer(0., 0.02)
      layer = Sequential()
      layer.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same', kernel_initializer=initializer,
                                       use_bias=False))
      layer.add(layers.BatchNormalization())
      if apply_dropout:
          layer.add(layers.Dropout(0.5))
      layer.add(layers.ReLU())
      return layer
  
  
  def UNet(image_size, num_classes):
      inputs = Input(shape=image_size + (3,))
  
      base_model = applications.MobileNetV2(input_shape=image_size + (3,), include_top=False)
      layer_names = [
          'block_1_expand_relu',
          'block_3_expand_relu',
          'block_6_expand_relu',
          'block_13_expand_relu',
          'block_16_project',
      ]
      base_model_outputs = [base_model.get_layer(name).output for name in layer_names]
      down_stack = Model(inputs=base_model.input, outputs=base_model_outputs)
      down_stack.trainable = False
  
      up_stack = [
          upsample(512, 3),
          upsample(256, 3),
          upsample(128, 3),
          upsample(64, 3)
      ]
  
      skips = down_stack(inputs)
      x = skips[-1]
      skips = reversed(skips[:-1])
  
      for up, skip in zip(up_stack, skips):
          x = up(x)
          x = layers.Concatenate()([x, skip])
  
      outputs = layers.Conv2DTranspose(filters=num_classes, kernel_size=3, strides=2, padding='same')(x)
  
      return Model(inputs, outputs)
  

  To load the images and masks for training, I built an image loader inherits from keras.Sequnce.

  class ImageLoader(utils.Sequence):
  
      def __init__(self, batch_size, img_size, img_paths, mask_paths):
          self.batch_size = batch_size
          self.img_size = img_size
          self.img_paths = img_paths
          self.mask_paths = mask_paths
  
      def __len__(self):
          return len(self.mask_paths) // self.batch_size
  
      def __getitem__(self, idx):
          i = idx * self.batch_size
          batch_img_paths = self.img_paths[i:i + self.batch_size]
          batch_mask_paths = self.mask_paths[i:i + self.batch_size]
  
          x = np.zeros((self.batch_size,) + self.img_size + (3,), dtype='float32')
          for j, path in enumerate(batch_img_paths):
              img = utils.load_img(path, target_size=self.img_size)
              img = utils.img_to_array(img)
              x[j] = img
  
          y = np.zeros((self.batch_size,) + self.img_size + (1,), dtype='uint8')
          for j, path in enumerate(batch_mask_paths):
              img = utils.load_img(path, target_size=self.img_size, color_mode='grayscale')
              img = utils.img_to_array(img)
              # [0, 255] -> [0, 1]
              img //= 255
              y[j] = img
  
          return x, y
  

  In my segmentation problem, all the labels are in the range [0, 1]. However, when I try to compile and then fit the model using Adam optimizer, Sparse categorical cross entropy loss and metric tf.keras.metrics.MeanIoU, I encountered with the following problem:

  Node: 'confusion_matrix/assert_non_negative_1/assert_less_equal/Assert/AssertGuard/Assert'
  2 root error(s) found.
    (0) INVALID_ARGUMENT:  assertion failed: [`predictions` contains negative values.  ] [Condition x >= 0 did not hold element-wise:] [x (confusion_matrix/Cast:0) = ] [-1 -1 -1...]
       [[{{node confusion_matrix/assert_non_negative_1/assert_less_equal/Assert/AssertGuard/Assert}}]]
       [[confusion_matrix/assert_less_1/Assert/AssertGuard/pivot_f/_31/_67]]
    (1) INVALID_ARGUMENT:  assertion failed: [`predictions` contains negative values.  ] [Condition x >= 0 did not hold element-wise:] [x (confusion_matrix/Cast:0) = ] [-1 -1 -1...]
       [[{{node confusion_matrix/assert_non_negative_1/assert_less_equal/Assert/AssertGuard/Assert}}]]
  

  At first, I used accuracy as a metrics for training and I didn't encounter this problem, however when I changed to MeanIoU, this problem happened. Does anyone know how to fix this problem? Thank you very much!

  UPDATE: I've searched on StackOverflow and found this question about a similar error, however the fix mentioned in that link (reduce learning rate) doesn't work in my case.
• Pytorch model object has no attribute 'predict' BERT

  I had train a BertClassifier model using pytorch. After creating my best.pt I would like to make in production my model and using it to predict and classifier starting from a sample, so I resume them from the checkpoint. Otherwise after put it in evaluation and freeze model, I use .predict to make in work on my sample but I'm encountering this Attribute Error. I had also inizialize it before calling the checkpoint. When I am wrong? Thank you for your help!

  def save_ckp(state, is_best, checkpoint_path, best_model_path):
      """
      function created to save checkpoint, the latest one and the best one. 
      This creates flexibility: either you are interested in the state of the latest checkpoint or the best checkpoint.
      state: checkpoint we want to save
      is_best: is this the best checkpoint; min validation loss
      checkpoint_path: path to save checkpoint
      best_model_path: path to save best model
      """
      f_path = checkpoint_path
      # save checkpoint data to the path given, checkpoint_path
      torch.save(state, f_path)
      # if it is a best model, min validation loss
      if is_best:
          best_fpath = best_model_path
          # copy that checkpoint file to best path given, best_model_path
          shutil.copyfile(f_path, best_fpath)
  
  def load_ckp(checkpoint_fpath, model, optimizer):
      """
      checkpoint_path: path to save checkpoint
      model: model that we want to load checkpoint parameters into       
      optimizer: optimizer we defined in previous training
      """
      # load check point
      checkpoint = torch.load(checkpoint_fpath)
      # initialize state_dict from checkpoint to model
      model.load_state_dict(checkpoint['state_dict'])
      # initialize optimizer from checkpoint to optimizer
      optimizer.load_state_dict(checkpoint['optimizer'])
      # initialize valid_loss_min from checkpoint to valid_loss_min
      valid_loss_min = checkpoint['valid_loss_min']
      # return model, optimizer, epoch value, min validation loss 
      return model, optimizer, checkpoint['epoch'], valid_loss_min.item()
  
  #Create the BertClassfier class
  class BertClassifier(nn.Module):
      """Bert Model for Classification Tasks."""
      def __init__(self, freeze_bert=True):
          """
           @param    bert: a BertModel object
           @param    classifier: a torch.nn.Module classifier
           @param    freeze_bert (bool): Set `False` to fine-tune the BERT model
          """
          super(BertClassifier, self).__init__()
          
          .......
          
      def forward(self, input_ids, attention_mask):
          ''' Feed input to BERT and the classifier to compute logits.
           @param    input_ids (torch.Tensor): an input tensor with shape (batch_size,
                         max_length)
           @param    attention_mask (torch.Tensor): a tensor that hold attention mask
                         information with shape (batch_size, max_length)
           @return   logits (torch.Tensor): an output tensor with shape (batch_size,
                         num_labels) '''
           # Feed input to BERT
          outputs = self.bert(input_ids=input_ids,
                               attention_mask=attention_mask)
           
           # Extract the last hidden state of the token `[CLS]` for classification task
          last_hidden_state_cls = outputs[0][:, 0, :]
   
           # Feed input to classifier to compute logits
          logits = self.classifier(last_hidden_state_cls)
   
          return logits
  
  def initialize_model(epochs):
      """ Initialize the Bert Classifier, the optimizer and the learning rate scheduler."""
      # Instantiate Bert Classifier
      bert_classifier = BertClassifier(freeze_bert=False)
  
      # Tell PyTorch to run the model on GPU
      bert_classifier = bert_classifier.to(device)
  
      # Create the optimizer
      optimizer = AdamW(bert_classifier.parameters(),
                        lr=lr,    # Default learning rate
                        eps=1e-8    # Default epsilon value
                        )
  
      # Total number of training steps
      total_steps = len(train_dataloader) * epochs
  
      # Set up the learning rate scheduler
      scheduler = get_linear_schedule_with_warmup(optimizer,
                                                  num_warmup_steps=0, # Default value
                                                  num_training_steps=total_steps)
      return bert_classifier, optimizer, scheduler
      
  
  def train(model, train_dataloader, val_dataloader, valid_loss_min_input, checkpoint_path, best_model_path, start_epochs, epochs, evaluation=True):
  
      """Train the BertClassifier model."""
      # Start training loop
      logging.info("--Start training...\n")
  
      # Initialize tracker for minimum validation loss
      valid_loss_min = valid_loss_min_input 
  
  
      for epoch_i in range(start_epochs, epochs):
          # =======================================
          #               Training
          # =======================================
          # Print the header of the result table
          logging.info((f"{'Epoch':^7} | {'Batch':^7} | {'Train Loss':^12} | {'Val Loss':^10} | {'Val Acc':^9} | {'Elapsed':^9}"))
  
          # Measure the elapsed time of each epoch
          t0_epoch, t0_batch = time.time(), time.time()
  
          # Reset tracking variables at the beginning of each epoch
          total_loss, batch_loss, batch_counts = 0, 0, 0
  
          # Put the model into the training mode
          model.train()
  
          # For each batch of training data...
          for step, batch in enumerate(train_dataloader):
              batch_counts +=1
              # Load batch to GPU
              b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)
  
              # Zero out any previously calculated gradients
              model.zero_grad()
  
              # Perform a forward pass. This will return logits.
              logits = model(b_input_ids, b_attn_mask)
  
              # Compute loss and accumulate the loss values
              loss = loss_fn(logits, b_labels)
              batch_loss += loss.item()
              total_loss += loss.item()
  
              # Perform a backward pass to calculate gradients
              loss.backward()
  
              # Clip the norm of the gradients to 1.0 to prevent "exploding gradients"
              torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
  
              # Update parameters and the learning rate
              optimizer.step()
              scheduler.step()
  
              # Print the loss values and time elapsed for every 20 batches
              if (step % 500 == 0 and step != 0) or (step == len(train_dataloader) - 1):
                  # Calculate time elapsed for 20 batches
                  time_elapsed = time.time() - t0_batch
  
                  # Print training results
                  logging.info(f"{epoch_i + 1:^7} | {step:^7} | {batch_loss / batch_counts:^12.6f} | {'-':^10} | {'-':^9} | {time_elapsed:^9.2f}")
  
                  # Reset batch tracking variables
                  batch_loss, batch_counts = 0, 0
                  t0_batch = time.time()
  
          # Calculate the average loss over the entire training data
          avg_train_loss = total_loss / len(train_dataloader)
  
          logging.info("-"*70)
          # =======================================
          #               Evaluation
          # =======================================
          if evaluation == True:
              # After the completion of each training epoch, measure the model's performance
              # on our validation set.
              val_loss, val_accuracy = evaluate(model, val_dataloader)
  
              # Print performance over the entire training data
              time_elapsed = time.time() - t0_epoch
              
              logging.info(f"{epoch_i + 1:^7} | {'-':^7} | {avg_train_loss:^12.6f} | {val_loss:^10.6f} | {val_accuracy:^10.6f} | {time_elapsed:^9.2f}")
  
              logging.info("-"*70)
          logging.info("\n")
  
  
           # create checkpoint variable and add important data
          checkpoint = {
              'epoch': epoch_i + 1,
              'valid_loss_min': val_loss,
              'state_dict': model.state_dict(),
              'optimizer': optimizer.state_dict(),
          }
          
          # save checkpoint
          save_ckp(checkpoint, False, checkpoint_path, best_model_path)
          
          ## TODO: save the model if validation loss has decreased
          if val_loss <= valid_loss_min:
              print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,val_loss))
              # save checkpoint as best model
              save_ckp(checkpoint, True, checkpoint_path, best_model_path)
              valid_loss_min = val_loss
      
      logging.info("-----------------Training complete--------------------------")
  
  def evaluate(model, val_dataloader):
      """After the completion of each training epoch, measure the model's performance on our validation set."""
      # Put the model into the evaluation mode. The dropout layers are disabled during the test time.
      model.eval()
  
      # Tracking variables
      val_accuracy = []
      val_loss = []
  
      # For each batch in our validation set...
      for batch in val_dataloader:
          # Load batch to GPU
          b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)
  
          # Compute logits
          with torch.no_grad():
              logits = model(b_input_ids, b_attn_mask)
  
          # Compute loss
          loss = loss_fn(logits, b_labels)
          val_loss.append(loss.item())
  
          # Get the predictions
          preds = torch.argmax(logits, dim=1).flatten()
  
          # Calculate the accuracy rate
          accuracy = (preds == b_labels).cpu().numpy().mean() * 100
          val_accuracy.append(accuracy)
  
      # Compute the average accuracy and loss over the validation set.
      val_loss = np.mean(val_loss)
      val_accuracy = np.mean(val_accuracy)
  
      return val_loss, val_accuracy
  
  bert_classifier, optimizer, scheduler = initialize_model(epochs=n_epochs)
  train(model = bert_classifier ......)
  
  
  
  bert_classifier, optimizer, scheduler = initialize_model(epochs=n_epochs)
  model, optimizer, start_epoch, valid_loss_min = load_ckp(r"./best_model/best_model.pt", bert_classifier, optimizer)
  
  model.eval()
  model.freeze()
  
  sample = {
    "seq": "ABCDE",}
  
  predictions = model.predict(sample)
  

  AttributeError: 'BertClassifier' object has no attribute 'predict'
  
• Where does transformers save microsoft/DialoGPT-small?

  when I run the code

  from transformers import AutoModelForCausalLM, AutoTokenizer
  import torch
  SML=input("small,medium or large: ").lower()#I pick small
  model_name = "microsoft/DialoGPT-"+SML
  tokenizer = AutoTokenizer.from_pretrained(model_name)
  model = AutoModelForCausalLM.from_pretrained(model_name)
  

  It downloads microsoft/DialoGPT-small from somewhere on the internet and downloads it to my computer but I don`t know where the saved file is. Anyone have any idea where this file is saved?

  #EDIT#

  I found the below code in \Lib\site-packages\transformers\models\dialogpt\convert_dialogpt_original_pytorch_checkpoint_to_pytorch.py but have no idea what it means.

  def convert_dialogpt_checkpoint(checkpoint_path: str, pytorch_dump_folder_path: str):
      d = torch.load(checkpoint_path)
      d[NEW_KEY] = d.pop(OLD_KEY)
      os.makedirs(pytorch_dump_folder_path, exist_ok=True)
      torch.save(d, os.path.join(pytorch_dump_folder_path, WEIGHTS_NAME))
  
  if __name__ == "__main__":
      parser = argparse.ArgumentParser()
      parser.add_argument("--dialogpt_path", default=".", type=str)
      args = parser.parse_args()
      for MODEL in DIALOGPT_MODELS:
          checkpoint_path = os.path.join(args.dialogpt_path, f"{MODEL}_ft.pkl")
          pytorch_dump_folder_path = f"./DialoGPT-{MODEL}"
          convert_dialogpt_checkpoint(
              checkpoint_path,
              pytorch_dump_folder_path,
          )
  
• what is the correct transformer for generating text out of image features

  I am working on a project about image captioning. I wanna make a full transformer model using SWIN as the encoder and for the decoder part I need another transformer to learn the captions in the COCO 2017 dataset and make text using features extracted from each image. the problem is I do not know if this is a good idea although I have found two papers with fully transformer architecture. they used only one encoder and a custom decoder not BERT or .. so I do not know and I can not decide which language transformer is good to use for the purpose.
• Why does tokeniser break down words that are present in vocab

  In my understanding, what tokeniser does is that, given each word, the tokeniser will break down the word into sub-words only the word is not present in the tokeniser.get_vocab()

  def checkModel(model):
      tokenizer = AutoTokenizer.from_pretrained(model)
  
      allList = []
      for word in tokenizer.get_vocab():
          word = word.lower()
  
          tokens = tokenizer.tokenize(word)
          try:
              if word[0]!='#' and word[0]!='[' and tokens[0] != word:
  
                  allList.append((word, tokens))
                  print(word, tokens)
          except:
              continue 
      return allList
  
  checkModel('bert-base-uncased')
  # ideally should return an empty list
  

  However, what I have observed is that some models on huggingface will break down words into smaller pieces even if the word is present in the vocab.

  checkModel('emilyalsentzer/Bio_ClinicalBERT')
  
  output: 
  welles ['well', '##es']
  lexington ['le', '##xing', '##ton']
  palestinian ['pale', '##st', '##inian']
  ...
  elisabeth ['el', '##isa', '##beth']
  alexander ['ale', '##xa', '##nder']
  appalachian ['app', '##ala', '##chia', '##n']
  mitchell ['mit', '##chel', '##l']
  ...
    
  4630 # tokens in vocab got broken down, not supposed to happen
  

  I have checked a few models of this behaviour, was wondering why is this happening?
• Use 1 tokenizer or 2 tokenizers for translation task?

  I’ve seen several tutorials about seq2seq tasks like translation. They usually use 2 tokenizers trained on corpus, one for source language and the other for target language. However, in huggingface’s translation task example, they just use one tokenizer for 2 languages. I wonder which is the better way, 1 tokenizer or 2 tokenizers? If i use 2 tokenizers then the output classes would be smaller and may be it can eliminate some tokens that target language doesn’t have, thus, improve the result or it is okay to use one tokenizer and the performance is still the same? Please, help me, thanks In advance!