cuDNN failing to initialize possible version mismatch

See also questions close to this topic

• MySQL Has gone away during executemany on after moving to VPS

  I'm trying to move some services to my VPS. It's multiple python scripts that is scraping from an API and feeding data to an MySQL database. Most of it is working flawlessly after the move, but this one script has a single insert that fails on the new MySQL-installation.

  It's trying to insert 100 lines in a 2.5gb table. It's returning with error 2006, 'MySQL server has gone away'. It's not a big insert, Python says its about 90kB. I can't believe that it's timing out cause I get the error within 10-15seconds of launching the script, which spends most of that time doing http-requests before attempting the insert. Heres what I've done:

  1. Verified that both MySQL-databases are set up with the same timeouts and allowed packet size. I've also tried increasing both on the new VPS.
  2. The same exact script works on my old database.
  3. I've tried dividing the insert into smaller parts, and it's still shutting down the connection.
  4. I've tried running the script from my old server, so the Python env. is identical. It's connecting fine and doing other inserts/updates.
  5. I can log into the mysql with ssh and have tried a simple 1-line insert into the table and that works.

  It's working well updating 100 lines in a different table, which is smaller (600MB). This is why I suspect that my VPS somehow cant handle the 2.5GB table. The VPS has less RAM than my previous server. But it shouldn't really open the entire table to INSERT, should it? The VPS has 4gb while my old server had 8gb. I'm never above 25% memory usage on my old server.

  Here's the function after I've tried making smaller parts of it:

  def updateComplete(completeupdate, LIST_LENGTH=10):
      logging.debug(f"updateComplete has been called for {completeupdate}")
      START_INDEX = 0 
      cursor = db.cursor()
      while completeupdate[START_INDEX:START_INDEX+LIST_LENGTH]:
          cursor.executemany("""INSERT INTO userscomplete
                              (timer, userid, username, userlevel, userrank, battlestats,
                              forumposts, karma, age, role, statu, description, jobposition, jobposition2,
                              companyid, factionposition, factionid, factiondays, inactivedays,
                              networth, propertyvalue, attackswon, attackslost, moneymugged, drugsused,
                              maxlife, defendslost, defendswon, undefeated, awards) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, \
                              %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, \
                              %s, %s)""", completeupdate)
          START_INDEX+=LIST_LENGTH
      db.commit()
  

  wait_timeout = 28800 max_allowed_packet = 104857600

  EDIT: Forgot to note that the old server is MySQL v. 8.0.20 and the new is 8.0.23.
• Doesn't add all elements that are not in dictionary values

  I'm iterating over the list and supposedly add element to dictionary that doesn't occur in dictionary as value, therefore I would expect output:

  {'key': 1, 'key2': 3, 'key3': 2, 'key4':4, 'key5':5}
  

  Code:

  di = {'key':1, 'key2':4}
  
  li=[1,1,1,2,4,3,5]
  
  b = sum(1 for key in di if key.startswith('key')) # check how many keys starts with 'key'
  
  for i in li:                #if element of list is not in dictionary key values 
      if i not in di.values():  #add it as value to 'key+b+1'
          di[f'key{b+1}']= i
  

  But the ouput I'm getting:

  {'key': 1, 'key2': 4, 'key3': 5}
  

  So as I see despite I'm telling Python to check elements in dict.values he's checking also keys or items.
• How to remove dynamically generated checkboxes?

  So I've got a pyqt5 GUI that generates checkboxes based on the titles of JSON files that are saved in a directory. My next step is making it so that when the users checks a box, it deletes the checkbox, and then the JSON file.

  Currently I generate the checkboxes as follows:

          for x in self.list:
              layout.addWidget(QCheckBox(x),row,0)
              row+=1
  

  Since I have not assigned each checkbox to an object, how would I go about checking the state of each box, and then assigning it an action?

  Since I'm learning to discuss what I was thinking:

  I was hoping I could use Signals to detect when a signal is transmitted from the overall widget, and then slot that with a function that deletes any file in the directory sharing the name of the object that generated the signal.

  Is there a better approach to generate the checkboxes as well?
• Mlflow and KerasTuner integration

  I am trying to integrate together KerasTuner and Mlflow. I'd like to record the loss at each epoch of each trial of Keras Tuner.

  My approach is:

  class MlflowCallback(tf.keras.callbacks.Callback):
      
      # This function will be called after each epoch.
      def on_epoch_end(self, epoch, logs=None):
          if not logs:
              return
          # Log the metrics from Keras to MLflow     
          mlflow.log_metric("loss", logs["loss"], step=epoch)
      
  
  from kerastuner.tuners import RandomSearch
  
  with mlflow.start_run(run_name="myrun", nested=True) as run:
    
    tuner = RandomSearch(
        train_fn,
        objective='loss',
        max_trials=25, 
    )
    tuner.search(train,
                validation_data=validation, 
                validation_steps=validation_steps,
                steps_per_epoch=steps_per_epoch, 
                epochs=5, 
                callbacks=[MlflowCallback()]
    )
  

  However, the loss values are reported (sequentially) in one single experiment. Is there a way to record them independently?
• save deep learning model with tf not pickle

  Would anyone have a tip for saving a sequential deep learning model made with tf.keras using tf.keras save_model method?

  I have an issue with reusing some old code where this used to work:

  # save model to file
  pickle.dump(model, open("darth-vader-model.pkl", "wb"))
  
  # save all of our data structures
  pickle.dump({'words':words, 'classes':classes, 'train_x':train_x, 'train_y':train_y}, open("darth-vader-data.pkl", "wb"))
  

  Revisiting this code this now errors out on:

  TypeError                                 Traceback (most recent call last)
  <ipython-input-13-5b1bb51d3ad3> in <module>
        1 # save model to file
  ----> 2 pickle.dump(model, open("darth-vader-model.pkl", "wb"))
        3 
        4 #model.save('./model')
  
  TypeError: cannot pickle 'weakref' object
  

  SO I have been trying to read about a solution just using tensorflow directly to save models instead of pickle method but I cannot figure out how to replicate the code above especially the #save all of our data structures

  Would anyone have any tips for how I could revise this for tf only save_model? Its a ML model for ntkl chat bot related processes.

  In tf should I be using a model.save('/tmp/model') or a

  tf.keras.models.save_model(
      model, filepath, overwrite=True, include_optimizer=True, save_format=None,
      signatures=None, options=None, save_traces=True
  )
  

  to replicate the pickle dump process? The process is for chat bot purposes, bag of words, ntlk use...
• Fail to find the dnn implementation. [[{{node CudnnRNN}}]]

  This error appears in my code.

  I was looking for help in Internet. Then I discovered that my cuda compiler is:

  nvcc --version
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2018 NVIDIA Corporation
  Built on Sat_Aug_25_21:08:01_CDT_2018
  Cuda compilation tools, release 10.0, V10.0.130
  

  However, if I write nvidia-smi, the following is shown:

  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |                               |                      |               MIG M. |
  |===============================+======================+======================|
  |   0  GeForce RTX 206...  Off  | 00000000:01:00.0  On |                  N/A |
  | 31%   33C    P8    11W / 184W |   7962MiB /  7979MiB |      3%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
                                                                                 
  +-----------------------------------------------------------------------------+
  | Processes:                                                                  |
  |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
  |        ID   ID                                                   Usage      |
  |=============================================================================|
  |    0   N/A  N/A      1291      G   /usr/lib/xorg/Xorg                 40MiB |
  |    0   N/A  N/A      1333      G   /usr/bin/gnome-shell               57MiB |
  |    0   N/A  N/A      1616      G   /usr/lib/xorg/Xorg                124MiB |
  |    0   N/A  N/A      1751      G   /usr/bin/gnome-shell               78MiB |
  |    0   N/A  N/A      2096      G   ...gAAAAAAAAA --shared-files       57MiB |
  |    0   N/A  N/A      2369      G   ...ulen/anaconda3/bin/python        5MiB |
  |    0   N/A  N/A      2492      C   ...ulen/anaconda3/bin/python     7529MiB |
  |    0   N/A  N/A      2707      G   ...AAAAAAAA== --shared-files       61MiB |
  +-----------------------------------------------------------------------------+
  

  Could this be the problem? I thought I only have installed CUDA 10.0
• how to install cudnn with conda installed cudatoolkit?

  Now I have used conda to install cudatoolkit=11.0, then how should I install cudnn which is downloading from Nvidia?

  I find my cudatoolkit files didn't inconsistent with the cudnn=8.0.5 here is my toolkit flies

  

  here is my cudnn flies

  
• How correctly set CuDNNLSTM Network

  I've a problem setting and running a training model using CuDNNLSTM from Tensorflow 2.4. If I use normal LSTM it works fine, is really slower but it starts correctly its training phase.

  My dataset is read from .csv file and is composed in this way:

  1. It is 1D array of 90 bytes
  2. 7 target labels
  3. 90 features (cols) normalized 0-1
  4. 2000 rows for each class

  This is my test network:

  def lstm(num_classes):
      model = models.Sequential()
      model.add(CuDNNLSTM(128,batch_size=16, input_shape=input_shape, return_sequences=True))
      model.add(Dropout(0.2))
      model.add(BatchNormalization())
      model.add(Dense(128, activation='relu'))
      model.add(Dense(12, activation='relu'))
      model.add(Dense(num_classes, activation='softmax')) 
      return model
  

  This is my attempt to preprocess data and start training phase:

  num_classes = 7
  input_shape = (90, 1)
  batch_size = 16
  epochs = 50
  
  df = pd.read_csv("Dataset/train.csv", header=0)
  df["generic_label"] = pd.Categorical(df["generic_label"])
  df["generic_label"] = df.generic_label.cat.codes
  Y = df.pop("generic_label")
  
  x_train_val, x_test, y_train_val, y_test = train_test_split(np.asarray(df), np.asarray(Y), test_size=0.2)
  x_train, x_val, y_train, y_val = train_test_split(x_train_val, y_train_val, test_size=0.4)
  
  x_train = np.reshape(x_train, (x_train.shape[0],) + input_shape)
  x_val = np.reshape(x_val, (x_val.shape[0],) + input_shape)
  x_test = np.reshape(x_test, (x_test.shape[0],) + input_shape)
  y_train = to_categorical(y_train, num_classes)
  y_val = to_categorical(y_val, num_classes)
  y_test = to_categorical(y_test, num_classes)
  
  model = lstm(num_classes)
  model.compile(
      optimizer=keras.optimizers.Adam(0.001),
      loss="categorical_crossentropy",
      metrics=["accuracy", tfa.metrics.F1Score(num_classes)],
  )
  model.summary()
  model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=(x_val, y_val))
  

  This is error output:

  ValueError: Shapes (16, 7) and (16, 90, 7) are incompatible

  I'm sure I'm doing a wrong preprocessing, indeed I'm missing timesteps (should be 5 rows) but I really don't know how to do it correctly.

  If I replace LSTM instead of CuDNNLSTM it works fine
• CUDA version crash

  I installed CUDA 11.2 and realised that's not the version I wanted... Then I uninstall it via Control Panel "Uninstall a Program" and uninstall all CUDA 11.2 stuff(but I didn't uninstall those Nvidia Nsight stuff)

  After that I install CUDA 10.2, but as I key in below 2 commans, both showed me different CUDA verisons. I'm using Windows 10, with GeForce GTX 1650.

  > nvcc --version
  
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2019 NVIDIA Corporation
  Built on Wed_Oct_23_19:32:27_Pacific_Daylight_Time_2019
  Cuda compilation tools, release 10.2, V10.2.89
  

  But with

  >nvidia-smi
  
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 461.09       Driver Version: 461.09       CUDA Version: 11.2     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |                               |                      |               MIG M. |
  |===============================+======================+======================|
  |   0  GeForce GTX 1650   WDDM  | 00000000:01:00.0 Off |                  N/A |
  | N/A   38C    P8     8W /  N/A |    134MiB /  4096MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  
  +-----------------------------------------------------------------------------+
  | Processes:                                                                  |
  |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
  |        ID   ID                                                   Usage      |
  |=============================================================================|
  |  No running processes found                                                 |
  +-----------------------------------------------------------------------------+
  

  10.2 and 11.2... Any idea on this?

  Thank you!!