How can I learn and understand machine learning

See also questions close to this topic

• Estimating CATE using Causal Forest with Python

  I am looking for a way to use causal forest with Python (not R) in order to estimate the conditional average treatment effect (CATE).

  I couldn't find any library to support such a thing on Python.

  I want the split rule of each tree to maximize the variance of the CATE, instead of maximizing the varaince of the outcome Y (as done in the ordinary regression trees).

  I would be really glad to be directed to the correct Python library or to the right algorithm.

  Thank you very much for your help!
• how can i train my chatbot using Tensorflow?

  Is it possible to train my kivy chatbot using TensorFlow?

  Here is how I code my bot until now, but I want to make it using an algorithm for training the user's inputs. my chatbot is for cryptocurrency prices and informs users about the creators of these cryptocurrencies. here is a part of the code :

  def answer(self, text, *args):
          # text is the user's message
  
          if text == 'hi' or text == 'hey' or text == 'hello':
              response = "Hey. I am Bitbot for more information type info."
              self.add_message(response, 'left', self.bot_message_color),
  
          elif text == 'info':
              response = "Bitbot can answer question like:\n" " - what is bitcoin price? for cryptocurrency price \n" " - what is bitcoin? for theritical aspect \n" " - website. for open website of cryptoccurency trading \n" " - what is the change of percentage for bitcoin for 1 hour? for getting the percentage of changing in hour \n" " - what is the change of percentage for bitcoin for 24 hours?  for getting the percentage of changing for 24hours \n" " - what is the change of percentage for bitcoin for one month? for getting the percentage of changing for one month \n"
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'website'or text=='navigate me to the website' or text=='provide me a website':
              webbrowser.open(r"https://coinmarketcap.com/")
              response = "hope that help."
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'how are you?':
              response = "I am fine thank you"
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'bye' or text == 'cao' or text == 'goodbye':
              response = "Bye hope to see you again"
              self.add_message(response, 'left', self.bot_message_color),
  
              ## THEORITICAL ASPECT
          elif text == 'what is bitcoin?':
              response = "Bitcoin is a cryptocurrency invented in 2008 by an unknown person or group of people using the name Satoshi Nakamoto. The currency began use in 2009 when its implementation was released as open-source software."
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'what is ethereum?':
              response = "Ethereum is a decentralized, open-source blockchain featuring smart contract functionality. Ether is the native cryptocurrency of the platform. It is the second-largest cryptocurrency by market capitalization, after Bitcoin. Ethereum is the most actively used blockchain"
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'what is litecoin?':
              response = "Litecoin is a peer-to-peer cryptocurrency and open-source software project released under the MIT/X11 license. Litecoin was an early bitcoin spinoff or altcoin, starting in October 2011. In technical details, Litecoin is nearly identical to Bitcoin."
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'what is xpr?':
              response = "XPR is often referred to as 'real cryptocurrency'; it is a token that uses Ripple network to enable money transfers between various currencies. Most clearing systems nowadays use US dollars as a base currency for conversion. "
              self.add_message(response, 'left', self.bot_message_color),
  
          elif text == 'Satoshi Nakamoto' or text== 'Satoshi' or text== 'bitcoin creator' or text=='who create bitcoin?':
              response = "Satoshi Nakamoto is the name used by the presumed pseudonymous person or persons who developed bitcoin, authored the bitcoin white paper, and created and deployed bitcoin's original reference implementation. As part of the implementation, Nakamoto also devised the first blockchain database.In the process, Nakamoto was the first to solve the double-spending problem for digital currency using a peer-to-peer network. Nakamoto was active in the development of bitcoin up until December 2010.Many people have claimed, or have been claimed, to be Nakamoto. "
              self.add_message(response, 'left', self.bot_message_color),
  
          elif text == 'Gavin Wood'or text=='Gavin' or text=='ethereum creator' or text=='who create ethereum?':
              response = "Gavin Wood is a British computer programmer, co-founder of Ethereum and creator of Polkadot. He invented Solidity and wrote the Yellow Paper specifying the Ethereum Virtual Machine. Wood served as the Ethereum Foundation's first chief technology officer. After leaving Ethereum in 2016, he co-founded Parity Technologies (formerly Ethcore), which develops core infrastructure for Ethereum, Bitcoin, Zcash and Polkadot. "
              self.add_message(response, 'left', self.bot_message_color),
  
          elif text == 'Charlie Lee'or text=='Charlie' or text=='litecoin creator' or text=='who create litecoin?':
              response = "Charles Lee is a computer scientist, best known as the creator of Litecoin. He serves as the managing director of the Litecoin Foundation.As of July 2013, he also worked for Coinbase."
              self.add_message(response, 'left', self.bot_message_color),
  
          elif text == 'Jed McCaleb'or text=='Jed' or text=='xpr creator' or text=='who create xpr?':
              response = "Jed McCaleb is an American programmer and entrepreneur. He is a co-founder and the CTO of Stellar.Prior to co-founding Stellar, McCaleb founded and served as the CTO of the company Ripple until 2013. McCaleb is also known for creating Mt. Gox, and the peer-to-peer eDonkey and Overnet networks as well as the eDonkey2000 application."
              self.add_message(response, 'left', self.bot_message_color),
  
              ## cryptoccurency prices
          elif text == 'what is bitcoin price?'or text=='bitcoin price'or text=='price of bitcoin'or text=='btc' or text=='BTC':
              response = "The Bitcoin price is " + "{:.2f}$".format(float(get_cmc_data('BTC')))
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'what is ethereum price?'or text=='ethereum price'or text=='price of ethereum'or text=='eth' or text=='ETH':
              response = "The Ethereum price is " + "{:.2f}$".format(float(get_cmc_data('ETH')))
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'what is litecoin price?'or text=='litecoin price'or text=='price of litecoin'or text=='ltc' or text=='LTC':
              response = "The Litecoin price is " + "{:.2f}$".format(float(get_cmc_data('LTC')))
              self.add_message(response, 'left', self.bot_message_color),
          elif text == 'what is xpr price?'or text=='xpr price'or text=='price of xpr'or text=='xpr' or text=='XPR':
              response = "The XPR price is " + "{:.2f}$".format(float(get_cmc_data('XPR')))
              self.add_message(response, 'left', self.bot_message_color)
  
• Finding pattern of unkown size

  Is it possible to find a pattern of unknown size e.g.

  12312 24312 -> A

  12412 24312 -> B

  123412 24112 -> B

  What I am looking for is a method to predict the pattern of class A which means by combining both sequence changes of position 3 in both sequences of class B, results in class A instead of class B.

  The size of the pattern is unknown. Is it possible to find a pattern (not class label) with machine learning or deep learning?
• deep learning for holes in audio

  I need to make a deep learning model which finds holes in audio files (which can occurred due to bad internet connection).

  I thought about taking a model for single-object localization in images, and change it to find holes in the audio.

  Im pretty new to this so I have a few questions

  1. Is it really the best way to achieve it?
  2. If so, what representation of the audio should I use? spectrogram or a vector of the frequencies?
  3. If it is not the best way, than what is the best way to achieve it?
  4. Is there a familiar model for this problem I can use?
• Input 0 of layer dense is incompatible with the layer: expected axis -1 of input shape to have value 8192 but received input with shape (None, 61608)

  I am trying to create an image processing CNN. I am using VGG16 to speed up some of the learning process. The creation of my CNN below works to the point of training and saving the model & weights. The issue occurs when I try to run a predict function after loading in the model.

  image_gen = ImageDataGenerator()
  train = image_gen.flow_from_directory('./data/train', class_mode='categorical', shuffle=False, batch_size=10, target_size=(151, 136))
  val = image_gen.flow_from_directory('./data/validate', class_mode='categorical', shuffle=False, batch_size=10, target_size=(151, 136))
  
  pretrained_model = VGG16(include_top=False, input_shape=(151, 136, 3), weights='imagenet')
  pretrained_model.summary()
  
  vgg_features_train = pretrained_model.predict(train)
  vgg_features_val = pretrained_model.predict(val)
  
  train_target = to_categorical(train.labels)
  val_target = to_categorical(val.labels)
  
  model = Sequential()
  model.add(Flatten())
  model.add(Dense(100, activation='relu'))
  model.add(Dropout(0.5))
  model.add(BatchNormalization())
  model.add(Dense(2, activation='softmax'))
  
  model.compile(optimizer='rmsprop', metrics=['accuracy'], loss='categorical_crossentropy')
  
  target_dir = './models/weights-improvement'
  if not os.path.exists(target_dir):
    os.mkdir(target_dir)
  
  checkpoint = ModelCheckpoint(filepath=target_dir + 'weights-improvement-{epoch:02d}-{val_accuracy:.2f}.hdf5', monitor='val_accuracy', verbose=1, save_best_only=True, mode='max')
  callbacks_list = [checkpoint]
  
  model.fit(vgg_features_train, train_target, epochs=100, batch_size=8, validation_data=(vgg_features_val, val_target), callbacks=callbacks_list)
  
  model.save('./models/model')
  model.save_weights('./models/weights')
  

  I have this predict function, that I would like to load in an image, and then return the categorisation of this image that the model gives.

  from keras.preprocessing.image import load_img, img_to_array
  def predict(file):
    x = load_img(file, target_size=(151,136,3))
    x = img_to_array(x)
    print(x.shape)
    print(x.shape)
    x = np.expand_dims(x, axis=0)
    array = model.predict(x)
    result = array[0]
    if result[0] > result[1]:
      if result[0] > 0.9:
        print("Predicted answer: Buy")
        answer = 'buy'
        print(result)
        print(array)
      else:
        print("Predicted answer: Not confident")
        answer = 'n/a'
        print(result)
    else:
      if result[1] > 0.9:
        print("Predicted answer: Sell")
        answer = 'sell'
        print(result)
      else:
        print("Predicted answer: Not confident")
        answer = 'n/a'
        print(result)
  
    return answer
  

  The issue I am experiencing is when I run this predict function, I get the following error.

    File "predict-binary.py", line 24, in predict
      array = model.predict(x)
    File ".venv\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1629, in predict    
      tmp_batch_outputs = self.predict_function(iterator)
    File ".venv\lib\site-packages\tensorflow\python\eager\def_function.py", line 828, in __call__       
      result = self._call(*args, **kwds)
    File ".venv\lib\site-packages\tensorflow\python\eager\def_function.py", line 871, in _call
      self._initialize(args, kwds, add_initializers_to=initializers)
    File ".venv\lib\site-packages\tensorflow\python\eager\def_function.py", line 725, in _initialize    
      self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
    File ".venv\lib\site-packages\tensorflow\python\eager\function.py", line 2969, in _get_concrete_function_internal_garbage_collected
      graph_function, _ = self._maybe_define_function(args, kwargs)
    File ".venv\lib\site-packages\tensorflow\python\eager\function.py", line 3361, in _maybe_define_function
      graph_function = self._create_graph_function(args, kwargs)
    File ".venv\lib\site-packages\tensorflow\python\eager\function.py", line 3196, in _create_graph_function
      func_graph_module.func_graph_from_py_func(
    File ".venv\lib\site-packages\tensorflow\python\framework\func_graph.py", line 990, in func_graph_from_py_func
      func_outputs = python_func(*func_args, **func_kwargs)
    File ".venv\lib\site-packages\tensorflow\python\eager\def_function.py", line 634, in wrapped_fn     
      out = weak_wrapped_fn().__wrapped__(*args, **kwds)
    File ".venv\lib\site-packages\tensorflow\python\framework\func_graph.py", line 977, in wrapper      
      raise e.ag_error_metadata.to_exception(e)
  ValueError: in user code:
  
      .venv\lib\site-packages\tensorflow\python\keras\engine\training.py:1478 predict_function  *       
          return step_function(self, iterator)
      .venv\lib\site-packages\tensorflow\python\keras\engine\training.py:1468 step_function  **
          outputs = model.distribute_strategy.run(run_step, args=(data,))
      .venv\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:1259 run
          return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
      .venv\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2730 call_for_each_replica 
          return self._call_for_each_replica(fn, args, kwargs)
      .venv\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3417 _call_for_each_replica
          return fn(*args, **kwargs)
      .venv\lib\site-packages\tensorflow\python\keras\engine\training.py:1461 run_step  **
          outputs = model.predict_step(data)
      .venv\lib\site-packages\tensorflow\python\keras\engine\training.py:1434 predict_step
          return self(x, training=False)
      .venv\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:1012 __call__
          outputs = call_fn(inputs, *args, **kwargs)
      .venv\lib\site-packages\tensorflow\python\keras\engine\sequential.py:375 call
          return super(Sequential, self).call(inputs, training=training, mask=mask)
      .venv\lib\site-packages\tensorflow\python\keras\engine\functional.py:424 call
          return self._run_internal_graph(
      .venv\lib\site-packages\tensorflow\python\keras\engine\functional.py:560 _run_internal_graph      
          outputs = node.layer(*args, **kwargs)
      .venv\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:998 __call__
          input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
      .venv\lib\site-packages\tensorflow\python\keras\engine\input_spec.py:255 assert_input_compatibility
          raise ValueError(
  
      ValueError: Input 0 of layer dense is incompatible with the layer: expected axis -1 of input shape to have value 8192 but received input with shape (None, 61608)
  

  I'm assuming I need to change something between the Flatten() and Dense() layers of my model, but I'm not sure what. I attempted to add model.add(Dense(61608, activation='relu)) between these two as that seemed to be what was suggested in another post I saw (cannot find link now), but it lead to the same error. (I tried it with 8192 instead of 61608 as well). Any help is appreciated, thanks.

  EDIT #1:

  Changing the model creation/training code as I think it was suggested by Gerry P to this

      img_shape = (151,136,3)
  base_model=VGG19( include_top=False, input_shape=img_shape, pooling='max', weights='imagenet' )
  x=base_model.output
  x=Dense(100, activation='relu')(x)
  x=Dropout(0.5)(x)
  x=BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001)(x)
  output=Dense(2, activation='softmax')(x)
  model=Model(inputs=base_model.input, outputs=output)
  
  image_gen = ImageDataGenerator()
  train = image_gen.flow_from_directory('./data/train', class_mode='categorical', shuffle=False, batch_size=10, target_size=(151, 136))
  val = image_gen.flow_from_directory('./data/validate', class_mode='categorical', shuffle=False, batch_size=10, target_size=(151, 136))
  
  vgg_features_train = base_model.predict(train)
  vgg_features_val = base_model.predict(val)
  
  train_target = to_categorical(train.labels)
  val_target = to_categorical(val.labels)
  
  model.compile(optimizer='rmsprop', metrics=['accuracy'], loss='categorical_crossentropy')
  
  model.fit(vgg_features_train, train_target, epochs=100, batch_size=8, validation_data=(vgg_features_val, val_target), callbacks=callbacks_list)
  

  This resulted in a different input shape error of File "train-binary.py", line 37, in <module> model.fit(vgg_features_train, train_target, epochs=100, batch_size=8, validation_data=(vgg_features_val, val_target), callbacks=callbacks_list) ValueError: Input 0 is incompatible with layer model: expected shape=(None, 151, 136, 3), found shape=(None, 512)
• result of the stacked autoencoder - lstm

  I trained (stacked autoencoder-LSTM) on the dataset that contains normal traffic many times, and every time I changed the batch size and the epoch, I save the models for testing as model1 model2 model3 ...etc.

  sequence_autoencoder_semi_history = sequence_autoencoder_semi.fit(X_train_sequence, X_train_sequence,
                       epochs=40,
                       batch_size=20,
                       shuffle=False,
                       validation_data=(Validation_X_sequence,
                       Validation_X_sequence),
                       callbacks=callbacks_list)
  

  At the testing, I used other datasets - 7 files- contain normal and abnormal data. I run them separately. The results are different, so when I used the test files for prediction on model 1, I got each file's results. When I run them on model 2, some files' results improved, and other files gave us worse than the previous model1 and so on.

  I need a scientific explanation answer, is that related to data size only?
• Automated Data Extraction from Financial Reports (PDF/Images)

  I am currently researching about products that can help convert unstructured data into tabular format. For instance, I am imagining an application which ingests company finanacial / annual reports, extracts the tables and data points required in a tabular foramt. The entire process should be a batch operation. The machine should be able to learn with time so as to minimize ambiguity. Now the annual reports will vary from one another, the machine should be intelligent enough to strandardzie everything. Obviously some human intervention is required for tranining unknown / low confidence data points. I know there exists SaaS products. Any leads is appreciated.
• How can we detect if a product on an online platform is Fake or Original through AI?

  How can we identify whether a product present online is real or fake? Are there any current technologies which uses AI or other algorithm to detect such products?

  I have tried using Image API but, was unable to identify counterfeit product.
• Is there any kaggle competition or deep learning model for image sequence prediction? (Ex. video next frame prediction)

  I'm searching some information for the image sequence prediction.
  Now I face a problem that predict the future image from the past. (per a day)
  (I just know few of things like CNN-LSTM. But it's used 3~4 years before as i know)
  I wonder to find new approach or model. But it's hard to search for beginner because i don't have any keyword...
  If someone tell me some kaggle competition or model related on the problem, I can study that.