Keyerror when processing pandas dataframe

See also questions close to this topic

• Any efficient way to compare two dataframes and append new entries in pandas?

  I have new files which I want to add them to historical table, before that, I need to check new file with historical table by comparing its two column in particular, one is state and another one is date column. First, I need to check max (state, date), then check those entries with max(state, date) in historical table; if they are not historical table, then append them, otherwise do nothing. I tried to do this in pandas by group-by on new file and historical table and do comparison, if any new entries from new file that not in historical data, then add them. Now I have issues to append new values to historical table correctly in pandas. Does anyone have quick thoughts?

  My current attempt:

  import pandas as pd 
    
  src_df=pd.read_csv("https://raw.githubusercontent.com/adamFlyn/test_rl/main/src_df.csv")
  hist_df=pd.read_csv("https://raw.githubusercontent.com/adamFlyn/test_rl/main/historical_df.csv")
  picked_rows = src_df.loc[src_df.groupby('state')['yyyy_mm'].idxmax()]
  

  I want to check picked_rows in hist_df where I need to check by state and yyyy_mm columns, so only add entries from picked_rows where state has max value or recent dates. I created desired output below. I tried inner join or pandas.concat but it is not giving me correct out. Does anyone have any ideas on this?

  Here is my desired output that I want to get:

  import pandas as pd
  desired_output=pd.read_csv("https://raw.githubusercontent.com/adamFlyn/test_rl/main/output_df.csv")
  

  
• How to bring data frame into single column from multiple columns in python

  I have data format in these multiple columns. So I want to bring all 4 columns of data into a single column.

  YEAR Month pcp1 pcp2 pcp3 pcp4              
  1984    1   0   0   0   0
  1984    2   1.2 0   0   0
  1984    3   0   0   0   0
  1984    4   0   0   0   0
  1984    5   0   0   0   0
  1984    6   0   0   0   1.6
  1984    7   3   3   9.2 3.2
  1984    8   6.2 27.1    5.4 0
  1984    9   0   0   0   0
  1984    10  0   0   0   0
  1984    11  0   0   0   0
  1984    12  0   0   0   0
  
• Exclude Japanese Stopwords from File

  I am trying to remove Japanese stopwords from a text corpus from twitter. Unfortunately the frequently used nltk does not contain Japanese, so I had to figure out a different way.

  This is my MWE:

  import urllib
  from urllib.request import urlopen
  import MeCab
  import re
  
  # slothlib
  slothlib_path = "http://svn.sourceforge.jp/svnroot/slothlib/CSharp/Version1/SlothLib/NLP/Filter/StopWord/word/Japanese.txt"
  sloth_file = urllib.request.urlopen(slothlib_path)
  
  # stopwordsiso
  iso_path = "https://raw.githubusercontent.com/stopwords-iso/stopwords-ja/master/stopwords-ja.txt"
  iso_file = urllib.request.urlopen(iso_path)
  stopwords = [line.decode("utf-8").strip() for line in iso_file]
  
  stopwords = [ss for ss in stopwords if not ss==u'']
  stopwords = list(set(stopwords))
  
  text = '日本語の自然言語処理は本当にしんどい、と彼は十回言った。'
  tagger = MeCab.Tagger("-Owakati")
  tok_text = tagger.parse(text)
  
  ws = re.compile(" ")
  words = [word for word in ws.split(tok_text)]
  if words[-1] == u"\n":
      words = words[:-1]
  ws = [w for w in words if w not in stopwords]
  
  print(words)
  print(ws)
  

  Successfully Completed: It does give out the original tokenized text as well as the one without stopwords

  ['日本語', 'の', '自然', '言語', '処理', 'は', '本当に', 'しんどい', '、', 'と', '彼', 'は', '十', '回', '言っ', 'た', '。']
  ['日本語', '自然', '言語', '処理', '本当に', 'しんどい', '、', '十', '回', '言っ', '。']
  

  ---

  There is still 2 issues I am facing though:

  a) Is it possible to have 2 stopword lists regarded? namely iso_file and sloth_file ? so if the word is either a stopword from iso_file or sloth_file it will be removed? (I tried to use line 14 as stopwords = [line.decode("utf-8").strip() for line in zip('iso_file','sloth_file')] but received an error as tuple attributes may not be decoded

  b) The ultimate goal would be to generate a new text file in which all stopwords are removed.

  I had created this MWE

  ### first clean twitter csv
  import pandas as pd
  import re
  import emoji
  
  df = pd.read_csv("input.csv")
  
  def cleaner(tweet):
      tweet = re.sub(r"@[^\s]+","",tweet) #Remove @username 
      tweet = re.sub(r"(?:\@|http?\://|https?\://|www)\S+|\\n","", tweet) #Remove http links & \n
      tweet = " ".join(tweet.split())
      tweet = ''.join(c for c in tweet if c not in emoji.UNICODE_EMOJI) #Remove Emojis
      tweet = tweet.replace("#", "").replace("_", " ") #Remove hashtag sign but keep the text
      return tweet
  df['text'] = df['text'].map(lambda x: cleaner(x))
  df['text'].to_csv(r'cleaned.txt', header=None, index=None, sep='\t', mode='a')
  
  ### remove stopwords
  
  import urllib
  from urllib.request import urlopen
  import MeCab
  import re
  
  # slothlib
  slothlib_path = "http://svn.sourceforge.jp/svnroot/slothlib/CSharp/Version1/SlothLib/NLP/Filter/StopWord/word/Japanese.txt"
  sloth_file = urllib.request.urlopen(slothlib_path)
  
  #stopwordsiso
  iso_path = "https://raw.githubusercontent.com/stopwords-iso/stopwords-ja/master/stopwords-ja.txt"
  iso_file = urllib.request.urlopen(iso_path)
  stopwords = [line.decode("utf-8").strip() for line in iso_file]
  
  stopwords = [ss for ss in stopwords if not ss==u'']
  stopwords = list(set(stopwords))
  
  with open("cleaned.txt",encoding='utf8') as f:
      cleanedlist = f.readlines()
      cleanedlist = list(set(cleanedlist))
  
  tagger = MeCab.Tagger("-Owakati")
  tok_text = tagger.parse(cleanedlist)
  
  ws = re.compile(" ")
  words = [word for word in ws.split(tok_text)]
  if words[-1] == u"\n":
      words = words[:-1]
  ws = [w for w in words if w not in stopwords]
  
  print(words)
  print(ws)
  

  While it works for the simple input text in the first MWE, for the MWE I just stated I get the error

  in method 'Tagger_parse', argument 2 of type 'char const *'
  Additional information:
  Wrong number or type of arguments for overloaded function 'Tagger_parse'.
    Possible C/C++ prototypes are:
      MeCab::Tagger::parse(MeCab::Lattice *) const
      MeCab::Tagger::parse(char const *)
  

  for this line: tok_text = tagger.parse(cleanedlist) So I assume I will need to make amendments to the cleanedlist?

  I have uploaded the cleaned.txt on github for reproducing the issue: [txt on github][1]

  Also: How would I be able to get the tokenized list that excludes stopwords back to a text format like cleaned.txt? Would it be possible to for this purpose create a df of ws? Or might there even be a more simple way?

  Sorry for the long request, I tried a lot and tried to make it as easy as possible to understand what I'm driving at :-)

  Thank you very much! [1]: https://gist.github.com/yin-ori/1756f6236944e458fdbc4a4aa8f85a2c
• Is there a way to use mutual information as part of a pipeline in scikit learn?

  I'm creating a model with scikit-learn. The pipeline that seems to be working best is:

  1. mutual_info_classif with a threshold
  2. PCA
  3. LogisticRegression

  I'd like to do them all using sklearn's pipeline object, but I'm not sure how to get the mutual info classification in. For the second and third steps I do:

  pca = PCA(random_state=100)
  lr = LogisticRegression(random_state=200)
  pipe = Pipeline(
      [
          ('dim_red', pca),
          ('pred', lr)
      ]
  )
  

  But I don't see a way to include the first step. I know I can create my own class to do this, and I will if I have to, but is there a way to do this within sklearn?
• Slicing an image into color based layers with Sklearn

  I have an image. I applied KMeans color clustering on this image so I need to present only purple clusters on image and orange clusters on different image. How do I need to do that?
• How to project a new data row onto PCA space using dudi.mix in R?

  I have a mixed dataset (comprising continuous, ordinal and nominal variables) that is high-dimensional (with more variables than rows). I want to perform a mixed data PCA using the dudi.mix() function in the R package ade4. After PCA, I want to project a new supplementary row onto the PCA space, i.e. find its coordinates in the PCA coordinate system. I tried the suprow() function in ade4 but it gives me the following error message: “Not yet implemented for 'dudi.mix'. Please use 'dudi.hillsmith'.” I don’t want to use the dudi.hillsmith() function because I think it only allows for mixed continuous and nominal variables, but my dataset comprises continuous, nominal and ordinal variables, and my understanding is that dudi.mix() is the correct function to use in this case.

  Is there an alternative way how I can project a new row onto the PCA space generated by dudi.mix()?

  Below an example:

  # load ade4 package
  library(ade4)
  
  # a high-dimensional mixed dataset with 11 rows and 13 variables
  dat <- data.frame(
  a=as.numeric(c(2.5,0.5,2.2,1.9,3.1,2.3,2.0,1.0,1.5,1.1,3.4)),
  b=as.numeric(c(2.4,0.7,2.9,2.2,3.0,2.7,1.6,1.1,1.6,0.9,3.1)),
  c=as.numeric(c(1.3,1.1,2.4,3.1,2.2,1.3,1.5,1.8,1.1,0.5,3.8)),
  d=as.numeric(c(1.9,0.9,2.1,2.3,2.8,1.9,1.9,1.3,2.9,0.8,2.9)),
  e=as.numeric(c(2.2,1.2,2.5,2.9,1.9,3.1,2.1,0.9,1.8,0.9,2.8)),
  f=as.factor(c(0,0,0,0,1,0,1,1,1,0,1)),
  g=as.factor(c(0,1,0,0,1,0,1,0,1,0,0)),
  h=as.factor(c(1,1,1,1,0,1,0,0,0,1,1)),
  i=as.factor(c(1,0,0,0,0,1,0,1,0,0,0)),
  j=as.ordered(c(0,1,0,2,3,4,0,1,2,4,2)),
  k=as.ordered(c(1,2,1,3,4,4,1,2,2,3,3)),
  l=as.ordered(c(0,1,1,2,3,2,0,1,1,3,1)),
  m=as.ordered(c(0,0,1,2,1,2,2,1,0,2,1)))
  
  # first 10 rows are used for PCA
  dat.1 <- dat[1:10,]
  
  # the 11s row should be projected onto the PCA space
  dat.2 <- dat[11,]
  
  # pca on dat.1 with 9 kept axes (i.e. number of rows - 1)
  pca.res <- dudi.mix(df=dat.1, scann=FALSE, nf = 9)
  
  # my attempt to project dat.2 onto pca.res fails
  suprow(x=pca.res, Xsup=dat.2)
  
• Seurat - cannot plot the same dimplot again

  I am trying to rewrite the code of this paper: https://doi.org/10.1038/s42003-020-0837-0
  I have written the code step-by-step based on the instructions mentioned in the methods section. But after clustering, for plotting the clusters by dimplot, I receive a dissimilar plot compared to the same plot in the paper.
  I wonder what is the problem? I have tailored every parameter to receive the same plot but it hasn't worked yet.
  Graph of the paper
  My graph
  Please help me to solve this issue.