Forecasting values across categories using Prophet and Python

See also questions close to this topic

• Comparing list values and storing new ones in a separate list

  import csv
  
  with open("DADSA RESIT CWK JULY 2018.csv", newline='') as f:
      r = csv.reader(f)
      database = list(r)
  
  del database[0]
  names = []
  names.append([])
  
  def fillnames(d, n):
      for j in n:
          for i in d:
              if d[i][0] == n[j][0] and d[i][1] == n[j][1]:
                  n[i][2] = n[i][2]+1
              else:
                  names.append([d[i][0], d[i][1], 0])
  
  fillnames(database, names)
  
  for i in names:
      print(i)
  

  The code I have here is me scanning in a csv file into a list. I then want to count how many entries share the same name, by scanning each new name into a separate list, then incrementing the number found every time I find a new one. Every time I run this code it returns "TypeError: list indices must be integers or slices, not list."
• Changing font of a list python

  Say I have some code that makes a list into a 4 by 4 1d array:

  nlist = [2,2,4,8, 0,0,0,0, 0,0,0,0, 0,0,0,0]
  def drawBoard():
      count = 0
      for i in range(16):
          print(nlist[i], end = ' ')
          count += 1
          if count == 4:
              print("")
              count = 0
      print("") 
  drawBoard()
  

  How can I change all the fonts in this list into size 26. I tried doing font = 'times 26' but I don't know where to put it or if that command needs tkinter.
• Get mouse coordinates without clicking in matplotlib

  In a matplotlib plot, how can I continuously read the coordinates of the mouse when it is moved, but without waiting for clicks? This is possible in matlab, and there is a mpld3 plugin to do almost exactly what I want, but I can't see how to actually access the coordinates from it. There is also the package mpldatacursor, but this seems to require clicks. Searching for things like "matplotlib mouse coordinates without clicking" did not yield answers.

  Answers using additional packages such as mpld3 are fine, but it seems like a pure matplotlib solution should be possible.
• Azure aztk workers not executing task

  When executing a spark task using aztk cluster submit, only the driver node is executing the task. If I check cluster status using aztk spark cluster get, only the master is running while the others are idle.

  Additionally by using ssh and viewing spark Web UI, all the workers are visible yet all are unused while the task is running. Even viewing the event graph, it shows that the task is not being split, and viewing the time taken to complete the task, there is no difference between 3 nodes and 5 nodes in the cluster.

  Is there any reason why this is happening. As a data store I am retrieving the data from wasb Azure storage.
• Collecting Spark RDD returns different results everytime

  I am working with Spark's Mllib using PySpark, with Jupyter Notebook. I trained a random forest classifier for a multiclass problem

  model = RandomForest.trainClassifier(transformed_df, numClasses=8,seed=123, categoricalFeaturesInfo={}, numTrees=150, featureSubsetStrategy="auto", impurity="gini",maxDepth=30)
  

  Next, I used the trained classifier to provide predictions for the dataset

  predictions = model.predict(transformed_df.map(lambda x: x.features))
  

  Now, when I collect the predictions using "predictions.collect()", the results are different every time I run the cell. I hoped that specifying the random seed in the model would help, but there was no difference.

  Any suggestions?
• On my Mac, hadoop 3.1.0 finds it native libraries, but spark 2.3.1 does not

  Intro

  I know that the answer in 99% of cases to this error message:

  WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  

  Is simply "It's just a warning, don't worry about it", and then sometimes followed by "Just download the libraries, compile them and point HADOOP_HOME to this folder and add $HADOOP_HOME/bin/native to your LD_LIBRARY_PATH"

  That's what I did, but I'm still getting the error, and after two days of googling I'm starting to feel there's something really interesting to learn if I manage to fix this, There is currently a strange behaviour that I do not understand, hopefully we can work through this together.

  Ok, so here's what's up:

  Hadoop finds the native libraries

  Running a hadoop checknative -a gives me this:

  dds-MacBook-Pro-2:~ Rkey$ hadoop checknative -a
  2018-07-15 16:18:25,956 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
  2018-07-15 16:18:25,959 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
  2018-07-15 16:18:25,963 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
  Native library checking:
  hadoop:  true /usr/local/Cellar/hadoop/3.1.0/lib/native/libhadoop.dylib
  zlib:    true /usr/lib/libz.1.dylib
  zstd  :  false 
  snappy:  true /usr/local/lib/libsnappy.1.dylib
  lz4:     true revision:10301
  bzip2:   false 
  openssl: false build does not support openssl.
  ISA-L:   false libhadoop was built without ISA-L support
  2018-07-15 16:18:25,986 INFO util.ExitUtil: Exiting with status 1: ExitException
  

  There are some errors here, which might be the cause, but most importantly for now this line is present:

  hadoop:  true /usr/local/Cellar/hadoop/3.1.0/lib/native/libhadoop.dylib
  

  When I start my hadoop cluster, this is how it looks:

  dds-MacBook-Pro-2:~ Rkey$ hstart
  Starting namenodes on [localhost]
  Starting datanodes
  Starting secondary namenodes [dds-MacBook-Pro-2.local]
  Starting resourcemanager
  Starting nodemanagers
  

  No warnings. I downloaded the hadoop source and built it myself. Before I did that, there where "Cannot find native library"-warnings on starting hadoop.

  However, spark does not find the native libraries

  This is how it looks when I run pyspark:

  dds-MacBook-Pro-2:~ Rkey$ pyspark
  Python 3.7.0 (default, Jun 29 2018, 20:13:53) 
  [Clang 8.0.0 (clang-800.0.42.1)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  2018-07-15 16:22:22 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  Setting default log level to "WARN".
  To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
  Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /__ / .__/\_,_/_/ /_/\_\   version 2.3.1
        /_/
  

  This is where our old friend makes a re-appearence:

  WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  

  I find this very strange, because I know for a fact it uses the same hadoop that I can start on my own without any warnings. There are no other hadoop installations on my computer.

  Clarifications

  I downloaded the non-hadoop version of apache-spark from their website called "Pre-build with user-provided Apache Hadoop". This was then put in my Cellar-folder just because I did not want to re-link everything.

  As for variables, this is my ~/.profile

  export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home
  export OPENSSL_ROOT_DIR=/usr/local/Cellar/openssl/1.0.2o_2
  
  export PYSPARK_PYTHON=python3
  export PYSPARK_DRIVER_PYTHON=python3
  export HADOOP_HOME=/usr/local/Cellar/hadoop/3.1.0
  export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
  export SPARK_HOME=/usr/local/spark
  
  export PATH=$SPARK_HOME/bin:$PATH
  
  export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
  
  alias hstart="$HADOOP_HOME/sbin/start-dfs.sh;$HADOOP_HOME/sbin/start-yarn.sh"
  alias hstop="$HADOOP_HOME/sbin/stop-dfs.sh;$HADOOP_HOME/sbin/stop-yarn.sh"
  

  And here are my additions to spark-env.sh:

  export SPARK_DIST_CLASSPATH=$(hadoop classpath)
  
  export LD_LIBRARY_PATH=/usr/local/Cellar/hadoop/3.1.0/lib/native/:$LD_LIBRARY_PATH
  

  This is how the folder /usr/local/Cellar/hadoop/3.1.0/lib/native looks:

  

  The question

  How is it that hadoop can start locally without giving a warning that it's missing its libraries, and goes through the checknatives -a command showing that it finds the native libraries, but when the same hadoop is launched through pyspark I'm suddenly given this warning again?

  Update 16/7

  I recently made a discovery. The standard version of this classic error message looks like this:

  WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  

  This is actually different, as my error message instead says NativeCodeLoader:60, with 60 instead of 62. This points towards my theory that it is not really the hadoop libraries missing, but some native libraries that hadoop is using that are missing. This is why hadoop can launch without warnings, but pyspark that likely tries to use more native libraries from hadoop, launches with warnings.

  This is still just a theory, and until I remove all warnings from the checknative -a call I do not know.

  Update 15/7

  Currently trying to remove the "WARN bzip2.Bzip2Factory:"-warning from hadoop checknative -a, perhaps this might remove the warning when launching pyspark.
• Running into "Py4JJavaError" while converting list to Dataframe using

           Summary: Running into "Py4JJavaError" while converting list to Dataframe using
  

  Python, Jupyter notebook Key: SPARK-24612 URL: https://issues.apache.org/jira/browse/SPARK-24612 Project: Spark Issue Type: Question Components: PySpark Affects Versions: 2.3.1 Environment: >python --version

  Python 3.6.5 :: Anaconda, Inc.

  java -version

  java version "1.8.0_144" Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

  jupyter --version

  4.4.0

  conda -V

  conda 4.5.4

  spark-2.3.0-bin-hadoop2.7 Reporter: A B

  rdd=sc.parallelize([[1,"Alice",50],[2,"Bob",80]])

  rdd.collect() [[1,"Alice",50],[2,"Bob",80]]

  However, when i run df=rdd.toDF() i run into the following error: Any help resolving this error is greatly appreciated.

  full link here http://mail-archives.apache.org/mod_mbox/spark-issues/201806.mbox/%3CJIRA.13167277.1529535154000.212161.1529535180018@Atlassian.JIRA%3E
• Displaying HTML content with R in Databricks

  I'm trying to figure out how to display HTML content in Databricks using htmlwidgets and other related libraries.

  I've found this example which would have answered my questions, except that following these steps did not help me. In fact, I can't even see the HTML plots on this very page.

  Here is my own notebook where I tried to recreate something similar but with no luck.

  If you can't see the notebook, here is a brief summary of what happens there:

  1. I install pandoc and htmlwidgets
  2. I load htmlwidgets and change one of the functions in the htmlwidgets package, as recommended by Databricks
  3. I install plotly and leaflet which I would like to use to generate my plots
  4. I load all these libraries and build the plots (1 using ggplotly and 1 using leaflet)

  I do not get any errors and it looks like databricks recognises that a plot should be generated, but instead of a plot, I see nothing but blank, white space. Leaflet is a bit different, because when I generate the plot, I'm prompted to download the resulting HTML file which displays the correct graphics, but it's not showing it in the databricks notebook. I do not get a single error throughout this process.

  Any ideas on how to fix this?

  Many thanks!
• How to detect Databricks environment programmatically

  I'm writing a spark job that needs to be runnable locally as well as on Databricks.

  The code has to be slightly different in each environment (file paths) so I'm trying to find a way to detect if the job is running in Databricks. The best way I have found so far was to look for a "dbfs" directory in the root dir and if it's there then assume it's running on Databricks. This doesn't feel like the right solution. Does anyone have a better idea?
• See different parameters' prediction errors from Spark MLlib's CrossValidator (in Scala)

  I've recently gotten into using Spark's MLlib and I've had some success using Spark's CrossValidator to tune my model's parameters. But I think it would be helpful to start visualizing the model selection process.

  I eventually want to create graphs like this:

  But since I'm going to use Databricks, I only need a DataFrame with an error and a parameter column to start, and can then call DataBricks's display() function on it to visualize it.

  So if we start from MLlib's CrossValidation example snippet here:
  (for simplicity, I removed hashingTF.numFeatures from the ParamGrid)

  import org.apache.spark.ml.Pipeline
  import org.apache.spark.ml.classification.LogisticRegression
  import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
  import org.apache.spark.ml.feature.{HashingTF, Tokenizer}
  import org.apache.spark.ml.linalg.Vector
  import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}
  import org.apache.spark.sql.Row
  
  // Prepare training data from a list of (id, text, label) tuples.
  val training = spark.createDataFrame(Seq(
    (0L, "a b c d e spark", 1.0),
    (1L, "b d", 0.0),
    (2L, "spark f g h", 1.0),
    (3L, "hadoop mapreduce", 0.0),
    (4L, "b spark who", 1.0),
    (5L, "g d a y", 0.0),
    (6L, "spark fly", 1.0),
    (7L, "was mapreduce", 0.0),
    (8L, "e spark program", 1.0),
    (9L, "a e c l", 0.0),
    (10L, "spark compile", 1.0),
    (11L, "hadoop software", 0.0)
  )).toDF("id", "text", "label")
  
  // Configure an ML pipeline, which consists of three stages: tokenizer, hashingTF, and lr.
  val tokenizer = new Tokenizer()
    .setInputCol("text")
    .setOutputCol("words")
  val hashingTF = new HashingTF()
    .setInputCol(tokenizer.getOutputCol)
    .setOutputCol("features")
  val lr = new LogisticRegression()
    .setMaxIter(10)
  val pipeline = new Pipeline()
    .setStages(Array(tokenizer, hashingTF, lr))
  
  // We use a ParamGridBuilder to construct a grid of parameters to search over.
  // With 2 values for lr.regParam, this grid will have 2 parameter settings for CrossValidator to choose from.
  val paramGrid = new ParamGridBuilder()
    .addGrid(lr.regParam, Array(0.1, 0.01))
    .build()
  
  // We now treat the Pipeline as an Estimator, wrapping it in a CrossValidator instance.
  // This will allow us to jointly choose parameters for all Pipeline stages.
  // A CrossValidator requires an Estimator, a set of Estimator ParamMaps, and an Evaluator.
  // Note that the evaluator here is a BinaryClassificationEvaluator and its default metric
  // is areaUnderROC.
  val cv = new CrossValidator()
    .setEstimator(pipeline)
    .setEvaluator(new BinaryClassificationEvaluator)
    .setEstimatorParamMaps(paramGrid)
    .setNumFolds(2)  // Use 3+ in practice
    .setParallelism(2)  // Evaluate up to 2 parameter settings in parallel
  
  // Run cross-validation, and choose the best set of parameters.
  val cvModel = cv.fit(training)
  
  // Prepare test documents, which are unlabeled (id, text) tuples.
  val test = spark.createDataFrame(Seq(
    (4L, "spark i j k"),
    (5L, "l m n"),
    (6L, "mapreduce spark"),
    (7L, "apache hadoop")
  )).toDF("id", "text")
  
  // Make predictions on test documents. cvModel uses the best model found (lrModel).
  cvModel.transform(test)
    .select("id", "text", "probability", "prediction")
    .collect()
    .foreach { case Row(id: Long, text: String, prob: Vector, prediction: Double) =>
      println(s"($id, $text) --> prob=$prob, prediction=$prediction")
    }
  

  If I want to find the prediction error of different parameter settings, how would I get that from (I assume) either cv (the CrossValidator) or cvModel (the CrossValidatorModel?)

  Bonus question: How could I do the same thing for 2 sets of parameters, like if we added cross validation on lrModel.elasticNetParam?
• R crashes when prophet model run in a loop

  R crashes when prophet model run in a loop and the termination of the loop is not in a particular fashion i.e. it crashes R randomly.

  for(j in 1:length(unique_code_train))
  {
  train1<- filter(train, CODE==unique_code_train[j])
  train1<-arrange(train1, BILLDATE)
  train1<-data.table(train1)
  names(train1)<-c("ds","y") # billdate= ds , sum(saleamt)= y
  m<-prophet(train1 ,changepoint.range = 1.0, seasonality.mode = 
  'multiplicative', holidays = holidays)
  future <- make_future_dataframe(m, periods = 26)
  forecast <- predict(m, future)
  forecast$yhat<-as.integer(forecast$yhat)
  }
  
• Removing outliers by filtering values in R

  i have a dataframe like this :

           ds        y
  1   2015-12-31 35.59050
  2   2016-01-01 28.75111
  3   2016-01-04 25.53158
  4   2016-01-06 17.75369
  5   2016-01-07 29.01500
  6   2016-01-08 29.22663
  7   2016-01-09 29.05249
  8   2016-01-10 27.54387
  9   2016-01-11 28.05674
  10  2016-01-12 29.00901
  11  2016-01-13 31.66441
  12  2016-01-14 29.18520
  13  2016-01-15 29.79364
  14  2016-01-16 30.07852
  

  i'm trying to create a loop that remove the rows which values in the 'ds' column are above 34 or below 26, because there is where my outliers are:

  for (i in grupo$y){if (i < 26) {grupo$y[i] = NA}}
  

  i tried this to remove those below 26, i don't get any errors, but those rows won't go.

  Any suggestions about how to remove those outliers??

  Thanks in advance