Elasticsearch Hadoop installation on top of hadoop?

See also questions close to this topic

• Running Kibana 6.x.x from source

  I'm trying to run Kibana by directly downloading it and running from the source. I downloaded v6.2.4 from their official site, updated the kibana.yml file to point to the right ES cluster and ran ./bin/kibana but it's consistently throwing an error.

  $ ./kibana
  FATAL CLI ERROR Error: Cannot find module './$data'
      at Function.Module._resolveFilename (module.js:547:15)
      at Function.Module._load (module.js:474:25)
      at Module.require (module.js:596:17)
      at require (internal/module.js:11:18)
      at Object.<anonymous> (/home/kibana632/usr/share/kibana/node_modules/ajv/lib/ajv.js:10:23)
      at Module._compile (module.js:652:30)
      at Module._extensions..js (module.js:663:10)
      at Object.require.extensions.(anonymous function) [as .js] (/home/kibana632/usr/share/kibana/node_modules/babel-register/lib/node.js:152:7)
      at Module.load (module.js:565:32)
      at tryModuleLoad (module.js:505:12)
      at Function.Module._load (module.js:497:3)
      at Module.require (module.js:596:17)
      at require (internal/module.js:11:18)
      at Object.<anonymous> (/home/kibana632/usr/share/kibana/node_modules/schema-utils/dist/validateOptions.js:15:12)
      at Module._compile (module.js:652:30)
      at Module._extensions..js (module.js:663:10)
  
  

  I tried reinstalling node_modules but no help. I also tried running kibana v6.3.2 but it throws the same error.

  I'm using centOS 7.4 and ES v6.2.4
• Understanding Azure SQL Server External Tables

  We are trying to create a cross-database query using Azure's preview Elastic Query. So we will be creating an External Table to make these queries happen.

  Unfortunately, I have some apprehension about how the queries will be executed. I don't want a query or stored procedure to fail at run-time because the database connection fails. I just don't understand how the External Tables work.

  Azure's External Table docs has good information on how to query and create the table. I just can't find information that specifically spells out how the data exists.

  Oracle's version of external tables are just flat files that are referenced. SQL*Loader loads data from external files into tables of an Oracle database. I couldn't find any documentation about Azure doing the same. (Is it implied that they are the same? Is that a stupid question?)

  If it is this way (external flat files), when the external table gets updated, does SQL Server update the flat files so our external table stays up to date? Or will I have to delete/create the link again every time I want to run the query for up to date information?
• Elasticsearch - This search doesn't return the correct results after upgrading from 6.7 to 7.0

  {  
    "query":{  
      "constant_score":{  
        "filter":{  
          "bool":{  
            "should":{  
              "terms":{  
                "field_a":[  
                  "value1", "value2"
                ]
              }
            },
            "must":{  
              "term":{  
                "field_b":"value"
              }
            }
          }
        }
      }
    }
  }
  

  This search was supposed to return the results containing value1 or value2 in field_a and value in field_b.

  So similar to this MySQL query:

  SELECT * FROM table WHERE field_a IN ('value1', 'value2') AND field_b = value
  

  After the upgrade it will return all the results where field_b = value. The first part of the query is completely ignored.
  Any suggestions for a fix?
• How to install hadoop reusable on a local network?

  For a multi-node image cluster on an institute, we have sevaral laptops and machines and we want to create a hadoop cluster with hbase on the top for indexing the data/images.

  I have tried some VMare and docker solutions, but the most tutorials are deprecated and very complicated. The sandboxes are not recommended for a productional usage.

  I have now, 3 macbook pros with 16gb ram, 2 win pcs with 8gb ram.

  I want to take a macbook as a masternode and the other machines as slaves / datanodes.

  What's the best way for us to end up with a cluster where we can read and write very large image files? The cluster will set up in a local network.

  Thanks in advice.
• What is the flow of classes for Hadoop to check for stragglers and speculate?

  From my limited knowledge of Hadoop, I understand that the TaskTracker is in charge of Map and Reduce tasks. I am attempting to modify the decision to speculate slow tasks by (1) checking for stragglers, (2) identifying the reason tasks are straggling, and then (3) deciding whether to speculate slow tasks.

  I know that you can modify limits such as duration and number of speculative tasks by modifying parameters such as mapreduce.map.speculative and mapreduce.reduce.speculative. However, I wanted to modify the method of speculating stragglers.

  I'm poring over the java classes in the source code to find where all this happens but having a hard time with the immensity of the code base. Primarily, I'm looking at classes in the 'mapreduce' directory (though I believe mapred is similar, just outdated) of the code located in the GitHub:

  https://github.com/apache/hadoop/tree/6d0bffe17eadedd60d4599427248b0db4a7c5502/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce

  Is this the right folder to look in? Any help pointing to the right classes for this progression would be greatly appreciated!!
• Run MapReduce natively on Kubernetes

  Kubernetes(k8s) is now becomming popular in modern computing resource management. Some frameworks(Apache Spark, Tensorflow...) provide native support to run on k8s. However, Hadoop-mapreduce seems to be an exception and the community is not interested in this issue. Some people deploy YARN on k8s and submit MR jobs to YARN-on-k8s, but this is not a graceful usage of k8s, since the advantages of k8s are not utilized. Is there any effort out there to run MR natively on k8s, that is, replace YARN with k8s completely?