Elasticsearch Hadoop installation on top of hadoop?

2018-04-17 05:35 Vishal imported from Stackoverflow

I install Hadoop and ELK stack. now i want to install ESHADOOP on a top of hadoop so i can fetch data from elastic search and store in Es-hadoop. please if any one have the idea how to do it tell me?

See also questions close to this topic

AWS Serverless Lambda Function: No module by name elasticsearch error
I am new to using Lambda in AWS and currently trying to execute some of the sample scripts given in different blogs to test how this all works in a server less environment. One of the examples that I am trying to work on is to create a lambda function that gets triggered whenever an object is added to s3 bucket and stores the metadata of the object in elasticsearch.

For this purpose, I have created an elasticsearch domain, s3 bucket and started writing the lambda function.

I am using the inline code editor given in the lambda console, and have selected python3.6 as my runtime. The following is the code:
```
from __future__ import print_function
from pprint import pprint
import boto3
import json
from elasticsearch import Elasticsearch, RequestsHttpConnection


def lambda_handler(event, context):
    imageid =  event['Records'][0]['s3']['object']['key']
    print(imageid)
```
This code is yet to push data to elastic search, which I will write later. However, I am getting the 'no module found` error for elasticsearch import.
The following is the error log when i test the above code.
```
Response:
{
  "errorMessage": "Unable to import module 'lambda_function'"
}

Request ID:
"a26249c3-00e2-11e9-83b0-ffc7719455cf"

Function Logs:
START RequestId: a26249c3-00e2-11e9-83b0-ffc7719455cf Version: $LATEST
Unable to import module 'lambda_function': No module named elasticsearch

END RequestId: a26249c3-00e2-11e9-83b0-ffc7719455cf
REPORT RequestId: a26249c3-00e2-11e9-83b0-ffc7719455cf  Duration: 0.42 ms   Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 25 MB  
```
I was under the impression the server less environment by default load all the requisite libraries as they are invoked. However, when I looked up for more info on this error, I found some of them specified that source should be specified in zip file, while others indicate that the zip file is When you use the ZipFile property to specify your function's source code and that function interacts.

I find this confusing. Do I have to install the elasticsearch module in a serverless environment or load it as a zip file? How do I get the lambda function talk to elastic search.
Streaming aws access logs from s3 to elasticsearch

All elb and alb access logs are streamed to one s3 bucket directly, I'm trying to write a lamda function to parse through each folder, get the zipped log file, unzip it and stream the contents to elasticsearch. This has to be done for multiple folders as there is one folder per elb/alb in the same bucket. Following the aws example gets logs from the root level, but i need to parse though all folders. any example on how to do this will be appreciated, thanks!

Spring - Elasticsearch and coreNLP dependency problem

I'm developing a web application using JHipster (Spring boot + Angular). Also in the app i have spring-data-elasticsearch:2.1.11.RELEASE.

I want to add coreNLP package using maven. This package includes also org.apache.lucene version(4.10.3) while elasticsearch uses lucene(5.5.4). I tried to view dependecy tree using mvn dependency:tree -Dverbose -Dincludes=org.apache.lucene commnand.

The output is:

 \- org.elasticsearch:elasticsearch:jar:2.4.6:compile (version managed from 2.4.0)
[INFO] |        +- (org.apache.lucene:lucene-core:jar:5.5.4:compile - omitted for conflict with 4.10.3)
[INFO] |        +- org.apache.lucene:lucene-backward-codecs:jar:5.5.4:compile
[INFO] |        |  \- (org.apache.lucene:lucene-core:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        +- (org.apache.lucene:lucene-analyzers-common:jar:5.5.4:compile - omitted for conflict with 4.10.3)
[INFO] |        +- (org.apache.lucene:lucene-queries:jar:5.5.4:compile - omitted for conflict with 4.10.3)
[INFO] |        +- org.apache.lucene:lucene-memory:jar:5.5.4:compile
[INFO] |        |  \- (org.apache.lucene:lucene-core:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        +- org.apache.lucene:lucene-highlighter:jar:5.5.4:compile
[INFO] |        |  +- (org.apache.lucene:lucene-analyzers-common:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        |  +- (org.apache.lucene:lucene-core:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        |  +- (org.apache.lucene:lucene-join:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        |  +- (org.apache.lucene:lucene-memory:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        |  \- (org.apache.lucene:lucene-queries:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        +- (org.apache.lucene:lucene-queryparser:jar:5.5.4:compile - omitted for conflict with 4.10.3)
[INFO] |        +- org.apache.lucene:lucene-suggest:jar:5.5.4:compile
[INFO] |        |  +- (org.apache.lucene:lucene-analyzers-common:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        |  +- (org.apache.lucene:lucene-core:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        |  +- org.apache.lucene:lucene-misc:jar:5.5.4:compile
[INFO] |        |  \- (org.apache.lucene:lucene-queries:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        +- org.apache.lucene:lucene-join:jar:5.5.4:compile
[INFO] |        |  +- (org.apache.lucene:lucene-core:jar:5.5.4:compile - omitted for duplicate)
[INFO] |        |  \- org.apache.lucene:lucene-grouping:jar:5.5.4:compile
[INFO] |        \- org.apache.lucene:lucene-spatial:jar:5.5.4:compile
[INFO] |           +- (org.apache.lucene:lucene-core:jar:5.5.4:compile - omitted for duplicate)
[INFO] |           +- (org.apache.lucene:lucene-misc:jar:5.5.4:compile - omitted for duplicate)
[INFO] |           +- (org.apache.lucene:lucene-queries:jar:5.5.4:compile - omitted for duplicate)
[INFO] |           \- org.apache.lucene:lucene-spatial3d:jar:5.5.4:compile
[INFO] \- edu.stanford.nlp:stanford-corenlp:jar:3.9.1:compile
[INFO]    +- org.apache.lucene:lucene-queryparser:jar:4.10.3:compile
[INFO]    |  +- (org.apache.lucene:lucene-core:jar:4.10.3:compile - omitted for duplicate)
[INFO]    |  +- (org.apache.lucene:lucene-queries:jar:4.10.3:compile - omitted for duplicate)
[INFO]    |  \- org.apache.lucene:lucene-sandbox:jar:4.10.3:compile
[INFO]    +- org.apache.lucene:lucene-analyzers-common:jar:4.10.3:compile
[INFO]    |  \- (org.apache.lucene:lucene-core:jar:4.10.3:compile - omitted for duplicate)
[INFO]    +- org.apache.lucene:lucene-queries:jar:4.10.3:compile
[INFO]    |  \- (org.apache.lucene:lucene-core:jar:4.10.3:compile - omitted for duplicate)
[INFO]    \- org.apache.lucene:lucene-core:jar:4.10.3:compile

After some searching in the stack-overflow, I found this. So I followed this technique and i modified my pom.xml like this:

modified pom.xml

<!--https://mvnrepository.com/artifact/edu.stanford.nlp/stanford-corenlp-->
        <dependency>
            <groupId>org.apache.lucene</groupId>
            <artifactId>lucene-core</artifactId>
            <version>5.5.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.lucene</groupId>
            <artifactId>lucene-analyzers-common</artifactId>
            <version>5.5.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.lucene</groupId>
            <artifactId>lucene-queries</artifactId>
            <version>5.5.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.lucene</groupId>
            <artifactId>lucene-queryparser</artifactId>
            <version>5.5.4</version>
        </dependency>
        <dependency>
            <groupId>edu.stanford.nlp</groupId>
            <artifactId>stanford-corenlp</artifactId>
            <version>3.9.2</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.lucene</groupId>
                    <artifactId>lucene-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.lucene</groupId>
                    <artifactId>lucene-analyzers-common</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.lucene</groupId>
                    <artifactId>lucene-queries</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.lucene</groupId>
                    <artifactId>lucene-queryparser</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

After that, my code compiles and starts but when I try to use CoreNLP library in a Service, it throws me

java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3$ExtendableMessageOrBuilder

So, i added to the pom.xml

<!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->
            <dependency>
                <groupId>com.google.protobuf</groupId>
                <artifactId>protobuf-java</artifactId>
                <version>2.6.1</version>
                <scope>compile</scope>
            </dependency>

After that, the exception did not disappeared, but when I was executing for the second time the same request that gave me the above exception(without rerun) , the error becomes this (awkward):

java.lang.NoClassDefFoundError: edu/stanford/nlp/pipeline/CoreNLPProtos$Document

What else can I do?

my App Class

import edu.stanford.nlp.simple.*;
...
public String transformSentence(String sentence) {
    log.debug("Request to transform sentence: {}", sentence);
    Document document = new Document(sentence);
    Sentence sent = document.sentence(0);

    List<String> tags = sent.posTags();
    System.out.println("Available tags: " + tags);
}

Hadoop : HDFS space quota

Does there any way to check if there is any space quota applied on a specific HDFS directory ?

I cannot find any kind of command that give such information here

Archiving Flat Files in HDFS to Gzip

We have 50 TB of file in HDFS and we have planned to archive these files to Gzip format. These files are available in different folders/partitions.

1. So what is the best possible way to archive it?
2. Is it possible to archive the partitions along with the files?
3. Is there any other formats that can be prefered in place of Gzip?

Hbase config override for invoking shell

I have hbase 1.1.x installed on Mapr M7. Log property fs.mapr.trace is enabled as debug mode. Now I want to override the property to open hbase shell without changing config file. Hbase shell -D it is available in 1.4x version and higher. I have tried below command in hbase shell but it’s not working

@shell.hbase.configuration.setChar(‘fs.mapr.trace’, INFO) NameError: uninitialized constant INFO

I have tried with .setString and .settoString but getting error

NoMethodError

Hbase shell -d is able to enable debug mode but I’m not getting anywhere where I can open it in info mode

Related questions

Elasticsearch Hadoop installation on top of hadoop?

See also questions close to this topic