Elasticsearch Hadoop installation on top of hadoop?

2018-04-17 05:35 Vishal imported from Stackoverflow

I install Hadoop and ELK stack. now i want to install ESHADOOP on a top of hadoop so i can fetch data from elastic search and store in Es-hadoop. please if any one have the idea how to do it tell me?

See also questions close to this topic

Failed to find geo_point field using @Spatial from hibernate search

I need to implement a solution that aims to filter one of my search query using location. You will find here my entity and how I used @Spatial annotation :

@Entity
@Indexed
@Spatial(spatialMode = SpatialMode.RANGE)
@Table(name = "ORGANIZATION", uniqueConstraints = { @UniqueConstraint(columnNames = { "CODE" }) })
public class Organization implements Serializable, FileEntity {

    ...

    @Latitude
    @Column(name = "LATITUDE")
    private Double latitude;

    @Longitude
    @Column(name = "LONGITUDE")
    private Double longitude;

   ...

}

Indexing does not figure any errors, here's the result I found using elasticsearch querying :

GET http://localhost:9201/com.supralog.lexis.model.organization.organization
{
    "com.supralog.lexis.model.organization.organization" : {
        "aliases" : { },
        "mappings" : {
        "com.supralog.lexis.model.organization.Organization" : {
            "properties" : {
                "_hibernate_default_coordinates" : {
                    "properties" : {
                        "lat" : {
                            "type" : "float"
                        },
                        "lon" : {
                            "type" : "float"
                        }
                    }
                },
                ...
            }
        }
    }
}

GET http://localhost:9201/com.supralog.lexis.model.organization.organization/_search?from=0&size=1
{
    "took" : 1,
    "timed_out" : false,
    "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
    },
    "hits" : {
       "total" : 15628,
       "max_score" : 1.0,
       "hits" : [
           {
               "_index" : "com.supralog.lexis.model.organization.organization",
               "_type" : "com.supralog.lexis.model.organization.Organization",
               "_id" : "...",
               "_score" : 1.0,
               "_source" : {
                   ...
                   "_hibernate_default_coordinates" : {
                       "lat" : 49.1886203,
                       "lon" : -0.38740259999997306
                   },
                   ...
               }
           }
       ]
   }
}

After checking indexation looks OK, I tried to query all Organization objects within a given radius of 100km :

    final Coordinates coordinates = Point.fromDegrees(form.getLatitude(), form.getLongitude());
    final String search = StringUtils.join(terms, " ");

    final FullTextSession fullTextSession = Search.getFullTextSession(sessionFactory.getCurrentSession());
    final QueryBuilder queryBuilder = fullTextSession.getSearchFactory().buildQueryBuilder()
            .forEntity(Organization.class).get();

    final org.apache.lucene.search.Query elasticQuery = queryBuilder.spatial().within(100,Unit.KM).ofCoordinates(coordinates).createQuery();

    final FullTextQuery fullTextQuery = fullTextSession.createFullTextQuery(elasticQuery, Organization.class);
    fullTextQuery.setMaxResults(form.getMaximumNumberOfResult());
    fullTextQuery.setProjection(FullTextQuery.THIS, FullTextQuery.SCORE);

And my problem is here, when I try to execute this query, I'm having the following return statement :

Request: POST /com.supralog.lexis.model.organization.organization/_search with parameters {from=0, size=50}
Response: 400 'Bad Request' with body 
{
    "error": {
        "root_cause": [
            {
                "type": "query_shard_exception",
                "reason": "failed to find geo_point field [_hibernate_default_coordinates]",
                "index_uuid": "phOfJTOyRvetHyZrfeUmrA",
                "index": "com.supralog.lexis.model.organization.organization"
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "com.supralog.lexis.model.organization.organization",
                "node": "9DCzSp6kS5KGtMq6tzywzg",
                "reason": {
                    "type": "query_shard_exception",
                    "reason": "failed to find geo_point field [_hibernate_default_coordinates]",
                    "index_uuid": "phOfJTOyRvetHyZrfeUmrA",
                    "index": "com.supralog.lexis.model.organization.organization"
                }
            }
        ]
    },
    "status": 400
}

To fix it, I tried to set a name to @Spatial record, I tried to make my entity implements Coordinates, etc. However I'm always having the same result. It looks like hibernate-search is not indexing my location as a geo_point, reason why it's failing on querying...

Do you have any idea on what I missed in documentation ?

Versions used :: hibernate : 5.3 ; hibernate-search : 5.10 ; elasticsearch : 5.6

Problem with ElasticsearchRepository handling in springBoot application
I have made an entity in spring boot using @Document and @Entity annotation. Code goes like :-

// Code
```
@Entity
@Document(indexName = "content", type = "doc")
public class Content implements Serializable{
/**
*
*/
private static final long serialVersionUID = 1L;
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long contentId;
@Field(type = Date)
@Column(updatable=false, insertable=false)
private LocalDateTime dateAdded;
@Field(type = Date)
@Column(updatable=false, insertable=false)
private LocalDateTime dateModified;

...... With getter and setter basically. Normal POJO basically.
}
```
And i have made the repository using :-

//Code
```
@Repository
public interface ElasticSearch extends ElasticsearchRepository<Content, Long>{

@Query("{"bool": {"must": [{"match": {"authors.name": "?0"}}]}}")
Page findByAuthorsNameUsingCustomQuery(String name, Pageable pageable);

@Query("{"bool": {"must": {"match_all": {}}, "filter": {"term": {"tags": "?0" }}}}")
Page vf(String tag, Pageable pageable);

@Query("{"bool": {"must": {"match": {"authors.name": "?0"}}, "filter": {"term": {"tags": "?1" }}}}")
Page findByAuthorsNameAndFilteredTagQuery(String name, String tag, Pageable pageable);
}
```
But as soon as I am running my spring boot application my system is giving error like :-

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'contentController': Unsatisfied dependency expressed through field 'contentSearchService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'contentSearchImpl': Unsatisfied dependency expressed through field 'contentSearch'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contentSearch': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Failed to create query for method public abstract java.lang.Object org.springframework.data.elasticsearch.repository.ElasticsearchRepository.index(java.lang.Object)! No property index found for type Content! at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:586) ~[spring-beans-5.0.9.RELEASE.jar:5.0.9.RELEASE]

at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90) ~[spring-beans-5.0.9.RELEASE.jar:5.0.9.RELEASE]

I have tried many things. Can someone please help with this.

How to configure Elasticsearch own logging?

I centralize all my systems / applications logs in Elasticsearch and one of the most prolific sources is Elasticsearch itself.

I thought that I could lower the verbosity in /etc/elasticsearch/log4j2.properties to warnings only by changing all the levels there to warn (such as logger.action.level = warn) but I keep on getting the log below at each query.

Where can I disable this kind of operational logs?

{
  "type": "response",
  "@timestamp": "2018-09-21T10:45:35Z",
  "tags": [],
  "pid": 30232,
  "method": "post",
  "statusCode": 200,
  "req": {
    "url": "/logtrail/search",
    "method": "post",
    "headers": {
      "host": "10.200.0.1:5601",
      "connection": "keep-alive",
      "content-length": "587",
      "accept": "application/json, text/plain, */*",
      "origin": "http://10.200.0.1:5601",
      "kbn-version": "6.2.4",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
      "dnt": "1",
      "content-type": "application/json;charset=UTF-8",
      "referer": "http://10.200.0.1:5601/app/logtrail",
      "accept-encoding": "gzip, deflate",
      "accept-language": "fr-FR,fr;q=0.9,en-XA;q=0.8,en;q=0.7,en-US;q=0.6"
    },
    "remoteAddress": "10.200.0.2",
    "userAgent": "10.200.0.2",
    "referer": "http://10.200.0.1:5601/app/logtrail"
  },
  "res": {
    "statusCode": 200,
    "responseTime": 18,
    "contentLength": 9
  },
  "message": "POST /logtrail/search 200 18ms - 9.0B"
}

How to place HDFS file blocks with same / shared partitioning applied for different files / tables on same Data Node

I have two big tables partitioned by date column. They are saved as parquet files in hdfs. Every partition is divided by blocks of 64 MB and replicated 3 times accross the cluster machines. To optimize join operation I want to place the same date partitions on the same machines (any join key value is placed in one date partition only).

In Spark there is Partitioner object which can help to distribute blocks of different RDDs accross the cluster. Its pretty similar to my question but I'm afraid that after saving these RDD's file blocks may be shuffled by hdfs mechanism. Explaned: RDD is Spark instance and df method saveAsTable(...) calls (I suppose) some low-level functions which choose data nodes and replicate the data.

Can anyone help me to know if the blocks of my tables are distributed the right way?

Hadoop sort and group keys differently

I am doing a word count, so the mapper returns key and value pairs

zz 1
zz 1 
b 1
c 1

and my reducer adds them up together

b 1
c 1
zz 2

but I want the keys to be sorted by length (decreasing)

zz 2
b 1
c 1

So I create a Comparator for the word length

public static class LengthComparator extends WritableComparator {
    @Override
    @Override
    public int compare(WritableComparable a, WritableComparable b) {
        String w1 = a.toString();
        String w2 = b.toString();

        if (w1.length == w2.length) return 0;

        return w1.length() > w2.length() ? -1 : 1;
    }
}

then set with

 job.setSortComparatorClass(LengthComparator.class);

now my output is this

 zz 2
 b 2

so i try this

public static class LengthComparator extends WritableComparator {
    @Override
    @Override
    public int compare(WritableComparable a, WritableComparable b) {
        String w1 = a.toString();
        String w2 = b.toString();

        return w1.length() > w2.length() ? -1 : 1;
    }
}

now my output is

 zz 1
 zz 1
 b 1
 c 1

How do I make it such that the keys are grouped by the key but the output is sorted by the length of the word?

Is reducer bottleneck in MR framework

I want to understand what to do in the following case. For example, I have 1TB of text data, and lets assume that 900GB of it is the word "Hello". After each map operation, i will have a collection of key-value pairs of <"Hello",1>.

But as I said, this is a huge collection, 900GB and as I understand , the reducer gets all of it and will crush.

My reducer RAM is of 80GB only.

Will the reducer really crush ?? In other words is reducer the bottleneck of horizontal scaling ?

Related questions

Elasticsearch Hadoop installation on top of hadoop?

See also questions close to this topic