Process 1TB data in few minutes without using hadoop, spark and elastic search

Is there a solution to process 1TB data in few minutes without using hadoop, spark and elastic search ?

Because i am facing problem while using spark and elastic search.

While using Spark: I am using Yarn. Dr.Who launches many jobs and my actual job which i want to be processed is stucked in accepted phase.

While using elastic search: I am not getting any java reference to code a program. Even though i have tried official JAVA API of elastic.co but it didn't worked some classes are not recognized.

1 answer

  • answered 2018-08-14 18:54 Abhinav

    I can't tell you about Elastic Search but the Yarn problem, I have a resolution for that.

    This should help:

    PROBLEM Customer unable to see logs via Resource Manager UI due to incorrect permissions for the default user dr.who.

    RESOLUTION Customer changed the following property in core-site.xml to resolve the issue. Other values such as hdfs or mapred also resolve the issue. If the cluster is managed by Ambari, this should be added in Ambari > HDFS > Configurations >Advanced core-site > Add Property

    hadoop.http.staticuser.user=yarn
    

    Even if you're not using Ambari, then also you can add this property to core-site.xml.