Elasticsearch aggregations - OR in buckets

2021-01-11 05:24 Josef Zamrzla imported from Stackoverflow

Say I have 5 docs:

{
  "owner": "joe",
  "color": "black"
},
{
  "owner": "joe",
  "color": "red"
},
{
  "owner": "joe",
  "color": "blue"
},
{
  "owner": "jack",
  "color": "black"
},
{
  "owner": "jack",
  "color": "white"
}

and aggregations:

{
  aggs: {
    owner: {
      "terms": {
        "field": "owner"
      }
    },
    color: {
      "terms": {
        "field": "color"
      }
    }
  }
}

to aggregate docs by owner and color.

If I run match all query I got:

owner
joe: 3
jack: 2

color
black: 2
red: 1
blue: 1
white: 1

What I want to achieve is: if I filter docs by owner: joe I want to get 3 docs where owner is joe, the color aggregation:

color
black: 1
red: 1
blue: 1

BUT I'd like to get the owner aggregation:

owner
joe: 3 [selected]
jack: 2 [possible to extend]

So get the number of other buckets that can be selected to extend the final result. So something like "OR" between the buckets.

How can I achieve this?

2 answers

answered 2021-01-11 05:40 ESCoder

As far as I can understand, you want to aggregate on the owner as well as color (where the owner is equal to joe) You can use filter aggregation to achieve your required use case -

{
  "size": 0,
  "aggs": {
    "owner": {
      "terms": {
        "field": "owner.keyword"
      }
    },
    "filtered_aggregation": {
      "filter": {
        "term": {
          "owner": "joe"
        }
      },
      "aggs": {
        "color": {
          "terms": {
            "field": "color.keyword"
          }
        }
      }
    }
  }
}

Search Result:

"aggregations": {
    "owner": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "joe",
          "doc_count": 3
        },
        {
          "key": "jack",
          "doc_count": 2
        }
      ]
    },
    "filtered_aggregation": {
      "doc_count": 3,
      "color": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "black",
            "doc_count": 1
          },
          {
            "key": "blue",
            "doc_count": 1
          },
          {
            "key": "red",
            "doc_count": 1
          }
        ]
      }
    }
  }

answered 2021-01-11 05:54 Val

The usual way to achieve this is by using a post_filter. The query below will return:

only joe's colors (using filtered_colors)
only joe's documents (using post_filter)
all owners that you can filter on (using a all_owners)

Query:

POST owners/_search
{
  "aggs": {
    "filtered_colors": {
      "filter": {
        "term": {
          "owner.keyword": "joe"
        }
      },
      "aggs": {
        "color": {
          "terms": {
            "field": "color.keyword"
          }
        }
      }
    },
    "all_owners": {
      "terms": {
        "field": "owner.keyword"
      }
    }
  },
  "post_filter": {
    "term": {
      "owner.keyword": "joe"
    }
  }
}

See also questions close to this topic

Logstash throws [message=>"undefined method `update' for nil:NilClass"] when Elasticsearch index template defined in logstash

I am trying to load data from sql server to elastic search using logstash. For index creation i am using index template that is specified in the config file of logstash.

Output field of logstash.config :

 output{ 
    stdout { codec => json_lines } 
    elasticsearch { 
                    hosts => ["localhost:9200"] 
                                             document_id => "%{[id]}" 
                                             manage_template => true                     
                                             template => "C:/Users/Lich/Documents/logstash-7.10.1-windows-x86_64/logstash-7.10.1/logstash_test.json" 
                                             template_name => "logstash_test" 
                                             template_overwrite => true 
            } 
    }

Error while running logstash :

    [2021-01-25T19:48:28,232][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]} 
[2021-01-25T19:48:28,298][INFO ][logstash.outputs.elasticsearch][main] Using mapping template from {:path=>"C:/Users/Lich/Documents/logstash-7.10.1-windows-x86_64/logstash-7.10.1/logstash_test.json"} 
[2021-01-25T19:48:28,370][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/Users/Lich/Documents/logstash-7.10.1-windows-x86_64/logstash-7.10.1/sampleconf.conf"], :thread=>"#<Thread:0x797ce1a8 run>"} 
[2021-01-25T19:48:28,509][ERROR][logstash.outputs.elasticsearch][main] Failed to install template. {:message=>"undefined method `update' for nil:NilClass", :class=>"NoMethodError", :backtrace=>["C:/Users/Lich/Documents/logstash-7.10.1-windows-x86_64/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:42:in `add_ilm_settings_to_template'", "C:/Users/Lich/Documents/logstash-7.10.1-windows-x86_64/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:15:in `install_template'", "C:/Users/Lich/Documents/logstash-7.10.1-windows-x86_64/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:218:in `install_template'", "C:/Users/Lich/Documents/logstash-7.10.1-windows-x86_64/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:49:in `block in setup_after_successful_connection'"]} 
[2021-01-25T19:48:28,546][INFO ][logstash.outputs.elasticsearch][main] Creating rollover alias <logstash-{now/d}-000001> 
[2021-01-25T19:48:29,463][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.08} 
[2021-01-25T19:48:29,802][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"} 
[2021-01-25T19:48:29,888][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

Index template configuration:

{ 
   "index_patterns":"*-logstash_test-*", 
   "priority":1, 
   "template":{ 
      "settings":{ 
            "analysis":{ 
               "analyzer":{ 
                  "test_autocomplete":{ 
                     "tokenizer":"test_tokenizer" 
                  } 
               }, 
               "tokenizer":{ 
                  "test_tokenizer":{ 
                     "custom_token_chars":"'-", 
                     "max_gram":"3", 
                     "min_gram":"3", 
                     "token_chars":[ 
                         
                     ], 
                     "type":"ngram" 
                  } 
               } 
            }, 
         "mappings":{ 
            "properties":{ 
               "test_id":{ 
                  "type":"text", 
                  "analyzer":"test_autocomplete" 
               } 
            } 
         } 
      } 
   } 
}

Although it fails the index gets created with the data, but the analyzers specified in the settings{..} does not reflect. I am new to ELK stack and not sure on why this is happening.

Elasticsearch Version : 7.10.1

Logstash Version : 7.10.1

Kubernetes event logs to elasticsearch
I'm trying to forward kubernetes-event logs to elasticsearch using fluentd.I currently use fluent/fluentd-kubernetes-daemonset:v1.10.1-debian-elasticsearch7-1.0as container image to forward my application logs to elasticsearch cluster.I've searched enough & my problem is that this image doesn't have enough documentation as to accomplishing this task(i.e; forward kubernetes event related logs).

I've found this plugin from splunk which has desired output but this has overhead like :
- add above plugin's gem to bundler.
- install essential tools like make etc.
- install the plugin .
Sure I can do above steps using init-container, but above operations are adding ~200MB to disk space .I'd like to know if it can be accomplished with smaller footprint or other way.

Any help is appreciated.

Thanks.
Scaling down ElasticSearch: what happens with closed indices?
To scale down ElasticSearch, I usually disable allocation on a node, wait for shards to relocate, and then remove the node, e.g.:
```
curl -X PUT "localhost:9200/_cluster/settings" -d'{
  "transient" : {
    "cluster.routing.allocation.exclude._name" : "node-4"
  }
}'
```
As I understand, open indices' shards will be relocated to other nodes... but what happens to closed indices?

A closed index is, per my understanding, ignored on snapshots... so it makes me think if it will be ignored when setting an allocation exclusion as well.

After excluding allocation to a node, I went to check its disk usage... and it seems like it still has a lot of things there:
```
$ du -hs /usr/share/elasticsearch/nodes/0/
131G    /usr/share/elasticsearch/nodes/0/
```
So, my questions are:
- What happens to closed indices when I exclude a node from allocation?
- How safe it is to do this if I don't want to loose those closed indices (for whatever reason)?
How do I aggregate a bit field in Elasticsearch?
I have a 60-bit (minutes in an hour) field that I would like to aggregate with a bit-wise or operator so that the end result is a bit set if any of the contributing values had that bit set.

The representation of the bit field isn't determined yet, but it would be nice to have it be somewhat compact as there are a lot of records entering the aggregation.

Say we have three documents with a bitfield taking the binary values: 0001, 1001, 1100. A bit-wise or aggregation would combine them to the value 1101 - a bit is true if that bit is set in any value. Much like a sum aggregation but bit-wise operation instead.

I've considered a bit-position array (position present if bit is set) but it gets a little verbose.

Scripted metric aggregation could be possible but I'm a little at a loss of how to implement it.

Scripted metric agg would look something like this broken painless code:
```
"aggs": {
    "profit": {
      "scripted_metric": {
        "init_script": "minagg = 0L", 
        "map_script": "minagg = minagg | minutes",
     }
   }
 }
```
Thanks for looking.

Elastic search query Pagination help on Aggregations

I have tried the below query for the Pagination on Aggregations but not working properly.

I Am getting the error "reason": "[40:7] [terms] unknown field [from], parser not found"

{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "answer.keyword": "UNHANDLED"
          }
        },
        {
          "term": {
            "source.keyword": "QUAL2"
          }
        }
      ]
    }
  },
  "aggs": {
    "MyBuckets": {
      "terms": {
        "field": "question.keyword",
        "order": {
          "_count": "asc"
        },
        "size": "10"
      },
      "aggs": {
        "MyBuckets": {
          "terms": {
            "field": "timestamp",
            "order": {
              "_count": "asc"
            },
            "size": "3",
            "from": 8
          }
        }
      }
    }
  }
}

How to write a elastic search query to get the list of 50 questions with the date and the count
I need to use 2 below MUST conditions while writing the elastic search query

MUST - ("source.keyword": "SONAX1")

MUST - ("answer.keyword": "UNHANDLED")

Required fields ( Questions & timestamp & aggregation count) & SIZE = 50 records needed

My timestamp is in epoch format and while displaying the records need to show in the date format.

Below is the query Tried

{

"query":{ "bool": { "must": { "term": { "answer.keyword": "UNHANDLED" } }, "must": { "term": { "source.keyword": "sonax" } } } }, "aggs": { "MyBuckets": { "terms": { "field": "question.keyword",”timestamp”, "sort":{ "_timestamp": "desc" "_source": { "includes": [ "source":"question.keyword",”timestamp”,

}, "size": "50" } } } }

Below is the errors:
1. Duplicate Key must syntax error
Please check this: some synatx is missing

Related questions

Elasticsearch aggregations - OR in buckets

2 answers

See also questions close to this topic