suggestions on fulltext search or already existing search algorithms

See also questions close to this topic

• create string r using string p and q, C++, Data Structures, Strings

  In this question we have given 3 strings and we nee dto count possible number of ways to create string r using string p and q

  
• Writing multithreaded code to generate incremental index

  I want to write a code in following scenario : we want to design an invoice generator.this invoice generator also generates an incremental invoice no .Also note that for some reasons we dont use something like sql Auto Increment here and we want to use C# (I wrote the increment logic). since several users can send requests at a time it means that we dont have to give the same invoice no to them (I used lock for it and I wrote it).

  Now i have written increment logic. and i take care of multiple users requests with lock . and I wrote CRUD for mongodb database.

  My problem is that my increent logic is naive. each time i find maximum and i increment by one which is not scalable. and another problem is that when we delete a record we don't want the invoice no to be repeated and since we find max if the deleted record had the max value then the invoice no would be duplicate and this is something we don't want to have.

  I really don't know how to handle the problem. specially in case of deletion should i store max values in a file or should i change increment logic completely?

  i appreciate any help
• Efficiently and constantly reorganize searchable data based on access frequency

  I want to be able to organize data for efficiency and constantly update the order of that data based on frequency of access, relevancy, and accuracy.

  For example, given a list L:

  L = [1, 'apple', 'dog', $2.99, 'Jim', 'carrot', 'peach', 5, $7.49, 'Susan', 'Cat']
  

  If a user searches for the integer 1, then peach, then 5 I'd like the list to reorganize itself based on all previous searches like so:

  L = [1, 5, $2.99, $7.49, 'peach', 'apple', 'carrot', 'dog', 'Jim', 'Susan', 'Cat']
  

  Giving priority to any exact matches, any data with numbers as those were searched more frequently, then fruit, food, and everything else maintaining its previous order.

  I've considered using an intensity weight as the key inside a dictionary instead:

  L = {10:1, 9:5, 8:$2.99, 7:$7.49, 6:'peach', 4:'apple', 3:'carrot', 2:'dog', 1:'Jim', 0:'Susan', 0:'Cat'}
  

  And then updating the keys with different weights.

  What will likely work better for my needs is something dictionary/json related as my dataset is greater than 500,000 data points and growing. Storing the data in a different manner that doesn't require loading the entire dataset into memory for every search would also be helpful.
• Setting A $_GET Value From A Previously Defined Variable - PHP

  I have a search form in the header of every page on a site that submits info to another page when undertaking a search query via the form's action attribute, namely a page called search.php and it takes the value of the search <input> field and stores it as a variable called searchQuery. Using PHP this searches a MySQL database. All of this works as expected.

  I would like the value of the search input field to be placed in the URL of the page the search is processed on, again namely search.php.

  I understand from other questions on SO that you can't add variables to a form's action attribute. If I add the key of the query string in the action attribute, the form still works OK:

  action="search.php?keyword="
  

  From this I thought I would be able to then set the $_GET value on the search.php page by using:

  if(isset($_POST['search-submit'])) {
  
      // make this variable available before HTML output
      $searchQuery = htmlspecialchars($_POST['search']);
  
      $_GET['keyword'] = $searchQuery;
  }
  

  This however does not work, although it doesn't throw any errors in my PHP error logs.

  Because I'm running the script on specific page allocated by the action attribute, I don't think I need to use a $_SESSION variable?

  HTML Form

  Here is the form that is on every page in the header

  <form action="search.php?keyword=" method="POST">
      <label for="search">Enter Search</label>
      <input type="search" name="search" id="search">
      <button type="submit" name="search-submit" id="search-submit">Search</button>
  </form>
  
• What are the approaches for recognizing search groups inside a single search field?

  There is a text field and the user is expected to enter a multiple-word search phrase, which can be words from the following search groups: 1) brand, 2) model, 3) color, 4) condition, 5) year. Below are examples of searches with division into groups:

  • Early Rider Belter Trail 16 2020 New Grey: -------------------------> 1) Early Rider 2) Belter Trail 16 3) Grey 4) New 5) 2020
  • De Rosa Planet 2 9.8 2010 Used Beige: ----------------------------->1) De Rosa 2) Planet 2 9.8 3) Beige 4) Used 5) 2010
  • De Rosa King XS 2019 Previous year model Orange: ----------->1) De Rosa 2) King XS 3) Orange 4) Previous year model 5) 2019
  • Dedacciai Strada Scuro Rs Black Edition Used Anthracite: -----> 1) Dedacciai Strada 2) Scuro Rs Black Edition 3) Anthracite 4) Used 5)

  Not all search groups may be present in a single search field. Can be, for example, only <model and year> or <model and color> or <model, condition and year>.

  What technical means (algorithms, services, approaches) exist, besides neural networks, that allow you to recognize a single string search field and divide the string into named groups if these named groups are known in advance (brand, model, condition, color)?

  Important note:

  1. We have formed registers for some search groups: color, condition.
  2. There is no problem to recognize the year in a single search string and free the string from it.
  3. The table with models is quite large: it contains tens of thousands of records. Searching for words inside <brand, model> on such an amount I see suboptimal.
• AWS Neptune FTS returns IOException on every federated queries to OpenSearch

  I am trying to implement full text search on AWS Neptune (engine 1.0.4.2) with AWS OpenSearch.

  A GET amazon_neptune/_search on OpenSearch returns:

  {
    "took": 5,
    "timed_out": false,
    "_shards": {
      "total": 2,
      "successful": 2,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": {
        "value": 100000,
        "relation": "gte"
      },
      "max_score": 1.0,
      "hits": [
        {
          "_index": "amazon_neptune",
          "_type": "_doc",
          "_id": "1234567890",
          "_score": 1.0,
          "_source": {
            "entity_id": "aeeA6GHI6ZvK4zOP",
            "document_type": "rdf-resource",
            "predicates": {
              "https://schema.org/description": [
                {
                  "value": "value"
                }
              ]
            }
          }
        }
      ]
    }
  }
  

  Now, when trying to execute federated query using this SPARQL:

  PREFIX foaf: <http://xmlns.com/foaf/0.1/>
  PREFIX neptune-fts: <http://aws.amazon.com/neptune/vocab/v01/services/fts#>
  PREFIX hint: <http://aws.amazon.com/neptune/vocab/v01/QueryHints#>
  
  SELECT ?res WHERE {
    SERVICE neptune-fts:search {
      neptune-fts:config neptune-fts:endpoint 'vpc-{url}.eu-central-1.es.amazonaws.com' .
      neptune-fts:config neptune-fts:queryType 'term' .
      neptune-fts:config neptune-fts:field 'Neptune#fts.document_type' .
      neptune-fts:config neptune-fts:query "rdf-resource" .
      neptune-fts:config neptune-fts:return ?res .
    }
  }
  

  ...or Lucene queries like:

  SELECT * WHERE {
    SERVICE neptune-fts:search {
      neptune-fts:config neptune-fts:endpoint 'https://vpc-{url}.eu-central-1.es.amazonaws.com' .
      neptune-fts:config neptune-fts:queryType 'simple_query_string' .
      neptune-fts:config neptune-fts:query "predicates.\\schema\\description.value:value" .
      neptune-fts:config neptune-fts:return ?res .
    }
  }
  

  No matter what query I use, I end up with this error:

  {
    "code": "BadRequestException",
    "detailedMessage": "An IOException happened while fetching data from ES",
    "requestId": "{id}"
  }
  

  I have tried different variations of the SPARQL federated query, I always end up with the "An IOException happened while fetching data from ES".

  So, what's going on here?

  Thanks in advance.
• How to do partial text search in Cloudsearch

  I have an indexed field of type "text" called obj_title in cloudsearch. It's value is

  queryevents@abc.com
  

  If I search for ts or *ts* or events or *events*. I get no results. I get results if I search for string queryevents though.

  I have tried Simple/Lucene/DisMax query parsers. What do I need to change in order to support Cloudsearch returning results on ts or *ts* or events or *events*

  I don't see option to have NgramFilter or EdgeNgramTokenizer field type in Cloudsearch. What am I missing?
• sqlite3 fts5 contentless or content=external table, how store and read a non-FTS column value

  I am working with an SQLite3 database. And I want to create an FTS5 content-less (content="") or content="external table" table to save space and prevent duplication of data (which occurs if I don't use a content-less or external-content table).

  My scenario =>

  tableA (textColumn, idColumn)
  tableB (textColumn, idColumn)
  tableFTS5 (textColumn, textIdColumn, tableNameColumn)
  

  What I do =>

  I insert data into tableA and tableB. Then I read all rows from tableA and insert into tableFTS5 table as

  tableFTS5.textColumn = tableA.textColumn, 
  tableFTS5.textIdColumn = tableA.idColumn,
  tableFTS5.tableNameColumn = "tableA"
  

  I repeat this for tableB also with tableFTS5.tableNameColumn = "tableB".

  What I need is =>

  I want to query tableFTS5 for some text and I want to retrieve just tableFTS5.textIdColumn and tableFTS5.tableNameColumn. With these two column values in hand, I can retrieve the original data from tableA or tableB.

  Is it possible to retrieve these columns from tableFTS5 if tableFTS5 is a content-less or content="external" table, maybe with some triggers or any other method ?

  My source of info about this Link to SQLite3 FTS5 docs
• MySQL fulltext index: how to disable stopwords

  Many users are going to search my InnoDB MySQL database for terms like "The Gap", "BT group" or "IG Index" etc. I don't want to annoy them with "zero results".

  My code is like this:

  SELECT COUNT(1) FROM `myTable` WHERE MATCH(tagline) AGAINST('+the +gap' IN BOOLEAN MODE)
  

  One approach would be to remove all stopwords from the queries through PHP, but then I would need to find the full list of stopwords.

  I decided to just remove stopwords, following the steps here:

  I added these lines to my.cnf file:

  innodb_ft_min_token_size=2
  ft_min_word_len=2
  ft_stopword_file = ""
  

  I emptied my table, ran this SQL in PHPMyAdmin: SET @@SESSION.innodb_ft_enable_stopword = 'OFF'; and rebuilt the indexes and added new data.

  I am still getting zero results for short terms like "the" and so on.
• MySQL Match Against: easiest way to deal with stopwords

  I get zero results from this query:

  SELECT COUNT(1) FROM `myTable` WHERE MATCH(tagline) AGAINST(' +IT professional' IN BOOLEAN MODE)
  

  I get 92 from this:

  SELECT COUNT(1) FROM `myTable` WHERE `tagline` LIKE '%IT professional%'
  

  I prefer the first query because MATCH against an index is 80% faster with my setup. Obviously "IT" is a stopword and causing problems. What is the easiest and most expedient way of dealing with this problem if I want the functionality of the second query and the speed of the first one?