Replies: 4 comments 9 replies
-
|
I'm also considering indexing only the readibility extracted content's text instead of every text that appear in a document. This cuts all the header/footer/sidebar contents, so the remaining text is relevant and super clean. However in rare cases it can exclude valid content as well. |
Beta Was this translation helpful? Give feedback.
-
|
When I read the first post of this discussion, I immediately thought of Mozilla's Readability.js :) I'm glad to see that you already implemented it in 078548b, nice! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, in my tests it produces drastically less false-positives (reindex required to test it). I'm also rewriting the query building from scratch and I think there is a big improvement in the quality of the results after these changes. Everybody, please test it and tell how it feels. |
Beta Was this translation helpful? Give feedback.
-
|
I don't like that search expects whole words. In search engines like DuckDuckGo or Google this normally happens only when I put the word in quotes, but Hister splits my query into words and then searches for whole words it seems, also quoting phrases did not work. I briefly tried https://pagefind.app/ and like how it works, e.g. I can search for I see that Hister is using https://blevesearch.com/, maybe making it work more like existing search engines work is a matter of configuring tokenizers. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe focus on making English search good first? The problem is not language detection, but that English search is not good yet and requires whole words. Without complicated language detection, check for |
Beta Was this translation helpful? Give feedback.
-
|
My suggested solution would make english work out of box if you add it as a new language. I would not make english default because not all the users want to search in english. I think it's a better default to provide a language-agnostic basic indexer and everybody can configure their preferred languages on top of it, no? What is your opinion @link2xt ? |
Beta Was this translation helpful? Give feedback.
-
|
I don't suggest defaulting to English, but if the page says it is in English I think it should be indexed using specialized tokenizer, so out of the box Hister will index technical documentation etc. properly. Most users are likely some sort of programmers, likely Go programmers, at this point, just because of where the app is discussed and how it is installed, so focusing on indexing technical documentation, visited stackoverflow pages and so on makes more sense. And ideally I hope the app does not require manual configuration to make any language work, and especially English should not default to basic tokenizer with the default configuration. |
Beta Was this translation helpful? Give feedback.
-
|
So if I understand correctly, you suggest dynamically adding new language indexes when we detect a document sent to the indexer that is in a language that we don't already have index for, right? This is possible to implement and sounds like a user-friendly and transparent solution. |
Beta Was this translation helpful? Give feedback.
-
|
@link2xt i've added language specific indexers. I'm still evaluating the results, but I can confirm that english stemming works. |
Beta Was this translation helpful? Give feedback.
-
[...]
The underlying problem is : generated HTML is not database/indexer friendly by nature (it's been done for programmers to please humans) Two other (obvious) answers to ugly-HTML problem (aside from readability.js):
The benefit of APIs is that once authentication is resolved, structured data resolves both the problem of ugly-HTML and crawler-blocking. |
Beta Was this translation helpful? Give feedback.
-
That's why we should write content specific extractors. I've already started to work on an architecture that makes it possible to extract semantically more meaningful data from specific websites using custom domain specific content extraction.
What endpoint would solve these problems? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
The number of irrelevant results for a given search query increases as the index grows. I'd like to prevent (or at least reduce) this behavior.
First, we need to identify the root-causes. As I see:
Please share your observations about other possible issues that can reduce search quality.
Second, we need solutions for these issues:
What do you think about these solutions? Do you have any better or complementary ideas?
Beta Was this translation helpful? Give feedback.
All reactions