Menu
Search

Forums

Forum Index

October 22, 2021
Gravatar of Adam D RuppePosted by Adam D Ruppe
in reply to drug		Reply
Adam D Ruppe
Gravatar of Adam D Ruppe
Posted in reply to drug


Reply
On Friday, 22 October 2021 at 11:54:44 UTC, drug wrote:
> Me too. One package can have 1000 downloads/month but it can be mechanical dependency performing trivial work.

Yeah, most the top ones are just dependencies of each other.

Also note that "downloads" here is just any time a thing asks for the download url. CI things for certain packages account for 90%+ of these requests.

I kinda wanna track the dependency thing, gonna make a local database....
October 22, 2021
Gravatar of SealabJasterPosted by SealabJaster
in reply to WebFreak001		Reply
SealabJaster
Gravatar of SealabJaster
Posted in reply to WebFreak001


Reply

On Friday, 22 October 2021 at 11:47:09 UTC, WebFreak001 wrote:

>

If you, the person reading this, are motivated in pushing through a search improvement in DUB, you can check that PR for hints where to start, but you should really create a new PR instead and let MongoDB do the work or introduce some other indexed search framework.

Meilisearch has acceptable results, and attempts to fix typos as well. Seems lightweight as well.

I kind of liked the results postgres was giving me as well, but I doubt it does anything about typos. And you have to be a bit specific. And of course dub doesn't use postgres >x3

I wanted to add Elasticsearch/Opensearch into the test as well, but couldn't be bothered since it seemed redundant.

In other words, if anyone's gonna do this, a dedicated search engine would likely be the way forward?

October 22, 2021
Gravatar of ImperatornPosted by Imperatorn
in reply to WebFreak001		Reply
Imperatorn
Gravatar of Imperatorn
Posted in reply to WebFreak001


Reply

On Friday, 22 October 2021 at 11:47:09 UTC, WebFreak001 wrote:

>

On Friday, 22 October 2021 at 11:17:11 UTC, SealabJaster wrote:

>

[...]

that PR is completely different and removes the internal MongoDB search, replacing it with "packageName.canFind(query)" - that is not the place you would want to extend this search into.

[...]

It was mainly done as the current search is unusable as everyone knows.

October 22, 2021
Gravatar of WebFreak001Posted by WebFreak001
in reply to Imperatorn		Reply
WebFreak001
Gravatar of WebFreak001
Posted in reply to Imperatorn


Reply

On Friday, 22 October 2021 at 12:48:19 UTC, Imperatorn wrote:

>

On Friday, 22 October 2021 at 11:47:09 UTC, WebFreak001 wrote:

>

On Friday, 22 October 2021 at 11:17:11 UTC, SealabJaster wrote:

>

[...]

that PR is completely different and removes the internal MongoDB search, replacing it with "packageName.canFind(query)" - that is not the place you would want to extend this search into.

[...]

It was mainly done as the current search is unusable as everyone knows.

I think it's ok for searching for functionality - if you start searching for package names it becomes worse, especially if they are made up words or contain punctuation.

Your improvement is great and I would love to put it in, but the implementation is not good as it is right now:

  • it's getting worse results if you typo or have different tense or form of words
  • it first fetches all the documents into memory and then filters on them (which will take longer and longer the more packages we have)
  • the PR is mixed with unrelated deprecation fixes which should really be done separately (but should definitely be done!)
October 22, 2021
Gravatar of ImperatornPosted by Imperatorn
in reply to WebFreak001		Reply
Imperatorn
Gravatar of Imperatorn
Posted in reply to WebFreak001


Reply

On Friday, 22 October 2021 at 13:43:17 UTC, WebFreak001 wrote:

>

On Friday, 22 October 2021 at 12:48:19 UTC, Imperatorn wrote:

>

[...]

I think it's ok for searching for functionality - if you start searching for package names it becomes worse, especially if they are made up words or contain punctuation.

Your improvement is great and I would love to put it in, but the implementation is not good as it is right now:

  • it's getting worse results if you typo or have different tense or form of words
  • it first fetches all the documents into memory and then filters on them (which will take longer and longer the more packages we have)
  • the PR is mixed with unrelated deprecation fixes which should really be done separately (but should definitely be done!)

Yeah, it's a proof of concept for someone to continue on. More to break the status quo

« First   ‹ Prev
1 2
Next ›   Last »