How do I get a list of all packages (Github URL) available at code.dlang.org?

I could download individual pages with wget --recursive code.dlang.org but I wonder if there is a better solution.

On Sunday, 9 September 2018 at 00:20:04 UTC, void wrote:
> How do I get a list of all packages (Github URL) available at code.dlang.org?
>
> I could download individual pages with wget --recursive code.dlang.org but I wonder if there is a better solution.

https://code.dlang.org/packages/index.json

https://code.dlang.org/api/packages/[package name]/info

On Sunday, 9 September 2018 at 01:30:14 UTC, Neia Neutuladh wrote:
> On Sunday, 9 September 2018 at 00:20:04 UTC, void wrote:
>> How do I get a list of all packages (Github URL) available at code.dlang.org?
>>
>> I could download individual pages with wget --recursive code.dlang.org but I wonder if there is a better solution.
>
> https://code.dlang.org/packages/index.json
>
> https://code.dlang.org/api/packages/[package name]/info


Thanks.

I downloaded 100 packages from code.dlang.org and search for "*doc*" and "*example*"

The results:

13 packages with "*doc*"
41 packages with "*example*"
55 packages with neither
8 packages with both

On Saturday, September 8, 2018 8:36:26 PM MDT void via Digitalmars-d wrote:
> On Sunday, 9 September 2018 at 01:30:14 UTC, Neia Neutuladh wrote:
> > On Sunday, 9 September 2018 at 00:20:04 UTC, void wrote:
> >> How do I get a list of all packages (Github URL) available at
> >> code.dlang.org?
> >>
> >> I could download individual pages with wget --recursive code.dlang.org but I wonder if there is a better solution.
> >
> > https://code.dlang.org/packages/index.json
> >
> > https://code.dlang.org/api/packages/[package name]/info
>
> Thanks.
>
> I downloaded 100 packages from code.dlang.org and search for "*doc*" and "*example*"
>
> The results:
>
> 13 packages with "*doc*"
> 41 packages with "*example*"
> 55 packages with neither
> 8 packages with both

What would you expect that to tell you? ddoc doesn't require any kind of doc folder (though some projects would have one for custom ddoc files that then affect how the documentation looks), and usually, the best way to handle examples is to use ddoc-ed unittest blocks. So, while in some cases, it would be appropriate to have some sort of additional documentation or examples separate from the source code, in general, there isn't much need for it. So, I don't know what information anyone would expect to get from whether or not a project has any folders with doc or example in their name.

- Jonathan M Davis

On Sunday, 9 September 2018 at 06:32:39 UTC, Jonathan M Davis wrote:
> What would you expect that to tell you? ddoc doesn't require any kind of doc folder (though some projects would have one for custom ddoc files that then affect how the documentation looks), and usually, the best way to handle examples is to use ddoc-ed unittest blocks. So, while in some cases, it would be appropriate to have some sort of additional documentation or examples separate from the source code, in general, there isn't much need for it. So, I don't know what information anyone would expect to get from whether or not a project has any folders with doc or example in their name.
>
> - Jonathan M Davis

It would tell you who thought their projects were complex enough to need separate prose documentation and separate examples, and used the most obvious directory names.

Which is potentially interesting. It's a rather narrow result, though.

On Sunday, 9 September 2018 at 06:32:39 UTC, Jonathan M Davis wrote:
> On Saturday, September 8, 2018 8:36:26 PM MDT void via Digitalmars-d wrote:
>> On Sunday, 9 September 2018 at 01:30:14 UTC, Neia Neutuladh wrote:
>> > On Sunday, 9 September 2018 at 00:20:04 UTC, void wrote:
>> >> [...]
>> >
>> > https://code.dlang.org/packages/index.json
>> >
>> > https://code.dlang.org/api/packages/[package name]/info
>>
>> Thanks.
>>
>> I downloaded 100 packages from code.dlang.org and search for "*doc*" and "*example*"
>>
>> The results:
>>
>> 13 packages with "*doc*"
>> 41 packages with "*example*"
>> 55 packages with neither
>> 8 packages with both
>
> What would you expect that to tell you? ddoc doesn't require any kind of doc folder (though some projects would have one for custom ddoc files that then affect how the documentation looks), and usually, the best way to handle examples is to use ddoc-ed unittest blocks. So, while in some cases, it would be appropriate to have some sort of additional documentation or examples separate from the source code, in general, there isn't much need for it. So, I don't know what information anyone would expect to get from whether or not a project has any folders with doc or example in their name.
>
> - Jonathan M Davis

Script updated now searches for the string "unittest" on package directory (*.d files only).

Result:
48 packages with "unittest"

That means there is ~= 50% chance an user should read the library source in order to use it. To be fair I should manually check if README.md is decent enough.

On Monday, 10 September 2018 at 00:56:37 UTC, void wrote:
> On Sunday, 9 September 2018 at 06:32:39 UTC, Jonathan M Davis wrote:
>>>
>>> I downloaded 100 packages from code.dlang.org and search for "*doc*" and "*example*"
>
> Script updated now searches for the string "unittest" on package directory (*.d files only).
>
> Result:
> 48 packages with "unittest"

Ok, this is really a scary result.
Is really more than half the packages around not using unittests?!?
I can't believe it...

On Monday, 10 September 2018 at 00:56:37 UTC, void wrote:
> On Sunday, 9 September 2018 at 06:32:39 UTC, Jonathan M Davis wrote:
>> On Saturday, September 8, 2018 8:36:26 PM MDT void via Digitalmars-d wrote:
>>> On Sunday, 9 September 2018 at 01:30:14 UTC, Neia Neutuladh wrote:
>>> > On Sunday, 9 September 2018 at 00:20:04 UTC, void wrote:
>>> >> [...]
>>> >
>>> > https://code.dlang.org/packages/index.json
>>> >
>>> > https://code.dlang.org/api/packages/[package name]/info
>>>
>>> Thanks.
>>>
>>> I downloaded 100 packages from code.dlang.org and search for "*doc*" and "*example*"
>>>
>>> The results:
>>>
>>> 13 packages with "*doc*"
>>> 41 packages with "*example*"
>>> 55 packages with neither
>>> 8 packages with both
>>
>> What would you expect that to tell you? ddoc doesn't require any kind of doc folder (though some projects would have one for custom ddoc files that then affect how the documentation looks), and usually, the best way to handle examples is to use ddoc-ed unittest blocks. So, while in some cases, it would be appropriate to have some sort of additional documentation or examples separate from the source code, in general, there isn't much need for it. So, I don't know what information anyone would expect to get from whether or not a project has any folders with doc or example in their name.
>>
>> - Jonathan M Davis
>
> Script updated now searches for the string "unittest" on package directory (*.d files only).
>
> Result:
> 48 packages with "unittest"
>
> That means there is ~= 50% chance an user should read the library source in order to use it. To be fair I should manually check if README.md is decent enough.

I'm sorry, but it's not even close to accurate, because some libraries has documents on additional websites that has examples and no examples directly in the source code using "standard unittests" - This is true for most big libraries / frameworks.

You can't really measure such a thing accurately.

On Monday, 10 September 2018 at 11:20:55 UTC, bauss wrote:
> I'm sorry, but it's not even close to accurate, because some libraries has documents on additional websites that has examples and no examples directly in the source code using "standard unittests" - This is true for most big libraries / frameworks.

Having checked out recent versions of all dub projects...

1300 dub projects with at least one release.

631 projects have example or test files / folders.

723 have unittests. (Or at least mention unittests somewhere in their source code. A `version (unittest) static assert(0)` would have counted. Or a comment saying that this project really should have at least one unittest. It's the thought that counts, right?)

452 have a file or directory with 'test' in the name.

352 have a file or directory with 'example' in the name.

953 have unittest or tests or examples.

That's about 19 out of every 26 packages.

> You can't really measure such a thing accurately.

It's quite possible that every package that lacks an example or test project instead has an external test suite. That's very unlikely, though. It's rather likely that about 25% of dub packages with at least one release have no tests.

Probably quite a lot of the untested packages have only one release and no readme.