February 26, 2013
On 02/26/2013 04:59 AM, Dicebot wrote:
> I agree with all but small comment on unit tests : current approach
> makes it really easy to start adding tests for projects that do not have
> them and this is huge. So having "unittest" blocks themselves is really
> a success feature. Tightly coupling handling of this blocks to compiler
> is an issue though.

Couldn't have said it better myself.
February 26, 2013
On Tuesday, 26 February 2013 at 19:53:11 UTC, Walter Bright wrote:
> On 2/25/2013 11:56 PM, foobar wrote:
>> DDoc isn't part of the language but rather part of the compiler, nevertheless it
>> has its downsides.  [...]
>> unittest is worse,
>
> I think you're missing something gigantic. Before D had ddoc, the documentation for Phobos was TERRIBLE - it was mostly missing, and the rest would describe something that had no resemblance to what the code did. Adding Ddoc completely revolutionized this. It's like night and day. Sure, you can pick at Ddoc's flaws all day, but without Ddoc, the Phobos documentation would have remained utter s**t.
>
> Yes, one could use Doxygen. One could hope an up-to-date version exists on all the platforms D is on. One could nag people to use it. One could argue with people who wanted to use a different doc generator. And one could look at typical C and C++ projects, which use no documentation generator at all, and pretty much have no documentation or have documentation as bad as the pre-Ddoc Phobos docs.
>
> Having Ddoc always there, always installed, always up to date, with literally zero effort, tips the balance. It gets used. It raised the bar on what is acceptable D code - it looks wrong without Ddoc documentation. By tipping the balance I mean it *revolutionized* D code.
>

All of the above describes the benefits of having standardized documentation and I agree with that. That has nothing to do with DDoc's specific design compared to other similar efforts. A quick survey of languages shows that Ruby, Python, Java, C#, and many others all have the same benefits but non has the doc generator built into the compiler/vm with all the problems this entails.

> The same goes for unittest. How many C/C++ projects have you run across that have unit tests? Again, yes, you can use 3rd party tools (of which there are a plethora). You can try to use multiple libraries that use different unit test frameworks. You can look at Phobos before unittest and see that it was pretty much completely untested.
>
> Unittest in the language, always there, always installed, zero effort, completely changed the game. I'm very pleased at the depth and breadth of unittests in Phobos. I have no doubt that would not have happened without unittest.
>
> Sure, you can pick all day at the flaws of unittest, but you'd be missing the point - without builtin unittest, there'd be nothing to pick at, because people would not have unit tests.
>

Same as above. You compare again to C++ and ignore the provably successful models of _many_ other languages. Ruby for instance really shines in this regard as its community is very much oriented towards TDD. Java has such a successful model with its JUnit that it inspired a whole bunch of clones for  other languges and you completely ignore this. Instead you discuss the design of a new car based on experiences of horseback riders.

>
>> Additional such problems - the AA issue which has been going own for years now.
>> The endless discussions regarding tuples.
>> It seems that D strives to bloat the language with needless features that really
>> should have been standardized in the library and on the other hand tries to put
>> in the library things that really ought to be built into the language to benefit
>> from proper integration and syntax.
>
> A little history is in order here. AA's were built in to the language from the beginning, a result of my experience with how incredibly useful they were in javascript. This was many years before D had templates. There was no other way at the time to implement them in a nice manner (try doing it in C, for example).
>
> D's improving generics has enabled them to be redone as library features.
>

I'm familiar with the history of AAs in D and how they came to be this horrible mess. Yet, templates in D are ancient news by now and the problem hadn't been fixed and not due to lack of effort. The problem is again - applying common c++ wisdom and trying to maintain inconsistent semantics.


>
>> The latest case was the huge properties debate and its offshoots regarding ref
>> semantics which I didn't even bother participate in. Bartosz developed an
>> ownership system for D to address all the safety issues raised by ref *years
>> ago* and it was rejected due to complexity. Now, Andrei tries to achieve similar
>> safety guaranties by giving ref the semantics of borrowed pointers. It all seems
>> to me like trying to build an airplane without wings cause they are too complex.
>> Rust on the other hand already integrated an ownership system and is already far
>> ahead of D's design. D had talked about macros *years ago* and rust already
>> implemented them.
>
> Bartosz' ownership system was intended to support multithreaded programming. It was and still is too complicated. I've been working on another design which should serve the purpose and will need nearly zero effort from the programmer and it won't break anything. There was some discussion last fall on the n.g. about it.
>
>
>>> We do have a significantly better D culture than the C++ one. For example, C++
>>> relies heavily and unapologetically on convention for writing correct, robust
>>> code. D eschews that, and instead is very biased towards mechanical verification.
>> I call bullshit. This is an half hearted intention at best.
>> @safe has holes in it,
>
> Yes, and those are bugs, and we have every intention of fixing all of them.
>
>
>> integers has no overflow checks,
>
> This has been discussed ad nauseum. To sum up, adding overflow checks everywhere would seriously degrade performance. Yet you can still have overflow checking integers if you build a library type to do it. See std.halffloat for an example of how to do it. It fits in with your suggestion that things that can be done in the library, should be done in the library.

Yes, but this conflicts with your statement of intention towards verification machinery for safety. Of course I can implement whatever I need myself but what ensures safety is the fact that the default is safe and the language forces me to be explicit when I choose to sacrifice safety for other benefits. You see, *defaults matter*. I can use an option type in D and still get null pointer segfaults whereas in Rust I cannot get a null value without an explicit option type which I am forced to check.
Another case in point - both Germany and Austria have an option to sign an organ donor card when getting a driver's license. Germany requires to tick the box if you want to join the program, Austria requires to tick the box if you do *not* want to join. Austria has a much higher percentage of organ donors.

>
>> ref also has holes,
>
> Yes, and we are actively working to fix them.
>
>
>> Not only D has null pointer bugs but they also cause segfaults.
>
> D now has all the features to create a library type NotNull!T, which would be a pointer type that is guaranteed to be not null.

irrelevant. See previous comment.

>
>
>>>> In fact there are many such "not c++"
>>>> features in D and which is why I find other languages such as rust a *much*
>>>> better design and it evolves much faster because it is designed in terms of -
>>>> what we want to achieve, how best to implement that.
>>>
>>> How does rust handle this particular issue?
>
> I presume rust does not have an official answer to the debug conditional issue and leaves it up to the user?

I'm not sure rust has such a feature as it is much more functional in style than D. If I'm not mistaken, "debug" in Rust is used as one of the logging macros.
February 26, 2013
On 2/26/13 3:51 PM, foobar wrote:
> On Tuesday, 26 February 2013 at 10:59:41 UTC, Dicebot wrote:
>> I agree with all but small comment on unit tests : current approach
>> makes it really easy to start adding tests for projects that do not
>> have them and this is huge. So having "unittest" blocks themselves is
>> really a success feature. Tightly coupling handling of this blocks to
>> compiler is an issue though.
>
> Again, this is a completely superfluous feature. D already has
> annotations (took only several years to convince Walter to add them)
> which are more flexible and much better suited for this.
>
> @unittest // <- this is a unit-test function
> void mySuperDuperTestFunction(...);
>
> There is no benefit in having all those special case features in the
> language which have all sorts of integration issues yet deny the
> usefulness of a more general solution generally accepted in the
> programming world, successfully used in many languages and thus also
> familiar to programmers coming from those languages.

Agreed, but it does happen often that a language feature is later superseded by a generalization thereof.

Andrei
February 26, 2013
On 2/26/2013 2:10 PM, foobar wrote:
> All of the above describes the benefits of having standardized documentation and
> I agree with that. That has nothing to do with DDoc's specific design compared
> to other similar efforts. A quick survey of languages shows that Ruby, Python,
> Java, C#, and many others all have the same benefits but non has the doc
> generator built into the compiler/vm with all the problems this entails.

Building ddoc into the compiler means it has access to the semantic information that compiler provides, and it uses that information. If it is not built in to the compiler, then the options are:

1. require the user to type the information in twice
2. add parsing and semantic analysis capability to the doc generator

I find (1) to be an unacceptable user experience, and (2) to be not viable given our limited resources.

BTW, Javadoc apparently can only generate HTML. As Andrei has demonstrated, Ddoc can generate html, pdf, and ebooks without changing the Ddoc comments. I'm curious what fundamental advantage you believe Javadoc has over Ddoc.


> Same as above. You compare again to C++ and ignore the provably successful
> models of _many_ other languages. Ruby for instance really shines in this regard
> as its community is very much oriented towards TDD. Java has such a successful
> model with its JUnit that it inspired a whole bunch of clones for  other
> languges and you completely ignore this. Instead you discuss the design of a new
> car based on experiences of horseback riders.

They're not that much different - except in one significant aspect. JUnit (for example) has a plethora of websites devoted to tutorials, cookbooks, how-tos, best practices, etc. D unittest is so simple there is no need for that. Anyone can be up and using D unittests in 2 minutes. The how-to is one slide in a presentation about D. Nobody is going to write a book about D unittest:

    http://www.amazon.com/JUnit-Action-Second-Petar-Tahchiev/dp/1935182021

That's a 450 page, $30 book on JUnit, in its second edition because it's so complicated.

I believe the success of unittests in D speak to the value of making it so simple to use.

That said, with the new UDA in D, you (or anyone else) can write a "DUnit" and present it to the community. I doubt you'll find it worth the effort, though, as unittests work very well.


>The problem is again - applying common c++
> wisdom and trying to maintain inconsistent semantics.

I'm curious how you ascribe builtin AA's, which C++ does not have and will never have, to applying C++ wisdom. As I said, D's AA's were inspired by my experience with Javascript's AA's.


>>> integers has no overflow checks,
>> This has been discussed ad nauseum. To sum up, adding overflow checks
>> everywhere would seriously degrade performance. Yet you can still have
>> overflow checking integers if you build a library type to do it. See
>> std.halffloat for an example of how to do it. It fits in with your suggestion
>> that things that can be done in the library, should be done in the library.
> Yes, but this conflicts with your statement of intention towards verification
> machinery for safety.

Designing *anything* is a compromise among mutually incompatible goals. In no way are we going to say principle A, worship A, use A as a substitute for thought & judgement, and drive right over the cliff with a blind adherence to A.

For example, try designing a car where safety is the overriding concern. You can't design it, and if you could you couldn't manufacture it, and if you could manufacture it you couldn't drive it an inch.

All language design decisions are tradeoffs. (And to be pedantic, integer overflows are not a memory safety issue. Neither are null pointers. You can have both (like Java does) and yet still have a provably memory safe language.)


> Of course I can implement whatever I need myself but what
> ensures safety is the fact that the default is safe and the language forces me
> to be explicit when I choose to sacrifice safety for other benefits. You see,
> *defaults matter*.

I agree they matter. And I doubt any two people will come up with the same list of what should be the default. Somebody has to make a decision. Again, tradeoffs.

February 27, 2013
On 2/26/13 4:23 PM, foobar wrote:
> I don't get what fault you find in the binary NOT operation.
> Regardless, my post wasn't really about specific features, (Walter
> actually mentioned those) but rather about the general design philosophy
> of D which I find lacking. Yes, it is obviously true that Rust has it
> own share of faults, The difference is what *principals* they use to
> address those faults and evolve the language.
> Rust is written in Rust, thus the developers themselves feel all the
> shortcomings, they also listen to their users and they strive to find
> the best way to express the semantics they want in the possibly simplest
> yet readable syntax possible. They think positive and build on their
> vision, whereas D thinks negatively based on C++'s vision.
> D exists for more than a decade and all it provides is slightly less
> hackish C++.
> At first, I dismissed Rust for having poor syntax ("ret" really? Are we
> back to assembly?) but lo and behold, in a very short time they
> considerably improved the syntax. D still argues about the exact same
> issues from several years ago, as if it's stuck in a time loop. This to
> me shows a lack of direction. I expected thing to improve lately with
> all those newly minted release process discussions and such, but alas
> the attitude hasn't shifted at all.

I understand how you see it, and honestly could see it from a mile. When a post (a) cherry-picks all negatives and (b) has a final tone that mentions no possible solution - it's a foregone conclusion that no amount of explaining, amending, arguing, etc. will improve the poster's outlook.

I could say e.g. "Well I think things have changed, and look we've turned the bug trend curve (http://goo.gl/kf4ZC) which is unprecedented, fixed some incomplete features, break new records on bug fixes with each release, and have a big conference coming." - to which I have no doubt it's possible to concoct a negative answer.

So I understand you have a negative outlook on D. That's entirely fine, as is mentioning it on the forum. The only thing I'd like you to understand and appreciate is that we who work on D are doing our best to find solutions to the various problems in front of us, and in quite a literal sense we don't know how to do any better. The constructive thing I'm getting out of this is that we could use some more radicalization - try things that push stronger against our comfort zone. I have a few in mind but it's too early to discuss them publicly.


Andrei
February 27, 2013
On Tuesday, 26 February 2013 at 23:44:32 UTC, Walter Bright wrote:
> As Andrei has demonstrated, Ddoc can generate html, pdf, and ebooks without changing the Ddoc comments.

This is not true in general because ddoc doesn't properly encode its output for different formats (it doesn't even get html right!)

http://dlang.org/ddoc.html

===
Embedded HTML

HTML can be embedded into the documentation comments, and it will be passed through to the HTML output unchanged. However, since it is not necessarily true that HTML will be the desired output format of the embedded documentation comment extractor, it is best to avoid using it where practical.
===


This "feature" is why I haven't bothered documenting my html library: the html examples are incorrectly displayed in the output!

BTW, using $(LT) and $(GT) is both hideously ugly and still wrong because other output formats need different characters escaped/encoded. It obviously isn't practical to macroize every character just in case.


The easiest way to fix this would be to use the ESCAPES capability ddoc already has over ALL the comment data (except Macros: sections) before doing anything else.

This will ensure all data is properly encoded while still keeping all the macro functionality.

Of course, it will kill the embedded html misfeature, which even the documentation, as seen above, admits is a bad idea to use anyway!
February 27, 2013
On 2/26/2013 4:08 PM, Adam D. Ruppe wrote:
> On Tuesday, 26 February 2013 at 23:44:32 UTC, Walter Bright wrote:
>> As Andrei has demonstrated, Ddoc can generate html, pdf, and ebooks without
>> changing the Ddoc comments.
>
> This is not true in general because ddoc doesn't properly encode its output for
> different formats

ddoc relies on using the macros to encode for different formats. Setting up the macros right is something for the user, although the default is for html.


> (it doesn't even get html right!)

The default setup should generate standard html. If the html is wrong, that should be a bug report in bugzilla, not an indictment of the approach.


> ===
> Embedded HTML
>
> HTML can be embedded into the documentation comments, and it will be passed
> through to the HTML output unchanged. However, since it is not necessarily true
> that HTML will be the desired output format of the embedded documentation
> comment extractor, it is best to avoid using it where practical.
> ===
>
>
> This "feature" is why I haven't bothered documenting my html library: the html
> examples are incorrectly displayed in the output!

Yes, if you write incorrect html in the ddoc comments, they'll just get passed through to the output. I don't think that is a fault with ddoc, though.


> Of course, it will kill the embedded html misfeature, which even the
> documentation, as seen above, admits is a bad idea to use anyway!

It's not actually a feature of ddoc at all. Ddoc just transmits its input to its output, expanding macros along the way.
February 27, 2013
On Wednesday, 27 February 2013 at 01:03:44 UTC, Walter Bright wrote:
> It's not actually a feature of ddoc at all. Ddoc just transmits its input to its output, expanding macros along the way.

The problem with that is it makes it extremely inconvenient to document a html library, and *impossible* to output correct data in formats unknown to the original author. You say outputting to different formats is a selling point, but that just isn't true.

Consider this simple comment:

/// Given "<b>text</b>", returns "text"

The most correct output for html is:

Given &quot;&lt;b&rt;text&lt;/b&rt;&quot;, returns &quot;text&quot;


Now, suppose we want our ddoc to output... say, json, just to pull an easy to show example. It *should* be:

"Given \"<b>text<\/b>\", returns \"text\""

In json, the quotes need to be escaped and it is common to also escape the forward slash, so that's what we want. In html, we encode quotes differently and should also encode < and >.


But, there's no way to actually do that with ddoc's macros. If we wanted our ddoc to output json, we'd have to write:

Given $(QUOTE)<b>text<$(SLASH)b>

well you get the idea. And now if we want it to work for both html AND json, we're writing:

$(QUOTE)$(LT)b$(GT)text$(LT)$(SLASH)b$(GR)


And obviously, that's absurd. What was a readable comment (which ddoc is supposed to aspire to) is now a hideous mess of macros.



So, you say "nobody would output ddoc to json", but this is just one easy example with different encoding rules than html. LaTeX I'm pretty sure needs you to escape the backslash. I'm not sure, I've barely used latex, so I wouldn't use those macros in my own code.

What if somebody else wants to apply his set of latex macros to my code file expecting it to just work? It probably won't.



And, again, this is very easy to solve in the vast majority of cases: put that ESCAPES macro to good use by running it over the input data ASAP. Then the data will be properly encoded for whatever formats we need, without needing these bad preemptive macros.
February 27, 2013
On Wed, Feb 27, 2013 at 02:28:44AM +0100, Adam D. Ruppe wrote:
> On Wednesday, 27 February 2013 at 01:03:44 UTC, Walter Bright wrote:
> >It's not actually a feature of ddoc at all. Ddoc just transmits its input to its output, expanding macros along the way.

Alas, the real world is not quite that simple. What if the output requires certain characters to be represented differently? There is currently no way to tell ddoc, the characters \{_^}% are metacharacters in the output format, and you need to escape them using macros $(BACKSLASH), $(OPENBRACE), ... etc.. You do NOT want to require the user to manually write these macros in the comment, because then it becomes completely unreadable, as Adam pointed out.

But currently, if you want to target *multiple* output formats (say, you want to produce both HTML docs for your website and PDF via LaTeX for a printed documentation), then you have no choice but write macros for the union of all metacharacters in all output formats. At which point, you might as well say that every character should be written as a macro, which is clearly ludicrous.


[...]
> And now if we want it to work for both html AND json, we're writing:
> 
> $(QUOTE)$(LT)b$(GT)text$(LT)$(SLASH)b$(GR)
> 
> 
> And obviously, that's absurd. What was a readable comment (which ddoc is supposed to aspire to) is now a hideous mess of macros.
> 
> So, you say "nobody would output ddoc to json", but this is just one easy example with different encoding rules than html. LaTeX I'm pretty sure needs you to escape the backslash. I'm not sure, I've barely used latex, so I wouldn't use those macros in my own code.

LaTeX is a far more complex beast than one might imagine at first glance. Not only backslashes need to be escaped, under standard conditions you need to also escape:

	$ { } ~ -- --- & _ ^ \ %

(Yes, multi-character sequences are included.)

Not to mention that D source code is UTF-8, but standard LaTeX is not, which means characters like ΓΌ need to be output as \"u instead. Furthermore, there are very precise spacing rules, like \ is required following a '.' if it's not a sentence break (e.g., in "Mr. Appleseed"), otherwise the output formatting may have the wrong spacing.

Now I said standard conditions. There's math mode, which you will end up in if you leave stray $'s lying around (but which, say, math-oriented software might want to actually use for typesetting equations and the like in the documentation), which has a *different* set of metacharacters, and all *spaces* must be escaped.

The only thorough solution for a multi-target ddoc comment currently is to write the entire comment in macros. This is an unreasonable expectation, which also detracts greatly from one's desire to actually use ddoc for this purpose.


> What if somebody else wants to apply his set of latex macros to my code file expecting it to just work? It probably won't.
> 
> And, again, this is very easy to solve in the vast majority of cases: put that ESCAPES macro to good use by running it over the input data ASAP. Then the data will be properly encoded for whatever formats we need, without needing these bad preemptive macros.

The ESCAPES macro will not completely solve the problem with formats like LaTeX, but it will help greatly. It will also nudge me slightly in the direction of embracing ddoc from my current position on the fence. ;-)


T

-- 
Computers are like a jungle: they have monitor lizards, rams, mice, c-moss, binary trees... and bugs.
February 27, 2013
On Tuesday, 26 February 2013 at 23:44:32 UTC, Walter Bright wrote:
> On 2/26/2013 2:10 PM, foobar wrote:
>> All of the above describes the benefits of having standardized documentation and
>> I agree with that. That has nothing to do with DDoc's specific design compared
>> to other similar efforts. A quick survey of languages shows that Ruby, Python,
>> Java, C#, and many others all have the same benefits but non has the doc
>> generator built into the compiler/vm with all the problems this entails.
>
> Building ddoc into the compiler means it has access to the semantic information that compiler provides, and it uses that information. If it is not built in to the compiler, then the options are:
>
> 1. require the user to type the information in twice
> 2. add parsing and semantic analysis capability to the doc generator
>
> I find (1) to be an unacceptable user experience, and (2) to be not viable given our limited resources.
>
> BTW, Javadoc apparently can only generate HTML. As Andrei has demonstrated, Ddoc can generate html, pdf, and ebooks without changing the Ddoc comments. I'm curious what fundamental advantage you believe Javadoc has over Ddoc.

This is not true.

Javadoc uses a plugin architecture known as doclet.

http://docs.oracle.com/javase/7/docs/technotes/guides/javadoc/doclet/overview.html

There are quite a few plugins available, this one for example generates UML diagrams from JavaDoc comments.

http://code.google.com/p/apiviz/

--
Paulo