LOL! This is terrible!

The performance is not so important for me, but this report moves me to
tears.
What is a bottleneck?

--
SHOO

(2010/11/14 21:55), Jacob Carlborg wrote:
> A benchmark has shown that the Tango XML module is a lot faster than many of the XML libraries out there, specially than the Phobos module. So I would say that D could do better than other libraries at least when it comes to speed.
>
> http://dotnot.org/blog/archives/2008/03/10/xml-benchmarks-updated-graphs-with-rapidxml/
>
> On 14 nov 2010, at 10:44, Russel Winder wrote:
>
>> On Sun, 2010-11-14 at 15:35 +0900, SHOO wrote:
>>> About a problem of std.xml.
>>
>> Potentially n00b comment but . . .
>>
>> The Python community seem to be giving up on having their own library for processing XML and are using bindings to libxml2 and libxslt.  Is it likely that a D source XML library can do better than libxml2 and libxslt or might it be better just to create a D idiomatic binding?
>>
>> --
>> Russel.
>> =============================================================================
>> Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
>> 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at russel.org.uk
>> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>

I'm not sure whether this is the main performance issue, but last time I debugged through std.xml (yes, it's used in Visual D, aswell as std.json), I noticed that it suffers badly under bugs

http://d.puremagic.com/issues/show_bug.cgi?id=2356 http://d.puremagic.com/issues/show_bug.cgi?id=4298

because constant arrays with hundreds of elements are created on the stack whenever a character lookup is done. Fortunately, the files where it is used in Visual D are rather small.

Rainer

SHOO wrote:
> LOL! This is terrible!
> 
> The performance is not so important for me, but this report moves me to
> tears.
> What is a bottleneck?
> 
> -- 
> SHOO
> 
> (2010/11/14 21:55), Jacob Carlborg wrote:
>> A benchmark has shown that the Tango XML module is a lot faster than many of the XML libraries out there, specially than the Phobos module. So I would say that D could do better than other libraries at least when it comes to speed.
>>
>> http://dotnot.org/blog/archives/2008/03/10/xml-benchmarks-updated-graphs-with-rapidxml/
>>

On Mon, 15 Nov 2010 00:10:08 +0900, SHOO <zan77137 at nifty.com> wrote:

> LOL! This is terrible!
>
> The performance is not so important for me, but this report moves me to tears.

I agree. The most important of XML is human readable and structured format.
I never use XML in performance needed situation.
I want efficient memory management and usable APIs.

Of course, faster implementation is good :)

> What is a bottleneck?
>
> --
> SHOO
>
> (2010/11/14 21:55), Jacob Carlborg wrote:
>> A benchmark has shown that the Tango XML module is a lot faster than many of the XML libraries out there, specially than the Phobos module. So I would say that D could do better than other libraries at least when it comes to speed.
>>
>> http://dotnot.org/blog/archives/2008/03/10/xml-benchmarks-updated-graphs-with-rapidxml/
>>
>> On 14 nov 2010, at 10:44, Russel Winder wrote:
>>
>>> On Sun, 2010-11-14 at 15:35 +0900, SHOO wrote:
>>>> About a problem of std.xml.
>>>
>>> Potentially n00b comment but . . .
>>>
>>> The Python community seem to be giving up on having their own library
>>> for processing XML and are using bindings to libxml2 and libxslt.  Is
>>> it
>>> likely that a D source XML library can do better than libxml2 and
>>> libxslt or might it be better just to create a D idiomatic binding?
>>>
>>> --
>>> Russel.
>>> =============================================================================
>>> Dr Russel Winder      t: +44 20 7585 2200   voip:
>>> sip:russel.winder at ekiga.net
>>> 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at russel.org.uk
>>> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
>>> _______________________________________________
>>> phobos mailing list
>>> phobos at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/phobos
>>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

On 11/14/10 4:25 AM, SHOO wrote:
> Perhaps it has a problem in license of libxml2 and libxslt.
> We may include it in etc, but it does not become the Boost license.
> And I don't need such libraries for a *standard*.

If the license of third-party libraries is okay and the libraries themselves are easily redistributable or ubiquitous, I don't see anything wrong with providing a wrapper for them with the standard distribution in etc/. If the wrapper is substantial, then it could be moved to std/.

Andrei

On 11/14/10 11:09 AM, Masahiro Nakagawa wrote:
> On Mon, 15 Nov 2010 00:10:08 +0900, SHOO <zan77137 at nifty.com> wrote:
>
>> LOL! This is terrible!
>>
>> The performance is not so important for me, but this report moves me to tears.
>
> I agree. The most important of XML is human readable and structured format.
> I never use XML in performance needed situation.
> I want efficient memory management and usable APIs.
>
> Of course, faster implementation is good :)

The problem is that readability and structure are subjective topics, whereas performance is easily measurable and comparable.

Andrei

I don't know why Phobos is so slow but what I've heard Tango is fast because it never copies the input buffer, what I've heard.

On 14 nov 2010, at 16:10, SHOO wrote:

> LOL! This is terrible!
> 
> The performance is not so important for me, but this report moves me to tears. What is a bottleneck?
> 
> --
> SHOO
> 
> (2010/11/14 21:55), Jacob Carlborg wrote:
>> A benchmark has shown that the Tango XML module is a lot faster than many of the XML libraries out there, specially than the Phobos module. So I would say that D could do better than other libraries at least when it comes to speed.
>> 
>> http://dotnot.org/blog/archives/2008/03/10/xml-benchmarks-updated-graphs-with-rapidxml/
>> 
>> On 14 nov 2010, at 10:44, Russel Winder wrote:
>> 
>>> On Sun, 2010-11-14 at 15:35 +0900, SHOO wrote:
>>>> About a problem of std.xml.
>>> 
>>> Potentially n00b comment but . . .
>>> 
>>> The Python community seem to be giving up on having their own library for processing XML and are using bindings to libxml2 and libxslt.  Is it likely that a D source XML library can do better than libxml2 and libxslt or might it be better just to create a D idiomatic binding?
>>> 
>>> --
>>> Russel.
>>> =============================================================================
>>> Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
>>> 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at russel.org.uk
>>> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
>>> _______________________________________________
>>> phobos mailing list
>>> phobos at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/phobos
>> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

-- 
/Jacob Carlborg

Dmitry Olshansky <dmitry.olsh at gmail.com> wrote:
> On 14.11.2010 11:05, SHOO wrote:
> > I think that it is important that we prioritize our to-do list.
> Strongly agree. It's even better if it's public list so that all users aware of it.

How about the trac wiki or ticket on dsource?


Shin

I think the Tango parser is fast for two reasons. First, it doesn't allocate at all and second, it doesn't decode the string data--that's left up to the user. Even XercesC converts data to a specific encoding before the user sees it (some variant of wchar_t I believe).  I don't know how Tango handles XML streams in different encodings, but perhaps it's a template on char type?

Sent from my iPhone

On Nov 14, 2010, at 10:39 PM, Jacob Carlborg <doob at me.com> wrote:

> I don't know why Phobos is so slow but what I've heard Tango is fast because it never copies the input buffer, what I've heard.
> 
> On 14 nov 2010, at 16:10, SHOO wrote:
> 
>> LOL! This is terrible!
>> 
>> The performance is not so important for me, but this report moves me to tears. What is a bottleneck?
>> 
>> --
>> SHOO
>> 
>> (2010/11/14 21:55), Jacob Carlborg wrote:
>>> A benchmark has shown that the Tango XML module is a lot faster than many of the XML libraries out there, specially than the Phobos module. So I would say that D could do better than other libraries at least when it comes to speed.
>>> 
>>> http://dotnot.org/blog/archives/2008/03/10/xml-benchmarks-updated-graphs-with-rapidxml/
>>> 
>>> On 14 nov 2010, at 10:44, Russel Winder wrote:
>>> 
>>>> On Sun, 2010-11-14 at 15:35 +0900, SHOO wrote:
>>>>> About a problem of std.xml.
>>>> 
>>>> Potentially n00b comment but . . .
>>>> 
>>>> The Python community seem to be giving up on having their own library for processing XML and are using bindings to libxml2 and libxslt.  Is it likely that a D source XML library can do better than libxml2 and libxslt or might it be better just to create a D idiomatic binding?
>>>> 
>>>> --
>>>> Russel.
>>>> =============================================================================
>>>> Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
>>>> 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at russel.org.uk
>>>> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
>>>> _______________________________________________
>>>> phobos mailing list
>>>> phobos at puremagic.com
>>>> http://lists.puremagic.com/mailman/listinfo/phobos
>>> 
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
> 
> -- 
> /Jacob Carlborg
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

These are the modules that couldn't be moved to the Boost license for one reason or another.  They should all be removed or replaced:

base64 - Chris Miller
loader - Matthew Wilson
md5 - RSA Data Security
openrj - Matthew Wilson
perf - Matthew Wilson
socket - Chris Miller
socketstream - Chris Miller
stream - Pavel Minayev

On Nov 14, 2010, at 12:05 AM, SHOO wrote:

> I think that it is important that we prioritize our to-do list.
> 
> Therefore I think that firstly we should make a list. Next, it is necessary for us to clarify the problem that each item has.
> 
> I show following list the thing which I hit on:
> - std.stream, I/O (replace, enhance)
> - std.xml (replace?)
> - std.json (replace?)
> - std.datetime (replace, enhance)
> - scope/RAII (replace, enhance)
> - std.scoket / asio (replace)
> - std.event (enhance)
> - std.serialize (enhance)
> - documents (enhance)
> - std.process (enhance)
> - std.path, std.file (enhance)
> - pure (apply)
> - nothrow (apply)
> - @safe/@trusted/@system (apply)
> - shared (enhance, bug fix)
> - GC (enhance)
> - std.container (enhance)
> - opDollars (enhance, apply)
> - and some voted bugs (bug fix)
> (I only enumerate of the list at this stage, and omit the detailed explanation.)
> 
> Are there items else?
> 
> --
> SHOO
> 
> (2010/11/14 14:52), Jonathan M Davis wrote:
>> We have several modules in Phobos which are supposedly going to be deprecated in favor of better implementations (std.stream, std.xml, std.json, etc). As I understand it, this is primarily because the code isn't being maintained, is poorly designed for D2 (possibly because it isn't range-centric or just hasn't been updated with D2-only features), and/or lacks a maintainer/champion. In addition to that, there's various types of functionality which should probably be in Phobos but haven't been done yet.
>> 
>> The Phobos developers only have so much time on their hands, and some portion of this kind of work is going to need to be done by people who are not currently on the Phobos team. That, and we seem to be adopting the idea that the ideal situation is for each module to have a "champion" of sorts who is behind the module, working to fix bugs on it and make it better.
>> 
>> So, I was wondering if what we should do is figure out what some of the modules are that we want in Phobos - and in particular the ones currently in Phobos which need to be overhauled - and then post on the main D list looking for people willing to take them on. We don't want to a flood of code that needs to be reviewed for inclusion in Phobos, but if we want to get a lot of this stuff done, we need more people working on it - particularly people who are really looking to focus on it and champion it.
>> 
>> So, I'm suggesting that we identify the top priority module which aren't likely to be done by Phobos developers any time soon and see if we can get others in the D community to do them. In particular, it's a problem that we have several modules which we intend to replace. The longer that we wait, the more code that will be written using the old modules, and the more code which will break when they get replaced.
>> 
>> - Jonathan M Davis
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

In the trunk, I already replaced base64 module.


Masahiro

On Tue, 16 Nov 2010 06:42:09 +0900, Sean Kelly <sean at invisibleduck.org> wrote:

> These are the modules that couldn't be moved to the Boost license for one reason or another.  They should all be removed or replaced:
>
> base64 - Chris Miller
> loader - Matthew Wilson
> md5 - RSA Data Security
> openrj - Matthew Wilson
> perf - Matthew Wilson
> socket - Chris Miller
> socketstream - Chris Miller
> stream - Pavel Minayev
>
> On Nov 14, 2010, at 12:05 AM, SHOO wrote:
>
>> I think that it is important that we prioritize our to-do list.
>>
>> Therefore I think that firstly we should make a list. Next, it is necessary for us to clarify the problem that each item has.
>>
>> I show following list the thing which I hit on:
>> - std.stream, I/O (replace, enhance)
>> - std.xml (replace?)
>> - std.json (replace?)
>> - std.datetime (replace, enhance)
>> - scope/RAII (replace, enhance)
>> - std.scoket / asio (replace)
>> - std.event (enhance)
>> - std.serialize (enhance)
>> - documents (enhance)
>> - std.process (enhance)
>> - std.path, std.file (enhance)
>> - pure (apply)
>> - nothrow (apply)
>> - @safe/@trusted/@system (apply)
>> - shared (enhance, bug fix)
>> - GC (enhance)
>> - std.container (enhance)
>> - opDollars (enhance, apply)
>> - and some voted bugs (bug fix)
>> (I only enumerate of the list at this stage, and omit the detailed
>> explanation.)
>>
>> Are there items else?
>>
>> --
>> SHOO
>>
>> (2010/11/14 14:52), Jonathan M Davis wrote:
>>> We have several modules in Phobos which are supposedly going to be
>>> deprecated in
>>> favor of better implementations (std.stream, std.xml, std.json, etc).
>>> As I
>>> understand it, this is primarily because the code isn't being
>>> maintained, is
>>> poorly designed for D2 (possibly because it isn't range-centric or
>>> just hasn't
>>> been updated with D2-only features), and/or lacks a
>>> maintainer/champion. In
>>> addition to that, there's various types of functionality which should
>>> probably
>>> be in Phobos but haven't been done yet.
>>>
>>> The Phobos developers only have so much time on their hands, and some
>>> portion of
>>> this kind of work is going to need to be done by people who are not
>>> currently on
>>> the Phobos team. That, and we seem to be adopting the idea that the
>>> ideal
>>> situation is for each module to have a "champion" of sorts who is
>>> behind the
>>> module, working to fix bugs on it and make it better.
>>>
>>> So, I was wondering if what we should do is figure out what some of
>>> the modules
>>> are that we want in Phobos - and in particular the ones currently in
>>> Phobos
>>> which need to be overhauled - and then post on the main D list looking
>>> for
>>> people willing to take them on. We don't want to a flood of code that
>>> needs to be
>>> reviewed for inclusion in Phobos, but if we want to get a lot of this
>>> stuff done,
>>> we need more people working on it - particularly people who are really
>>> looking
>>> to focus on it and champion it.
>>>
>>> So, I'm suggesting that we identify the top priority module which
>>> aren't likely
>>> to be done by Phobos developers any time soon and see if we can get
>>> others in
>>> the D community to do them. In particular, it's a problem that we have
>>> several
>>> modules which we intend to replace. The longer that we wait, the more
>>> code that
>>> will be written using the old modules, and the more code which will
>>> break when
>>> they get replaced.
>>>
>>> - Jonathan M Davis
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos