View mode: basic / threaded / horizontal-split · Log in · Help
March 13, 2011
std.xml: Why is it so slow? Is there anything else wrong with it?
There seems to be a consensus around here that Phobos needs a good XML 
module, and that std.xml doesn't cut it, at least partly due to 
performance issues.  I have no clue how to write a good XML module from 
scratch.  It seems like noone else is taking up the project either. 
This leads me to two questions:

1.  Has anyone ever sat down and tried to figure out **why** std.xml is 
so slow?  Seriously, if noone's bothered to profile it or read the code 
carefully, then for all we know there might be some low hanging fruit 
and it might be an afternoon of optimization away from being reasonably 
fast.  Basically every experience I've ever had suggests that, if a 
piece of code has not already been profiled and heavily optimized, at 
least a 5-fold speedup can almost always be obtained just by optimizing 
the low-hanging fruit.  (For example, see my recent pull request for the 
D garbage collector.  BTW, if excessive allocations are a contributing 
factor, then fixing the GC should help with XML, too.)

If the answer is no, this hasn't been done, please post some canned 
benchmarks and maybe I'll take a crack at it.

2.  What other major defects/design flaws, if any, does std.xml have?

In other words, how are we really so sure that we need to start from 
scratch?
March 13, 2011
Re: std.xml: Why is it so slow? Is there anything else wrong with it?
Am 13.03.2011 05:34, schrieb dsimcha:
> There seems to be a consensus around here that Phobos needs a good XML
> module, and that std.xml doesn't cut it, at least partly due to
> performance issues. I have no clue how to write a good XML module from
> scratch. It seems like noone else is taking up the project either. This
> leads me to two questions:
>

Isn't Tomek Sowiński working on it?

> 1. Has anyone ever sat down and tried to figure out **why** std.xml is
> so slow? Seriously, if noone's bothered to profile it or read the code
> carefully, then for all we know there might be some low hanging fruit
> and it might be an afternoon of optimization away from being reasonably
> fast. Basically every experience I've ever had suggests that, if a piece
> of code has not already been profiled and heavily optimized, at least a
> 5-fold speedup can almost always be obtained just by optimizing the
> low-hanging fruit. (For example, see my recent pull request for the D
> garbage collector. BTW, if excessive allocations are a contributing
> factor, then fixing the GC should help with XML, too.)
>
> If the answer is no, this hasn't been done, please post some canned
> benchmarks and maybe I'll take a crack at it.
>
> 2. What other major defects/design flaws, if any, does std.xml have?
>
> In other words, how are we really so sure that we need to start from
> scratch?

(These questions should probably discusses nevertheless)

Cheers,
- Daniel
March 13, 2011
Re: std.xml: Why is it so slow? Is there anything else wrong with it?
Do we want to take a look at libxml, or are there legal issues with that?
March 13, 2011
Re: std.xml: Why is it so slow? Is there anything else wrong with it?
On Saturday 12 March 2011 20:39:31 Daniel Gibson wrote:
> Am 13.03.2011 05:34, schrieb dsimcha:
> > There seems to be a consensus around here that Phobos needs a good XML
> > module, and that std.xml doesn't cut it, at least partly due to
> > performance issues. I have no clue how to write a good XML module from
> > scratch. It seems like noone else is taking up the project either. This
> 
> > leads me to two questions:
> Isn't Tomek Sowiński working on it?

Yes.

> > 1. Has anyone ever sat down and tried to figure out **why** std.xml is
> > so slow? Seriously, if noone's bothered to profile it or read the code
> > carefully, then for all we know there might be some low hanging fruit
> > and it might be an afternoon of optimization away from being reasonably
> > fast. Basically every experience I've ever had suggests that, if a piece
> > of code has not already been profiled and heavily optimized, at least a
> > 5-fold speedup can almost always be obtained just by optimizing the
> > low-hanging fruit. (For example, see my recent pull request for the D
> > garbage collector. BTW, if excessive allocations are a contributing
> > factor, then fixing the GC should help with XML, too.)
> > 
> > If the answer is no, this hasn't been done, please post some canned
> > benchmarks and maybe I'll take a crack at it.
> > 
> > 2. What other major defects/design flaws, if any, does std.xml have?
> > 
> > In other words, how are we really so sure that we need to start from
> > scratch?

As I understand it, one of the main issues is that std.xml is delegate-based. I 
don't know how well it does with slicing and avoiding copying strings, but one 
of the biggest advantages that D has is its array slicing. And taking full 
advantage of that and avoiding string copying is one of - if not _the_ best - 
way to make std.xml lightning fast.

In any case, there was a discussion about std.xml recently, and the consensus 
was that we should just throw it out rather than leave it there and have people 
complain about how bad Phobos' xml module is.

As Daniel pointed out, Tomek Sowiński is currently working on a new std.xml. I 
don't know how far along he is or when he expects it to be done, but supposedly 
he's working on it and sometime reasonably soon we should have a new std.xml to 
review.

We are definitely _not_ going to be working on improving the current std.xml 
though. I think that the only reason that it's still there is that Andrei didn't 
get around to throwing it out before the last release (or at least deprecating 
it). That's definitely what he wants to do, and the consensus was in favor of 
that decision.

- Jonathan M Davis
March 13, 2011
Re: std.xml: Why is it so slow? Is there anything else wrong with it?
On Sat, 2011-03-12 at 23:34 -0500, dsimcha wrote:
> There seems to be a consensus around here that Phobos needs a good XML 
> module, and that std.xml doesn't cut it, at least partly due to 
> performance issues.  I have no clue how to write a good XML module from 
> scratch.  It seems like noone else is taking up the project either. 

I just worry that creating a whole self-standing library is a waste of
time when wrapping libxml2 and libxslt gets a fast XML subsystem for
free.  This is the direction Python has gone. cf.  the lxml package to
replace ElementTree.  The elephant in the room is of course W3C DOM.
Everyone believes they have to have an implementation, but no-one then
uses it.

> This leads me to two questions:
> 
> 1.  Has anyone ever sat down and tried to figure out **why** std.xml is 
> so slow?  Seriously, if noone's bothered to profile it or read the code 
> carefully, then for all we know there might be some low hanging fruit 
> and it might be an afternoon of optimization away from being reasonably 
> fast.  Basically every experience I've ever had suggests that, if a 
> piece of code has not already been profiled and heavily optimized, at 
> least a 5-fold speedup can almost always be obtained just by optimizing 
> the low-hanging fruit.  (For example, see my recent pull request for the 
> D garbage collector.  BTW, if excessive allocations are a contributing 
> factor, then fixing the GC should help with XML, too.)
> 
> If the answer is no, this hasn't been done, please post some canned 
> benchmarks and maybe I'll take a crack at it.
> 
> 2.  What other major defects/design flaws, if any, does std.xml have?
> 
> In other words, how are we really so sure that we need to start from 
> scratch?

Excellent question.  Especially given the existence of libxml2 and
libxslt.

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
March 13, 2011
Re: std.xml: Why is it so slow? Is there anything else wrong with it?
On Sunday 13 March 2011 01:11:05 Russel Winder wrote:
> On Sat, 2011-03-12 at 23:34 -0500, dsimcha wrote:
> > There seems to be a consensus around here that Phobos needs a good XML
> > module, and that std.xml doesn't cut it, at least partly due to
> > performance issues.  I have no clue how to write a good XML module from
> > scratch.  It seems like noone else is taking up the project either.
> 
> I just worry that creating a whole self-standing library is a waste of
> time when wrapping libxml2 and libxslt gets a fast XML subsystem for
> free.  This is the direction Python has gone. cf.  the lxml package to
> replace ElementTree.  The elephant in the room is of course W3C DOM.
> Everyone believes they have to have an implementation, but no-one then
> uses it.
> 
> > This leads me to two questions:
> > 
> > 1.  Has anyone ever sat down and tried to figure out **why** std.xml is
> > so slow?  Seriously, if noone's bothered to profile it or read the code
> > carefully, then for all we know there might be some low hanging fruit
> > and it might be an afternoon of optimization away from being reasonably
> > fast.  Basically every experience I've ever had suggests that, if a
> > piece of code has not already been profiled and heavily optimized, at
> > least a 5-fold speedup can almost always be obtained just by optimizing
> > the low-hanging fruit.  (For example, see my recent pull request for the
> > D garbage collector.  BTW, if excessive allocations are a contributing
> > factor, then fixing the GC should help with XML, too.)
> > 
> > If the answer is no, this hasn't been done, please post some canned
> > benchmarks and maybe I'll take a crack at it.
> > 
> > 2.  What other major defects/design flaws, if any, does std.xml have?
> > 
> > In other words, how are we really so sure that we need to start from
> > scratch?
> 
> Excellent question.  Especially given the existence of libxml2 and
> libxslt.

Well, Tom is working a new std.xml regardless, but I would fully expect a 
properly implemented xml library in D to cream something like libxml. D's 
slicing abilities give it a _huge_ advantage when it comes to stuff like parsing. 
libxml isn't going to be able to take advantage of that. Tango's XML parser is 
_extremely_ fast ( http://dotnot.org/blog/archives/2008/03/12/why-is-dtango-so-
fast-at-parsing-xml/ ), and one of the biggest reasons for that is D's slicing 
abilities. Parsing is one place where D should be able to seriously shine and is 
_definitely_ one of the places where we _don't_ want to wrap a C library if we 
don't have to.

But regardless, a new std.xml is in the works, and hopefully it'll be up for 
review within the next couple of months (I have no idea how fast Tom is making 
progress on it though).

- Jonathan M Davis
Top | Discussion index | About this forum | D home