Since my project nonagon isn't open-source (though that may change in the next few months), I distribute it as a lib with "headers" which just have declarations for all the classes and functions and whatnot.  In order to generate these "headers," I basically just have to remove all function bodies and replace them with semicolons.

The problem is that I did this manually at first, and it took me maybe 5-10 minutes to do it.  Now that nonagon has grown so much, It would probably take me close to an hour to replace all the function bodies with semicolons.

I wrote a "dumb" tool which basically counts the number of left and right braces, and based on the nesting level, it either outputs the text it reads or it doesn't.  The problem with this is that once I start adding things in like version blocks and nested classes, it'll start stripping things out that it's not supposed to.  It also doesn't strip out global-level function bodies.

Is there any kind of lexer/parser tool for D that will allow me to quickly strip out the function bodies?  Surely someone has run across this problem before.  Normally, I'd look into using the frontend, but, well, it's completely undocumented and written in C++.  And I really don't feel like spending a week and a half figuring out how to use the thing and then writing a tool.

Wasn't there some kind of D lexer tool written in D several months ago?  I can't find the NG thread..

Jarrett Billingsley wrote:
> The problem is that I did this manually at first, and it took me maybe 5-10 minutes to do it.  Now that nonagon has grown so much, It would probably take me close to an hour to replace all the function bodies with semicolons.

True. The approach available with current tools (i.e. stripping the function bodies manually) is not an option. Because the current approach is so difficult there are people that think it's *technically impossible* to create closed-source libraries with D (thus abandoning the language) and I don't blame them.

> Is there any kind of lexer/parser tool for D that will allow me to quickly strip out the function bodies?  Surely someone has run across this problem before.

Surely we have :) This has been discussed in the NG from time to time
and sometimes rather philosophical elements creeped into the discussion.
 The following is a summary of most common opinions, or at least my
perception of them.

The open source maniac: We don't need that kind of abomination. Let some company create a commercial tool for people who need it. If you're not going to share your source, you don't deserve to use free tools. Developers who are working for free shouldn't waste their time programming a tool to help creating nonfree/closed-source software.

The pragmatic: This kind of tool would be very useful but open source developers have no time to create one and commercial developers aren't interested in making tools for a language with tiny user (=customer) base.

The D evangelist: surely D is suitable for closed-source develpoment too, it's only a matter of time before someone will create this kind of tool. It's even possible now! You just have to go through living hell to process your thousands if not hundreds of thousands lines of code and cheerfully descend to the maintenance hell that surely awaits anyone who keeps two sets of source code with manual synchronization.

--
Niko Korhonen
SW Developer

"Niko Korhonen" <niktheblak@hotmail.com> wrote in message news:ddaing$2ops$1@digitaldaemon.com...
> The open source maniac: We don't need that kind of abomination. Let some company create a commercial tool for people who need it. If you're not going to share your source, you don't deserve to use free tools. Developers who are working for free shouldn't waste their time programming a tool to help creating nonfree/closed-source software.

Haha, those crazy Linuxheads ;)

What I'm thinking is that it would probably not be too difficult a feature to implement in the _compiler_.  Think about it - creating a symbol table that just holds names and signatures is exactly what the compiler does when it imports a module.  Outputting the symbol table in a readable format wouldn't be too much of a stretch from there.

One solution suggests it’s self (if I have time I might try implementing it, later this week maybe).

Use a stack to keep track of the state in each level of nesting. Whenever you find a '{' go back and check to se what type of block it is, push your current state, change your state and keep going. Whenever you find a '}' pop off a state. You would need to delay output (or backup occasionally) but it shouldn’t be that bad. With a little more work it could even handle braces in quotes.



In article <ddah3h$2mn5$1@digitaldaemon.com>, Jarrett Billingsley says...
>
>Since my project nonagon isn't open-source (though that may change in the next few months), I distribute it as a lib with "headers" which just have declarations for all the classes and functions and whatnot.  In order to generate these "headers," I basically just have to remove all function bodies and replace them with semicolons.
>
>The problem is that I did this manually at first, and it took me maybe 5-10 minutes to do it.  Now that nonagon has grown so much, It would probably take me close to an hour to replace all the function bodies with semicolons.
>
>I wrote a "dumb" tool which basically counts the number of left and right braces, and based on the nesting level, it either outputs the text it reads or it doesn't.  The problem with this is that once I start adding things in like version blocks and nested classes, it'll start stripping things out that it's not supposed to.  It also doesn't strip out global-level function bodies.
>
>Is there any kind of lexer/parser tool for D that will allow me to quickly strip out the function bodies?  Surely someone has run across this problem before.  Normally, I'd look into using the frontend, but, well, it's completely undocumented and written in C++.  And I really don't feel like spending a week and a half figuring out how to use the thing and then writing a tool.
>
>Wasn't there some kind of D lexer tool written in D several months ago?  I can't find the NG thread..
>
>

> Normally, I'd look into using the frontend, but, well, it's completely undocumented and written in C++.  And I really don't feel like spending a week and a half figuring out how to use the thing and then writing a tool.

The dmdfe "tool starter kit" might make writing the tool a little easier -
but one would still have to write some C++ to get it working. Also dmfe is
probably a few release behind by now. The dsource page is
http://www.dsource.org/projects/dmdfe/
I also remember someone writing a header-generator but I don't have any
links handy. Hopefully they are still around or someone remembers more.
Check the wiki or Google or something...

good luck

I'm working on just that tool right now.  For me it is the first phase of building a tool that creates scripting language bindings for D code.  What I have now is pretty basic, but it is starting to get there.  If you want to take what I already have and run with it, just ask & I'll post it here.
So far I
 - recognise the version statement, cutting out versions that are not set & adding versions that are set
 - leave struct and class bodies
 - strip bodies from regular functions and member functions

The code is very simple & probably quite broken in places, so it is at your own risk :)

Brad

On Tue, 2005-08-09 at 18:41 +0300, Niko Korhonen wrote:
> The open source maniac: We don't need that kind of abomination. Let some company create a commercial tool for people who need it. If you're not going to share your source, you don't deserve to use free tools. Developers who are working for free shouldn't waste their time programming a tool to help creating nonfree/closed-source software.
> 

I'm a F/OSS evangelist, and I definitely disagree.  A tool like this would even be useful for open-source projects.  Most people don't have the full source code for libraries on their machines, so why should they with D.  Eventually, when a library like Mango is distributed in binary form, libmango.a along with the code-stripped source should be distributed, not the full source.

I won't comment about whether or not closed-source guys deserve to use open-source stuff, however.  No need for a political debate.  I will say that not all F/OSS people are RMS, however.


-John Demme

In article <ddah3h$2mn5$1@digitaldaemon.com>, Jarrett Billingsley says...
>Is there any kind of lexer/parser tool for D that will allow me to quickly strip out the function bodies?  Surely someone has run across this problem before.  Normally, I'd look into using the frontend, but, well, it's completely undocumented and written in C++.  And I really don't feel like spending a week and a half figuring out how to use the thing and then writing a tool.

You might have to do that if you really don't want to release the source.

>Wasn't there some kind of D lexer tool written in D several months ago?  I can't find the NG thread..

There's a bunch of different tools that do different things. The tool "digc" might help you out. It has a function called "strip". It's mentioned here: http://www.prowiki.org/wiki4d/wiki.cgi?ReferenceForTools

This might be the NG post you're thinking of: http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/11075

jcc7

Jarrett Billingsley wrote:

> Since my project nonagon isn't open-source (though that may change in the next few months), I distribute it as a lib with "headers" which just have declarations for all the classes and functions and whatnot.  In order to generate these "headers," I basically just have to remove all function bodies and replace them with semicolons.
> 
> The problem is that I did this manually at first, and it took me maybe 5-10 minutes to do it.  Now that nonagon has grown so much, It would probably take me close to an hour to replace all the function bodies with semicolons.
> 
> I wrote a "dumb" tool which basically counts the number of left and right braces, and based on the nesting level, it either outputs the text it reads or it doesn't.  The problem with this is that once I start adding things in like version blocks and nested classes, it'll start stripping things out that it's not supposed to.  It also doesn't strip out global-level function bodies.
> 
> Is there any kind of lexer/parser tool for D that will allow me to quickly strip out the function bodies?  Surely someone has run across this problem before.  Normally, I'd look into using the frontend, but, well, it's completely undocumented and written in C++.  And I really don't feel like spending a week and a half figuring out how to use the thing and then writing a tool.

digc has a tool for stripping function bodies.  The problem is that because of D's VERY complex versioning and debugging syntax there is no way to handle stripping robustly without understanding the entire language; it must parse the whole thing.  Which is a pickle, since D is not simple to parse at all and some of its syntax is so subtle that you practically need to depend upon DMD.  I think Ben has the right idea.

I've attached the code that's currently in digc which parses D into a syntax tree.  I haven't looked at it in months so it might not work with various language features; there are so damned many of them.  At least usage is simple:

char [] dstrip (char [] filename, char [] data,
     bit [char []] versions = null, int version_level = 0,
     bit [char []] debugs = null, uint debug_level = 0)
{
     class_lexer lexer = new class_lexer ();
     class_parser parser = new class_parser (lexer);

     lexer.source = type_marker (filename, data);

     type_printer printer;
     class_module mod = parser.parse_module ();

     printer.versions = versions;
     printer.version_level = version_level;
     printer.debugs = debugs;
     printer.debug_level = debug_level;
     printer.load_default_versions ();
     printer.skip_bodies = true;
     printer.pretty = true;
     printer.collapse_versions = true;
     return mod.toString (printer);
}

I think it's been mentioned in this thread that hacking DMDFE may be a way to go.  Another route would be to hack an existing utility that already understands version statements well: take a look at the source for 'build' over on dsource.org.  At the very least, it's already aware of brackets and strings, which would be the biggest hurdles for writing your own tool.

Good luck.

- EricAnderton at yahoo