The original "Phobos is cool" thread encouraged me to share my experience.

D has been one of my two main languages for about 5 years now. I use it almost daily in my work where it has proved very useful. Yet I don't think a case such as mine is discussed often so I thought I would take the time to explain how and why I use D. I don't expect anything in return, but I wonder how many users have a similar use of the language and whether this user base is significant enough to warrant consideration.

But first of all, why should you care?

You probably shouldn't. I'm not a software developer, my company's business doesn't rely on D in any way, I'll never write a killer app or case study and even if I were to share a library it probably wouldn't be maintained more than a week. I'm not someone that'll bring much value back in the language.

I'm a security consultant. My job is to find security issues and exploit them. I'm sure most of you know about pentesters yet my job goes a bit beyond common penetration testing. I review websites and networks but also attack electronic devices, perform malware analysis, source code review, cryptographic review, human manipulation, lockpicking... Anything goes.

In that context I very often need to write dedicated tools. I don't have the luxury of time: since most missions take about a week I feel lucky if I get 2 days to write an exploit. This means that I need to prototype a solution very quickly. For that, the goto language in my field is Python (Go is coming big recently). Python is my other language of predilection and fast scripting is where it absolutely shines and given its large library support (both standard and community based) it is often the right tool for the job. However I often end up needing more performances or low level access and optimizing Python is hard.

For example when trying to exploit a tight race condition I need all the speed I can get, especially when that race condition happens in a chip on which I'm directly connected. Similarily for things like bruteforcing running 10% faster may be the difference between finding an access in time or not.

The difference with D is that I can start very scripty, using no explicit type, then adding details in once I have a strong skeleton. I know that style is often dismissed by more industrial users but it's one of the core reasons why I use D.

I use almost exclusively UFCS function chains when prototyping because they fit the way I think perfectly. UFCS is absolutely a killer feature for me from a design perspective. If I use something else it is either because I'm optimizing (I found that basic for loops often yield better performances than long templated UFCS chains) or because I'm porting another program to D.

Since I don't choose the technologies I'm facing I end up constantly butting heads against new formats or protocols. In these cases I often port part of a parsing library to D or use a C one directly if available.

D is perfect to use C libraries or port libraries from other languages.

I rarely get enough time to port a full library but the fact that D supports all paradigms really shines there since it enables me to port programs without rethinking their architecture. It's rarely optimal but it's a huge time saver as a first step.

I also get to use that at a low level. For example the other day I was studying a ransomware. I needed to do some statistics over a large amount of files, started decompiling the malware code by hand translating it into C-style D, then hit a function that was not fun to reverse (often the case in cryptographic code). I took the disassembly, stuck it into an asm{} statement, tweaked it just enough to get it working and continued decompiling the rest without needing to worry about that function anymore.

I could have done it in another way, but this meant that I didn't need to change language at any time, it was all in D, I could optimize it all as needed at the end. This is especially important since I need to balance this low-level code with the high-level contexts in which they are found (and said context rarely makes it easy for me).

There are some tools that I use often and wrote myself in D (generally variants of "we need to try every element of this list as quickly as possible"). In general it is painless and efficient. I've even had a few surprises of code that I prototyped in D, reformated in C-style D for performance, translated in C for even more performance, discovered that pure C was slower than my C-style DMD generated code (?) so rolled back to D.

The only time where I had some issues were that time I wrote a Windows trojan in D. I needed to do some pretty convoluted stuff to load my code in just the right way to avoid antiviruses and it wasn't fun to do it in D.  Cross-compiling this proved another hurdle. It would have been easier in C++ directly. But I can't very well blame anyone not to make malware development easier and none of the issues I had should impact normal development.

Most of the code I write exploits a specific cryptographic mistake, buffer overflow or race condition for a specific client so I have nothing left to publish. As I said at the beginning, even though I write D code daily I am just not working on projects that are likely to ever bring back to the community. And, for all the code I write, I'm not a developper. I wouldn't know how to manage a project that lasts more than 3 days honnestly. That doesn't mean that I'm not serious about code quality (I spend days reviewing code and explaining to programmers how to make it better after all) but I don't contribute code to open-source projects. I'm just a selfish user minding his own business.

As for issues... (What? It wouldn't be a correct forum post without issue...)

The low number of native libraries is not ideal. Yes, I can use a C library or port it from another language but I've generally got less than two days to work on a project. Any time spent on this is time not spent on actually solving the problem.

For the same reason I am a huge advocate of battery-included stdlib. I know many are of the opinion of moving as much as possible from Phobos into dub.  For me any library that isn't in dub is one more library I have to evaluate, try, keep track of when they update. It's lost time. Huge projects can have a mindset of "we'll manage our own version anyway" but my projects are written in a few days and are useful a week or two at most. So go Phobos, go! We also need lots of libraries but that is the kind of things that is built upon a strong standard library, not at its expense.

Then there's the omnipresent negativity of a huge chunk of the community...  it's rough. I used to read the forums daily until I noticed that it stressed me a lot, made me feel angry and sad and that it wasn't the place I wanted to be. Now I mostly keep an eye on Announce. I have no idea how this can be fixed though so I'll leave it at that. I completely assume the irony of posting this here.

All in all I've tried hard to find a better language than D for my work, but its design choices make it a really good fit for my purpose. The main reason why I still use lots of Python is because of its library support and because I can expect my coworkers and clients to be able to run it without issue or hour-long explainations. I hope it stays that way even though I understand why most trends I see on the forum target more "industrial" users (by which I mean working on month or year-long software development projects in a professionnal setting). I have no expectations from that post, but I wonder how many others use D everyday at work in a similar fashion.

On Monday, 29 June 2020 at 09:09:14 UTC, Cym13 wrote:
> The original "Phobos is cool" thread encouraged me to share my experience.
>
> [...]

Thank you for sharing, that's refreshing!

On Monday, 29 June 2020 at 09:09:14 UTC, Cym13 wrote:
> The original "Phobos is cool" thread encouraged me to share my experience.

Happy to read this.

I love phobos, and think it should grow and include more stuff.

What libs are you missing?

On Monday, 29 June 2020 at 09:09:14 UTC, Cym13 wrote:
> I'm a security consultant. My job is to find security issues and exploit them. I'm sure most of you know about pentesters yet my job goes a bit beyond common penetration testing. I review websites and networks but also attack electronic devices, perform malware analysis, source code review, cryptographic review, human manipulation, lockpicking... Anything goes.


Super interesting thanks!

On Monday, 29 June 2020 at 09:09:14 UTC, Cym13 wrote:
> The original "Phobos is cool" thread encouraged me to share my experience.
>
> [...]

Interesting and thank you for it. This could be a nice blog post, imho.

On Monday, 29 June 2020 at 09:09:14 UTC, Cym13 wrote:
> I'm a security consultant. My job is to find security issues and exploit them. I'm sure most of you know about pentesters yet my job goes a bit beyond common penetration testing. I review websites and networks but also attack electronic devices, perform malware analysis, source code review, cryptographic review, human manipulation, lockpicking... Anything goes.

It would be cool to see a blog post about this.

Matheus.

On Monday, 29 June 2020 at 11:18:03 UTC, Guillaume Piolat wrote:
> On Monday, 29 June 2020 at 09:09:14 UTC, Cym13 wrote:
>> I'm a security consultant. My job is to find security issues and exploit them. I'm sure most of you know about pentesters yet my job goes a bit beyond common penetration testing. I review websites and networks but also attack electronic devices, perform malware analysis, source code review, cryptographic review, human manipulation, lockpicking... Anything goes.
>
>
> Super interesting thanks!

I'll bite:

I've been defending:
   - unsafe by default,
   - throw by default,
   - impure by default,
   - mutable by default

for years. To me those defaults are the right ones, and it warms me up that, as it seems, your D usage would be made _almost impossible_ if the permissive default changed to a strict one.

I've been in a similar situation where you have to create an internal tool as fast as possible because of limited political support, and all those features  - that are completely rational and acceptable in the right context - may reveal a liability in the trenches.

So I hope people will keep the important use cases of _bad programs_, may I say in mind when improving D.

On Monday, 29 June 2020 at 12:18:02 UTC, Guillaume Piolat wrote:
>
> I'll bite:
>
> I've been defending:
>    - unsafe by default,
>    - throw by default,
>    - impure by default,
>    - mutable by default
>
> for years. To me those defaults are the right ones, and it warms me up that, as it seems, your D usage would be made _almost impossible_ if the permissive default changed to a strict one.
>
> I've been in a similar situation where you have to create an internal tool as fast as possible because of limited political support, and all those features  - that are completely rational and acceptable in the right context - may reveal a liability in the trenches.
>
> So I hope people will keep the important use cases of _bad programs_, may I say in mind when improving D.

I have mixed feelings about that. On one hand I absolutely agree with you, if these features were in place D would probably not be nearly as useful to me as it is now.

On the other hand, I spend my days exploiting security issues and I think that these features would be effective to fight real bugs that have real consequences. I don't want to be suject to these limitations, but if more people were subjected to them I would have a harder time finding bugs which is good.

It wouldn't solve all issues of course (most issues are at a much higher level such as forgetting to protect a webpage with authentication) but it would help.

(Even if I'd be stuck)

On Monday, 29 June 2020 at 09:09:14 UTC, Cym13 wrote:
> The original "Phobos is cool" thread encouraged me to share my experience.

Thanks a lot for sharing ! It's great to see a balanced post by a serious user.

You mentioned a few times that you don't bring much value back to the language. I don't really think that matters, as it's just natural that most people use tools without contributing to them, even in OSS. As someone having invested a ridiculous amount of my time in D, I surely don't blame anyone for not choosing the same path. But if you even mentioned D to a couple of your colleagues, that's a good contribution, in my book.

> The difference with D is that I can start very scripty, using no explicit type, then adding details in once I have a strong skeleton. I know that style is often dismissed by more industrial users but it's one of the core reasons why I use D.

At one of the DConf, there was a panel asking: "What do you love about D ?"
Suggestions were made, and people voted, unfortunately I cannot seem to find the talk anymore. If memory serves me well, what made it to the top of the list was "Fast prototyping". And as an industry user (and a programmer by trade), I find the same advantages apply: We're working on a blockchain project, and the ability to prototype quickly, and iterate through multiple ideas / improvements without too much trouble is amazing.
I'm curious, given your use case, how did you feel about @safe by default ?

> The only time where I had some issues were that time I wrote a Windows trojan in D. I needed to do some pretty convoluted stuff to load my code in just the right way to avoid antiviruses and it wasn't fun to do it in D.  Cross-compiling this proved another hurdle. It would have been easier in C++ directly. But I can't very well blame anyone not to make malware development easier and none of the issues I had should impact normal development.

If I understand correctly, you're using Linux + DMD ? Have you tried LDC ? It should make those use cases much easier (past the initial getting used to it, but you can just `alias dmd = ldmd2`).

> The low number of native libraries is not ideal. Yes, I can use a C library or port it from another language but I've generally got less than two days to work on a project. Any time spent on this is time not spent on actually solving the problem.

Have you tried https://github.com/jacob-carlborg/dstep ? Did you encounter any issue ?

> For the same reason I am a huge advocate of battery-included stdlib. I know many are of the opinion of moving as much as possible from Phobos into dub.  For me any library that isn't in dub is one more library I have to evaluate, try, keep track of when they update. It's lost time. Huge projects can have a mindset of "we'll manage our own version anyway" but my projects are written in a few days and are useful a week or two at most. So go Phobos, go! We also need lots of libraries but that is the kind of things that is built upon a strong standard library, not at its expense.

Out of curiosity, if you have used Go extensively, was the solution they came up with (Github URL) good enough for you use case ?

On Monday, 29 June 2020 at 12:18:02 UTC, Guillaume Piolat wrote:
>
> I'll bite:
>
> I've been defending:
>    - unsafe by default,
>    - throw by default,
>    - impure by default,
>    - mutable by default
>
> for years. To me those defaults are the right ones, and it warms me up that, as it seems, your D usage would be made _almost impossible_ if the permissive default changed to a strict one.
>
> I've been in a similar situation where you have to create an internal tool as fast as possible because of limited political support, and all those features  - that are completely rational and acceptable in the right context - may reveal a liability in the trenches.
>
> So I hope people will keep the important use cases of _bad programs_, may I say in mind when improving D.

This is absolutely true. These are the reason why D is useful to me.

That said, I have mixed feelings since it's also true that, should these features be introduced by default, they would help with many bugs I see that cause security issues. As much as I like my job I would be happier if it weren't needed.

It wouldn't fix everything since the most common issues are at a much higher level (such as forgetting to protect a web page with authentication). It would help with memory corruptions and possibly race conditions since it would incentivise safer use of multithreading.

I'd be stuck though. Please don't. I don't want to write Go :p