On 9/17/20 8:07 PM, wjoe wrote:
> 
> Not a reply to this post in particular but to all the ones I've read so far.
> 
> If I understand correctly. Vibe parses the form data and writes all files to disk. Where to ?

See the code here: https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d#L311

> Can I configure it ? I don't want libraries to just write data to my file systems without me setting this up. Nowhere did I find this behavior described in the docs.

No, not at the moment. Which is why I was saying, it could be an enhancement request to vibe.

> And if not, how is data processed with a 10mb file upload followed by a few number fields ?

All the data is processed before the accessor to the form data or the file data. It HAS to be this way, as the data is still on the incoming network socket.

> It needs to read all of the file data to get to the other data fields, doesn't it ?

Yes

> 
> I'm sorry this is completely counter intuitive. I can understand the memory/security risks and all but I have no intention to hack, DOS or however else disrupt my private server in my private network with garbage data. I just want to get the data in a byte[].

Again, enhancement request needed. The code currently is hard-coded to write to disk.

> 
> Why does the lib not simply reject files that are unreasonably (configurable) big ?

If you had 1000 requests being processed simultaneously, and each of those requests provided 10MB files, then you now need potentially 10GB of RAM to hold those requests. This doesn't scale when the application is unknown to vibe.

But again, solved with an enhancement that allows you to process the data in your code. I'll file the enhancement request for you, as I think it's a nice addition.

> Writing files to disk in order to then needing to copy them somewhere else or to read them back into memory for further processing sounds, above all else, incredibly inefficient.

Agreed. In my case, it was an actual copy, as the location of the stored data was on a different filesystem than the temporary files.

> I might not even want to keep the file and drop it.

Yep, it's a waste in that case.

> 
> I guess it's no problem to parse the data myself, but then what's the point in using a framework ?

Agreed.

> Are there other frameworks besides vibe that can do what I want ?

In D, I'm not sure what the other frameworks do. I believe there are others if you search on code.dlang.org, you should be able to find some.

> I'm sorry for the rant, developing this kind of software is a pain in the drain and stresses me out to no end. It sucked hard in the past with php and decades later with python, ruby, D, you name it it still sucks ;)

web development sucks in general ;) Yet, we all still do it.

-Steve

On 9/18/20 8:39 AM, Steven Schveighoffer wrote:
> But again, solved with an enhancement that allows you to process the data in your code. I'll file the enhancement request for you, as I think it's a nice addition.

https://github.com/vibe-d/vibe.d/issues/2478

-Steve

On Friday, 18 September 2020 at 12:39:43 UTC, Steven Schveighoffer wrote:
> On 9/17/20 8:07 PM, wjoe wrote:
>> [...]
>
> See the code here: https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d#L311
>
>> [...]
>
> No, not at the moment. Which is why I was saying, it could be an enhancement request to vibe.
>
>> [...]
>
> All the data is processed before the accessor to the form data or the file data. It HAS to be this way, as the data is still on the incoming network socket.
>
>> [...]
>
> Yes
>
>> [...]
>
> Again, enhancement request needed. The code currently is hard-coded to write to disk.
>
>> [...]
>
> If you had 1000 requests being processed simultaneously, and each of those requests provided 10MB files, then you now need potentially 10GB of RAM to hold those requests. This doesn't scale when the application is unknown to vibe.
>
> But again, solved with an enhancement that allows you to process the data in your code. I'll file the enhancement request for you, as I think it's a nice addition.
>
>> [...]
>
> Agreed. In my case, it was an actual copy, as the location of the stored data was on a different filesystem than the temporary files.
>
>> [...]
>
> Yep, it's a waste in that case.
>
>> [...]
>
> Agreed.
>
>> [...]
>
> In D, I'm not sure what the other frameworks do. I believe there are others if you search on code.dlang.org, you should be able to find some.
>
>> [...]
>
> web development sucks in general ;) Yet, we all still do it.
>
> -Steve

I was a little tired yesterday when I read the replies. After a couple hours of sleep and reading them again it makes a lot more sense to me.

Also your suggestion about a direct call to vibe.inet.webform.parseFormData with a specific handler for files and form data is a lot better than access via byte[].

Thanks for your help. Very much appreciated :)

On Friday, 18 September 2020 at 12:58:29 UTC, Steven Schveighoffer wrote:
> On 9/18/20 8:39 AM, Steven Schveighoffer wrote:
>> But again, solved with an enhancement that allows you to process the data in your code. I'll file the enhancement request for you, as I think it's a nice addition.
>
> https://github.com/vibe-d/vibe.d/issues/2478
>
> -Steve

Awesome! Thanks a ton :)

On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:
> On Thursday, 17 September 2020 at 22:33:46 UTC, Steven Schveighoffer wrote:
>> On 9/17/20 6:13 PM, aberba wrote:
>>> On Thursday, 17 September 2020 at 21:57:37 UTC, Steven Schveighoffer wrote:
>>>> On 9/17/20 1:08 PM, wjoe wrote:
>>>>> [...]
>>>>
>>>> the `files` property actually does the processing only when you call it.
>>>>
>>>> If you access the `bodyReader` property directly, you can process that data yourself. You can even register a web interface function with an `InputStream` parameter type, and it will be bound to the body data.
>>> 
>>> I'm not sure I understand how to do this and parser the files in memory.
>>
>> So an HTTP request with form data will come in with the headers parsed, but the data is still on the network stream.
>>
>> The first time you access `files`, it processes the stream data, and splits it into form data and file data, saves the files, and then gives you back the file dictionary so you can use them.
>>
>> If instead, you access `bodyReader`, YOU get to process the form data and file data.
>>
>>>>
>>>> I've done this with my REST interface, though that's not form data.
>>>>
>>>> That's not a great API, though. I would love to see vibe.d allow a direct call to vibe.inet.webform.parseFormData with a specific handler for files and form data.
>>> Can we file an issue for this? Because I'm very interested in having this resolved
>>
>> You can always file an issue! https://github.com/vibe-d/vibe.d/issues
>>
>> There may already be one in there.
>>
>>> There's potential to results in out of memory condition. Its a know issues. A complete parser (like multer in nodejs) allowance you to limit file size as well for error handling.
>>
>> Meh, this is D :) we should be able to just process the data and do whatever we want with it. What I would like to see is vibe provide the parsing of form data, and just give me the data as it comes (kind of like a SAX parser). Maybe just a property in the HTTPServerRequest that I can set that says "use this callback when you get Form File data".
>>
>>>> I've done this with my REST interface, though that's not form data.
>>> 
>>> Can you share your code for this?
>>
>> Heh, this is not form data, it's just file data, raw on the stream. So I have a function like:
>>
>> ```
>> class FileRestImpl
>> {
>>     @path(":cat/:id/:uuid/upload")
>>     @getAuth
>>     void postUpload(HTTPServerResponse res, string _cat, int _id, string _uuid, InputStream stream, Nullable!string md5sum, NRMAuthInfo _authInfo)
>>     {
>>         ...
>>     }
>> }
>> ```
>>
>> You can see, I take an InputStream as a parameter -- the data comes in there. I just read it and save it to a file (in the correct location) anyway, verifying the md5sum is valid.
>>
>> -Steve
>
> Not a reply to this post in particular but to all the ones I've read so far.
>
> If I understand correctly. Vibe parses the form data and writes all files to disk. Where to ?
> Can I configure it ? I don't want libraries to just write data to my file systems without me setting this up. Nowhere did I find this behavior described in the docs.
> And if not, how is data processed with a 10mb file upload followed by a few number fields ?
> It needs to read all of the file data to get to the other data fields, doesn't it ?
>
> I'm sorry this is completely counter intuitive. I can understand the memory/security risks and all but I have no intention to hack, DOS or however else disrupt my private server in my private network with garbage data. I just want to get the data in a byte[].

That's what I was trying to answer. When Steve said meh, he probably didn't get what I said. Probably its because of my typos.

This sort of convenience and productivity benefit is part of why I use Node.Js in the job when I need to get things done....and not D yet. There are several pieces and bits you can't write yourself when working on projects.

In this case you want to get the file(s) in memory...in the form of bytes (or buffer) and probably set a file size limit. Its all doable through a library but such a library doesn't exist in D yet. At least not that I know of.

Its why I mentioned that multer[1] in Node.Js able to do that...hence the advantage. Its built for the express framework...meaning such library can be built to work with vibe.d. Not everything can be built into vibe.d..and I think that'll even make it bloated for other uses case. Its need an ecosystem of third-party libraries.

In the case of the vibe.d form, data and files are handled using this implementation[2] so its a reference to such a form parser implementation...with support for a storage parameter for either a MemoryStore or SessionStore. Multer does it pretty cleanly.

1. Multer: https://www.npmjs.com/package/multer
2. https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d


>
> Why does the lib not simply reject files that are unreasonably (configurable) big ?
> Writing files to disk in order to then needing to copy them somewhere else or to read them back into memory for further processing sounds, above all else, incredibly inefficient.
> I might not even want to keep the file and drop it.

Not implemented yet.

>
> I guess it's no problem to parse the data myself, but then what's the point in using a framework ?

Vibe.d still lacks many things I personally need... there's simply not enough ecosystem third-party libraries.

>
> Are there other frameworks besides vibe that can do what I want ?

I don't think so. Vibe.d is the one with most features. But Does it need to be part of the framework itself?

>
> I'm sorry for the rant, developing this kind of software is a pain in the drain and stresses me out to no end. It sucked hard in the past with php and decades later with python, ruby, D, you name it it still sucks ;)

Its sucks but the absence of libraries makes it suck more. Its only a matter of time...as long as more people try vibe.d and develop tools around it.

I love D but I also know its not ready for every task...unless you're willing to write some things yourself.

On Friday, 18 September 2020 at 22:02:07 UTC, aberba wrote:
> In this case you want to get the file(s) in memory...in the form of bytes (or buffer) and probably set a file size limit. Its all doable through a library but such a library doesn't exist in D yet. At least not that I know of.

I actually added *exactly* this to cgi.d in... 2010 if I remember right. I even kinda documented it: http://dpldocs.info/experimental-docs/arsd.cgi.Cgi.UploadedFile.contentInMemory.html

The threshold where it goes to a file is right now 10 MB and not configurable, but I've been meaning to go back and clean up this api a little. The max file size it accepts is configurable <http://dpldocs.info/experimental-docs/arsd.cgi.GenericMain.html> but the threshold where it goes from memory to file is not.

Though I should note that if vibe is putting it in a temporary file it probably realistically stays in memory anyway.... this whole thing might be about nothing since the OS is pretty good about optimizing temp files.

Just I have bigger things to take care of right now (or should I say smaller things?!)

On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:

> Are there other frameworks besides vibe that can do what I want?

Just FYI, there is also:

https://code.dlang.org/packages/hunt-framework


I never used myself, you need to investigate.

On Friday, 18 September 2020 at 22:02:07 UTC, aberba wrote:
>> [...]
>
> That's what I was trying to answer. When Steve said meh, he probably didn't get what I said. Probably its because of my typos.
>
> This sort of convenience and productivity benefit is part of why I use Node.Js in the job when I need to get things done....and not D yet. There are several pieces and bits you can't write yourself when working on projects.
>
> In this case you want to get the file(s) in memory...in the form of bytes (or buffer) and probably set a file size limit. Its all doable through a library but such a library doesn't exist in D yet. At least not that I know of.
>

Yes it would be convenient and thanks for the links but since I'm not familiar with Node.js at all parsing the input myself will be better than getting into Node.


> Vibe.d still lacks many things I personally need... there's simply not enough ecosystem third-party libraries.

My first impression of vibe.d is that the author(s) made a presumptions about the one way things should be done(tm) and if that's not your use case too bad for you.
That's where most of my displeasure of using vibe.d comes from and that those aren't described in the documentation - not so much because of the things that aren't implemented (yet).

Handling file uploads is one example, another is command line arguments.
The presumption here is there is vibe and there can only be vibe. It takes them from Runtime.args. Duh?
This didn't make sense to me until I saw example where the initialization of vibe was done in a module constructor.
Looks cool on a cursory look but is incredibly impractical if you don't want to do it that way. And that's the gist - vibe.d is incredibly impractical if your use case doesn't exactly match the author(s) assumptions of the one way to do things(tm).
And there are issues with that, too. E.g. in the example the server will be started before command line args are completely processed. That means if there are wrong or missing or extra args the application aborts and leaks OS resources.
The way I would have implemented it is to take args from Runtime.args by default but allow passing an args[] to a vibe args-handler. I could then parse whatever args I'm interested in in my favorite way and pass the rest to vibe.

But you can handle those with vibe getOption or some such - why don't you want to do comandline args processing with vibe you ask ?
Because 1. there's std.getopt which is awesome, and
2. If I don't depend on vibe to parse them and I have a build configuration with version="NO_SERVER" I don't even need to link against vibe and its dependencies.

By using vibe I feel like I need to bend myself like a snake and jump through the hoops of vibe's one way to do it. You save a lot of time here and there and then you lose half a day because of some stupid road block like the above.

On Friday, 18 September 2020 at 13:13:16 UTC, wjoe wrote:
> On Friday, 18 September 2020 at 12:58:29 UTC, Steven Schveighoffer wrote:
>> On 9/18/20 8:39 AM, Steven Schveighoffer wrote:
>>> But again, solved with an enhancement that allows you to process the data in your code. I'll file the enhancement request for you, as I think it's a nice addition.
>>
>> https://github.com/vibe-d/vibe.d/issues/2478
>>
>> -Steve
>
> Awesome! Thanks a ton :)

My preferable way to handle such thing is: convert incoming data into input range of immutable(ubyte)[] and let user to consume this range (and handle it as it wish - transform, store in memory as is, or dump to disk)

On Saturday, 19 September 2020 at 11:11:21 UTC, ikod wrote:
> On Friday, 18 September 2020 at 13:13:16 UTC, wjoe wrote:
>> On Friday, 18 September 2020 at 12:58:29 UTC, Steven Schveighoffer wrote:
>>> On 9/18/20 8:39 AM, Steven Schveighoffer wrote:
>>>> But again, solved with an enhancement that allows you to process the data in your code. I'll file the enhancement request for you, as I think it's a nice addition.
>>>
>>> https://github.com/vibe-d/vibe.d/issues/2478
>>>
>>> -Steve
>>
>> Awesome! Thanks a ton :)
>
> My preferable way to handle such thing is: convert incoming data into input range of immutable(ubyte)[] and let user to consume this range (and handle it as it wish - transform, store in memory as is, or dump to disk)

sorry, this looks exactly like it described in proposal (and this is how requests works).