I found this [1] but unfortunately the post this refers to is a dead link and the content, unfortunately, didn't tell me anything that I didn't already find in the docs.

What I can get from the form is the form fields with content, the field name for the file upload and the file name. But the file name is useless to me because I need the file content.

Where is it stored ?

[1] https://aberba.com/2017/multiple-file-upload-in-vibe-d/

On Thursday, 17 September 2020 at 16:00:33 UTC, wjoe wrote:
> I found this [1] but unfortunately the post this refers to is a dead link and the content, unfortunately, didn't tell me anything that I didn't already find in the docs.
>
> What I can get from the form is the form fields with content, the field name for the file upload and the file name. But the file name is useless to me because I need the file content.
>
> Where is it stored ?
>
> [1] https://aberba.com/2017/multiple-file-upload-in-vibe-d/

hi, you can access HTTPServerRequest.files which contains the uploaded file.

Example in real code: https://github.com/WebFreak001/ImageUploader/blob/master/source/app.d#L141-L159

Documentation: https://vibed.org/api/vibe.inet.webform/FilePart

Note: the file is only downloaded from the client / stored on disk once you access the files or the form property, though this isn't documented.

I don't believe the temp file behavior is customizable as it is hardcoded here to write to a temporary file (which is called on form parsing): https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d#L311

However if you really wanted to (but I'd advise against it) you could parse the form data from the HTTPServerRequest.bodyReader yourself

On Thursday, 17 September 2020 at 16:32:55 UTC, WebFreak001 wrote:
> On Thursday, 17 September 2020 at 16:00:33 UTC, wjoe wrote:
>> I found this [1] but unfortunately the post this refers to is a dead link and the content, unfortunately, didn't tell me anything that I didn't already find in the docs.
>>
>> What I can get from the form is the form fields with content, the field name for the file upload and the file name. But the file name is useless to me because I need the file content.
>>
>> Where is it stored ?
>>
>> [1] https://aberba.com/2017/multiple-file-upload-in-vibe-d/
>
> hi, you can access HTTPServerRequest.files which contains the uploaded file.
>
> Example in real code: https://github.com/WebFreak001/ImageUploader/blob/master/source/app.d#L141-L159
>
> Documentation: https://vibed.org/api/vibe.inet.webform/FilePart
>
> Note: the file is only downloaded from the client / stored on disk once you access the files or the form property, though this isn't documented.

Every post or example I found copies the file, like your code does, too. Why is that ? The content of the file upload is embedded in the form data. There's no need for temporary files or copying at all.

On top of that, if I upload a file to a server which is on a different PC on a different file system, how am I supposed to read the file from disk on a remote file system ?

This makes no sense.

What I want is something like this:

> ~$ cat /var/log/temperatures.log
>
> temp_1=22;temp_2=28
> temp_1=21;temp_2=25


> ~$ curl -F "temperature_log=@/var/log/temperatures.log" 192.168.1.1:20222/temperature_upload

> ~$ nc -l 127.0.0.1 20222
>
> POST /temperature_upload HTTP/1.1
> Host: 192.168.1.1:20222
> User-Agent: curl/7.72.0
> Accept: */*
> Content-Length: 245
> Content-Type: multipart/form-data; boundary=------------------------c73c71472ff9e7d5
>
> --------------------------c73c71472ff9e7d5
> Content-Disposition: form-data; name="temperature_log"; filename="/var/log/temperatures.log"
> Content-Type: application/octet-stream
>
> temp_1=22;temp_2=28
> temp_1=21;temp_2=25
>
> --------------------------c73c71472ff9e7d5--


void upload(HttpRequest.. req, blah)
{
   auto file = "temperature_log" in req.files;
   if (file) {
      string temp_data_raw = file.data;
      assert ("temp_1=22;temp_2=28\ntemp_1=21;temp_2=25" == temp_data_raw);
   }
}

On Thursday, 17 September 2020 at 16:32:55 UTC, WebFreak001 wrote:
> On Thursday, 17 September 2020 at 16:00:33 UTC, wjoe wrote:
>> I found this [1] but unfortunately the post this refers to is a dead link and the content, unfortunately, didn't tell me anything that I didn't already find in the docs.
>>
>> What I can get from the form is the form fields with content, the field name for the file upload and the file name. But the file name is useless to me because I need the file content.
>>
>> Where is it stored ?
>>
>> [1] https://aberba.com/2017/multiple-file-upload-in-vibe-d/
>
> hi, you can access HTTPServerRequest.files which contains the uploaded file.
>
> Example in real code: https://github.com/WebFreak001/ImageUploader/blob/master/source/app.d#L141-L159
>
> Documentation: https://vibed.org/api/vibe.inet.webform/FilePart
>
> Note: the file is only downloaded from the client / stored on disk once you access the files or the form property, though this isn't documented.
>
> I don't believe the temp file behavior is customizable as it is hardcoded here to write to a temporary file (which is called on form parsing): https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d#L311
>
> However if you really wanted to (but I'd advise against it) you could parse the form data from the HTTPServerRequest.bodyReader yourself


Yeah I think what he wants is a way to write the file into an in-memory buffer. This is especially necessary it some environments where you can write to disk at all.


How that's done, even in node.js is to use a middleware for parsing library to parse the headers in such way. I often use multer (which is also based on busybody) to do that in nodejs.

Unfortunately I haven't used D to that extent to need such a thing... I in never hit that problem.

I wonder if using a psuedo file handler will work for the in-memory buffer thing.

On 9/17/20 1:08 PM, wjoe wrote:
> Every post or example I found copies the file, like your code does, too. Why is that ? The content of the file upload is embedded in the form data. There's no need for temporary files or copying at all.
> 
> On top of that, if I upload a file to a server which is on a different PC on a different file system, how am I supposed to read the file from disk on a remote file system ?
> 
> This makes no sense.
> 
> What I want is something like this:
> 
>> ~$ cat /var/log/temperatures.log
>>
>> temp_1=22;temp_2=28
>> temp_1=21;temp_2=25
> 
> 
>> ~$ curl -F "temperature_log=@/var/log/temperatures.log" 192.168.1.1:20222/temperature_upload
> 
>> ~$ nc -l 127.0.0.1 20222
>>
>> POST /temperature_upload HTTP/1.1
>> Host: 192.168.1.1:20222
>> User-Agent: curl/7.72.0
>> Accept: */*
>> Content-Length: 245
>> Content-Type: multipart/form-data; boundary=------------------------c73c71472ff9e7d5
>>
>> --------------------------c73c71472ff9e7d5
>> Content-Disposition: form-data; name="temperature_log"; filename="/var/log/temperatures.log"
>> Content-Type: application/octet-stream
>>
>> temp_1=22;temp_2=28
>> temp_1=21;temp_2=25
>>
>> --------------------------c73c71472ff9e7d5--
> 
> 
> void upload(HttpRequest.. req, blah)
> {
>     auto file = "temperature_log" in req.files;
>     if (file) {
>        string temp_data_raw = file.data;
>        assert ("temp_1=22;temp_2=28\ntemp_1=21;temp_2=25" == temp_data_raw);
>     }
> }
> 

the `files` property actually does the processing only when you call it.

If you access the `bodyReader` property directly, you can process that data yourself. You can even register a web interface function with an `InputStream` parameter type, and it will be bound to the body data.

I've done this with my REST interface, though that's not form data.

That's not a great API, though. I would love to see vibe.d allow a direct call to vibe.inet.webform.parseFormData with a specific handler for files and form data.

I think you can agree that it's not feasible to store arbitrary sized file contents in memory. But it certainly can provide a mechanism to handle it as it's read.

-Steve

On Thursday, 17 September 2020 at 21:57:37 UTC, Steven Schveighoffer wrote:
> On 9/17/20 1:08 PM, wjoe wrote:
>> [...]
>
> the `files` property actually does the processing only when you call it.
>
> If you access the `bodyReader` property directly, you can process that data yourself. You can even register a web interface function with an `InputStream` parameter type, and it will be bound to the body data.

I'm not sure I understand how to do this and parser the files in memory.

>
> I've done this with my REST interface, though that's not form data.
>
> That's not a great API, though. I would love to see vibe.d allow a direct call to vibe.inet.webform.parseFormData with a specific handler for files and form data.
Can we file an issue for this? Because I'm very interested in having this resolved

>
> I think you can agree that it's not feasible to store arbitrary sized file contents in memory. But it certainly can provide a mechanism to handle it as it's read.
>
> -Steve

There's potential to results in out of memory condition. Its a know issues. A complete parser (like multer in nodejs) allowance you to limit file size as well for error handling.

> I've done this with my REST interface, though that's not form data.

Can you share your code for this?

On 9/17/20 6:13 PM, aberba wrote:
> On Thursday, 17 September 2020 at 21:57:37 UTC, Steven Schveighoffer wrote:
>> On 9/17/20 1:08 PM, wjoe wrote:
>>> [...]
>>
>> the `files` property actually does the processing only when you call it.
>>
>> If you access the `bodyReader` property directly, you can process that data yourself. You can even register a web interface function with an `InputStream` parameter type, and it will be bound to the body data.
> 
> I'm not sure I understand how to do this and parser the files in memory.

So an HTTP request with form data will come in with the headers parsed, but the data is still on the network stream.

The first time you access `files`, it processes the stream data, and splits it into form data and file data, saves the files, and then gives you back the file dictionary so you can use them.

If instead, you access `bodyReader`, YOU get to process the form data and file data.

>>
>> I've done this with my REST interface, though that's not form data.
>>
>> That's not a great API, though. I would love to see vibe.d allow a direct call to vibe.inet.webform.parseFormData with a specific handler for files and form data.
> Can we file an issue for this? Because I'm very interested in having this resolved

You can always file an issue! https://github.com/vibe-d/vibe.d/issues

There may already be one in there.

> There's potential to results in out of memory condition. Its a know issues. A complete parser (like multer in nodejs) allowance you to limit file size as well for error handling.

Meh, this is D :) we should be able to just process the data and do whatever we want with it. What I would like to see is vibe provide the parsing of form data, and just give me the data as it comes (kind of like a SAX parser). Maybe just a property in the HTTPServerRequest that I can set that says "use this callback when you get Form File data".

>> I've done this with my REST interface, though that's not form data.
> 
> Can you share your code for this?

Heh, this is not form data, it's just file data, raw on the stream. So I have a function like:

```
class FileRestImpl
{
    @path(":cat/:id/:uuid/upload")
    @getAuth
    void postUpload(HTTPServerResponse res, string _cat, int _id, string _uuid, InputStream stream, Nullable!string md5sum, NRMAuthInfo _authInfo)
    {
        ...
    }
}
```

You can see, I take an InputStream as a parameter -- the data comes in there. I just read it and save it to a file (in the correct location) anyway, verifying the md5sum is valid.

-Steve

On Thursday, 17 September 2020 at 22:33:46 UTC, Steven Schveighoffer wrote:
> On 9/17/20 6:13 PM, aberba wrote:
>> On Thursday, 17 September 2020 at 21:57:37 UTC, Steven Schveighoffer wrote:
>>> On 9/17/20 1:08 PM, wjoe wrote:
>>>> [...]
>>>
>>> the `files` property actually does the processing only when you call it.
>>>
>>> If you access the `bodyReader` property directly, you can process that data yourself. You can even register a web interface function with an `InputStream` parameter type, and it will be bound to the body data.
>> 
>> I'm not sure I understand how to do this and parser the files in memory.
>
> So an HTTP request with form data will come in with the headers parsed, but the data is still on the network stream.
>
> The first time you access `files`, it processes the stream data, and splits it into form data and file data, saves the files, and then gives you back the file dictionary so you can use them.
>
> If instead, you access `bodyReader`, YOU get to process the form data and file data.
>
>>>
>>> I've done this with my REST interface, though that's not form data.
>>>
>>> That's not a great API, though. I would love to see vibe.d allow a direct call to vibe.inet.webform.parseFormData with a specific handler for files and form data.
>> Can we file an issue for this? Because I'm very interested in having this resolved
>
> You can always file an issue! https://github.com/vibe-d/vibe.d/issues
>
> There may already be one in there.
>
>> There's potential to results in out of memory condition. Its a know issues. A complete parser (like multer in nodejs) allowance you to limit file size as well for error handling.
>
> Meh, this is D :) we should be able to just process the data and do whatever we want with it. What I would like to see is vibe provide the parsing of form data, and just give me the data as it comes (kind of like a SAX parser). Maybe just a property in the HTTPServerRequest that I can set that says "use this callback when you get Form File data".
>
>>> I've done this with my REST interface, though that's not form data.
>> 
>> Can you share your code for this?
>
> Heh, this is not form data, it's just file data, raw on the stream. So I have a function like:
>
> ```
> class FileRestImpl
> {
>     @path(":cat/:id/:uuid/upload")
>     @getAuth
>     void postUpload(HTTPServerResponse res, string _cat, int _id, string _uuid, InputStream stream, Nullable!string md5sum, NRMAuthInfo _authInfo)
>     {
>         ...
>     }
> }
> ```
>
> You can see, I take an InputStream as a parameter -- the data comes in there. I just read it and save it to a file (in the correct location) anyway, verifying the md5sum is valid.
>
> -Steve

Not a reply to this post in particular but to all the ones I've read so far.

If I understand correctly. Vibe parses the form data and writes all files to disk. Where to ?
Can I configure it ? I don't want libraries to just write data to my file systems without me setting this up. Nowhere did I find this behavior described in the docs.
And if not, how is data processed with a 10mb file upload followed by a few number fields ?
It needs to read all of the file data to get to the other data fields, doesn't it ?

I'm sorry this is completely counter intuitive. I can understand the memory/security risks and all but I have no intention to hack, DOS or however else disrupt my private server in my private network with garbage data. I just want to get the data in a byte[].

Why does the lib not simply reject files that are unreasonably (configurable) big ?
Writing files to disk in order to then needing to copy them somewhere else or to read them back into memory for further processing sounds, above all else, incredibly inefficient.
I might not even want to keep the file and drop it.

I guess it's no problem to parse the data myself, but then what's the point in using a framework ?

Are there other frameworks besides vibe that can do what I want ?

I'm sorry for the rant, developing this kind of software is a pain in the drain and stresses me out to no end. It sucked hard in the past with php and decades later with python, ruby, D, you name it it still sucks ;)

Still, thanks a lot for all the replies everybody. Very much appreciated :)

On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:
> And if not, how is data processed with a 10mb file upload followed by a few number fields ?
> It needs to read all of the file data to get to the other data fields, doesn't it ?

Yes and no

On Friday, 18 September 2020 at 11:44:39 UTC, Atwork wrote:
> On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:
>> And if not, how is data processed with a 10mb file upload followed by a few number fields ?
>> It needs to read all of the file data to get to the other data fields, doesn't it ?
>
> Yes and no

Consider this:

> ~$ curl -F "temperature_log=@/var/log/temperatures.log" -F "field1=a" -F "field2+foo" 192.168.1.1:20222/temperature_upload

> ~$ nc -l 127.0.0.1 20222
>
> POST /temperature_upload HTTP/1.1
> Host: 192.168.1.1:20222
> User-Agent: curl/7.72.0
> Accept: */*
> Content-Length: 10486005
> Content-Type: multipart/form-data; boundary=------------------------c73c71472ff9e7d5
>
> --------------------------c73c71472ff9e7d5
> Content-Disposition: form-data; name="temperature_log"; filename="/var/log/temperatures.log"
> Content-Type: application/octet-stream
>
> temp_1=22;temp_2=28
> temp_1=21;temp_2=25
>
> [ ... 10 MB of data ... ]
>
> --------------------------c73c71472ff9e7d5
> Content-Disposition: form-data; name="field1"
>
> a
> --------------------------c73c71472ff9e7d5
> Content-Disposition: form-data; name="field2"
>
> foo
> --------------------------c73c71472ff9e7d5--

How is vibe going to extract field1 and field2 without processing the temperature_log field ?