Phobos currently has packages for working with various archives (Zlib/Gzip, Zip), and it's probably reasonable to expect that support for more archive formats will be added in future. Before any more are added, it would probably be beneficial to define an interface that any modules adding support for archives should obey, and having archive related modules under std.archive for organisational purposes probably wouldn't hurt.

I saw a suggestion for improved archive support on the D wiki's wish list, and the code link 'std.archive' module in the review queue now 404s, so it looks to be the case that there isn't currently an effort to implement an archiving package for Phobos. What I would propose for such a package is:

- Split archives in to two categories: stream-based (Gzip, Zlib, Bzip, etc) and file-based (Zip, Tar).
- Make support for creating archives optional (e.g. by having IStreamArchive and IWritableStreamArchive).
- Make compression-related functionality separate (e.g. through an ICompressableArchive(T) interface, where T is an enum specifying algorithms). I'm not sure on this one, so ideas are especially welcome for it.
- A hierarchy of exceptions for implementations under std.archive rather than the current Phobos norm of everything having its own exceptions.

I'd like to know whether there would be demand for such a package, and what the community would want in terms of the API and features. I haven't yet written any code for the package as the API is going to be one of the most important parts, and I thought it would be best to have agreement on what is wanted before starting on it.

On 5/17/15 9:32 AM, Liam McSherry wrote:
> I'd like to know whether there would be demand for such a package, and
> what the community would want in terms of the API and features. I
> haven't yet written any code for the package as the API is going to be
> one of the most important parts, and I thought it would be best to have
> agreement on what is wanted before starting on it.

That'd be great. A few thoughts: (a) should use ranges through and through; (b) might be interesting to allow libarchive http://www.libarchive.org as engine. That's BSD licensed, but we can interface with it. Or at least use it as reference for low-level design. -- Andrei

On Sunday, 17 May 2015 at 17:02:13 UTC, Andrei Alexandrescu wrote:
> On 5/17/15 9:32 AM, Liam McSherry wrote:
>> I'd like to know whether there would be demand for such a package, and
>> what the community would want in terms of the API and features. I
>> haven't yet written any code for the package as the API is going to be
>> one of the most important parts, and I thought it would be best to have
>> agreement on what is wanted before starting on it.
>
> That'd be great. A few thoughts: (a) should use ranges through and through; (b) might be interesting to allow libarchive http://www.libarchive.org as engine. That's BSD licensed, but we can interface with it. Or at least use it as reference for low-level design. -- Andrei

Ranges definitely.

Rather than add libarchive as a dependency, it might be something to port the libarchive code to D. That would require licensing sections of std.archive under BSD rather than Boost, though.

On Sunday, 17 May 2015 at 17:32:10 UTC, Liam McSherry wrote:
> On Sunday, 17 May 2015 at 17:02:13 UTC, Andrei Alexandrescu wrote:
>> On 5/17/15 9:32 AM, Liam McSherry wrote:
>>> I'd like to know whether there would be demand for such a package, and
>>> what the community would want in terms of the API and features. I
>>> haven't yet written any code for the package as the API is going to be
>>> one of the most important parts, and I thought it would be best to have
>>> agreement on what is wanted before starting on it.
>>
>> That'd be great. A few thoughts: (a) should use ranges through and through; (b) might be interesting to allow libarchive http://www.libarchive.org as engine. That's BSD licensed, but we can interface with it. Or at least use it as reference for low-level design. -- Andrei
>
> Ranges definitely.
>
> Rather than add libarchive as a dependency, it might be something to port the libarchive code to D. That would require licensing sections of std.archive under BSD rather than Boost, though.

It doesn't looks like a good idea to me (to port to D but more simply to use this lib), just look to this:

https://github.com/libarchive/libarchive/issues?q=is%3Aopen+is%3Aissue

over 100 open bug reports ! The maintainers of the D package would have to follow this from near and it couldn't be well kept in sync. with phobos.

On Sunday, 17 May 2015 at 18:01:42 UTC, wabi wrote:
>
> It doesn't looks like a good idea to me (to port to D but more simply to use this lib), just look to this:
>
> https://github.com/libarchive/libarchive/issues?q=is%3Aopen+is%3Aissue
>
> over 100 open bug reports ! The maintainers of the D package would have to follow this from near and it couldn't be well kept in sync. with phobos.

I would still think that a "native" D implementation would be preferable to a dependency, although having some implementations in D doesn't preclude using libarchive for others.

On 5/17/2015 10:02 AM, Andrei Alexandrescu wrote:
> On 5/17/15 9:32 AM, Liam McSherry wrote:
>> I'd like to know whether there would be demand for such a package, and
>> what the community would want in terms of the API and features. I
>> haven't yet written any code for the package as the API is going to be
>> one of the most important parts, and I thought it would be best to have
>> agreement on what is wanted before starting on it.
>
> That'd be great. A few thoughts: (a) should use ranges through and through; (b)
> might be interesting to allow libarchive http://www.libarchive.org as engine.
> That's BSD licensed, but we can interface with it. Or at least use it as
> reference for low-level design. -- Andrei

I've thought about making such a package many times. A couple "archives" nobody ever thinks about are .lib and .a archives. Interestingly, they work pretty much just like .zip files, except without compression.

In fact, if I was still working on linkers, I'd use .zip as the library format!

Of course, like idiots, programmers keep inventing (badly) new archive formats.

Oh, and lest I forget, it should not have a streaming interface. It should have a range interface.

On Sunday, 17 May 2015 at 20:03:02 UTC, Walter Bright wrote:
> I've thought about making such a package many times. A couple "archives" nobody ever thinks about are .lib and .a archives. Interestingly, they work pretty much just like .zip files, except without compression.
>
> In fact, if I was still working on linkers, I'd use .zip as the library format!
>
> Of course, like idiots, programmers keep inventing (badly) new archive formats.

I think .a just fell in to relative obscurity after .tar came about, especially since .tar can store much longer file names (100 bytes + 155 bytes of prefix/path, vs .a's 16 bytes). Having support for it in Phobos out-of-the-box would probably be useful for anyone writing a compiler/linker, though.

> Oh, and lest I forget, it should not have a streaming interface. It should have a range interface.

If this is in response to "stream-based," apologies, I mustn't have made it clear what I meant. "Stream" was used for want of a better word, as Zlib/Gzip/etc store data as one big "stream" of data, while Zip/Tar/etc structure their data in to files and the like.

On Sunday, 17 May 2015 at 16:32:31 UTC, Liam McSherry wrote:
> Phobos currently has packages for working with various archives (Zlib/Gzip, Zip), and it's probably reasonable to expect that support for more archive formats will be added in future. Before any more are added, it would probably be beneficial to define an interface that any modules adding support for archives should obey, and having archive related modules under std.archive for organisational purposes probably wouldn't hurt.
>
> I saw a suggestion for improved archive support on the D wiki's wish list, and the code link 'std.archive' module in the review queue now 404s, so it looks to be the case that there isn't currently an effort to implement an archiving package for Phobos. What I would propose for such a package is:
>
> - Split archives in to two categories: stream-based (Gzip, Zlib, Bzip, etc) and file-based (Zip, Tar).
> - Make support for creating archives optional (e.g. by having IStreamArchive and IWritableStreamArchive).
> - Make compression-related functionality separate (e.g. through an ICompressableArchive(T) interface, where T is an enum specifying algorithms). I'm not sure on this one, so ideas are especially welcome for it.
> - A hierarchy of exceptions for implementations under std.archive rather than the current Phobos norm of everything having its own exceptions.
>
> I'd like to know whether there would be demand for such a package, and what the community would want in terms of the API and features. I haven't yet written any code for the package as the API is going to be one of the most important parts, and I thought it would be best to have agreement on what is wanted before starting on it.

We need two things actually:
1) compress package with set of commonly used compression algorithms.
   std.alg.compress comes to mind as package name.
2) VFS module/package with set of interfaces and reference implementations for as many formats as possible.
3) It would be extremely useful to have something like Java SPI (http://en.wikipedia.org/wiki/Service_provider_interface) in D. All the VFS implementations would then follow this standard and give us truly runtime-replaceable components. More about this: http://resources.sei.cmu.edu/asset_files/TechnicalNote/2002_004_001_13958.pdf

On Tuesday, 19 May 2015 at 12:31:15 UTC, Dejan Lekic wrote:
> We need two things actually:
> 1) compress package with set of commonly used compression algorithms.
>    std.alg.compress comes to mind as package name.

A compression package would certainly be useful as the number of supported archives grows.

> 2) VFS module/package with set of interfaces and reference implementations for as many formats as possible.
> 3) It would be extremely useful to have something like Java SPI (http://en.wikipedia.org/wiki/Service_provider_interface) in D. All the VFS implementations would then follow this standard and give us truly runtime-replaceable components. More about this: http://resources.sei.cmu.edu/asset_files/TechnicalNote/2002_004_001_13958.pdf

I imagine a VFS interface and an interface for file-based archives like Zip/Tar/7z would be fairly similar. Support for virtual filesystems could probably be implemented as part of the archive package.

WRT runtime replaceability, the defining of a single set of exceptions (without making them too specific) for all std.archive submodules should help. The fact that there aren't any "standard" exceptions seems to come up a fair bit (DIP33, for example).

On 2015-05-17 18:32, Liam McSherry wrote:
> Phobos currently has packages for working with various archives
> (Zlib/Gzip, Zip), and it's probably reasonable to expect that support
> for more archive formats will be added in future. Before any more are
> added, it would probably be beneficial to define an interface that any
> modules adding support for archives should obey, and having archive
> related modules under std.archive for organisational purposes probably
> wouldn't hurt.
>
> I saw a suggestion for improved archive support on the D wiki's wish
> list, and the code link 'std.archive' module in the review queue now
> 404s, so it looks to be the case that there isn't currently an effort to
> implement an archiving package for Phobos. What I would propose for such
> a package is:
>
> - Split archives in to two categories: stream-based (Gzip, Zlib, Bzip,
> etc) and file-based (Zip, Tar).
> - Make support for creating archives optional (e.g. by having
> IStreamArchive and IWritableStreamArchive).
> - Make compression-related functionality separate (e.g. through an
> ICompressableArchive(T) interface, where T is an enum specifying
> algorithms). I'm not sure on this one, so ideas are especially welcome
> for it.
> - A hierarchy of exceptions for implementations under std.archive rather
> than the current Phobos norm of everything having its own exceptions.
>
> I'd like to know whether there would be demand for such a package, and
> what the community would want in terms of the API and features. I
> haven't yet written any code for the package as the API is going to be
> one of the most important parts, and I thought it would be best to have
> agreement on what is wanted before starting on it.

Support for RAR archives would be nice.

-- 
/Jacob Carlborg