[Issue 3191] std.zlib.UnCompress errors if buffer is reused

Apr 18, 2014

Walter Bright

Jul 11, 2014

Justin Whear

Jul 11, 2014

Jul 11, 2014

Feb 11, 2016

Feb 12, 2016

Feb 12, 2016

Sep 14, 2017

github-bugzilla@puremagic.com

Sep 14, 2017

github-bugzilla@puremagic.com

Oct 16, 2017

github-bugzilla@puremagic.com

https://issues.dlang.org/show_bug.cgi?id=3191 Walter Bright <bugzilla@digitalmars.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|1.046 |D1 --

https://issues.dlang.org/show_bug.cgi?id=3191 Justin Whear <justin@economicmodeling.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |justin@economicmodeling.com --- Comment #3 from Justin Whear <justin@economicmodeling.com> --- The DEFLATE decompression algorithm relies on the results of previous blocks, as it tries to reuse the encoding tree. From the RFC: "Note that a duplicated string reference may refer to a string in a previous block; i.e., the backward distance may cross one or more block boundaries. However a distance cannot refer past the beginning of the output stream." (http://www.w3.org/Graphics/PNG/RFC-1951#huffman) So I think the bug should be clarified to this function allowing block reuse in the first place. --

https://issues.dlang.org/show_bug.cgi?id=3191 --- Comment #4 from Justin Whear <justin@economicmodeling.com> --- I should note that some amount of block reuse is possible if the blocks are sufficiently distant; the maximum distance for a back reference is 32,768 bytes. --

https://issues.dlang.org/show_bug.cgi?id=3191 --- Comment #5 from Stewart Gordon <smjg@iname.com> --- (In reply to Justin Whear from comment #3) > The DEFLATE decompression algorithm relies on the results of previous blocks, as it tries to reuse the encoding tree. From the RFC: "Note that a duplicated string reference may refer to a string in a previous block; i.e., the backward distance may cross one or more block boundaries. If I'm not mistaken, deflate blocks are independent of chunks that the datastream may be split into for decompression. (OK, maybe "block" was the wrong word in my original bug report.) That said, (In reply to Justin Whear from comment #4) > I should note that some amount of block reuse is possible if the blocks are sufficiently distant; the maximum distance for a back reference is 32,768 bytes. This says to me that it is a window of 32768 bytes (or maybe 32769 or 32770 bytes) that needs to be kept in memory at a given time, regardless of block or chunk boundaries. So why did I find that alternating between buffers works even if the buffers are much smaller than this? (Indeed, I've a vague recollection of finding that it works if each of the buffers is a single byte.) --

February 11, 2016

[Issue 3191] std.zlib.UnCompress errors if buffer is reused

Posted by Andre

Permalink

Andre

Permalink

https://issues.dlang.org/show_bug.cgi?id=3191

Andre <alverste@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alverste@gmail.com

--- Comment #6 from Andre <alverste@gmail.com> ---
(In reply to Stewart Gordon from comment #5)
> (In reply to Justin Whear from comment #3)
> > The DEFLATE decompression algorithm relies on the results of previous blocks, as it tries to reuse the encoding tree.  From the RFC: "Note that a duplicated string reference may refer to a string in a previous block; i.e., the backward distance may cross one or more block boundaries.
> 
> If I'm not mistaken, deflate blocks are independent of chunks that the datastream may be split into for decompression.  (OK, maybe "block" was the wrong word in my original bug report.)
> 
> That said,
> (In reply to Justin Whear from comment #4)
> > I should note that some amount of block reuse is possible if the blocks are sufficiently distant; the maximum distance for a back reference is 32,768 bytes.
> 
> This says to me that it is a window of 32768 bytes (or maybe 32769 or 32770 bytes) that needs to be kept in memory at a given time, regardless of block or chunk boundaries.  So why did I find that alternating between buffers works even if the buffers are much smaller than this?  (Indeed, I've a vague recollection of finding that it works if each of the buffers is a single byte.)

To your knowledge, is this fixed in std/zlib or etc/c/zlib? Or did you use a workaround?

--

https://issues.dlang.org/show_bug.cgi?id=3191 Andre <alverste@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|nobody@puremagic.com |alverste@gmail.com --

https://issues.dlang.org/show_bug.cgi?id=3191 --- Comment #7 from github-bugzilla@puremagic.com --- Commits pushed to master at https://github.com/dlang/phobos https://github.com/dlang/phobos/commit/5cf20bd8773e0f746c74b19137a03d699cdfe28b Fixed issues 3191 and 9505 std.zlib.UnCompress.uncompress() now consumes as much of the input buffer as possible and extends / reallocates the output buffer accordingly It also sets inputEnded = 1 when Z_STREAM_END is returned from inflate() so that additional data after the compressed stream is not consumed https://github.com/dlang/phobos/commit/8e47bfc54c106b222835db3c3a37e81b52ab2f04 Merge pull request #5720 from kas-luthor/fix-zlib Fix zlib issues 3191, 9505 and 8779 merged-on-behalf-of: MetaLang <MetaLang@users.noreply.github.com> --

https://issues.dlang.org/show_bug.cgi?id=3191 --- Comment #8 from github-bugzilla@puremagic.com --- Commits pushed to stable at https://github.com/dlang/phobos https://github.com/dlang/phobos/commit/5cf20bd8773e0f746c74b19137a03d699cdfe28b Fixed issues 3191 and 9505 https://github.com/dlang/phobos/commit/8e47bfc54c106b222835db3c3a37e81b52ab2f04 Merge pull request #5720 from kas-luthor/fix-zlib --

Forums