Hacker News new | past | comments | ask | show | jobs | submit login

EDIT: I've seen the file you're referencing is 4Gi-Base64. This file is not very compressible (75% with gzip). It's possible that igzip is simply storing the file or some parts of it without compression. This explain why it can be faster that libdeflate, because in this case igzip is using memcpy at decompression.



That is not the case. I have compressed the file with gzip and then tested decompression with multiple gzip decompressors. I.e., they all are fed the same compressed binary. Furthermore, I have used rapidgzip --analyze, to print out all deflate blocks, their compression types, and other information and statistics. All of the blocks are Dynamic Huffman compressed blocks.

I have already incorporated the ISA-L Huffman decoder into rapidgzip but that did not give full ISA-L speed. Ergo, I think the last performance comes from the inflate loop (decode Huffman, decode distance code if necessary, resolve references, repeat). This part is written in Assembler and seems to do some kind of speculative prefetching, i.e., already get the next Huffman Code symbol assuming that the current symbol is a literal or something like that. It's quite interesting but I doubt, or rather, I already tried a bit and failed to reproduce this kind of prefetching inside the C++ code but failed to do so. The compiler is probably rearranging everything anyway.

rapidgzip --analyze 4GiB-base64.gz | tail -100

    == Benchmark Profile (Cumulative Times) ==

    readDynamicHuffmanCoding : 0.903727 s (3.38252 %)
    readData                 : 25.8139 s (96.6175 %)
    Dynamic Huffman Initialization in Detail:
        Read precode       : 0.00975829 s (1.07978 %)
        Create precode HC  : 0.0341786 s (3.78196 %)
        Apply precode HC   : 0.0667695 s (7.38823 %)
        Create distance HC : 0.114097 s (12.6252 %)
        Create literal HC  : 0.678924 s (75.1249 %)


    == Alphabet Statistics ==

    Precode  : 123274 duplicates out of 126748 (97.2591 %)
    Distance : 597 duplicates out of 126748 (0.471013 %)
    Literals : 13799 duplicates out of 126748 (10.887 %)

    == Precode Code Length Count Distribution ==

    16 |==================== (126748)

    == Distance Code Length Count Distribution ==

    30 |==================== (126748)

    == Literal Code Length Count Distribution ==

             259 |=============        (50635)
    2.601250e+02 |==================== (74281)
                 |                     (1809)
             262 |                     (23)


    == Encoded Block Size Distribution ==

    185942 bits |                     (1)
                |                     
                |                     
                |                     
                |                     
                |                     
                |                     (2)
    207391 bits |==================== (126745)


    == Decoded Block Size Distribution ==

    30579 Bytes |                     (1)
                |                     
                |                     
                |                     
                |                     
                |                     
                |                     (1)
    34112 Bytes |==================== (126746)


    == Compression Ratio Distribution ==

    1.314967e+00 Bytes |                     (9)
                       |                     (1327)
                       |======               (17906)
    1.315808e+00 Bytes |==================== (52698)
                       |================     (42890)
                       |====                 (10876)
                       |                     (1001)
    1.316891e+00 Bytes |                     (41)

    == Deflate Block Compression Types ==

    Dynamic Huffman : 126748


Thank for your analyse. It seems it's a corner case, benchmarking a base64 encoded random file is not a typical case. igzip has no more advantage above libdeflate. It has only fast compression at the levels 0,1,2 but with mediocre compression ratio.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: