<< ctxmodel.net

Here's the BWT output CM compression comparison, version 2.

o1rc9f added:

  • rc_sh2f bitwise 32-bit arithmetic coder added, but didn't change anything, so the planned implementation of 64-bit coder was cancelled;
  • Node2i update tables are now generated at runtime, and static tables which were used before had been removed... still this allowed to win only 50 bytes on calgary;
  • mixer limit masks were expanded (another -100 bytes);
  • some minor Node2i update modifications and a lot of various reformatting which didn't affect the compression.
o1rc9g added:
  • adding run-lengths to mixer contexts failed (adding it to counter contexts failed earlier);
  • using the o1rc9f for ranking and additional order0 for coding (secondary model)... not exactly failed, but its results were too pitiful to try adding it to the mix or something;
  • another order0 submodel was added and mixed with other two order0's before the mix with order1, but then it was turned into order1 by optimizer, and after tuning it shows better results than the order2 coder... Most funny thing though, is that this new order1 submodel is so "fast", that it practically assigns all the probability to the LES (last encountered symbol) in the context. And that looks like another proof of bitwise coding inefficiency at this type of data... guess its really the time to start writing an unary bytewise coder.

Note that the best BWT results on world95.txt from maximumcompression.com seem to be 467793 from UHBC and 457426 from M03.

Also paq8o6 -7 compresses calgary_bwt to 772989 and calgary_rbwt to 771273, so there're clearly some possibilities for improvement as paq8's preprocessing tricks don't apply to BWT output, and its highly improbable that order3+ contexts can be of any help.

o1rc8mix(o0',mix(o0,mix(o1',o1))) 793254
o1rc8ao1rc8 with parameters tuned to world95.bwt 792668
o1rc9mix(mix(o1,o1'),mix(o0,o0')), tuned to calgary.tar.bwt787211
o1rc9aimproved version 785782
o1rc9bcleaned o1rc9a, tuned to calgary.tar.rbwt 785643
o1rc9ffurther cosmetic changes 785456
o1rc9gmix(mix(mix(o1,o1'),o1''),mix(o0,o0')) 784037
o2rcAmix(o2,mix(mix(o1,o1'),mix(o0,o0')))<.b> 784956
o2rcA1cleaned o2rcA, tuned to calgary.tar.rbwt 784684

calgarycalgary.tar
calgary.segsegmented with Shkarin's seg_file (default level)
calgary.bwtusing BWT.cpp
calgary_rbwtbyte reverse + BWT
calgary_dc1Binder's DC 0.99.307: dc -a -d calgary calgary_dc1
calgary_dc2dc -d calgary calgary_dc2 (BWT + all filters enabled, but coding disabled)
calgary_dc2dc -a -d +feodxlb calgary calgary_dc3 (BWT + all filters enabled, but coding disabled)
calgary_seg_bwtseg_file + BWT.cpp + separate compression + rar -m0 archive
calgary_seg_rbwtseg_file + reverse + BWT.cpp + separate compression + rar -m0 archive
calgary_seg_dc1seg_file + DC's BWT + separate compression + rar -m0 archive
calgary_seg_dc2seg_file + DC's BWT&filters + separate compression + rar -m0 archive
world95world95.txt
book1from calgary.tar
wcc386wcc386.exe

Coder: source bcm002 fpaq0pv4B o1rc8 o1rc8a o1rc9 o1rc9a o1rc9b o1rc9f o1rc9g o2rcA o2rcA1
 
BOOK1 768771 212672 441300 350899 351415 348405 350784 350205 350079 350014 349041 348794
BOOK1_bwt 768771   243630 212418 212308 211708 211614 211586 211393 211172 211267 211256
BOOK1_dc1 768860   242948 212364 212310 211398 211419 211348 211137 210992 211021 210963
BOOK1_dc2 544443   238980 205887 205687 205383 206673 206191 206268 206011 205687 205511
BOOK1_dc3 772091   237730 206519 206467 205660 205476 205424 205253 205101 205148 205117
BOOK1_rbwt 768771   242878 212300 212215 211299 211341 211267 211047 210908 210937 210871
BOOK1_seg_bwt 768984   243838 212631 212545 211931 211808 211787 211599 211374 211472 211461
BOOK1_seg_dc1 769153   243198 212583 212552 211612 211611 211541 211331 211188 211219 211158
BOOK1_seg_dc2 544736   239216 206149 205951 205608 206883 206404 206484 206226 205907 205729
BOOK1_seg_dc3 772405   237972 206830 206720 205894 205688 205638 205468 205323 205360 205332
BOOK1_seg_rbwt 768989   243096 212507 212392 211521 211536 211464 211252 211115 211133 211075
 
calgary 3152896 791434 1680338 1259218 1262575 1249383 1255225 1254212 1254275 1254908 1251319 1250665
calgary_bwt 3152896   981490 795650 794616 789541 787668 787842 787756 786387 786639 786809
calgary_dc1 3152994   981060 793969 793145 787908 786499 786381 786177 784772 785662 785394
calgary_dc2 2778210   996936 799444 798445 793799 794435 793650 793798 791925 792893 792723
calgary_dc3 3167109   975626 789337 788501 783530 782035 781934 781720 780359 781275 781033
calgary_rbwt 3152896   979796 793254 792668 787211 785782 785643 785456 784037 784956 784684
calgary_seg_bwt 3154474   971622 769050 768648 764586 763643 763543 763290 761969 762460 762437
calgary_seg_dc1 3155863   971390 767789 767292 762935 762464 762199 761838 760626 761131 761041
calgary_seg_dc2 2156735   948284 743547 742888 739230 740469 739322 739346 737745 738293 737770
calgary_seg_dc3 3157830   965844 762759 762001 757623 756979 756744 756477 755277 755808 755548
calgary_seg_rbwt 3154511   970564 766977 766433 762086 761693 761426 761067 759872 760401 760247
 
wcc386 536624 282718 400420 303075 305538 305179 305572 305706 305827 306622 305067 306063
wcc386_bwt 536624   315546 282822 282616 281931 281598 281570 281584 281117 281388 281427
wcc386_dc1 536702   315300 282203 282055 281333 281063 281009 280902 280470 280821 280725
wcc386_dc2 551470   315098 272417 272113 270794 270676 270603 270549 270014 270468 270348
wcc386_dc3 536707   315304 282208 282002 281338 281068 281015 280907 280474 280825 280730
wcc386_rbwt 536624   315256 282120 281949 281278 281013 280959 280852 280419 280771 280675
wcc386_seg_bwt 537152   308582 277535 277529 276998 276671 276600 276562 276138 276433 276499
wcc386_seg_dc1 537604   308470 277154 277179 276702 276384 276313 276203 275817 276122 276043
wcc386_seg_dc2 524852   308180 268315 268098 267051 266862 266782 266695 266169 266644 266586
wcc386_seg_dc3 536938   308684 277369 277384 276914 276597 276526 276411 276020 276319 276243
wcc386_seg_rbwt 537165   308200 276950 276919 276449 276146 276078 275963 275575 275882 275802
 
world95 2988578 474298 1862146 1333942 1340040 1321571 1318986 1319200 1319125 1317509 1310936 1310059
world95_bwt 2988578   648752 478050 475989 471773 472751 472760 472574 471879 471968 472046
world95_dc1 2988672   647194 476701 474654 470068 470739 470707 470571 469832 469969 470201
world95_dc2 2339029   639862 462754 461250 456848 458155 457734 457757 456537 457287 457026
world95_dc3 2852581   637804 466585 464540 460243 460761 460781 460619 459917 460153 460282
world95_rbwt 2988578   647124 476693 474635 469909 470578 470522 470390 469640 469820 470046
world95_seg_bwt 2989034   667084 482645 480699 476495 477246 477266 477089 476399 476383 476516
world95_seg_dc1 2989415   665804 481004 478954 474361 474856 474836 474716 473918 474105 474336
world95_seg_dc2 2360995   661318 469762 468428 463915 465021 464589 464605 463347 464075 463753
world95_seg_dc3 2854179   656426 470939 468962 464738 465086 465103 464926 464205 464383 464558
world95_seg_rbwt 2989045   665540 480833 478723 474149 474658 474644 474519 473717 473912 474134


2013-08-15 03:50:20                 >
2014-11-26 21:12:23                 >
2015-01-11 20:10:53                 >

Write a comment:

Name: