CRC errors?

In response to a request from Volker, here are the results from using Unzip 6.0 on PersonalAlpha with the OpenVMS 8.3 Hobbyist CD. Note that I have tested cases where the ZIP archive is copied from the CD before Unzipping and also where Unzipping is performed directly from the CD media.

$ SH DEF
DKA100:[CRAP]
$
$ COPY DKA200:[KITS]BASIC016.ZIP;1 []
$
$ UNZIP -t BASIC016.ZIP
Archive: DKA100:[CRAP]BASIC016.ZIP;1
testing: basic016/documentation/ OK
testing: basic016/kit/ OK
testing: basic016/documentation/basic016_cover.ps OK
testing: basic016/documentation/basic016_cover.txt OK
testing: basic016/documentation/basic016_iguide.ps OK
testing: basic016/documentation/basic016_iguide.txt OK
testing: basic016/documentation/basic016_spd.ps OK
testing: basic016/documentation/basic016_spd.txt OK
testing: basic016/kit/basic016.a OK
testing: basic016/kit/basic016.b OK
testing: basic016/kit/basic016.c OK
testing: basic016/kit/basic016.d OK
testing: basic016/kit/basic016.e OK
testing: basic016/kit/basic016.f OK
testing: basic016/kit/basic016.g OK
testing: basic016/kit/basic016.h OK
testing: basic016/kit/basic016.i OK
No errors detected in compressed data of DKA100:[CRAP]BASIC016.ZIP;1.
$
$ UNZIP -t DKA200:[KITS]BASIC016.ZIP
Archive: DKA200:[KITS]BASIC016.ZIP;1
testing: basic016/documentation/ OK
testing: basic016/kit/ OK
testing: basic016/documentation/basic016_cover.ps OK
testing: basic016/documentation/basic016_cover.txt OK
testing: basic016/documentation/basic016_iguide.ps OK
testing: basic016/documentation/basic016_iguide.txt OK
testing: basic016/documentation/basic016_spd.ps OK
testing: basic016/documentation/basic016_spd.txt OK
testing: basic016/kit/basic016.a OK
testing: basic016/kit/basic016.b OK
testing: basic016/kit/basic016.c OK
testing: basic016/kit/basic016.d OK
testing: basic016/kit/basic016.e OK
testing: basic016/kit/basic016.f OK
testing: basic016/kit/basic016.g OK
testing: basic016/kit/basic016.h OK
testing: basic016/kit/basic016.i OK
No errors detected in compressed data of DKA200:[KITS]BASIC016.ZIP;1.
$

As you can see, there are no problems to report. I will perform the same test using FreeAXP release 283 and report back shortly.

Volker

Here is the same test performed with FreeAXP 283.

Note that I am using the exact same hardware and O/S that I used for this test on PersonalAlpha, which is Intel Core 2 Duo, LG DVD drive WinXP Pro SP3 32 bit.

It seems fairly conclusive from my test results that these CRC errors emanate from a bug in the FreeAXP emulator.

$ SH DEF
DKA100:[CRAP]
$
$ COPY DKA200:[KITS]BASIC016.ZIP;1 []
$ UNZIP -t BASIC016.ZIP
Archive: DKA100:[CRAP]BASIC016.ZIP;1
testing: basic016/documentation/ OK
testing: basic016/kit/ OK
testing: basic016/documentation/basic016_cover.ps OK
testing: basic016/documentation/basic016_cover.txt OK
testing: basic016/documentation/basic016_iguide.ps OK
testing: basic016/documentation/basic016_iguide.txt OK
testing: basic016/documentation/basic016_spd.ps bad CRC c174c123 (should
be 31b98e50)
testing: basic016/documentation/basic016_spd.txt OK
testing: basic016/kit/basic016.a OK
testing: basic016/kit/basic016.b bad CRC 7becf1cb (should be a444e794)
testing: basic016/kit/basic016.c bad CRC d4f8f664 (should be cef1e30f)
testing: basic016/kit/basic016.d
error: invalid compressed data to inflate
testing: basic016/kit/basic016.e OK
testing: basic016/kit/basic016.f bad CRC 92a0891b (should be 6f16cea2)
testing: basic016/kit/basic016.g OK
testing: basic016/kit/basic016.h
error: invalid compressed data to inflate
testing: basic016/kit/basic016.i
error: invalid compressed data to inflate
At least one error was detected in DKA100:[CRAP]BASIC016.ZIP;1.
$
$
$ UNZIP -t DKA200:[KITS]BASIC016.ZIP
Archive: DKA200:[KITS]BASIC016.ZIP;1
testing: basic016/documentation/ OK
testing: basic016/kit/ OK
testing: basic016/documentation/basic016_cover.ps OK
testing: basic016/documentation/basic016_cover.txt OK
testing: basic016/documentation/basic016_iguide.ps OK
testing: basic016/documentation/basic016_iguide.txt OK
testing: basic016/documentation/basic016_spd.ps bad CRC c174c123 (should
be 31b98e50)
testing: basic016/documentation/basic016_spd.txt OK
testing: basic016/kit/basic016.a OK
testing: basic016/kit/basic016.b bad CRC 7becf1cb (should be a444e794)
testing: basic016/kit/basic016.c bad CRC d4f8f664 (should be cef1e30f)
testing: basic016/kit/basic016.d
error: invalid compressed data to inflate
testing: basic016/kit/basic016.e OK
testing: basic016/kit/basic016.f bad CRC 92a0891b (should be 6f16cea2)
testing: basic016/kit/basic016.g OK
testing: basic016/kit/basic016.h
error: invalid compressed data to inflate
testing: basic016/kit/basic016.i
error: invalid compressed data to inflate
At least one error was detected in DKA200:[KITS]BASIC016.ZIP;1.
$

JohnC

JohnC,

thanks a lot. With this kind of cooperation, it looks like we've actually nailed the problem quickly !

Now it's up to Bruce and his engineering team to fix it...

Let me add one observation on your FreeAXP test:

I would NOT have expected the UNZIP test directly from the CD to fail, because I don't think UNZIP would actually be using large IO block sizes, but the 'corrupted' file data (from the COPY operation) may have still been in the XFC cache. Could you try again with MOUNT/NOCACHE DKA200: ? Or UNZIP from CD first (before COPY).

Volker.<hr>

With nocache, blocks are corrupted, but not zeroed.

For the file appdev010.zip, starting at block 118, two blocks at exactly 124 blocks apart are corrupted throughout the entire file.

The data pattern looks like nothing from the device was read for those blocks.

blocks 118, 119,
242. 243.
366, 367
490, 491
614, 615
738, 739
862, 863
986, 987
1110, 1111
1234, 1235
1358, 1359
1482, 1483
1606, 1607
1730. 1731
1854, 1855

John,

if using XFC, it will probably allocate demand-zero pages and fill the data read into those pages using 'DMA' (being emulated in FreeAXP). If skipping XFC (you can also reduce VCC_MAX_IO_SIZE to &lt; 124), COPY might present 'dirty' buffers, which are being filled by DMA. If the DMA algorithm has a problem, you might see the 'previous data' in those buffers.

you could also check the corrupted VBN numbers after COPY/BLOCK_SIZE=nnn by varying the IO block size. We know that nnn=63 works. When does the problem start, with nnn=64 ? What happens with nnn=127 ?

Now when the corrupted blocks do NOT contain ZEROs, but some data (after MOUNT/NOCACHE), can you find out, if this data is from 'other' blocks ? If so, from which ones ?

We know it's the 'big' reads from CD, which are causing the problem. Reading from the virtual disk is fine, right ? If you COPY a 'good' zip file from disk to disk, it should NOT get corrupted. You can do a SET FILE/NOCACHE first to eliminate XFC.

You can also trace the IOs to the CD by using LDDRIVER. It has a TRACE mode, which can trace all IOs. You create a LDA1: device, which sends all IOs straight to the real device (FROG$DKA200: - your CD).

$ ld connect/replace FROG$DKA200: lda1:
$ MOUNT/OVER=ID LDA1:
$ LD TRACE LDA1:
$ COPY LDA1:&lt;x,y&gt;some-file.ZIP other_disk:&lt;other_dir&gt;
$ LD SHOW/TRACE LDA1:
...
$ LD NOTRACE LDA1:
$ DISM LDA1:
$ LD DISC LDA1:

Volker.<hr>

Thanks for all the feedback and testing. We'll have a go at reproducing the problem locally.