Do some more file format spelunking

`WarHammer.ani` turns out to be a regular `obj` file; `WarHammer.idx`
is partially decoded, but I'm struggling to link it to the former in
a reasonable way at the moment.
This commit is contained in:
2020-04-15 00:27:43 +01:00
parent e2ad8f61c1
commit acb7882549
7 changed files with 413 additions and 196 deletions

305
doc/formats/ani.md Normal file
View File

@@ -0,0 +1,305 @@
# `Anim/WarHammer.ani`
This turns out to simply be an [`obj`](obj.md#WarHammer.ani) file. However, some
other files are implicated in its interpretation:
- `Data/AniObDef.dat`
- `Idx/WarHammer.idx`
Including comments, the former is 4098 lines, giving approx. 45 lines for each
of the ~188 characters in the `ani`. That doesn't seem many, and there's no
obvious correspondence between the commented-on names (`SMOKE01`?) and the
viewed frames... but then, I've not viewed all the frames.
Still, I think a focus in `WarHammer.idx` (1,880,078 bytes, binary, so around
10KiB per character, in theory) is more reasonable.
Here's a list of operations on the file when `WH40K_TD.EXE` is instructed to
place an Ultramarine squad:
<details>
```
# Operation 1
_llseek(34, 12, [12], SEEK_SET) = 0
read(34, "\0\30\0\0X\4\0\0\0\0\0\0", 12) = 12
# Operation 2
_llseek(34, 6144, [6144], SEEK_SET) = 0
read(34, "\2\6\1\0 L\0\0\r\0\0\0", 12) = 12
_llseek(34, 6156, [6156], SEEK_SET) = 0
read(34, "\2\6\2\0\202L\0\0\r\0\0\0", 12) = 12
_llseek(34, 6168, [6168], SEEK_SET) = 0
read(34, "\2\6\3\0\344L\0\0\r\0\0\0", 12) = 12
_llseek(34, 6180, [6180], SEEK_SET) = 0
read(34, "\2\6\4\0FM\0\0\r\0\0\0", 12) = 12
_llseek(34, 6192, [6192], SEEK_SET) = 0
read(34, "\2\6\5\0\250M\0\0\r\0\0\0", 12) = 12
# Operation 3
_llseek(34, 19880, [19880], SEEK_SET) = 0
read(34, "4\0@\0@\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(34, "\0\0\6\0\4\0\0\0\6\0\4\0\0\0\6\0\4\0\0\0\5\0\4\0\0\0\5\0\4\0\0\0"..., 78) = 78
# Operation 4: repeat 1
# Operation 5: repeat 2
# Operation 6: repeat 3
# Operation 7
_llseek(34, 24, [24], SEEK_SET) = 0
read(34, "0\212\1\0X\4\0\0\210&\0\0", 12) = 12
# Operation 8
_llseek(34, 100912, [100912], SEEK_SET) = 0
read(34, "\2\6\1\0P\276\1\0\r\0\0\0", 12) = 12
_llseek(34, 100924, [100924], SEEK_SET) = 0
read(34, "\2\6\2\0\262\276\1\0\r\0\0\0", 12) = 12
_llseek(34, 100936, [100936], SEEK_SET) = 0
read(34, "\2\6\3\0\24\277\1\0\r\0\0\0", 12) = 12
_llseek(34, 100948, [100948], SEEK_SET) = 0
read(34, "\2\6\4\0v\277\1\0\r\0\0\0", 12) = 12
_llseek(34, 100960, [100960], SEEK_SET) = 0
read(34, "\2\6\5\0\330\277\1\0\r\0\0\0", 12) = 12
# Operation 9
_llseek(34, 114648, [114648], SEEK_SET) = 0
read(34, "4\0@\0@\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
# Operation 10
_llseek(34, 114648, [114648], SEEK_SET) = 0
read(34, "4\0@\0@\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(34, "\0\0\6\0\4\0\0\0\6\0\4\0\0\0\6\0\4\0\0\0\5\0\4\0\0\0\5\0\4\0\0\0"..., 78) = 78
# Operation 11: repeat 7
# Operation 12: repeat 8
# Operation 13: repeat 9
# lots of repeats
```
</details>
So what are we doing here? What did we read? Here's another view of it:
```
# Operation 1: read 12 bytes from 0x0C
0000000C 00 18 00 00 58 04 00 00 00 00 00 00 ....X.......
# Operation 7: read 12 bytes from offset 0x18
00000018 30 8A 01 00 58 04 00 00 88 26 00 00 0...X....&..
# Operation 2: 5 times, read 12 bytes from offset 0x1800
00001800 02 06 01 00 20 4C 00 00 0D 00 00 00 .... L......
0000180C 02 06 02 00 82 4C 00 00 0D 00 00 00 .....L......
00001818 02 06 03 00 E4 4C 00 00 0D 00 00 00 .....L......
00001824 02 06 04 00 46 4D 00 00 0D 00 00 00 ....FM......
00001830 02 06 05 00 A8 4D 00 00 0D 00 00 00 .....M......
# Operation 3a: read 20 bytes from offset 0x4DA8
00004DA8 34 00 40 00 40 00 01 00 00 00 00 00 4.@.@.......
00004DB4 00 00 00 00 00 00 00 00 ........
# Operation 3b: read 78 more bytes
00004DBC 00 00 06 00 04 00 00 00 06 00 04 00
00004DC8 00 00 06 00 04 00 00 00 05 00 04 00
00004DD4 00 00 05 00 04 00 00 00 03 00 04 00
00004DE0 00 00 04 00 FC FF 00 00 05 00 FC FF
00004DEC 00 00 06 00 FC FF 00 00 05 00 FC FF
00004DF8 00 00 06 00 FC FF 00 00 06 00 FC FF
00004E04 00 00 00 00 00 00
# Operation 8: 5 times, read 12 bytes from offset 0x18A30
00018A30 02 06 01 00 50 BE 01 00 0D 00 00 00 ....P.......
00018A3C 02 06 02 00 B2 BE 01 00 0D 00 00 00 ............
00018A48 02 06 03 00 14 BF 01 00 0D 00 00 00 ............
00018A54 02 06 04 00 76 BF 01 00 0D 00 00 00 ....v.......
00018A60 02 06 05 00 D8 BF 01 00 0D 00 00 00 ............
# Operation 9a: read 20 bytes from offset 0x1BFD8
0001BFD8 34 00 40 00 40 00 01 00 00 00 00 00 4.@.@.......
0001BFE4 00 00 00 00 00 00 00 00 ........
# Operation 9b: read 78 more bytes
0001BFEC 00 00 06 00 04 00 00 00 06 00 04 00
0001BFF8 00 00 06 00 04 00 00 00 05 00 04 00
0001C004 00 00 05 00 04 00 00 00 03 00 04 00
0001C010 00 00 04 00 FC FF 00 00 05 00 FC FF
0001C01C 00 00 06 00 FC FF 00 00 05 00 FC FF
0001C028 00 00 06 00 FC FF 00 00 06 00 FC FF
0001C034 00 00 00 00 00 00
```
Operation 1 contains `[18 00]`. Operation 2, at `0x1800`, contains `[A8 4D]` -
and operation 3 starts at `0x4DA8`. The pattern repeats for operations
7 -> 8 -> 9.
Assumimg operations 1 & 7 represent one type of record, while 2 & 8 represent
another, that would give us a 511-entry directory starting at 0xC. We know the
first 4 bytes represent an offset to find the type of record represented by
2 & 8. We don't know about the other 8 bytes.
| Offset | Size | Meaning |
| ------ | ---- | ------- |
| 0 | 4 | Position of type 2 record(s) |
| 4 | ? | Unknown |
The records represented by operations 2 & 8 both read 5x 12-byte records, but it
looks like they're arranged in blocks of 8. Nothing in type 1 records indicates
how many type 2 records there are.
| Offset | Size | Meaning |
| ------ | ---- | ------- |
| 0 | 2? | `02 06` - ??? |
| 2 | 2? | `n xx` - ??? `n` counts from 1 to 8 |
| 4 | 4 | Position of type 3 record |
The two type 3a and 3b records are identical to each other, it's hard to know
what's what.
None of the unknown numbers, however sliced, seem to be sprite indices for
`Anim/WarHammer.ani`.
Here's a trace of both `idx` (fd 34) and `ani` (fd 14) files when placing a
single librarian for me to dig into in more detail.
<details>
```
read(14, "~\337\2\0 \0\0\0\360\373\26\0\20\374\26\0\245J\303\30\0\0\0\0\0\0\0\0(\222R\0", 32) = 32
_llseek(34, 132, [132], SEEK_SET) = 0
read(34, "0|\t\0\230\0\0\0\210\370\0\0", 12) = 12
_llseek(34, 132, [132], SEEK_SET) = 0
read(34, "0|\t\0\230\0\0\0\210\370\0\0", 12) = 12
_llseek(34, 621616, [621616], SEEK_SET) = 0
read(34, "\2\1\0013P\203\t\0\r\0\0\0", 12) = 12
_llseek(34, 621628, [621628], SEEK_SET) = 0
read(34, "\2\1\0023\262\203\t\0\r\0\0\0", 12) = 12
_llseek(34, 621640, [621640], SEEK_SET) = 0
read(34, "\2\1\0033\24\204\t\0\r\0\0\0", 12) = 12
_llseek(34, 621652, [621652], SEEK_SET) = 0
read(34, "\2\1\0043v\204\t\0\r\0\0\0", 12) = 12
_llseek(34, 621664, [621664], SEEK_SET) = 0
read(34, "\2\1\0053\330\204\t\0\r\0\0\0", 12) = 12
_llseek(34, 623832, [623832], SEEK_SET) = 0
read(34, "4\0@\0@\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
_llseek(14, 509440, [509440], SEEK_SET) = 0
read(14, "\fM\266\th\16\0\0", 8) = 8
_llseek(14, 509448, [509448], SEEK_SET) = 0
read(14, "t[\266\t\376\16\0\0", 8) = 8
_llseek(14, 509456, [509456], SEEK_SET) = 0
read(14, "rj\266\tg\17\0\0", 8) = 8
_llseek(14, 509464, [509464], SEEK_SET) = 0
read(14, "\331y\266\t\251\17\0\0", 8) = 8
_llseek(14, 509472, [509472], SEEK_SET) = 0
read(14, "\202\211\266\t\273\17\0\0", 8) = 8
_llseek(14, 509480, [509480], SEEK_SET) = 0
read(14, "=\231\266\t\10\20\0\0", 8) = 8
_llseek(14, 509488, [509488], SEEK_SET) = 0
read(14, "E\251\266\t\321\17\0\0", 8) = 8
_llseek(14, 509496, [509496], SEEK_SET) = 0
read(14, "\26\271\266\t\1\17\0\0", 8) = 8
_llseek(14, 509504, [509504], SEEK_SET) = 0
read(14, "\27\310\266\t\304\16\0\0", 8) = 8
_llseek(14, 509512, [509512], SEEK_SET) = 0
read(14, "\333\326\266\t\343\16\0\0", 8) = 8
_llseek(14, 509520, [509520], SEEK_SET) = 0
read(14, "\276\345\266\t\f\17\0\0", 8) = 8
_llseek(14, 509528, [509528], SEEK_SET) = 0
read(14, "\312\364\266\tA\17\0\0", 8) = 8
_llseek(14, 509536, [509536], SEEK_SET) = 0
read(14, "\v\4\267\t\246\17\0\0", 8) = 8
_llseek(14, 509440, [509440], SEEK_SET) = 0
read(14, "\fM\266\th\16\0\0", 8) = 8
_llseek(14, 164448540, [164448540], SEEK_SET) = 0
read(14, "\367\0\n\0015\0T\0\0\0\0\0P\16\0\0\324q;\1\0\0\0\0\200\23\207**+*+"..., 3688) = 3688
_llseek(14, 509448, [509448], SEEK_SET) = 0
read(14, "t[\266\t\376\16\0\0", 8) = 8
_llseek(14, 164452228, [164452228], SEEK_SET) = 0
read(14, "\365\0\10\0017\0W\0\0\0\0\0\346\16\0\0\324q;\1\0\0\0\0\200\25\3*\212+*,"..., 3838) = 3838
_llseek(14, 509456, [509456], SEEK_SET) = 0
read(14, "rj\266\tg\17\0\0", 8) = 8
_llseek(14, 164456066, [164456066], SEEK_SET) = 0
read(14, "\364\0\10\19\0Z\0\0\0\0\0O\17\0\0\324q;\1\0\0\0\0\200\30\201*\5+\200\33"..., 3943) = 3943
_llseek(14, 509464, [509464], SEEK_SET) = 0
read(14, "\331y\266\t\251\17\0\0", 8) = 8
_llseek(14, 164460009, [164460009], SEEK_SET) = 0
read(14, "\356\0\7\1B\0[\0\0\0\0\0\221\17\0\0\324q;\1\0\0\0\0\200\"\201*\200\37\0\200"..., 4009) = 4009
_llseek(14, 509472, [509472], SEEK_SET) = 0
read(14, "\202\211\266\t\273\17\0\0", 8) = 8
_llseek(14, 164464018, [164464018], SEEK_SET) = 0
read(14, "\356\0\n\1C\0\\\0\0\0\0\0\243\17\0\0\324q;\1\0\0\0\0\200#\3)\3+\200\32"..., 4027) = 4027
_llseek(14, 509480, [509480], SEEK_SET) = 0
read(14, "=\231\266\t\10\20\0\0", 8) = 8
_llseek(14, 164468045, [164468045], SEEK_SET) = 0
read(14, "\354\0\t\1C\0Z\0\0\0\0\0\360\17\0\0\324q;\1\0\0\0\0\200$\201*\200\36\0\200"..., 4104) = 4104
_llseek(14, 509488, [509488], SEEK_SET) = 0
read(14, "E\251\266\t\321\17\0\0", 8) = 8
_llseek(14, 164472149, [164472149], SEEK_SET) = 0
read(14, "\356\0\t\1?\0V\0\0\0\0\0\271\17\0\0\324q;\1\0\0\0\0\200\35\212&&H)*"..., 4049) = 4049
_llseek(14, 509496, [509496], SEEK_SET) = 0
read(14, "\26\271\266\t\1\17\0\0", 8) = 8
_llseek(14, 164476198, [164476198], SEEK_SET) = 0
read(14, "\366\0\10\0015\0[\0\0\0\0\0\351\16\0\0\324q;\1\0\0\0\0\200\20\201*\4+\206*"..., 3841) = 3841
_llseek(14, 509504, [509504], SEEK_SET) = 0
read(14, "\27\310\266\t\304\16\0\0", 8) = 8
_llseek(14, 164480039, [164480039], SEEK_SET) = 0
read(14, "\367\0\7\0013\0[\0\0\0\0\0\254\16\0\0\324q;\1\0\0\0\0\200\33\210+,+,,"..., 3780) = 3780
_llseek(14, 509512, [509512], SEEK_SET) = 0
read(14, "\333\326\266\t\343\16\0\0", 8) = 8
_llseek(14, 164483819, [164483819], SEEK_SET) = 0
read(14, "\370\0\7\1A\0_\0\0\0\0\0\313\16\0\0\324q;\1\0\0\0\0\200\34\203,,*\200\""..., 3811) = 3811
_llseek(14, 509520, [509520], SEEK_SET) = 0
read(14, "\276\345\266\t\f\17\0\0", 8) = 8
_llseek(14, 164487630, [164487630], SEEK_SET) = 0
read(14, "\370\0\7\1H\0`\0\0\0\0\0\364\16\0\0\324q;\1\0\0\0\0\200\35\201,\200*\0\200"..., 3852) = 3852
_llseek(14, 509528, [509528], SEEK_SET) = 0
read(14, "\312\364\266\tA\17\0\0", 8) = 8
_llseek(14, 164491482, [164491482], SEEK_SET) = 0
read(14, "\373\0\10\1;\0Z\0\0\0\0\0)\17\0\0\324q;\1\0\0\0\0\200\36\202+,\200\33\0"..., 3905) = 3905
_llseek(14, 509536, [509536], SEEK_SET) = 0
read(14, "\v\4\267\t\246\17\0\0", 8) = 8
_llseek(14, 164495387, [164495387], SEEK_SET) = 0
read(14, "\366\0\n\18\0Y\0\0\0\0\0\216\17\0\0\324q;\1\0\0\0\0\200\32\t+\205*\4,"..., 4006) = 4006
_llseek(34, 623832, [623832], SEEK_SET) = 0
read(34, "4\0@\0@\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
read(34, "\0\0\6\0\4\0\0\0\6\0\4\0\0\0\6\0\4\0\0\0\5\0\4\0\0\0\5\0\4\0\0\0"..., 78) = 78
_llseek(34, 132, [132], SEEK_SET) = 0
read(34, "0|\t\0\230\0\0\0\210\370\0\0", 12) = 12
_llseek(34, 621616, [621616], SEEK_SET) = 0
read(34, "\2\1\0013P\203\t\0\r\0\0\0", 12) = 12
_llseek(34, 621628, [621628], SEEK_SET) = 0
read(34, "\2\1\0023\262\203\t\0\r\0\0\0", 12) = 12
_llseek(34, 621640, [621640], SEEK_SET) = 0
read(34, "\2\1\0033\24\204\t\0\r\0\0\0", 12) = 12
_llseek(34, 621652, [621652], SEEK_SET) = 0
read(34, "\2\1\0043v\204\t\0\r\0\0\0", 12) = 12
_llseek(34, 621664, [621664], SEEK_SET) = 0
read(34, "\2\1\0053\330\204\t\0\r\0\0\0", 12) = 12
_llseek(34, 623832, [623832], SEEK_SET) = 0
read(34, "4\0@\0@\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
```
</details>
Notable is that we still load 5 type 2 records, even though there's just a
single librarian, and 8 compass points. Why 5?

View File

@@ -6,22 +6,18 @@ remake.
## Filesystem layout
* `Anim/`
* `WarHammer.ani` # Doesn't seem to be a RIFF file. 398M so very important.
* There's a pcx image header at `dd ... bs=1 skip=213` but it seems to be a false alert
* Hits for "AmigaOS bitmap font"... probably a false positive
* Lots of 8-byte reads when loading stuff in the mission editor
* Some ~4K reads, havent found one corresponding to a known format yet
* [✓] [`Anim/`](obj.md#WarHammer.ani)
* [`WarHammer.ani`](obj.md#WarHammer.ani)
* [`Assign/`](obj.md#assign)
* `*.asn` # Unknown, seems to be related to .obj files
* `Cursor/`
* `*.ani` # RIFF data
* `*.cur` # Presumably standard windows-format non-animated cursors
* `*.ani` # RIFF data, standard ANI format \o/
* [`Cursors.cur`](obj.md) # `obj` file containing pointers and drag elements
* `Data/`
* `*.dat` # plaintext files defining properties of objects. No single format
* **PARSED**
* `Accounting.dat` # key = value => internal/data/accounting.go
* `AniObjDef.dat` # animated object definitions
* [`AniObjDef.dat`](ani.mnu) # animated object definitions
* `GenericData.dat` # Generic Game Settings
* **TODO**
* `ChaNames.dat` # list of character names
@@ -63,24 +59,24 @@ remake.
* `wh40k.flt` # Audio filter(s?)
* [✓] [`Fonts/`](fonts.md)
* `cboxfont` # ???
* `*.fnt`
* `*.spr`
* `Idx/`
* `WarHammer.idx` # unknown, 1.8M
* [`*.fnt`](fonts.md)
* [`*.spr`](obj.md) # `obj` file
* [ ] [`Idx/`](ani.md)
* [`WarHammer.idx`](ani.md) # unknown, 1.8M
* [`Maps/`](maps.md)
* `*.MAP`
* `*.TXT`
* [`*.MAP`](maps.md)
* [`*.TXT`](maps.md)
* [`Menu/`](mnu.md) - UI element definitions
* `*.mni`
* `*.mnu`
* [`*.mni`](mnu.md) # Menu include file
* [`*.mnu`](mnu.md)
* [`*.obj`](obj.md)
* `Misc/`
* `occlusio.lis` # plain text, presumably occlusion mappings?
* [`MultiMaps/`](maps.md#multimaps)
* `*.MAP`
* `*.TXT`
* [`*.MAP`](maps.md)
* [`*.TXT`](maps.md)
* [✓] [`Obj/`](obj.md)
* `*.obj`
* [`*.obj`](obj.md)
* [✓] `Pic/`
* `*.pcx` # Standard .pcx format
* `RandomMaps/`
@@ -91,12 +87,12 @@ remake.
* `*.txt` # Seems to be a copy of one of Maps/*.txt
* [✓] [`Sets/`](sets.md)
* `Data.chk`
* `*.set`
* [`*.set`](sets.md)
* [✓] `SMK/`
* `*.smk` # Videos: RAD Game Tools Smacker Multimedia version 2
* [ ] `Sounds/`
* [ ] [`Sounds/`](sound.md)
* [`wh40k.ds`](sound.md)
* [ ] `Wav/`
* [ ] [`Wav/`](sound.md)
* [`*.wav`](sound.md)
Phew.

View File

@@ -206,8 +206,8 @@ in the CENTER position. Interesting.
| 0x0004 | x,y size (16 bits each) |
| 0x0008 | ? (blank in all cases so far)
| 0x000c | Size of remaining pixeldata |
| 0x0010 | Padding? |
| 0x0014 | Padding? |
| 0x0010 | Set in `WarHammer.ani` |
| 0x0014 | ? (blank in all cases so far) |
The volume represented by a cell is a little odd. We see three faces of a fake
3D volume of size 64x64x32(ish). This is presented in an isomorphic fashion, so
@@ -512,3 +512,60 @@ break *0x41DD10
This lets me focus very narrowly on what happens when loading sprites, and
might give clues.
## WarHammer.ani
This 400MiB file appears to be a standard object file, it's just very large.
The directory contains 188,286 sprites!
The first 1,064 sprites are all of the same Ultramarine, carryng a bolter. There
are eight "facing" orientations:
* North
* Northeast
* East
* Southeast
* South
* Southwest
* West
* Northwest
For each orientation, an action is pictured in a variable number of frames. The
final frame appears to be "stationary".
* Walk (13 frames)
* Run (9 frames)
* Crouch down (8 frames)
* Stand up (8 frames)
* Take aim (standing) (6 frames)
* Fire (standing) (6 frames)
* Relax aim (standing) (6 frames)
* Throw grenade (standing) (18 frames)
* Take aim (crouched) (5 frames)
* Fire (crouched) (5 frames)
* Relax aim (crouched) (5 frames)
* Throw grenade (crouched) (17 frames)
* Draw melee weapon (standing) (10 frames)
* Strike down with melee weapon (standing) (8 frames)
* Stab with melee weapon (standing) (9 frames)
Added together and multiplied by 87, that's 1064.
The next sprite is a walking-north action for an ultramarine with a flamer. The
total number of frames for this character is 1120 - 56 additional frames, or 7
per orientation. Could be an extra action, or an extra frame per action.
Also notable is that while the bolter showed muzzle flash in the animation, the
flamer only showed a tiny hint of fire. I think the animation for spewing flame
is held elsewhere.
I strongly suspect the actions and the number of frames in each action are
configurable. But where?
The field at 0x10 in the sprite header is set in `WarHammer.ani`, but not in
the other object files encountered so far. However, it seems to be
set statically to the bytes `[212 113 59 1]` for all of them.
Assuming ~1000 sprites per character, `WarHammer.ani` contains 188 characters.
Two other files have been implicated in animation - `Data/AniObDefs.dat` and
`Idx/WarHammer.idx`. More on those in `ani.md`.