lupine/ordoor
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Rework the .obj format documentation

Browse Source 
* Split container documentation (well-understood) from sprites
* Begin investigation into blank.obj vs. pillar.obj
This commit is contained in:
Nick Thomas
2018-03-20 02:20:02 +00:00
parent f1261d1e2f
commit 9b4e2b43e0
2 changed files with 110 additions and 179 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
doc/formats/img/blank.obj.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

289
doc/formats/obj.md
View File

@@ -16,7 +16,24 @@ Theory: .obj files were originally generated from `.flc` files. This is an
AutoDesk format for visual data, so this suggests the .obj files contain pixels
\o/
jungtil.asn references 18 frames (0-17):
`blank.asn` references 6 frames (0-5):
```
# single pixel tile
# transpix.obj/.asn
#/--> transpix.flc
#
0-5:DEF 1;
0-5:TYPE 13;
END OF FILE
```
`jungtil.asn` references 18 frames (0-17):
```
# jungle floor
@@ -53,197 +70,111 @@ jungtil.asn references 18 frames (0-17):
END OF FILE
```
## Structure
So it seems this visual data can have quite complicated attributes. At a minimum
we see:
Investigated by using strace on WH40K_TD.exe while starting up.
* `TYPE`
* `DEF`
* `DESTROY`
* `Dmg1Lnk`
It opens and reads jungtil.obj in 20 blocks, *after* reading jungtil.asn.
Presumably each is a frame in some format or another, but seemingly *not* FLC.
The `type` field **may** tell us what format each sprite is in.
Here's our list of reads:
## OBJ container structure
| Offset | Size | Gap | Purpose |
|--------|------|-----|---------|
| 0 | 32 | 0 | Main header
| 32 | 144 | 0 | Frame headers
| 176 | 463 | 0 | Frame 1
| 639 | 4466 | 0 | ... |
| 5105 | 4437 | 0 | |
| 9542 | 4468 | 0 | |
| 14010 | 4470 | 0 | |
| 18480 | 4461 | 0 | |
| 22941 | 4472 | 0 | |
| 27413 | 4464 | 0 | |
| 31877 | 4477 | 0 | |
| 36354 | 4473 | 0 | |
| 40827 | 4472 | 0 | |
| 45299 | 4437 | 0 | |
| 49736 | 463 | 0 | |
| 50199 | 4251 | 0 | |
| 54450 | 4350 | 0 | |
| 58800 | 4278 | 0 | |
| 63078 | 4329 | 0 | |
| 67407 | 4329 | 0 | Frame 18 |
`.obj` files represent visual data. They contain a number of sprites, which are
assigned attributes in `.asn` files and referenced from `.map` files.
20 reads in total.
The container format is worked out, but the per-sprite data is still unknown, so
I'm documenting the former here while still investigating the latter.
The following chunks are identical:
The file begins with a header, with all values 32-bit little-endians:
| Offset | Meaning |
| ------ | ------- |
| 0x0000 | Number of sprites |
| 0x0004 | Offset of sprite directory |
| 0x0008 | Size of sprite directory |
| 0x000c | Offset of the sprite data |
| 0x0010 | Size of the sprite data |
The sprite directory contains an 8-byte record per sprite. That record contains
two 32-bit little-endians:
| Offset | Meaning |
| ------ | ------- |
| 0x0000 | Relative offset of sprite in data block |
| 0x0004 | Size of sprite in data block |
For sanity checks, we can ensure that:
* The header, sprite directory and data blocks do not overlap or extend past the
end of the file
* None of the individual sprites overlap each other or extend past the end of
the data block
## Sprite structure
If the `type` field of an `.asn` field *does* tell us how to interpret sprite
data, I'll need to split this up per type. For now, I'm investigating a small
number of files in depth, and comparing across files in a shallow manner, so all
I need to do is manually select sets with the same type in the latter case.
First, the `blank.obj` file. `blank.asn` helpfully tells us that it's a single-
pixel tile and assigns it a type of 13. There are six sprites, all of which are
identical. Data dump:
```
176 and 49736 ( 463-byte frames)
63078 and 67407 (4329-byte frames)
-------------------------------------------------------------------------------
0 1 2 3 4 5 6 7 01234567 0 1 2 3 4 5 6 7
-------------------------------------------------------------------------------
0x0000 10 01 61 01 01 00 01 00 | a | 16 1 97 1 1 0 1 0
0x0008 00 00 00 00 03 00 00 00 | | 0 0 0 0 3 0 0 0
0x0010 00 00 00 00 00 00 00 00 | | 0 0 0 0 0 0 0 0
0x0018 01 fd 00 | | 1 253 0
```
Here's the hex of the first 32 bytes of jungtil.obj:
Total size is 27 bytes, so if there *is* only one pixel, it's tempting to
suggest that the first 24 bytes are header, and the final 3 bytes represent that
one pixel.
I think this is rendered as a 1px dot with the colour `#ff00ff` in WH40K_TD.exe:
![blank.obj as rendered by WH40K_TD.exe](img/blank.obj.png)
In the 64x64 tile, the dot is in the very centre.
The colour itself doesn't show up in the data directly, so it's not a simple RGB
array of pixels. WH40K_TD.exe is a 16-bit-colour application and may use a
palette, but these factoids don't help me make immediate sense of the data. If
there's a palette in the first 24 bytes, it's not obvious.
There are 45 `TYPE 13` .obj files in the game. Comparing the above with the
start of the second sprite in `pillar.obj`:
```
0 1 2 3 4 5 6 7 8 9 a b c d e f
0x0000 12 00 00 00 20 00 00 00 90 00 00 00 B0 00 00 00
0x0010 88 17 01 00 00 00 00 00 00 00 00 00 00 00 00 00
----------------------------------------------
0 1 2 3 4 5 6 7 01234567
----------------------------------------------
0x0000 fa 00 25 01 2e 00 48 00 | H | 250 0 37 1 46 0 72 0
0x0008 00 00 00 00 f4 0c 00 00 | | 0 0 0 0 244 12 0 0
0x0010 00 00 00 00 00 00 00 00 | | 0 0 0 0 0 0 0 0
0x0018 80 0f 90 1a 4b 1a 1b 4b | K K | 128 15 144 26 75 26 27 75
0x0020 1a 4c 1a 4b 19 19 49 19 | L K I | 26 76 26 75 25 25 73 25
0x0028 4a 18 4a 80 0f 00 80 0b | J J | 74 24 74 128 15 0 128 11
0x0030 98 1a 4c 1a 4c 1a 9e 1a | L L | 152 26 76 26 76 26 158 26
0x0038 4c 1b 4b 1a 1a 49 19 48 | L K I H | 76 27 75 26 26 73 25 72
0x0040 18 18 48 19 4b 19 4b 19 | H K K | 24 24 72 25 75 25 75 25
0x0048 9d 80 0b 00 80 08 9e 1a | | 157 128 11 0 128 8 158 26
0x0050 4a 1a 1b 4d 9e 1a 4c 1b | J M L | 74 26 27 77 158 26 76 27
0x0058 1b 4c 1b 4c 1a 1a 49 18 | L L I | 27 76 27 76 26 26 73 24
0x0060 48 17 49 18 19 4a 19 4b | H I J K | 72 23 73 24 25 74 25 75
0x0068 19 9d 4a 1a 49 80 08 00 | J I | 25 157 74 26 73 128 8 0
0x0070 80 07 a0 19 4b 1a 1b 4c | K L | 128 7 160 25 75 26 27 76
0x0078 1b 1b 4d 9e 1b 4c 1b 4d | M L M | 27 27 77 158 27 76 27 77
```
* Offset 0 = 0x12 = 18 = the total number of frames ?
* Offset 4 = 0x20 = 32 = the size of the this block ?
* Offset 8 = 0x90 = 144 = the size of the next block ?
* Offset c = 0xB0 = 176 = the offset of the third read, or the size of the first two blocks combined ?
Need to unpack a few more objs to check what these numbers do. They could all
be 32-bit LE integers too.
0x10-0x13 as a 32-bit LE is 71560.
Total file size (71736) - 32 - 144 = 71560 = size of all frames combined
Obj/j_tree02.obj doesn't seem to adhere to these rules at all...
Then the next 144 bytes:
```
0 1 2 3 4 5 6 7
0x0020 00 00 00 00 CF 01 00 00 = 0, 463
0x0028 CF 01 00 00 72 11 00 00 = 463, 4466
0x0030 41 13 00 00 55 11 00 00 = 4929, 4437
0x0038 96 24 00 00 74 11 00 00 = ...
0x0040 0A 36 00 00 76 11 00 00
0x0048 80 47 00 00 6D 11 00 00
0x0050 ED 58 00 00 78 11 00 00
0x0058 65 6A 00 00 70 11 00 00
0x0060 D5 7B 00 00 7D 11 00 00
0c0068 52 8D 00 00 79 11 00 00
0x0070 CB 9E 00 00 78 11 00 00
0x0078 43 B0 00 00 55 11 00 00
0x0080 98 C1 00 00 CF 01 00 00
0x0088 67 C3 00 00 9B 10 00 00
0x0090 02 D4 00 00 FE 10 00 00
0x0098 00 E5 00 00 B6 10 00 00
0x00A0 B6 F5 00 00 E9 10 00 00
0x00A8 9F 06 01 00 E9 10 00 00
```
8 bytes per frame (8*18=144).
First four bytes in each line are the position relative to the start of the
first frame (i.e., minus the size of the first two blocks).
Last four bytes in each line are the total size of the frame, header + data.
The first 24 bytes look quite different to the remainder of this file, lending
weight to the 24-byte header theory.
Then there are 18 blocks of varying size. These must be the frames. First 32
bytes of all 18 blocks has a lot of similarities:
```
0 1 2 3 4 5 6 7 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f (0c,0d) (18,19) (1a,1b) (1c,1d) (1e,1f)
00176.0463 D1 00 42 01 80 00 3F 00 00 00 00 00 B7 01 00 00 00 00 00 00 00 00 00 00 80 3E 04 1F 80 3E 00 80 439 16000 7940 16000 32768
00639.4466 D1 00 42 01 80 00 3F 00 00 00 00 00 5A 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 4442 16000 28036 28268 32798
05105.4437 D1 00 42 01 80 00 3F 00 00 00 00 00 3D 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 4413 16000 28036 28268 32798
09542.4468 D1 00 42 01 80 00 3F 00 00 00 00 00 5C 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 76 7E 80 4444 16000 28036 30316 32894
14010.4470 D1 00 42 01 80 00 3F 00 00 00 00 00 5E 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 4446 16000 28036 28268 32798
18480.4461 D1 00 42 01 80 00 3F 00 00 00 00 00 55 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 65 6E 1E 7C 80 4437 16000 25988 7790 32892
22941.4472 D1 00 42 01 80 00 3F 00 00 00 00 00 60 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 4448 16000 28036 28268 32798
27413.4464 D1 00 42 01 80 00 3F 00 00 00 00 00 58 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 76 76 1E 80 ... 16000 28036 30326 32798
31877.4477 D1 00 42 01 80 00 3F 00 00 00 00 00 65 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6C 6E 97 1C 80 16000 27780 38766 32796
36354.4473 D1 00 42 01 80 00 3F 00 00 00 00 00 61 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 16000 28036 28268 32798
40827.4472 D1 00 42 01 80 00 3F 00 00 00 00 00 60 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 16000 28036 28268 32798
45299.4437 D1 00 42 01 80 00 3F 00 00 00 00 00 3D 11 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 16000 28036 28268 32798
49736.0463 D1 00 42 01 80 00 3F 00 00 00 00 00 B7 01 00 00 00 00 00 00 00 00 00 00 80 3E 04 1F 80 3E 00 80 16000 7940 16000 32768
50199.4251 D1 00 42 01 80 00 3F 00 00 00 00 00 83 10 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 16000 28036 28268 32798
54450.4350 D1 00 42 01 80 00 3F 00 00 00 00 00 E6 10 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 16000 28036 28268 32798
58800.4278 D1 00 42 01 80 00 3F 00 00 00 00 00 9E 10 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 16000 28036 28268 32798
63078.4329 D1 00 42 01 80 00 3F 00 00 00 00 00 D1 10 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 16000 28036 28268 32798
67407.4329 D1 00 42 01 80 00 3F 00 00 00 00 00 D1 10 00 00 00 00 00 00 00 00 00 00 80 3E 84 6D 6C 6E 1E 80 16000 28036 28268 32798
IDENTICAL IDENTICAL IDENTICAL LE SIZE? IDENTICAL IDENTICAL
V 21102801 4128896 0
vv [209, 322] [128, 63] [0,0]
```
Remainder of each block is highly variable, so probably pixel data, and
(some of?) these 32 bytes are header data
bytes 4-7 could be dimension?
Tiles are 64x64 pixels (I think?), displayed in an isometric fashion, so I am
wrong or it's something else.
bytes 12 & 13 as LE 16-bit (or 12-15 as LE 32-bit) is always 24 bytes under full
block size.
Perhaps we have 24 bytes of header and that's the header-exclusive pixel data size?
Last 8 bits would then actually be the start of the image, but there's a lot
of similarity between them...
It smells a bit like a DIB:
| Offset | Size | Example | Purpose |
| ------ | ---- | ----------- | ------- |
| 0 | 2 | D1 00 (209) | ? |
| 2 | 2 | 41 01 (321) | ?
| 4 | 2 | 80 00 (128) | Width? |
| 6 | 2 | 3F 00 (63) | Height? |
| 8 | 4 | 00 00 00 00 | Could be pitch / stride? |
| 12 | 4 | D1 10 00 00 | Size of pixel data |
| 16 | 4 | 00 00 00 00 | ? |
| 20 | 4 | 00 00 00 00 | ? |
Now, how to convert the pixel data to an image?!?!
Average bits per pixel seems to be less-than-1, e.g. for crates.obj:
```
2018/03/16 21:21:37 CFrame 0: w=76 h=57 sz=3008 w*h=4332 bpp=0.6943674976915974
2018/03/16 21:21:37 CFrame 1: w=76 h=57 sz=2988 w*h=4332 bpp=0.6897506925207756
2018/03/16 21:21:37 CFrame 2: w=108 h=73 sz=5185 w*h=7884 bpp=0.6576610857432775
2018/03/16 21:21:37 CFrame 3: w=107 h=73 sz=5208 w*h=7811 bpp=0.6667520163871463
2018/03/16 21:21:37 CFrame 4: w=109 h=95 sz=7639 w*h=10355 bpp=0.7377112506035731
2018/03/16 21:21:37 CFrame 5: w=109 h=95 sz=7620 w*h=10355 bpp=0.7358763882182521
2018/03/16 21:21:37 CFrame 6: w=76 h=57 sz=2996 w*h=4332 bpp=0.6915974145891043
2018/03/16 21:21:37 CFrame 7: w=76 h=57 sz=2964 w*h=4332 bpp=0.6842105263157895
2018/03/16 21:21:37 CFrame 8: w=108 h=73 sz=5148 w*h=7884 bpp=0.6529680365296804
2018/03/16 21:21:37 CFrame 9: w=107 h=73 sz=5153 w*h=7811 bpp=0.659710664447574
2018/03/16 21:21:37 CFrame 10: w=109 h=95 sz=7423 w*h=10355 bpp=0.7168517624336069
2018/03/16 21:21:37 CFrame 11: w=109 h=95 sz=7487 w*h=10355 bpp=0.7230323515210043
```
So some form of compression? RLE?
Here's the WH40K_TD.EXE rendering:
![First first frames of jungtil.obj as rendered by WH40K_TD.exe](img/jungtil_drawn.png)
When it *does* draw them, they end up being 64x64 squares, so the possible
`width` (above) is definitely not.
A clue lies in the identical frames 0 and 12 in jungtil.obj - these are just 463
bytes, yet represent a 64x64 block (4096 bytes, assuming 1bpp). These are both
*empty frames*. One undamaged, one damaged, but identical pixel data when drawn
by WH40K_TD.exe
So a compression algorithm of some kind *must* have taken that 64x64x? block and
compressed it to just 463 bytes. But which one? `file` doesn't recognise any
magic bytes.
Perhaps there's a built-in assumption that these are all square, and 0x06 stores
the maximum X/Y coordinate (0 - 63 = 64). 0x04 then becomes something else - a
flag, perhaps? We see a stray 0x80 in the frame number sprite number field in
the .obj coordinate records; perhaps it says "this is an X type of sprite"?