Trim map based on trailer data

doc/formats/maps.md

@@ -392,8 +392,11 @@ well-aligned amount.
 Investigation has so far suggested the following:
 
 * `Cell[0]` seems related to doors and canisters. Observed:
+    * Nothing special: 0x38
+    * ???: 0x39
     * Imperial crate: 0x28
     * Door: 0xB8
+
 * `Cell[1]` seems related to special placeables (but not triggers). Bitfield. Observed:
     *  0x01: Reactor
     *  0x20: Door or door lock?
@@ -408,12 +411,12 @@ Investigation has so far suggested the following:
 * `Cell[7]` Object 2 (Right) Area (Sets/*.set lookup)
 * `Cell[6]` Object 2 (Right) Sprite + active flag
 * `Cell[9]` Object 3 (Center) Area (Sets/*.set lookup)
-* `Cell[10]` Object 3 (Right) Sprite + active flag
+* `Cell[10]` Object 3 (Center) Sprite + active flag
-* `Cell[11]` all 255?
+* `Cell[11]` all 255? Vehicle?
 * `Cell[12]` all 0?
 * `Cell[13]` all 0?
 * `Cell[14]` all 0?
-* `Cell[15]` shows squad positions, MP start positions, etc, as 0x04
+* `Cell[15]` shows squad positions, MP start positions, etc, as 0x04. Bitfield?
 
 Mapping the altar in Chapter01 to the map01 set suggests it's a palette entry
 lookup, 0-indexed. `U` debug in WH40K_TD.exe says the cell's `Object 3-Center`
@@ -515,10 +518,120 @@ Around 001841A0: mission objectives!
 00184240   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ................
 ```
 
-Since all the files are exactly the same length uncompressed, I'm going to
+Relative offsets from the start of the trailer, we have:
-assume these are all a fixed number of fixed-size records when looking into it.
 
+| Offset   | Text |
+| -------- | ---- |
+| `0xEE`   | Mania |
+| `0x78A`  | Dagon |
+| `0xE26`  | Nihasa |
+| `0x14C2` | Samnu |
+| `0x1b5e` | Bael |
+| `0x2896` | Gigamen |
+| `0x2f32` | Valefor |
+| `0x35ce` | Baalberith |
+| `0x3c6a` | Fenriz |
+| `0x4306` | #Character |
+| `0x49a2` | Apollyon |
 
+So there are 1692 bytes between each name (the names probably don't come at the
+start of each block, but it's still a useful stride). Presumably `#Character` is
+a space for one of the player characters, while the others specify an NPC placed
+on the map.
+
+There's 56 of these records between the first and last name we see - `Ahpuch`.
+
+Then there are a number of other strings that seem related to triggers / events,
+including lots that say `NO FILE`. The first two are 96 bytes apart; from then
+on they seem to be placed variably apart from each other; I've seen 96, 256, and
+352 byte offsets.
+
+At 0x20916 the mission objective is readable.
+
+At 0x2092a the mission description is readable.
+
+Generating another map with just 5 characters on it, things look different:
+
+* Trailer size is 13543 bytes
+* There are only 5 names
+* There are none of the trigger/event strings
+* Mission title is found at 0x2b93
+* Mission briefing is found at 0x2c92
+
+Since the trailer is a variable size, there must be a header that tells us how
+many of each type of record to read. Peeking at the differences in `vbindiff`:
+
+```
+Chapter01.MAP.Trailer                                                           
+0000 0000: 38 00 00 68 00 00 00 50  00 00 00 1A 00 00 00 14  8..h...P ........  
+0000 0010: 00 00 00 3A 00 00 00 00  38 25 00 04 00 00 00 00  ...:.... 8%......  
+0000 0020: 00 00 00 1A 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  
+0000 0030: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  
+0000 0040: 00 00 00 00 00 00 00 00  00 00 00 02 00 00 00 00  ........ ........  
+0000 0050: 00 00 00 32 00 00 00 00  00 00 00 00 00 00 00 00  ...2.... ........  
+
+TINYSQUAD.MAP.Trailer                                                           
+0000 0000: 38 00 00 4B 00 00 00 3C  00 00 00 37 00 00 00 28  8..K...< ...7...(  
+0000 0010: 00 00 00 05 00 00 00 00  2B 3A 00 04 00 00 00 05  ........ +:......  
+0000 0020: 00 00 00 01 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  
+0000 0030: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  
+0000 0040: 00 00 00 00 00 00 00 00  00 00 00 02 00 00 00 00  ........ ........  
+0000 0050: 00 00 00 32 00 00 00 00  00 00 00 00 00 00 00 00  ...2.... ........  
+```
+
+The size of the trailer for Chapter01 is 139,483 bytes, assuming it starts at
+`0x163890`. However, things may be a lot more sensible if we drop 3 bytes off
+the start of that to get the fields into little-endian alignment. Have I made a
+maths error above somewhere? Is it some sort of alignment thing? Do those 3
+bytes actually have meaning?
+
+Ignoring them for now, here's a first guess at a header:
+
+| Offset | Size | Meaning |
+| ------ | ---- | ------- |
+| 0      | 4    | Map maximum X + 1 |
+| 4      | 4    | Map maximum Y + 1 |
+| 8      | 4    | Map minimum X |
+| 12     | 4    | Map minimum Y |
+| 16     | 4    | Number of character records |
+| 20     | 4    | Padding? - invariant `00 00 00 00` |
+| 24     | 2    | ??? - varies. Seems related to character/squad position? |
+| 26     | 2    | ??? - invariant `00 04` |
+| 28     | 4    | ??? - varies (0 vs 5) |
+| 32     | 4    | Number of thingies (26 vs 1) |
+| 36     | 20   | Padding? |
+
+56 bytes of data is interesting because the value of that first, ignored byte is
+0x38 - perhaps it's a skip value + 2 bytes of padding? It's just weird. Keep
+ignoring it for now.
+
+0x4b contains the next non-null byte; is the gap between the the number of
+thingies, and it, padding? Minus a bit? 0x50 is another non-null byte. Then
+it's all zeroes until one byte before the first name at 0xee.
+
+It's hard to say where the alignment should be at this point. We need to compare
+more characters with each other. Other notes...
+
+Characters are organised into Squads somehow.
+
+Individual cells seem to have a flag to say "We have a character in us", but not
+the number for the character themselves, so the coordinates must be in the
+per-character records also. There are several candidates for this.
+
+Placing a single character at (64,49) causes those bytes to show up at four
+offsets - 0x18 (!), 0x1F4, 0x1F8, and 0x6C8.
+
+Generating a map with no characters at all, the trailer is 2,447 bytes, and the
+mission title starts at 0x3B (59). So we can say we have 20 bytes of padding as
+a first approximation?
+
+The "trailer trailer", for want of a better term, seems to be organised as:
+
+| Offset | Size | Meaning |
+| -----  | ---- | ------- |
+| 0      | 255  | Title   |
+| 255    | 2048 | Briefing |
+| 2304   | 85   | ??? - each byte is 1 or 0. Spaced so it may be partly uint32 |
 
 
 ## Soldiers At War
@@ -600,3 +713,15 @@ xxd -s 0xc0 -c 13 -l 260 -g 13 BIGGESTMAP.MAP
 000000e7: 00 00 85 01 00 00 00 00 00 ff 00 00 1f  .............
 # ...
 ```
+
+This can be interpreted more or less the same way as the Chaos Gate maps now,
+and the `soldiers-at-war` branch contains a hacked-up implementation that kind
+of works \o/.
+
+Does the same trailer apply? Seemingly not. Looking at `PARIS.MAP`, there's no
+similarity at first glance.
+
+However, I did manage to track down 4 32-bit ints inside the trailer, starting
+at `0x121ad1`, which specify dimensions of the map, at least. Perhaps the
+position has moved, but some of the data is the same? It's 3320 bytes into the
+trailer.

internal/maps/maps.go

@@ -15,8 +15,8 @@ import (
 )
 
 var (
-	expectedMagic         = []byte("\x08\x00WHMAP\x00")
+	expectedMagic         = []byte("\x15\x00AMB_MAP\x00")
-	expectedSetNameOffset = uint32(0x34)
+	expectedSetNameOffset = uint32(0x10)
 	notImplemented        = fmt.Errorf("Not implemented")
 )
 
@@ -25,41 +25,53 @@ const (
 	MaxLength = 100 // Y coordinate
 	MaxWidth  = 130 // X coordinate
 
-	CellSize = 16 // seems to be
+	CellSize = 13 // seems to be
 
-	cellDataOffset = 0x110 // definitely
+	cellDataOffset = 0xc0
 	cellCount      = MaxHeight * MaxLength * MaxWidth
 )
 
 type Header struct {
-	IsCampaignMap uint32 // Tentatively: 0 = no, 1 = yes
+	Magic   [10]byte // "\x15\x00AMB_MAP\x00"
-	MinWidth      uint32
+	SetName [8]byte  // Links to a filename in `/Sets/*.set`
-	MinLength     uint32
-	MaxWidth      uint32
-	MaxLength     uint32
-	Unknown1      uint32
-	Unknown2      uint32
-	Unknown3      uint32
-	Unknown4      uint32
-	Magic         [8]byte // "\x08\x00WHMAP\x00"
-	Unknown5      uint32
-	Unknown6      uint32
-	SetName       [8]byte // Links to a filename in `/Sets/*.set`
 	// Need to investigate the rest of the header too
+	IsCampaignMap byte
 }
 
-func (h Header) Width() int {
+type TrailerHeader struct {
-	return int(h.MaxWidth - h.MinWidth)
+	Discard1 [3]byte // No idea what this lot is
+	MaxWidth  uint32
+	MaxLength uint32
+	MinWidth  uint32
+	MinLength uint32
+
+	NumCharacters uint32
+
+	Unknown1 uint32
+	Unknown2 uint16
+	Unknown3 uint16
+	Unknown4 uint32
+	NumThingies uint32
+	Padding1 [20]byte
 }
 
-func (h Header) Length() int {
+type TrailerTrailer struct {
-	return int(h.MaxLength - h.MinLength)
+	Title    [255]byte
+	Briefing [2048]byte
+	Unknown1 [85]uint8 // Maybe? each contains either 0 or 1? Hard to say
 }
 
-func (h Header) Height() int {
+type Character struct {
-	return MaxHeight
+	Unknown1 uint32
 }
 
+type Characters []Character
+
+// TODO. These are triggers/reactors/etc.
+type Thingy struct {}
+
+type Thingies []Thingy
+
 func (h Header) MapSetName() string {
 	idx := bytes.IndexByte(h.SetName[:], 0)
 	if idx < 0 {
@@ -80,7 +92,7 @@ type ObjRef struct {
 
 // The index into a set palette to retrieve the object
 func (o ObjRef) Index() int {
-	return int(o.AreaByte)
+	return int(o.AreaByte & 0x7f)
 }
 
 func (o ObjRef) Sprite() int {
@@ -91,12 +103,13 @@ func (o ObjRef) Sprite() int {
 // The top bit seems to say whether we should draw or not.
 func (o ObjRef) IsActive() bool {
 	return (o.SpriteAndFlagByte & 0x80) == 0x80
-}
+} // PARIS is 78 x 60 x 7
-
+//          4E   3C   7
+/*
 type Cell struct {
 	DoorAndCanisterRelated    byte
-	DoorLockAndReactorRelated byte
+//	DoorLockAndReactorRelated byte
-	Unknown2                  byte
+//	Unknown2                  byte
 	Surface                   ObjRef
 	Left                      ObjRef
 	Right                     ObjRef
@@ -105,43 +118,60 @@ type Cell struct {
 	Unknown12                 byte
 	Unknown13                 byte
 	Unknown14                 byte
-	SquadRelated              byte
+//	SquadRelated              byte
+}*/
+
+type Cell struct {
+	Unknown1 byte
+	Surface  ObjRef
+	Left     ObjRef
+	Right    ObjRef
+	Center   ObjRef
+	Unknown2 [4]byte
+
+	/*
+	   	DoorAndCanisterRelated    byte
+	   //	DoorLockAndReactorRelated byte
+	   //	Unknown2                  byte
+	   	Surface                   ObjRef
+	   	Left                      ObjRef
+	   	Right                     ObjRef
+	   	Center                    ObjRef
+	   	Unknown11                 byte
+	   	Unknown12                 byte
+	   	Unknown13                 byte
+	   	Unknown14                 byte
+	   	SquadRelated              byte*/
 }
 
 func (c *Cell) At(n int) byte {
 	switch n {
 	case 0:
-		return c.DoorAndCanisterRelated
+		return c.Unknown1
 	case 1:
-		return c.DoorLockAndReactorRelated
-	case 2:
-		return c.Unknown2
-	case 3:
 		return c.Surface.AreaByte
-	case 4:
+	case 2:
 		return c.Surface.SpriteAndFlagByte
-	case 5:
+	case 3:
 		return c.Left.AreaByte
-	case 6:
+	case 4:
 		return c.Left.SpriteAndFlagByte
-	case 7:
+	case 5:
 		return c.Right.AreaByte
-	case 8:
+	case 6:
 		return c.Right.SpriteAndFlagByte
-	case 9:
+	case 7:
 		return c.Center.AreaByte
-	case 10:
+	case 8:
 		return c.Center.SpriteAndFlagByte
+	case 9:
+		return c.Unknown2[0]
+	case 10:
+		return c.Unknown2[1]
 	case 11:
-		return c.Unknown11
+		return c.Unknown2[2]
 	case 12:
-		return c.Unknown12
+		return c.Unknown2[3]
-	case 13:
-		return c.Unknown13
-	case 14:
-		return c.Unknown14
-	case 15:
-		return c.SquadRelated
 	}
 
 	return 0
@@ -150,15 +180,23 @@ func (c *Cell) At(n int) byte {
 // Cells is always a fixed size; use At to get a cell according to x,y,z
 type Cells []Cell
 
+// 6 Possibilities for being laid out in memory. Most likely:
+// XXYYZZ
+// OR
+// XYZXYZ
+
 func (c Cells) At(x, y, z int) Cell {
-	return c[(z*MaxLength*MaxWidth)+(y*MaxWidth)+x]
+	//	log.Printf("At (%v,%v,%v)=%v", x, y, z, x*y*z)
+	return c[(z*MaxLength*MaxWidth)+
+		(y*MaxWidth)+
+		x]
 }
 
 func (h Header) Check() []error {
 	var out []error
-	if h.IsCampaignMap > 1 {
+	//	if h.IsCampaignMap > 1 {
-		out = append(out, fmt.Errorf("Expected 0 or 1 for IsCampaignMap, got %v", h.IsCampaignMap))
+	//		out = append(out, fmt.Errorf("Expected 0 or 1 for IsCampaignMap, got %v", h.IsCampaignMap))
-	}
+	//	}
 
 	if bytes.Compare(expectedMagic, h.Magic[:]) != 0 {
 		out = append(out, fmt.Errorf("Unexpected magic value: %v", h.Magic))
@@ -170,16 +208,23 @@ func (h Header) Check() []error {
 type GameMap struct {
 	Header
 	Cells
-	// TODO: parse this into sections
+	TrailerHeader
+	Characters
+	Thingies
+	TrailerTrailer
+
+	// TODO: parse this into sections. This is the text that comes from the
+	// .TXT file. Maybe we don't need it at all since it should be duplicated in
+	// the trailer.
 	Text string
 }
 
 func (m *GameMap) Rect() image.Rectangle {
 	return image.Rect(
-		int(m.Header.MinWidth),
+		int(m.TrailerHeader.MinWidth),
-		int(m.Header.MinLength),
+		int(m.TrailerHeader.MinLength),
-		int(m.Header.MaxWidth),
+		int(m.TrailerHeader.MaxWidth-1),
-		int(m.Header.MaxLength),
+		int(m.TrailerHeader.MaxLength-1),
 	)
 }
 
@@ -286,5 +331,42 @@ func loadMapFile(filename string) (*GameMap, error) {
 		return nil, fmt.Errorf("Error parsing cells for %s: %v", filename, err)
 	}
 
+	// no gzip.SeekReader, so discard unread trailer bytes for now
+	if _, err := io.CopyN(ioutil.Discard, zr, int64(3320-2)); err != nil { // observed
+		return nil, err
+	}
+
+	if err := binary.Read(zr, binary.LittleEndian, &out.TrailerHeader); err != nil {
+		return nil, fmt.Errorf("Error parsing trailer header for %s: %v", filename, err)
+	}
+
+	log.Printf("Trailer Header: %#+v", out.TrailerHeader)
+	/*
+	// TODO: until we know how large each character record should be, we can't read this lot
+
+	out.Characters = make(Characters, int(out.TrailerHeader.NumCharacters))
+	if err := binary.Read(zr, binary.LittleEndian, &out.Characters); err != nil {
+		return nil, fmt.Errorf("Error parsing characters for %s: %v", filename, err)
+	}
+
+	out.Thingies = make(Thingies, int(out.TrailerHeader.NumThingies))
+	if err := binary.Read(zr, binary.LittleEndian, &out.Thingies); err != nil {
+		return nil, fmt.Errorf("Error parsing thingies for %s: %v", filename, err)
+	}
+
+	if err := binary.Read(zr, binary.LittleEndian, &out.TrailerTrailer); err != nil {
+		return nil, fmt.Errorf("Error parsing trailer trailer for %s: %v", filename, err)
+	}
+	log.Printf("Trailer Trailer: %s", out.TrailerTrailer.String())
+*/
 	return &out, nil
 }
+
+func (t *TrailerTrailer) String() string {
+	return fmt.Sprintf(
+		"title=%q briefing=%q rest=%#+v",
+		strings.TrimRight(string(t.Title[:]), "\x00"),
+		strings.TrimRight(string(t.Briefing[:]), "\x00"),
+		t.Unknown1,
+	)
+}