Howdy, There's quite a few tools here for modding ToEE that people have used to much success. However, many of them aren't really my style. I don't run Windows, and I'm used to command line tools and standard text editors from programming. So, I've started working on some of my own, and I'm linking them here in case anyone else finds them useful. They're on github at https://github.com/dolio/temple-tools Right now there are two tools. discern-dat - this is a dat extractor. You can either list or actually unpack the stuff in a .dat. I learned what the format was like by looking at the OpenTemple code. Also included is a formal grammar file that documents my understanding of the format (but it's basically a ZIP file with fewer features). transmute-tab-to-text - this is a tool for going back and forth between .tab files and a nicer textual representation. Right now it only supports help tables. The idea is that you can use a text editor to edit the files with, like, actual line breaks, and then use the tool to produce the .tab file that ToEE actually uses. And if you already have a .tab file you work on, you can produce a nicer text file version to start from. I haven't used either for much yet. The first one seems to work for unpacking all the ToEE dats. The second one I've run tests to make sure it understands most of the existing help file, but there might be kinks with actually using it to produce new help that I haven't encountered. Also, I've only used them on my system, so it's possible they behave in ways that won't ultimately work on other platforms. Cheers.
New tool: polymorph-mob This is aiming to be a tool for going back and forth between .mob files (which are binary) and a more tractable format. Right now I'm dumping out JSON. So far it can go from the binary format to the text format, but not backwards, because I'm still figuring out the details from various sources. One question I have is, are there corrupt mob files in the ToEE dats? I'm able to parse the majority of files (or fail in expected ways), but some just seem to not have enough bytes, or they have nonsensical values for embedded sanity checking information within the format. It's possible that either my understanding of the format, or my dat extractor are wrong somehow.
Agetian was able to get ToEE:WB to parse mobs to a better than 99% success rate, and using it I can only remember a handful of mobs over the years that didn't work (generally stuff embedded in sectors), so I don't think there is evidence of significant numbers of corrupt mob files. As you no doubt know, they can vary wildly in size since the GUIDs of associated mobs (like inventory) are written into the mob itself.
There aren't many that have too few bytes. I haven't looked at them all, but one I did look at seemed like it didn't have much fancy going on. I think it was a weapon or armor in Emridy Meadows. The more common problem goes like this... Every time an array field is stored, the mob format contains both the size of the array elements and how many elements there are. In theory, the first is redundant information, because you know what every sort of field is. Like, the abilities array is 32 bit numbers, so that first number should be 4 (4*8=32). Or, there are a lot of fields where that seemingly should be 8, because more complicated data is stored as an array of 64 bit numbers. ToEE:WB has a tendency to just ignore these (and other similar) numbers, because it knows what they should be. Sometimes it does use them when, I guess, it just parses the field as a binary blob. But I've found several cases where it doesn't care what they are. But I've been trying to make sure that they actually match the expected answer. And there are several that don't. Instead of 4 or 8 they are numbers in the thousands (or millions), which is clearly wrong. This can easily be a problem with my code. If I do something earlier incorrectly, I can become misaligned, so I'm reading the wrong stuff for the number. But, it could also be that some of the files have garbage here, and WB is ignoring the numbers because they're sometimes garbage, and the information isn't necessary. At this point I'm still operating under the assumption that I'm getting misaligned or similar. Some stuff I've been tweaking based on WB does seem like there might be an error of that sort in my code.
Okay, I finally figured it out. My problem was with an assumption from the WorldBuilder code. Lots of fields have an extra byte in front of them that WB just skips, because it thinks it's supposed to always be 1. When I started testing this, it turned out that it wasn't always 1, and all the files I was having problems with were the ones where it wasn't 1. ToEE also checks this byte to see if it's 0 or not, and bails out in the first case. But, importantly, that doesn't mean it's a bad file. What it is is a short circuit that encodes default 'null values' in one byte instead of a bunch. So, if you want some object property to just be the empty array, you use 1 byte for a 0 there, instead of like 20 bytes encoding a 0 element array in the normal format. But, if you don't know about this, and just skip past the byte and parse normally, you're misaligned for the rest of the file, hence my problems. With this there are 0 errors with mob files from either vanilla ToEE or the Co8 mod (so, over 16,000 test cases).
I've renamed the DAT tool to "craft-dat" because it now has a mode that can create files from a directory structure. I haven't done extensive testing, because I just got it working. However, the files it produces are at least good enough to repack ToEE1-4.dat and have the game load successfully (which required some unexpected tweaks, so it isn't a completely trivial test). It also seems to produce almost identical files to the datpack exe that Temple+ uses for its releases. I think all the differences are in garbage fields and the GUID (if I don't tell my tool to use the same one).
Okay, so, after a lot of puzzling around with crashes and such, I've finally managed to figure out an experiment people can try if they wish: running Co8 mods with the Temple+ module selector instead of TFE. This is not as automated or easy, so it's kind of pointless unless you just want to experiment. To follow these steps, you'll need to install `craft-dat` from temple-tools above (or perform similar steps with some other dat extractor/creator). You'll also need some other tools that I didn't make for some of the other steps. Get the Co8 module of your choice. I did this with the 'new content' version, but it should work with others. Get the .tfm file from the Co8 mod. This is a zip file containing the main mod data. On linux, I used a program called innoextract to unpack the Co8 installer, but you could also run the installer and the .tfm file will end up in your ToEE directory. Unzip the .tfm file to get the two main directories that make up the mod. One is the data files (which for vanilla are in ToEE1-4.dat) and the other has the module files (which for vanilla are in modules/ToEE.dat). Make dat versions of these directories (some of the commands needed might be a little different for KotB; not sure) For the data files, use `craft-dat fabricate -o ToEE5.dat data` For the module, use `craft-dat fabricate -o ToEE-co8-only.dat modules/ToEE` At least for the ToEE mods (not sure about KotB), just putting the Co8 module in a dat is not sufficient. Information from the original ToEE module needs to be included or the game will crash. So you need to create a combined module dat. Copy the `modules/ToEE.dat` file from your ToEE directory to wherever you built the `ToEE-co8-only.dat` file Run the command `craft-dat unite -o ToEE-co8.dat ToEE.dat ToEE-co8-only.dat` Copy `ToEE-co8.dat` to the `modules/` directory of your ToEE installation. Copy `ToEE5.dat` to the main directory, next to `ToEE1.dat` and so on. In the main directory, create a `TFE-X.ini` file that matches the module. If you're using the "New Content" version, that file has to say "New Content" on one of the lines. For KotB, it needs to say "Keep on the Borderlands". Temple+ uses this to figure out which module is being run (to load its own overrides). "Icewind Dale" and "Paladin" are also recognized. Use the Temple+ config program to select the ToEE-co8 module Copy `temple.dll` and `toee.exe` from the Co8 mod into your main ToEE directory. Note: if you do this step early, the Temple+ configurator will show an error, and not let you select modules, I think. After you have done all this, I think you should be able to launch the game, and you will be playing your chosen Co8 mod pack, but directly from .dat files, instead of using the ToEE `data/` and `modules/ToEE/` directories like are normally used. This is similar to how the original game works. In fact, if you replace the original `modules/ToEE.dat` with `ToEE-co8.dat` (giving the latter the original's name), then I think you will be able to play with `toee.exe` instead of T+. You still need the Co8 versions of the exe and dll to start in the shop map, though. Anyhow, I don't have any particular use for this. It's just a curiosity for those interested. As I mentioned it's not as convenient as TFE. For example, `ToEE5.dat` has some data specific to the mod, and there's no way to swap which dats at that level get loaded based on the T+ module selected (they're just all loaded). So you can't easily swap between multiple modules this way unless they all have the same `data` component. Also, I haven't really played the game much with it set up this way. So it's possible it will fail down the road. I'm not sure why it would exactly, but it's a possibility.
Only today I finally figured out the purpose of the thread's tools for DAT compiling; however, that does not mean that I understand how to follow the above instructions. I have downloaded Main.hs but no idea what to do with it after the first 3 steps (which I know how to do). I don't understand Step 4 nor Step 5.2. What is meant by Step 7 stating the file has to say "" on one of the lines? Thank you for explaining to a noob
