[Breakthrough] A 99.9% perfect object codec

So, I'm back in action and getting ready to make another breakthrough in ToEE modding.

Basically, I've already spent some time working on a new, flexible codec that could encode and decode files stored in object file format (in ToEE such files are the MOBs (mobile objects), static objects in sectors, player character files (ToEEPCs), the map cache, and possibly something else.

The new codec is able to read all properties from object files (429 out of 429, compared to only ~100 out of 429 in the ToEEWB codec), as well as write them out to disk.

The codec allows to operate with any number of objects at the same time, so it's possible to load a whole folder of MOB files from the disk into memory and work with them together (as opposed to only one object at a time in the WB codec). Also, it allows to write the objects back to disk in one operation.

The latest test I made right now on a folder with 132 mobile object files (.MOB format) -- from the "Nulb Exterior" -- showed that the codec can load ALL the objects together into an array perfectly in less than 0.8 seconds (34,077 bytes were read out of 34,077 - which means that there was no data loss during the process of reading).

Next tests will include testing the mass writeout of objects and data consistency (preliminary tests showed that the new codec does not corrupt even the most convoluted of objects when both loading and saving them).

Also, the codec can easily be configured (with very little effort) for use with:
- Mobile object files
- Static object data in sectors
- Player character files (ToEEPC)
- Game map cache
- Possibly other fields of use

The codec is also flexible enough so it can be configured to read and write Arcanum object files (if ever needed ), as well as any custom object file format structures you can ever think of (any custom structures based on the object file format).

So far we have the following comparison chart:

ToEEWB (reference codec that is currently in wide use):
- Works with one object at a time
- Can read, parse, and save approximately 100 fields out of 429 possible
- Can work only with mobile object files (.mob) and with sector-embedded data
- Loads and saves one object within approximately one second

New codec alpha 1 (codename Gaia):
- Works with multiple objects at a time, can easily handle thousands of objects together
- Can read, parse, and save all 429 out of 429 fields
- Can work with any imaginable object file format structure - mobile object files, sector-embedded data, game map cache, player character files, and possibly more
- Loads and saves about 150 objects within one second
- Can be easily configured for use with Arcanum objects or even custom objects in third party projects

So far there's only one problem with the codec: it can hardly be integrated with ToEEWB, so most likely once it's finished I will start a new tool that will specifically aim at working with all kinds of ToEE objects (serve as a universal viewer/editor for objects), which will at a later phase be integrated with ToEEWB (as a separate tool). This is not to be expected soon, but this is something I'm working at at the moment.

The codec itself may be made available later as a separate .DLL which can be used in third party .NET projects in order to add support for the object file format without having to worry about the complications of this data format.

I'll inform you of all progress in this area.

- Agetian

 

Great effort Ag! :notworthy

Does any of this shed any light on the problems in the Gnarley Forest map, with the embeded objects not loading properly on return visits? I would hypothesise its because they have to be embedded in a certain order, but thats just a guess.

 

Hey Ted,

Not really, I still don't understand why you're having an issue here (technically the sector file is intact and the objects are allocated correctly). This hardly has anything to do with the codec issues, maybe your hypothesis is correct - we'll have to check that out. Once I get some time to actually work with the game I'll check your issue out. Please PM me some extra information (how often it happens and whether it happens only with a certain category of objects or with all static objects regardless of their character).

A COUPLE GUESSES: It occured to me that in Arcanum the loading order actually mattered for objects integrated in sectors, maybe they should be sorted in the top left->bottom right order or something like that when embedded. Once again, this is just a theory. It looks like the objects embedded in original game sectors are sorted by the Y coodrinates (e.g. objects with smaller Y coord is embedded before an object with the bigger Y coord), and within the same Y coord they are sorted by an X coord. I'm not really sure if it's exactly what matters or not. However, in your sector file this order is reverse, so maybe that's what is troubling you. Try changing the order of the embedded objects (so that the fog comes before the door icon because its Y coord is smaller).

- Agetian

 

Thanks Ag! Glad to know I occasionally have some vague idea of how things work(though I was thinking it was the order in memory, that maybe they expected certain objects to be attached in a certain order - portals, scenery, chests etc). I'll give this a go. The only other time I have had this issue is with internal doors, I gave up trying to put more than one of them in a single sector.

 

Please give it a try. If this turns out to be correct, I'll make a utility that will sort the objects in sector files automatically.

- Agetian

 

OK, here's a screenie from an early test version of an application called ObjView that I write in order to test the object codec. It shows the main screen of a program that loads all MOB files from a folder and then gives a tree-like representation of all objects together with their properties and data. The thing is still not ready for a public demonstration, but it will soon be.

- Agetian

 

<cough> in like FLYNN.

The idea worked for the Gnarley Forest problem Ag, I will test it on some other issues I have in KotB over the next few days.

 

@ Shiningted: Gotcha, I already got the SectorSort utility ready, testing now and uploading within next few hours along with the ObjView thingie. It seems to solve all the issues (disappearing doors as well), so get ready for a couple new tools.

UPDATE: OK, uploaded. Seem to work fine for me, but feel free to provide any feedback, I can fix bugs if there are some.

- Agetian