MOB File Format Breakdown Thread

Discussion in 'General Modification' started by Agetian, Mar 20, 2005.

Remove all ads!
Page 4 of 5
  1. dulcaoin

    dulcaoin Established Member Veteran

    Joined:
    Mar 10, 2005
    Messages:
    213
    Likes Received:
    0
    *nod* Same here. :)

    (and that's totally what I was getting at -- I grok'ed that; I was referring to the programmer's tendency to appear all-knowing until s/he hits his/er limit and has to admit the need for help on something)

    from memory...

    1) start the tutorial, walk into the room with the chest (you need to be close enough to pick it up with the command we'll use)

    2) bring up the console

    3) a = game.obj_list_vicinity(game.party[0].location, OLC_ALL)

    a is now a list of all the objects near your location (a door, two rooms, you, the chest). The chest should be a[0], but make sure...

    4) a

    This lists the items in a. You'll see the names. a[0] should be the chest.

    You're done. a[0] is now the handle to the chest.

    a[0].location will give its location
    a[0].rotation is its rotation
    and so on.

    note: you could've used OLC_CONTAINERS (or OLC_CONTAINER -- again this is from memory) to limit the list to just the chest. But I used OLC_ALL when I did it. <shrug>

    -- dulcaoin
     
    dulcaoin, Apr 2, 2005
    #46
  2. Firestrand

    Firestrand Member

    Joined:
    Mar 11, 2004
    Messages:
    79
    Likes Received:
    0
    You know, I was wondering something... Where do the compiled game scripts live? Are they directly in the temple.dll? Or are they somewhere else I haven't found. I ask because it may be possible to extract those compiled python files, and decompile the bytecode back into some type of python.

    Just to be clear, I don't mean the .pyc files created when you have a custom script.

    -Firestrand
     
    Firestrand, Apr 2, 2005
    #47
  3. dulcaoin

    dulcaoin Established Member Veteran

    Joined:
    Mar 10, 2005
    Messages:
    213
    Likes Received:
    0
    BOOM!

    You've just restated my "Python God" hope exactly as I meant/conceived it.

    -- dulcaoin
     
    dulcaoin, Apr 2, 2005
    #48
  4. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    A friend of mine has decompiled temple.dll using IDA, and he found out that the commands that reside in the "game" package (e.g. game.obj_create) reside in temple.dll and are kinda built-in (written in C++ with calls to PyToEE22.dll). He also found that there are two commands - python_script_serialize and python_script_deserialize - that contain calls to cPickle. However, there seems to be no way to call the above things from the game...
    Unfortunately, my friend is unfamiliar with toee that much, and I'm not familiar with disassembly, so we kinda don't understand 50% of what each of us says ;-)
    Anyway, if you have special requests as to what to look for in the disassembly and what it should look like, let me know OK? Any ideas how we can trace the *.mob loading routine? Probably then we'll be able to see what calls are made and understand whether the pickling or some other form of serialization is used!

    P.S. Thanks for the info about the offsets!! I'll look into it some more, probably compare some stuff with protos.tab(?)

    - Agetian
     
    Last edited: Apr 2, 2005
    Agetian, Apr 2, 2005
    #49
  5. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    About pickling...

    It seems to me that the mob files are not entirely pickled or serialized in some other way, since this file format seems to be basically an update of the similar file format from Arcanum which didn't use python (AFAIK). They even have the same header (77 00 00 00 01 ...).

    Anyway, that's not the point.

    @ dulcaoin : how did you guess the names for the object properties? You mentioned that the offsets correspond to the structure of an object in memory. I'm wondering how you found out the names for object properties and how you were able to see how exactly they are aligned in memory... This information might be very helpful for future mob analysis.
    Thanks in advance.
    - Agetian
     
    Agetian, Apr 2, 2005
    #50
  6. dulcaoin

    dulcaoin Established Member Veteran

    Joined:
    Mar 10, 2005
    Messages:
    213
    Likes Received:
    0
    temple.dll has the names of the object fields in "reverse" order. Do a strings dump of that file. Then look for obj_f_begin (for instance), going backwards from there are the fields of an an object; first the general fields that all objects have, then fields for each type of obj (containers, portals, critters, pc's, npc's, etc.)

    When an object didn't use a full set of fields, they used "padding", and numbered them in order. That way you know the order. Also note that along with a "_begin", you'll see a matching "_end", so that way you know what fields a particular object has.

    That list has actually allowed me to come up with a LOT of new column titles for protos.tab columns, ones that are actually "just" guesses, but as it turns out, protos.tab follows that order really well, giving great clues to "missing" field names.

    -- dulcaoin
     
    dulcaoin, Apr 2, 2005
    #51
  7. Firestrand

    Firestrand Member

    Joined:
    Mar 11, 2004
    Messages:
    79
    Likes Received:
    0
    For those interested, here is the complete string dump from temple.dll.
    In order of calling reference, so take a look.

    I suspect that temple.dll calls fread from tio.dll then uses pickle to deserialize it.

    -Firestrand
     

    Attached Files:

    Firestrand, Apr 3, 2005
    #52
  8. dulcaoin

    dulcaoin Established Member Veteran

    Joined:
    Mar 10, 2005
    Messages:
    213
    Likes Received:
    0
    Looking again (computer's back up, dev environment back up, my protos editor back up), I see that 0c and 0d in a .mob file contain the proto id for the item. Looking at G_0247AC<...>.mob, I get 18 04 (418h = 1048d, and 1048d is the Tutorial Chest A)

    Looking at G_3A3B3D<...>.mob, there's a 61 38 in that spot, and 3861h is 14433d, and 14433d is the Giant Rat in protos.tab

    There's a possibility that a full longword was used for these, then 0c through 0f looks possible [frankly, my memory of how words are swapped in a longword with Intel is a little fuzzy].

    Agetian, have you gotten any further?

    Comparing just these two files, I notice the first 8 bytes look like a potential header. Then 08 and 09 seem to always match 18 and 19, which is curious. Somewhere in all that must be a byte count, or negative byte count, indicating the length of the following data. In both these two files, 0a is 52 and 0b is 00; strange that they'd match like that.

    00 4F looks like a popular number up top, too.

    -- dul
     
    dulcaoin, Apr 5, 2005
    #53
  9. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    Hi, dulcaoin!

    Yep, that was already known, but thanks for pointing it out again anyway. I also think that a dword (4 bytes) was used for it, because 0x000e and 0x000f are zeros in all files I've analyzed.

    Yep, noticed that too. It's funny that a couple days ago I tried plain removing a whole lot of crap from a .mob file, including the bytes you mentioned before (I just put 0x00h in all of 'em) and it didn't seem to stump the game in any way. Moreover, just removing MOST bytes after the header seemed purely normal, the object didn't lose its functionality. The "4F 00" thingies were also zeroed out, and it didn't seem to have any effect on the game. I wonder if all that stuff is one big leftover from Arcanum Worlded or something like that...

    I wish I had that file so I could post it here for further modding, I'll probably find it and post it later this day.

    P.S. About the header: right, the "77 00 00 00 01 00" thing-a-ma-bober in the beginning of any mob file is the header. From what I got out of tinkering with IDA (yep - my friend has it, it's awesome! :) it's the "object file format version" (well, at least the "77 00 00 00" part of it). By the way, it's the same for Arcanum mobfiles. Changing 0x77 into anything else causes an internal error in temple.dll with the text "invalid object file version" or somethin' like that.

    P.P.S. The offsets you provided last time as 'offset_x' and 'offset_y' don't seem to do the job, while 0x004B and 0x004F definitely contain X and Y coordinates of an object (tried to display the 'game.party[0].location' and then see what it corresponds to in the file). Btw, the offsets were already discovered before, so I'm just posting them here for reference...
    (curious: I wonder what the byte 0x004A (value=0x01 in our chest mob file) is doing there... changing it makes the game crash, but it's not the part of the coordinates).

    Right now I spend most of the time tinkering with the IDA and temple.dll. The whole thing can be metaphorically compared to banging one's head against a wall. :) I can't trap the mob loading routine into the debugger so far (it's kinda elusive ;-) but I already know a lot more about the temple.dll functioning that I did before (I outlined the gmesh/clipping loading routines etc., even though that's not THAT useful for now).

    Some more revelations about temple.dll: someone asked before where the built-in functions for toee.*, game.*, etc. reside. Well, they reside in temple.dll, they are written in C++ with references to PyToEE22.dll... (did I post that before? well, nevermind if i did...)

    And even some more revelations about MOBs (the chest stuff, y'know): in our tutorial map 1 chest, 0x006F and 0x0078 both store the number of items in a chest. And so does 0x00B4. So, if you are to change the number of item entries in a chest (the stuff that starts with "02 00 00 00 00 00 00 00" and ends with a 16-byte GUID - see 0x0080 for example) you have to change the above-mentioned offsets to the exact number of items in a chest. Maximum number of items in a chest seems to be 31 (read on to find out why).
    So, in our chest:

    0x006F == 0x02 (obvious, why - there's a longsword and a shield)
    0x0078 == 0x02 (for the same reason)
    0x00B4 == 0x03 (you might be asking yerself WTF already, but don't worry, I've got this covered).

    0x00B4 thru 0x00B7 contain the number of items in a chest, but as a dword that contains the value "2^N-1", where N is the number of items in a chest. So, we have two items in a chest, so we put 2*2-1 = 3 in there.
    If we had 8 items, we would do:

    2^8 - 1 = 255, or in the dword form:
    FF 00 00 00

    If we had 29 items, we would do FF FF FF 1F
    If we had 30 items, we would do FF FF FF 3F
    If we had 31 items, we would do FF FF FF FF
    If we had 32 items, we're outta space in our dword, sorry ;-)

    About the GUIDs: in our chest mob file, the item entries start at the offset 0x0080. However, this may be different in different files (so, I wonder if there's some entry in a mob file that defines where the item entries are). They go one after another without any breaks, and they have the following format:
    02 00 00 00 00 00 00 00 <GUID>

    where <GUID> is a 16-byte monstrosity containing the whole "G_xxxxxxxx_xxxx_xxxx_xxxxxxxxxxxx" thing for the items inside. The GUID is a little bit twisted, the first half of it seems reversed while the second part seems going from left to right. For example, for our chest, which is:

    G_0247AC97_992C_4CA6_B5F4_D29EB2CF755E.mob

    The GUID would be:

    97 AC 47 02 2C 99 A6 4C B5 F4 D2 9E B2 CF 75 5E
    (go figure - the GUID of the chest is stored at the 0x001C offset, starting with the regular 02 00 00.... thing). Note that the 0247AC97, 992C and 4CA6 parts of the GUID are written in reversed form, while the rest of it is left-to-right.

    Hope that helps!

    The bytes I'm MOST CURIOUS about right now are:
    0x0073 (and 0x0074) - seems to be a word/dword there, probably word
    0x007C (and 0x007D) - also a word?
    0x0046 (and 0x0047) - either a word, or 2 bytes?

    Those bytes definitely DO something, but any attempt at changing them fails with an error. I'd be glad if you could help me out here, dulcaoin. ;-)

    OK, g2g!
    Talk to ya later, I'll post that mobfile so you can see... If you have any other discoveries, feel free to post 'em - that'd be great!
    Take care!
    - Agetian

    IMPORTANT EDIT: Clarified some offsets with the chest items that were erraneous before!
     
    Last edited: Apr 6, 2005
    Agetian, Apr 6, 2005
    #54
  10. dulcaoin

    dulcaoin Established Member Veteran

    Joined:
    Mar 10, 2005
    Messages:
    213
    Likes Received:
    0
    When I reported offset_x and offset_y, I used the internal names that TOEE uses, I can't explain what they mean (why have a location AND offsets in the first place, after all). I used the console to look up the values, and then matched them to the file.

    I also found a "quicker" way to find them on an object in the console today, rather than use the obj_get_int() function; they're also surfaced as properties (but I don't have the names right in front of me, something like off_x and off_y, I think)...

    -- dulcaoin
     
    dulcaoin, Apr 6, 2005
    #55
  11. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    Hey, dulcaoin!
    Thanks for the tip, I'll check that out.
    BTW, I've clarified some offsets in my post above, so if you've read my post before you might want to re-check the offsets' stuff (oh, I probably just overexerted myself with the stuff so I started talking trash :)))

    - Agetian
     
    Agetian, Apr 6, 2005
    #56
  12. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    Hi again, dulcaoin!

    Could you, please, clarify a bit more what you mean? I just don't get it (the "surfaced as properties" part - does it mean we can somehow list it in the console among other properties?)

    BTW, I post the reconstructed mob file with the removed (zeroed out) bytes and with a legend (map) for it. You might want to check that out. The chest seems functionally identical to the original one, but it has a LOT of stuff set to 0x00 (check it out :)

    OK, l8r, man!
    - Agetian
     

    Attached Files:

    Agetian, Apr 6, 2005
    #57
  13. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    Hey, just wanted to point one more thing out - I've IDA'ed temple.dll some more and found out that the main MOB routine might reside at the following location (note that it's not *for sure*, but at least it seems to operate with mob files and is called once for every mob file when the game is in "Initializing sector data..." mode when it loads):

    0x10071350 (in memory)
    0x00071350 (in the DLL file itself)

    I don't possess enough assembly skills to discern what the routine does exactly, though (it's pretty big, with lot of variables, cross-references, etc.), I'll ask my friend to work on it a little bit, but anyway, if anyone here has some ASM skills he's welcome to try the above-mentioned routine out and see what it does!

    UPDATE: Confirmed: the above-mentioned function works with both the mob files and the weird-named files in the "maps" folder. In fact, it seems to load the stuff from mob files and write it down in the weird-named ones. It also seems to analyze the whole data in the mob files (??). The function is called at 0x10005556, with the parameter being passed to it through the EAX register that stores a pointer to a string containing the current map to analyze.

    Hey, ASM gods? 1337 h4xx0rz? Anyone out there to dig into it further? :)))
    - Agetian
     
    Last edited: Apr 6, 2005
    Agetian, Apr 6, 2005
    #58
  14. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    Hey everyone,
    I'd like to say that I'm going to be pretty busy next week, so it's unlikely that I'll post a whole lot of updates in this section. However, if some of you have time (Morpheus? dulcaoin? Someone else?), please check out some of my previous posts that contain a list of MOB file offsets that need investigation. If you guys could help with these offsets, I'd be very grateful. I would have checked it out myself but I don't really have time right now...
    Sorry for the delay...

    - Agetian
     
    Agetian, Apr 7, 2005
    #59
  15. dulcaoin

    dulcaoin Established Member Veteran

    Joined:
    Mar 10, 2005
    Messages:
    213
    Likes Received:
    0
    I have some corrections to make

    Code:
    offset
004b-0054     location: i64  Value is 1932735283676 (or x = 476, y = 450)
0053-0056     off_x: float  Value is 4.242641
0057-005a     off_y: float  Value is 1.414214
005b-005e     ????: value is -2.5293E-29 as float, 2415935536 as int
005f-0062     rotation: float  Value is 2.356194
0063-0066     radius: float Value is 36.1744
0067-006a     height: float  Value is 29.66554

00bc-00bf     container_inventory_source: int Value is 276
    Before, I was trying to treat these as integers, and getting weird results. Now I used an advanced hex editor, and cast them as float, and they exactly match the named object parameters (off_x, off_y, rotation, radius, height).

    This is really great news, as it means the "weird" float format is actually quite standard. We'll be able to directly read the values using C or Delphi, I bet.

    This does make for a strange potential bug. If I try to retrieve the value for offset_x as an integer, I get the value for off_y (cast as int), and if I try to get the value for offset_y as an integer, I get something very different than off_y. The int retriever might be a tad off in its address calculations, maybe.

    -- d
     
    dulcaoin, Apr 8, 2005
    #60
Page 4 of 5
Our Host!