Temple.DLL internal workings discussion

Sitra Achara has opened this thread (http://www.co8.org/forum/showthread.php?t=11463). I'm opening this one so that we can discuss the things that are going on there.

It was great to see someone digging around in temple.dll. I've done my share too so I'll get straight to the point

To me it looks like the function "PyObj_Get_Internal_Field_32bit" at address 1009E260 is used for retrieving single precision floats and not integers. They are 32 bit values too, but seeing the function's usage I've assumed it's all about fractional values.

I also believe that both that function and any function designated with "Get_Internal_Field" do not accept a Python object as a parameter. It is definitely some 64 bit value, but I am not sure what it is.

I have merged parts of Sitra Archara's work into my own and made it available here: (https://onedrive.live.com/?cid=28BA36BA0AE7093B&id=28BA36BA0AE7093B!109). It has the disadvantage of using a version of IDA which is newer and commercial. This means that my work can't be opened and explored in IDA's free version.

 

Welcome, chaos! The feeling's mutual

Makes sense about the that Get_ function being for floating points, as it's always used for geometric sizes which are indeed floating points. Thanks, I'll update the file and change the descriptions as appropriate.

Edit: actually, I was mistaken, it's NOT identical to the 32bit_Int function - I didn't notice that instead of

Code:

mov     eax, [esp+1Ch+pOut]

it has

Code:

fld     [esp+1Ch+pOut]

D'oh! Again, thanks. That's indeed floating point, and it's not even output to EAX like the other function. I'm not sure if it's single or double precision actually, the opcode is
D9 44 24 18
That modr/m notation has unfortunately managed to baffle me for now.


As for the Python object, I guess it's more precise to say that they accept a 64bit (Long) handle. The _Ext is just the MSB part - I'll update my terminology to reflect that. From that handle you can get the object pointer from PyObj_Get_Memory_Address (you can see this used inside the Get_Internal_Field functions).

 

You're most welcome

As I come to think about it - I am not entirely sure those are single precision. This might be something that needs to be looked at.

If my IDB file is of interest to you (being for IDA 6.6 and all) I can try and keep it up to date. I uploaded my new version just now.

In the meantime I am looking forward to your elaboration on Get_Memory_Address and the related memory structures. It's something I have stayed away from.

 

Unfortunately I don't have IDA 6.6, way to expensive for me. And I'd prefer to refrain from doing you-know-what.

As for the Get_Memory_Address, I'll crack it eventually...

 

Yeah, I know what you mean irate:

While you are at it, I will be digging in the code that handles and uses the .mes files. Having some strings really gives context on what's going on in the code.

 

I've been digging deep into startup / graphics code.

I currently have a version of ToEE that now runs on D3D9EX instead of D3D8. I can go ingame and so far, everything looks fine.

Open TODOs for me there:
- Borderless Window fullscreen (think instant tab-in/tab-out of fullscreen)
- Fixing Movie playback (the constant mode-switching)
- Find any missing direct3d8->direct3d9 translations
- See if it's possible to use a hardware cursor instead of the software one. This should fix hanging cursors when ToEE is lagging

 

We should probably also collect what else we want to change about the technical workings. Things I am thinking about:

ToEE currently uses directinput. I think it doesn't correctly handle mouse acceleration in full-screen. For a game that is controlled like a desktop application, it should simply use the mouse config as the Windows desktop.

 

Perhaps upgrading the Python version? IIRC newer python versions have a 'ctypes' module that allows more flexible script-side manipulation of data (e.g. using pointers).

Other than that I have a list of pet-peeve issues but they may be outside your focus.

 

Nah go ahead.

My idea wasn't even trying to "cheat" by using ctypes in python. I'd rather extend the list of exposed properties and APIs using the standard way. It shouldn't be too hard since the header files for Python are open source.

Usually ToEE gives Python a structure that looks like this:

{ "name_of_prop", function_to_get_value, function_to_set_value },
{ "name_of_prop2", function_to_get_value2, function_to_set_value2 },

etc. etc.

We should be able to just add more entries to those tables with some magic

That way we can directly add new properties to objects that ToEE exposes and write the actual getter-code in C.

 

Btw. the source code for Python 2.2 can be found here:
https://www.python.org/ftp/python/2.2.3/Python-2.2.3.tgz

Most of the interesting stuff is in Python.h. I think we can actually include those .h files in IDA (File -> Load -> Parse C Header) so IDA can understand the Python structs.

 

Hmm I'm getting an error trying to do that... is there a particular order the files should be loaded?

As it is, IDA kind of understands the python commands by itself, it may have something like that built-in.

I agree about extending Python to expose more properties, it shouldn't even be that hard, I'm guessing some copy-paste of existing internal functions + a bigass switch case table.

As for my pet peeves - I've mentioned some already (the wonky event system that causes things like Ready vs. Action stealing the next actor's movement, Spell casting interference, failure to run time events / scripts randomly, etc.), I'd also mention that:
1) The screen should stop scrolling in windowed mode when the mouse exists the screen (maybe the borderless windowed mode solves this?)
2) The infamous particle system bug - I'd like to solve the issue at the root rather than removing particles from MOBs as a workaround
There's probably more that I can't recall right now, but it's not like there's any shortage of work...

 

IDA is pretty picky about the header files and certain structures used within. I'll take a look. We probably should check out that collab plugin soon, hah

What you definetly can see is the use of functions that are exported by the python dll. Thos have names. But in certain cases the data structures passed into those python functions will not be properly named in IDA.

Yes, it should be something along those lines. Or we add our own PySet statements to modify the extend the global objects.

As for my pet peeves - I've mentioned some already (the wonky event system that causes things like Ready vs. Action stealing the next actor's movement, Spell casting interference, failure to run time events / scripts randomly, etc.), I'd also mention that:

Borderless windowed would be fullscreen

Is this only about rendering performance? Part of the beauty of porting ToEE to DX9 is that we get access to some pretty awesome graphics debugging and performance analysis tools.

 

If you recall, there's a bug that some particle systems cause massive slowdowns when they come into view - the infamous node bugs issue - you had mentioned that you ran into the same problem with Evil Temple. It's not so severe in modern systems (I rarely got more than a few seconds of hiccup on this computer), so in a sense I guess boosting performance could solve it, but it seems it's more of a bug-fixing issue.

 

The issue is that how 3D is implemented at a very basic level in ToEE is insanely inefficient on any GPU that came out in the last 10 years

They basically upload everything (except textures) again and again to the GPU on every frame. That means there is basically nil hardware acceleration for anything.

On a different note. I found some very complex functions and was puzzled. Turns out they are MSCVRT functions that IDA missed. Here are the ones I found (they are used a lot):

10EFB5A4 _errno
10255543 ; char *__cdecl strtok(char *strToken, const char *strDelimit)
102564A4 ; intptr_t __cdecl _findfirst(const char *lpFileName, _finddata_t *fileinfo)
(the above also needs the _finddata_t structure, but that is already available as a std struct)
10255D3B ; int __cdecl fprintf(FILE *, int, int)
10258A3D ; int __cdecl NMSG_WRITE(DWORD rtError)
10259AC5 _crt_codepage_related
10259C57 _setmbcp
1025A55C _floor_ceil_internal
1025AA88 _fltin2
1025B706 _sscanf_internal
1025C555 ; int __cdecl write(int fh, const void *buf, unsigned int cnt)
10261A71 __sopen_internal
1025D1B7 _floor_internal
1025F5A0 unknown_libname
1025B808 unknown_libname_0


(less important, but the more funcs we name, the better)
102570D8 ; int __cdecl printf_internal(FILE *, int, int)

Due to some inlining, a simple _isspace(ch) call gets mangled a lot.

If you see any calls to __isctype, that's the reason.
I added an enum with the flags to __isctype:

00000001 ; enum ISCTYPE (bitfield)
00000001 _UPPER = 1
00000002 _LOWER = 2
00000004 _DIGIT = 4
00000008 _SPACE = 8
00000010 _PUNCT = 10h
00000020 _CONTROL = 20h
00000040 _BLANK = 40h
00000080 _HEX = 80h
00008000 _LEADBYTE = 8000h

If it checks a variable cchMultiByte followed by a lookup into an array OR a call to __isctype, that was just one quick _isdigit (or similar) check in Troikas code.