Clipping

Hello everyone,

i searched through the forums, but found only posts stating, that the clipping file format (*.DAG) is unknown.

Is this information still current? Or has anyone made any progress on the DAG files?

I am trying to recreate a rendering engine that can use ToEE files and thus far i get:

(Click for bigger image)

Basically the tree in the middle should be hidden behind the house.

From what I've gathered, the DAG files are responsible for this in the original client.

If someone has *any* information on those files, feel free to post here. In the meantime I'll try my luck disasembling the temple.dll :/

Cu,
Storm

 

Allright, I figured it out:

The format is actually *very* simple. It contains downsampled geometry used to render the background map. Judging from the CLIP-TEST folder the Troika team planned to optimize this and reuse objects on the map initially. But in the end it seems like they just dumped the entire scene to a DAG file. At last for most maps there is only one and it contains the entire background geometry.

I wrote a little Blender import plugin to actually *view* the geometry and here are my results:

(Click for larger Version) - 5 points for anyone who guesses which map this is for

The Blender import plugin is attached to this post. I didn't test it with many DAGs however, so beware.

To use the plugin, get Blender (duh..) and copy the py file to:
%APPDATA%\Blender Foundation\Blender\.blender\scripts
(If you type this into the Run box or the windows explorer adress bar, it should take you to the correct directory)
Restart blender and ToEE(DAG) should show up in the File->Import list.

Cu,
Storm

 

Here is the 010 Hex Editor File Format Spec for the DAG Files:

Code:

struct DagHeader {
    float boundingCenterXOffset; // in world coordinates, subtract this from x pos
    float boundingCenterYOffset; // in world coordinates, subtract this from y pos
    float boundingCenterZOffset; // in world coordinates, add this to z pos
    float boundingHalfSidelength; // sidelength/2 of the bounding box (screen coordinates)
    int objectCount;
    int objectDataStart <format=hex>;
} header;

FSeek(header.objectDataStart);

struct DagObject {
    int vertexCount;
    int triangleCount;
    int vertexDataStart <format=hex>;
    int triangleDataStart <format=hex>;
    
    local int nextStruct = FTell();
    FSeek(vertexDataStart);
    struct DagVertex {
        float x;
        float y;
        float z;
    } vertices[vertexCount];

    FSeek(triangleDataStart);
    struct DagTriangle {
        short vertex1;
        short vertex2;
        short vertex3;
    } triangles[triangleCount];
} objects[header.objectCount];

As you can see, The header starts with four unknown floats.
I don't know what these are used for yet. If you know, let me know!

Cu,
Storm

 

Okay after some work I think I got usable data out of the cgf and cif files.

Basically the format seems to be:

CGF: Just an index of the DAG files used by the map.
int fileCount;
char filename[260][fileCount];

If i remember correctly, MAX_PATH on windows was defined as 260. Maybe they've used that.

In short: First 4 bytes is the number of DAG files referenced in the CGF file. After that you have just 260 byte long null-terminated strings until the end of the file (or exactly fileCount times, whatever you prefer).

The CIF files contain instantiations of the geometry referenced in the CGF file.

Code:

struct Header {
    int instanceCount;
} header;

struct Vector3 {
    float x;
    float y;
    float z;
};

struct Instance {
    int fileId;
    Vector3 position;
    Vector3 scale;
    float rotation;
} instances[header.instanceCount];

In Short: The file starts with the number of instances in the file. Each instance seems to start with an index into the CGF file which is followed by the position (Premultiplied by the side-length of a tile ~= 28), scale and rotation of the geometry in the world.

*EDIT*: I had to find out the hard way, that the scale applies to the screen coordinate space. So scaling on the X axis is NOT the isometric X axis, but the screen X axis.

Cu,
Storm

 

Okay,

I've implemented the formats above into my little engine and draw the clipping geometry as an overlay for the currently loaded map using the x/y position from the instance file (*.cif) and this is what I get:


(Click for bigger version)

In principle it seems right. The coordinate system I use may still have problems which explain the y-offset. Or maybe I missed an offset in the clipping information stuff.

Interestingly the X/Y vertex positions were inverted for the clipping geometry, but I don't know why yet. Maybe a different coordinate system is used for the clipping geometry (but since the geometry seems to be correct except for the y offset, i would doubt that).

So I am still working on it, but I'm making progress.

Cu,
Storm

*edit*: Progress update


(Click for bigger version)

*edit2*: To clarify the picture above: The yellow overlay is the 3d clipping geometry drawn using a semi-transparent yellow color over the background map. While it's rendered, it's also writing to the z-buffer. After that stage, the rest of the geometry is drawn (trees, etc.) and since the z-buffer contains the depth-information for the background-map now, trees seem to be hidden behind buildings, although the building is just part of the background map.

*edit3*: Another picture:


(Click for bigger version)

 

Hi,

If you have some way of modelling the 3d geometry in blender (or some other modelling tool) that can serve as your clipping geometry, exporting that into the correct format would be trivial. I still have to figure out some minor positioning issues, but that should be manageable. (I guess some of the unknown float's are x/y/z correction values in some way).

My personal goal however is to re-create the ToEE engine, not modify the original game files. But for clipping geometry, it should be possible to create custom clipping files with the information contained in this thread. (And possibly with positioning-help from my little engine project).

To sum it up: If you have a map without clipping, adding clipping could be possible if you have geometry matching the background of your map.

Cu,
Storm

*edit*: corrected parenthesis

 

When most people are new to the forums, and they say something along those lines, I go "When pigs fly". However, in under 10 posts and one thread you have impressed me enough to believe it.

Good luck DarkStorm. You seem to know what your doing and we could definitely use someone of your talents in the community.

 

This is the current state of affairs in my Engine:


At the moment I only place the instanced geometry from the DAG files in my scene at the x,y,z position given in the CIF file... There is an offset I am missing somewhere.

I managed to re-enable the built-in clipping debugging code in ToEE by writing to a certain memory location and managed to get a reference screenshot out of the original game:


As you can see I am off by a few pixels... I guess my next step is figuring out what exactly the unknown fields in the instance file do...

Cu,
Storm

*edit*: I don't know what the white circles are... That was another debugging flag I enabled in addition to the wireframe clipping one.

 

Erm... Well....

I re-interpreted the 3 unknowns as a scaling factor for the x, z and y axis (in that order) and here's what i got:


Well. As usual the ToEE coordinate system is a mess. Seems to look "pixel-perfect" now!

Cu,
Storm

 

WOW! :yikes:

Now we can go "behind" a building on a new map.

I wonder if there is a way to address images being hidden, as beneath the trees on the spider map in KotB.

 

Images being hidden? Sorry... I don't follow ;-)

I attached my current build of the prototype to this post. Please keep in mind:
a) This is my first c# project. I am more of a Java guy. I don't have any idea what the best way to package a C# program is ;-)
b) This is just a prototype to test out how to read and display the ToEE file formats.
c) Not optimized. This program leaks memory like crazy and in general textures once loaded are not unloaded (meaning: if you switch maps too often you'll run out of memory).

It shouldn't kill your computer however, so feel free to drop this into your ToEE directory and play around with it. It also includes an option to automatically take a screenshot and upload it to a free image hoster in case you want to post error screenshots here.

You do need .NET 2.0 (Thats what i selected as "target", I hope it actually makes a difference) and the SlimDX Redistributables (This is a DirectX Wrapper for .NET).

If you have any comments, let me know.

Cu,
Storm

*edit*: To navigate around the map, use the arrow keys on your keyboard.

 

I found out what the last unknown in the CIF is for: Rotation.

Basically it makes sense: The CIF file instantiates the geometry like a static object basically and then gives it a location, rotation and scale (although more fine-grained than the properties for a static object allow).

I updated my post above to reflect this. The only unknowns left are the first four floats in the DAG file. My personal guess still is, that they form some form of bounding box or volume... But I am not sure. Possible in screen coordinates though.

Cu,
Storm