Clipping tool

Is there a clipping tool available? I've searched the threads and it seems like clipping was only figured out relatively recently.

In a post from 2010, Gaer said:

"Yes, making tiles 'vison extender' greys out characters, but clipping does play a role. Without clipping (that's one thing we can't do ourselves) characters are either all grey, or not. (e.g. if you're standing behind a wall at all, you'll be grey, even if part of you is not behind the wall.) With clipping in use, the parts of characters behind walls get greyed out, and the parts that aren't behind walls don't. It's much better with clipping, more precise and convincing.​

Then there were some postings by Darkstorm, who seemed to crack the code, but I didn't see any follow up or a tool.

It seems like it has been figured out though, since the clipping as you walk behind buildings in Verbobonc is really perfect.

Any help would be appreciated as usual.


(Also, I am currently using the Extend flag on the sector tiles, and it's working very roughly, but only seems to make the object walking behind it grey out a little bit, where I usually see them become abut 75% transparent in other maps.)

 

Could anyone elaborate how this was done, for example with the Verbobonc maps?

I can see that the clipping mechanism is held in the clipping.cgf and clipping.cif files, which subsequently reference the DAG files in the art\clip folders. I can see by examining the clipping.cgf file that it lists the DAG files involved, and the clipping.cgf file (which is very small) somehow ties the whole thing together.

I admit I am getting a little out of my league with this particular part of the map making process, but excitingly it's the final thing I need to know to make all the maps I've created viable for an adventure I am modding up. The incredible World Builder, and all the other tools and tutorials, have done 90% of the work for me, and put the capability of doing this into the realm of possibility for a moderately skilled computer programmer like myself.

 

Ahh of course, I should have been able to figure this out earlier by seeing that all the clipping.cgf files in the Co8 locations have names of existing DAG files in them. And also that there are no newly created DAG files in the art\clip directory. That seems convenient, since with so many of them already created you can just build your custom art around the framework of a structure that already has the DAG file.

So basically, now I just need to figure out how that mysterious little clipping.cif file ties it all together. :yikes: Is it something that can be done without the app you mentioned, by hand?, even though it might take longer.

 

I can now use existing clipping data for my new maps. I've deciphered at least enough of the contents of the clipping files to overlay any clip from an existing ToEE map onto a newly created map. For example, if I want to grab The Waterside Hostel and put it onto a new map of a town I've made, I can use the clipping data from it's DAG file to make perfect clipping around it. I haven't seen this documented anywhere, so I'll share what I have explored and learned so far.

Please excuse/accept my somewhat verbose non-technical explanation of this process. It's been a while since I've programmed, and my knowledge of modern programming techniques and terminology is pretty much nonexistent.


There are only two files that need to be edited. Each map area has a unique copy of these two files in their respective directory along with their sector data and mobs:

1) clipping.cgf is pretty straightforward. This is an index of possible DAG files that can be referenced by the second file, clipping.cif, described below. The DAG files, located elsewhere, describe the mysterious clipping process for a particular image in the 2D art. I have not ventured into the DAG files themselves, nor do I intend to. There are enough of them to choose from and use for a new location.

clipping.cgf starts with a 4 byte header, in hex, designating the number of DAG files which will be listed in the file. Next, the path for each DAG file is then listed in 260 byte strings, which all look something like "art\clip\map-9-nulb-exterior\bsmith_logs.dag". It's convenient that the files are named explicitly in most cases so you can usually see what the DAG file is clipping. In this case, it's the pile of firewood outside of Otis's blacksmith shop in Nulb. It would suck if they were named Xc6ddga5x.dag.


2) clipping.cif is a bit more cryptic, and I still don't have a full understanding of everything in the file, but enough to use the clipping I need for a new map. Here's an example of a clipping.cif file. It's in hex, of course.

03000000
00000000 06A55F46 00000000 18F94846 0000803F 0000803F 0000803F 00000000
01000000 AA6F5146 00000000 BA8A4A46 0000803F 0000803F 0000803F 00000000
02000000 DF764C46 00000000 8ABA5446 0000803F 9D47813F 55B87E3F 00000000​

The first 4 bytes is a header, designating the number of clips which will be listed in the file. (Note that the number in the header must match the number of definitions listed after it, or a CTD will usually occur.)

The header is followed by a 32 byte definition for each 'clip' that will be put onto the map.

So what exactly is in those 32 bytes? I'm not sure of some of it, but I discovered by process of elimination how to alter it to get the clipping data onto my map. There is an old thread somewhere which took a guess at the structure of the files, and this helped me very much in my process, but it never explained what the numbers referred to exactly in terms of the map. Here's what I came up with:

Bytes 1,2,3,4: This is an index to the DAG file listed in clipping.cgf. So, the magical clipping information for this 'clip' will come from the DAG file indexed by this number in clipping.cgf. In the example above, the clip which starts with 00000000 will use the first DAG file listed in clipping.cgf, the line starting with 01000000 will us the second file listed, and the line starting with 02000000 will use the third file listed. (Note that you can use the same index multiple times, so if you wanted three piles of firewood on your map, and it was the third file in clipping.cgf, you would make three lines all starting with 0200000 in clipping.cif.)

Bytes 5,6,7,8: The X-coordinate on the map where the clipping data will be overlaid.

Bytes 9,10,11,12: The Z-coordinate on the map where the clipping data will be overlaid. This always seems to be 00000000.

Bytes 13,14,15,16: The Y-coordinate on the map where the clipping data will be overlaid.

Bytes 17 - 32: I never touch these when I copy a clip into my new clipping.cig file, and everything works fine. I think they deal with scaling and rotation. Quite often the are all zeros, or repeat the same numbers across many different clips.​

So for my purposes, to get the clipping aligned to the artwork, I only need to alter bytes 5-8 (X-coord), and 13-16 (Y-coord). The following is my interpretation as to how bytes 5-8 locate the X coordinate on the area map. The analysis is identical for bytes 13-16 which describe the Y-coordinate. The four bytes seem to construct a very large number, with byte 8 having the highest order of magnitude, and bytes 7, 6 and 5 getting progressively smaller.

Byte 5: Insignificant.

Byte 6: This is the one I tweak to get the clip exactly over the art. Changing it by 10 will only move the clip by a few pixels.

Byte 7: This is the one I tweak to get the clip close to the art. Changing this by a factor of 1 will shift it's location by about 2 normal game coordinates.

Byte 8: Almost always 46. If the clip moves to a remote part of a very large map, like the Church in Hommlet, it will change to 45.
​

Unfortunately the X/Y coordinates in the clipping file (bytes 5-8, and 13-16) are not the same scale as the X/Y coordinates that are used in most of the game for placing objects into sectors and such. So I tried to find a relation between the two sets of coordinates. The ratio of changing the value of byte 7 (for X) or byte 15 (for Y) seems to result in a change of about 2.29 X/Y sector coordinates on the map. There is a formula in step 4 below to convert them.


Here's how I overlay existing clipping data from a DAG file onto a new map:

1) Create the map: I create my new 2D art and make a map directory with sectors and lighting and mobs, as per usual. As a simple example, let's say I carefully photoshopped out the Waterside Hostel from Nulb and placed it on a clean map with a grassy area. You have to cut the edges carefully around the image that will have clipping data around it, or the clipping data won't line up right. Usually a pixel here and there won't matter, but it needs to be close.


2) Make clipping.cgf: In a new clipping.cgf file, add the path to the DAG file of the Waterside Hostel.

The first 4 bytes are the header, declaring that there is one DAG file, so 01000000 will be entered as HEX.

The next 160 bytes is the name to the file DAG file, which must be entered as TEXT (in Hex Workshop, enter it into the area on the right side). The program only uses the path and file name from this entry, so everything after '.dag' is ignored (I think), but it still must be padded with characters to make it 160 bytes.

art\clip\map-9-nulb-exterior\hostile.dag..........extend to 160 total characters.........​

3) Make clipping.cif: From the clipping.cif file in Map-9-Nulb-Exterior, I copy the relevant line into the new clipping.cif file for my area. It's line 19000000 (hex) in the Nulb file, since the DAG file for the hostile is indexed at 25 (19 hex) in Nulbs clipping.cgf file. It looks like this:

19000000 F7CB7746 00000000 C9684A46 0000803F 0000803F 0000803F 00000000​

As I described above, I'll want to alter bytes 1-4 (the index), 5-8 (the X-coords), and 13-16 (the Y-coords). The index is easy, since there's only one DAG file in my clipping.cgf right now, so I change it to 00000000. So bytes 1-4 are changed to 00000000, resulting in:

00000000 F7CB7746 00000000 C9684A46 0000803F 0000803F 0000803F 00000000​

4) Get the clip into the right sector: The X/Y coordinates for the clip (bytes 5-8, and 13-16) needs to be aligned precisely so that it lays over the picture of the Waterside Hostel on my new map. Well, as I was describing above, the normal game coordinates are a different scale than the coordinates used for the clipping data. After some mathematical exploration, this formula can be used for finding the values to enter into byte 7 and 14 based on the X/Y coordinates on your map. This will not put the clipping data precisely onto the Hostel, but... it will get it close, and from there I can move on to steps 5 and 6 which will get the clip precisely aligned.

Byte 7 = (X - 267) / 2.29 {then convert to hex}

Byte 13 = (Y - 280) / 2.29 {then convert to hex}​

Lets say I want to put the clip on my map somewhere around 480,480. This computes to byte 7 = 5D, and byte 13 = 57. So, to get the clip into the sector, which will get it close to the art of the Hostel, I change:

00000000 F7CB7746 00000000 C9684A46 0000803F 0000803F 0000803F 00000000​

to:

00000000 F7CB5D46 00000000 C9685746 0000803F 0000803F 0000803F 00000000​

5) Get the clip close: OK, now to align the clip very close to the Hostel. I fire up the game and enter the map, and I walk around my sector and can see myself randomly disappearing behind the clipping data for the Hostel. I can see it's close, but it's not in the right place. Well, I need to find a recognizable part of the clipping data, like a corner, by walking around and seeing myself disappear behind the clipping. After a while I can see the a corner of the clipping data is too far south and west. Since I need to adjust it north and east, the X and Y both need to be reduced, so I go back to the entry with the coordinates and and reduce the value of X (byte 7) and Y (byte 15) by a few. So I try changing it from:

00000000 F7CB5D46 00000000 C9685746 0000803F 0000803F 0000803F 00000000​

to:

00000000 F7CB5B46 00000000 C9685546 0000803F 0000803F 0000803F 00000000​

I then save clipping.cif, reload my map, and walk around again. After repeating this process a few times (it usally takes me about 5 adjustments), I have the clipping data within about 50 pixels of the picture.


6) Get the clip perfect: Finally, it's time to to align the clipping precisely onto the image. This is done by tweaking the more sensitive bytes (6 and 14). These are very precise. It takes changing the number by about 3 just to get a 1 pixel change.

00000000 F7CB5B46 00000000 C9685546 0000803F 0000803F 0000803F 00000000​

But similar to step 5 above, after about 5 adjustments and reloads, the Hostel (or whatever image is clipped) can now be walked around with pinpoint accuracy.

Note that creatures will disappear completely behind the clipping until the sector visability flag is set to "Extend" in the World Builder for each tile in that sector that is behind the Hostel. Once you do that, they will show up as vague grey shadows behind the clipping. But the beauty of this is how they appear when they are half way behind something, and especially things like statues and railings. It really gives a 3D realism to the map.


Lastly, if only one image needs clipping for a sector, it's actually easier just to copy the entire art and map data for that sector to the new map directory and build your new map from that image, and everything will line up correctly. But if you want to make a unique area with multiple clipped images on the same map, like an old haunted house from of the Hostel, and some creepy pillars in the yard taken from the Temple, and Burne's tower off in the woods, it has to be done by hand. Unless someone knows differently, and can speed up the process.


Well that's it. Like I said, it's a bit tedious, but after I did a few times it only takes about 15 minutes to do a new one. I can definitely see a tool that would update the X/Y in clipping.cif as you march around, and then reloading the map it works. Maybe something similar to the commands in ed.py which lay out sector data.

 

Maybe I have not read into this clipping as far as I need to understand
custom cliping plane models go,
But I think Darkstorm or someone out there had defined the
Clipping DATA DAG files data sizes header and such.


MY DAG SOURCE

IE CODE

Code:

//read header
typedef struct DagHeader {
    float CXOffset; // in world coordinates, subtract this from x pos
    float CYOffset; // in world coordinates, subtract this from y pos
    float CZOffset; // in world coordinates, add this to z pos
    float Halflength; // sidelength/2 of the bounding box (screen coordinates)
    int objectCount;
    int DStartOffset;
    
} DagHeader;

typedef struct DagObjectHead {
    int vCount;
    int tCount;
    int vDataStart ;//<format=hex>;
    int tDataStart ;//<format=hex>;
}DagObjectHead;

Now givin the actual abilty to code and extract this data from just knowing the data
in the file, I had successfully extracted these DAG files and could edit them.

I had not further adopted the tool to then Convert the files back into the game, because I am not quite sure if they have any use?

Would being able to edit the dag file clipping data, have any influence on the abilty
to improve map building?

example of
art\clip\Map-2-Hommlet-Exterior\well.DAG