Image Resolution Degradation With WorldBuilder's 2D Maps Recombining Feature

Discussion in 'Maps, Textures, and Graphics' started by Gaear, Mar 2, 2013.

Remove all ads!
  1. Gaear

    Gaear Bastard Maestro Administrator

    Joined:
    Apr 27, 2004
    Messages:
    11,029
    Likes Received:
    42
    :scratchhe

    The map art for the first floor of the inn has never been altered. There are override files in place in the mod for that map, but for a different reason than to change the art. There is one .jpeg chunk (00230020 to be specific) that is problematic. It is strangely larger in size than the others, although like the others, it is a 96 dpi image. If you recombine that map in WorldBuilder and then split it again however, WorldBuilder turns that chunk it into a 72 dpi image, for reasons I don't know. This all seems to be handled well internally with the vanilla image, but ironically, not with a WorldBuilder processed image, like yours.

    What I did for the mod was to save a version of 00230020 with the same exact specs, resolution, and relative data size of its neighbors, and then simply included all .jpeg chunks in that map in the mod for a total override. The reason for this is that I had fears that the game might not have been handling it well internally and possibly causing some of the issues we see with that map - long load times and maybe even the onset of exterior Hommlet jerkstop. I don't believe that's proven to be the case, but it's still probably best to have consistent .jpeg map chunks running. More to the point though, all of this results in no effective visual change for the user. None of the content of the image for that map was changed whatsoever, as you can see in the images below.

    Vanilla

    InnFirstVanilla_.jpg

    Co8

    InnFirstCo8_.jpg

    However, we do see the problem I described above arrive in a recombined version of your .jpeg chunks.

    Zoltec

    InnFirstZoltec_0.jpg

    And here is a highlighted version to show the specific chunk with issues. As you can see, it is too large to fit in its allotted square.

    InnFirstZoltec_1.jpg

    The game seems to resize it correctly in-game, but the condition is doubtfully ideal. The Co8 version of the file is 14 kb in size, 96 dpi. Your version is 85 kb in size, 72 dpi.

    Finally, here's a composite to make comparison easier.

    InnFirstComposite_.jpg

    The bottom line is that there is no need to either delete or replace the art for this map from the mod, and it might even be best if you did neither. Doing so will provide no visual change or 'restoration' whatsoever.

    All that said, I'm interested in your transparency fix for the black and white mystic garb. If it tests okay, we may use that for the mod if you don't mind.
     
  2. Zoltec

    Zoltec Pгōdigium

    Joined:
    Apr 22, 2012
    Messages:
    547
    Likes Received:
    16
    Re: Aziraxis Modification Pack

    Hi, Gaear. Thanks for being specific, I already tracked down the problem and made a revised chunk. The revised chunk altogether with the vanilla ones when recombined and split, does not show your findings anymore. And I lowered the quality of my chunk but retain the same quality texture, the current file size is 66.2kb oppose to the vanilla one which is 80kb++.

    Summary

    - The revised chunk retains the 96 dpi image when recombined and split in latest world builder, together with the vanilla ones.

    Opinion and Facts

    Hmm, Gaear. I assume your using a 1680x resolution that's why you can't see any drastic change to the image quality done by the World Builder, I'm using a 1024x768 resolution, and I really can see the quality of the image being brought down when being recombined and split in ToEE World Builder.

    Sorry for using an inappropriate word 'alter' going to retract that, what I meant of is the image quality is lowered when being recombined and split.

    Hopefully, this will fix the chunk problem and this kills your doubts about the 'weird ToEE engine image handling', while retaining the high quality texture of the Hommlet-Inn-First-Floor-Interior.

    Congratulations! =) Ah yes, go implement that garbfix into your mod, if you want revises, I can.
     

    Attached Files:

    Last edited: Mar 3, 2013
  3. Gaear

    Gaear Bastard Maestro Administrator

    Joined:
    Apr 27, 2004
    Messages:
    11,029
    Likes Received:
    42
    Re: Aziraxis Modification Pack

    The .jpeg chunks for the inn map in vanilla and Co8 are functionally identical. 00230021, for example, is a 256 x 256, 96 dpi, 8,931 byte image in Co8, and a 256 x 256, 96 dpi, 8,936 byte image in vanilla. See 100% side-by-side comparison below.

    chunks.jpg

    I don't believe that WB even does any real image processing. Its main function in the 2D Maps app is to simply split your existing map, along with its existing resolution and various other characteristics, into 256 x 256 sections, or to recombine existing 256 x 256 sections of maps into a large unified image for editing by the user in Photoshop or whatever. There is no resizing going on or any other editing to lighting, colors, etc. (the issues with the trouble chunk notwithstanding).

    Can you post some screens to show what you're talking about? I assume that on your 1024 x 768 display, the same side-by-side exercise as I showed above should reveal noticeable differences between the images. At any rate, if this were really an issue we could simply copy the .jpegs from vanilla (minus the goofy one) directly into the Co8 mod.
     
  4. Zoltec

    Zoltec Pгōdigium

    Joined:
    Apr 22, 2012
    Messages:
    547
    Likes Received:
    16
    Re: Aziraxis Modification Pack

    Hmm. In here, the extracted vanilla (00230021) sizes to 16,475 bytes while the co8 is 8,931 bytes. This is the evidence that the image quality is lowered by the WB tools.

    Actually, there is. Notice the differences with the image provided below. These are the chunks of Vanilla (First Image), Zoltec (Second Image), Co8 WB (Third Image).

    You will see the image quality lowered when processed by the World Builder tools i.e. (recombine,split function).

    Update: You'll want to download the .tga format in .zip file, to see and fully realise, what I'm into, and for more clear visuals.

    Like what I did to my revised chunk? Yes, it is. You may do it like that, thus, it retains almost the quality image of vanilla ones, more-over, does provide more quality visuals to the Co8 players.
     

    Attached Files:

  5. Gaear

    Gaear Bastard Maestro Administrator

    Joined:
    Apr 27, 2004
    Messages:
    11,029
    Likes Received:
    42
    Re: Aziraxis Modification Pack

    In where? I'm pulling 00230021 straight from the dats. It's 8,936 bytes.

    No, it's not. No visual difference that I can see, no data size difference, no indication from Agetian that he ever employed anything but the most basic functions for splitting and recombining.

    No offense Z but I guess I need some of what you're smoking. Looking at those images reasonably closely (e.g. not with a magnifying glass but with a studied eye), I can see no effective difference. I looked at the .tgas as well, still not seeing it. Can anybody else see what he's talking about?

    I think maybe you should try posting a screen capture to try to demonstrate what's happening in-game. Also, what other maps do you see this problem in?

    [edit]

    Okay, after scrutinizing the images very closely at 200% magnification in Photoshop, I do see a difference. The thing is, my WB and dat versions are identical, so it's still not WB. Where are you getting your vanilla chunks from, and what utility did you use to extract the dats?
     
  6. Zoltec

    Zoltec Pгōdigium

    Joined:
    Apr 22, 2012
    Messages:
    547
    Likes Received:
    16
    Re: Aziraxis Modification Pack

    I did get the chunks from the original .dat files of Temple of Elemental Evil (fresh install). I'm using ToEE DAT TOOL from Agetian to extract the contents of the .dat files.

    I'm not questioning his program, indeed he is very helpful to the Co8 community by developing the tool. But you see the original number of bytes is decreased when the vanilla one's chunks is <recombined into one single image>, then <split> the recombined chunks or single image, observe and you'll see the differences in vanilla chunks (untouched by WB) and the processed vanilla chunks by WB.

    No offense either, but please be decent, be a model to us, what you are telling me here is what you reflect, try to understand your fellow modders, I'm sharing my time here to help develop the game to transcend it to a more beautiful state. And personally I don't smoke, I'm not either one of what you really talking about. I maybe not trained to give consistent details, then that's my drawback, I'm just a human.

    Yeah, I can pin-point one particular map, and that's the tutorial map in the Underdark Quest - Verbobonc. I don't know if it has a texture re-done in the current Co8 v7.7. Well, let me see. I'll post the screenies later.

    What do you mean by 'my' WB? and dats are being identical to? Do you own an updated version made by Agetian other than hosted here? I think the image processing of the program (WB) is at fault here and I really can't blame the developer, instead this one will be a way to help, to improve the minor-consistencies found in the program one-step. Like what I stated above, I'm using ToEE DAT TOOL by Agetian and did get the chunks from the fresh install .dats of Temple of Elemental Evil. I'm using the latest World Builder.

    By the way, I have a copy of a specific original chunk named (00230021) it has exactly the same details on what I have told you earlier i.e. 16,475 bytes included in the vanillla Inn-First folder. Download, the file below. Its an eye opener. ;)

    Edit:

    And here is the original Co8 - InnFirst - processed by World Builder, for comparison purpose.
     

    Attached Files:

    Last edited: Mar 4, 2013
  7. Gaear

    Gaear Bastard Maestro Administrator

    Joined:
    Apr 27, 2004
    Messages:
    11,029
    Likes Received:
    42
    Re: Aziraxis Modification Pack

    Okay, I used Zane's. I wonder if that might be the real culprit, because ...

    ... that's exactly what I don't see. For me, the vanilla chunk from the dat is identical in size and image quality to the same vanilla chunk recombined and split by WB.

    Okay, but a little less with the lecturing please. Believe it or not, I'm trying to make your modding better by holding you to a higher standard.

    I will have to take a broad look at all our modded maps, because if image quality is suffering throughout, that will have to be dealt with. Unfortunately, in the case of all our custom maps, there is no other way to make a map besides splitting all those chunks by hand. That would be a daunting task to put it mildly, which is why I'm hoping the culprit is not WB here. The large exterior maps can contain hundreds if not thousands of chunks.

    I mean my WB and my dat versions - the chunks as taken straight from the dats and the chunks as split by WB.

    Me too. I'm using the same tools as you. The only difference is that I extracted the dats with Zane's dat extractor as opposed to the app in WB.

    I will have to try re-extracting the dats in WB and comparing them against those I got with Zane's.
     
  8. Gaear

    Gaear Bastard Maestro Administrator

    Joined:
    Apr 27, 2004
    Messages:
    11,029
    Likes Received:
    42
    Re: Aziraxis Modification Pack

    Okay, it's WB, specifically in the recombining step. :(

    I think I inadvertently had WB-split chunks in my first floor inn dat folder, which is why I couldn't see the difference initially.

    So anyway, being as the game doesn't seem to have any issues handling that one weird chunk in the inn itself, we'll just remove the override jpegs to go back to the better resolution for the mod. Have to look at a bunch of other maps too unfortunately.

    ~

    Also I split this off from your thread so as not to take it over, Z. You may want to repost your fixes and whatnot there.

    [edit]

    Additional info ...

    Upon looking more closely at things, I see that vanilla .jpeg chunks are actually 72 ppi, and for reasons that are not clear to me, WB resamples them to 96 ppi. In my admittedly limited understanding of DPI, I don't think this should affect anything, but who knows? Also, from what I can gather (according to some people at least), there is some inherent degree of image quality loss that occurs with opening and saving any .jpeg due to the format's lossy high compression characteristics.
     
  9. Gaear

    Gaear Bastard Maestro Administrator

    Joined:
    Apr 27, 2004
    Messages:
    11,029
    Likes Received:
    42
    Okay so far I have 'restored' the inn and the other three Hommlet interiors that were flipped horizontally for accuracy purposes. In the case of the latter three I just rebuilt the maps with flipped vanilla .jpeg chunks and left WB-recombining out of the equatiion. Also re-did Emridy Meadows, with the only override files being the chunks where the fissures are.

    Unfortunately, in the case of anything that was customized from original ToEE art (which is basically everything, like the fissures above), the customization was done off of a WB-recombined version of the map. So we will be stuck with the image quality loss for those, as I'm not going to re-do all the art in the mod for the next 10 years.
     
  10. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    JPEG, being a lossy format, is pretty difficult to preserve absolutely intact when different compression schemes and internal algorithms are used [when storing the original image and when resaving it in a different program]. There is nothing in ToEEWB that would purposefully downsample the recombined images or store them at a deliberately lower quality. All ToEEWB does is load up the images, put them all on a combined canvas, and save them back in the same basic format (jpeg, that is). The process of splitting is much the same. The loss in quality comes from two facts: the first is that .NET Framework might utilize a somewhat different approximation algorithm for pixels than originally used, and the second (I believe, more influential) is that the JPEG file format, being lossy, always utilizes at least some approximation to reduce the file size. As such, when resaving a modified JPEG file (split/recombined), some loss is bound to happen because we are talking about reapproximating an already approximated picture (and using a different algorithm, that is). I wish there were something I could do, but sadly I don't think I can - not unless ToEE used a different format (lossless, such as PNG) for its maps.

    - Agetian
     
  11. Gaear

    Gaear Bastard Maestro Administrator

    Joined:
    Apr 27, 2004
    Messages:
    11,029
    Likes Received:
    42
    Could WB convert the files to bitmap, or simply operate with bitmaps or other lossless formats? (e.g., the user could convert vanilla chunks from JPEG to BMP, WB could recombine/split as BMP, and the user could reconvert the WB-processed BMPs to JPEGs.) When I manually work with the JPEG chunks and then manually resplit them one by one in Photoshop using the highest quality JPEG setting, I get no perceptible image degradation.

    To put it another way, I believe that if I open a vanilla JPEG map chunk in Photoshop and convert it to BMP, there is no image degredation. Likewise, if I convert the BMP back to JPEG using the highest quality setting, there is no image degradation. What this means ultimately is that the original JPEG and the reconverted JPEG would be functionally and visually identical; I know that a native BMP image would show a marked loss in quality if simply converted to JPEG, even at the highest quality setting, simply because as you say JPEG is a lossy format. (It's reason for existence is in fact to be compressed, much like an .mp3 audio file.)
     
  12. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    Sure thing, that sounds like a good idea! I'll post a version of ToEEWB later today that will output BMPs instead of JPGs (hope to find time... but I'm pretty sure I will), you can use that then to preserve quality within ToEEWB.

    - Agetian
     
  13. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    OK, posted an update to ToEEWB v2.0.5a that has a few extra buttons to recombine and split into the BMP format. Remember that it'll recombine from JPGs (the original game format) into a BMP, and will split from BMP to BMPs (so you'll have to manually convert those resplit BMPs into JPG for the game to use them; most advanced graphics editors, such as Photoshop and GIMP, should contain a way to batch-convert those tons of BMPs into JPEGs, but I haven't looked into that). Consider this release experimental, please let me know if it works for you and does the right thing - if not, I'll patch it up. :) Hope this helps!

    - Agetian
     
  14. Gaear

    Gaear Bastard Maestro Administrator

    Joined:
    Apr 27, 2004
    Messages:
    11,029
    Likes Received:
    42
    Sweet Ag, thanks. :thumbsup:

    I likely won't try it for a bit (busy with other stuff, v7.8 due next Sunday), but I'll let you know when I do.
     
  15. Agetian

    Agetian Attorney General Administrator

    Joined:
    Aug 14, 2004
    Messages:
    2,526
    Likes Received:
    0
    Sure thing! I'll go ahead and code the new TFE-X in the meantime (no way I can make it before v7.8, so it's likely to go into the next revision, e.g. v7.9). Good luck with everything, my friend! Let me know if you need any assistance, I'll try to check back regularly.

    - Agetian
     
Our Host!