After being in Development Hell for...I forgot how many months, Wesnoth-TC, the advanced Team-Colorizer version 1.5.0 is finished and released.

This version introduces a few changes in the output files' naming scheme, and adds support for palette switches a la ~PAL() image function from Wesnoth's engine (which I implemented, by the way). The old Makefile is gone and there's a normal autotools build system in place, which seems to work well on Linux systems, as well as Mac OS X and even Mingw32-based cross compiling environments!

Thanks to this improvement and a small help from loonycyborg, Wesnoth-TC also has a experimental “official” Win32 build this time, which I have successfully tested on Windows XP SP3, and Debian Squeeze using WINE. It was cross-compiled using the Mingw32 environment packages from Debian. There are no important functionality differences between the Win32 build and normal Unix-based builds. It remains a command-line application as it's supposed to be.

Download

• Source code: wesnoth-tc-1.5.0.tar.bz2 (149.7 KiB)
  (SHA1: 030b4584a52e41f9706f7e59fdd5a9896485ed83)
• Windows (Win32) binary: wesnoth-tc-1.5.0-win32.zip (373.3 KiB)
  (SHA1: 311adf05b3855bf74eb782fe020992e61aac80eb)

In an ideal world, the availability of a CLI Win32 binary would raise interest on this tool and someone would volunteer for working with me on a GUI front-end. That's very unlikely to happen in the real world, so I'm researching Qt4 and messing around with some ideas for a cross-platform front-end based on penguin's Mac OS X applet.

Instructions

Both distributions come with a README file, and the source code distro includes a INSTALL file with detailed instructions on configuring, compiling and installing wesnoth-tc. The Win32 binary distribution doesn't require any installation besides unpacking it into an appropriate directory — which you may optionally add to PATH.

EDIT: some people may have problems compiling under certain platforms or configurations because the configure script in this version enables strict mode by default, treating compiler warnings as errors. To solve this, pass the --disable-strict option to configure. Git master already solves this problem, plus a particular warning reported on Ubuntu 9.10.

Behind the scenes

As I mentioned before, I tried using gd for PNG manipulation but ultimately discovered that its limitations weren't nice for wesnoth-tc's particular case. I rewrote the PNG reading and writing code again afterwards, using the SDL_image library as a model instead.

The Win32 build doesn't have support for libpng exception handling (via longjmp/setjmp) because the compiler complains about it for some reason:

warning: variable ‘height’ might be clobbered by ‘longjmp’ or ‘vfork’

This shouldn't be a big deal since libpng won't complain unless someone feeds a corrupted or invalid PNG file to it. (EDIT: figured out the cause with some help, but haven't fixed it in Git master yet.) The other functionality change is the lack of support for checking file access permissions considering effective user and group ids, etc.

Supposedly, this release should work properly on PowerPC and other big-endian platforms. However, I don't have such a machine and therefore can't test.

My own software projects tend to be very much like pets to me. I take care of them, carry them with me anywhere if it's possible, I feel horribly sad when something bad happens to them and, even when I go mad at them for something, in a few hours we are together like a neat happy family again.

Invasion from the Unknown has been the Wesnoth add-on project of mine since around September 10th 2007 and it has evolved throughout time and endured 3 mainline development cycles introducing drastic game engine changes, receiving little automated help from the likes of wmllint. Instead, it has been kept on shape by me and a few people who have helped with the huge maintenance burden that this epic-length campaign is.

Inspired by what ESR has to say regarding migration from SVN to the Git version control system in practice, I decided to start “considering” actual options for abandoning that unsafe, annoying and apparently not very scalable centralized version control system known as SVN in one of the only remaining projects where SVN is still used and I'm involved as a developer. In this case, I'm the founder of this project, good old Wesnoth-UMC-Dev.

More specifically, I've decided to take a realistic look at the options as they present themselves — and since Git is a hot topic on both the mainline Wesnoth Project and Wesnoth-UMC-Dev, exploring our possibilities with it sounds like a good way to begin yet another “mayhem” series in this blog (oh boy, I need to categorize those).

Note that I'd normally have posted this in the wesnoth-umc-dev blog if it still existed — it wasn't being used by anyone and the information it contained has been moved and expanded upon in other web pages of our site. Blame AI0867 for that.

For a few years since ESR established the new mainline conventions in Wesnoth for tagging and adding new contributed music tracks, I've been in charge of helping with the process of importing tracks since ESR doesn't seem to be available for this anymore, and our “Lords” of Music never get familiarized with SVN to do it themselves. Normally I'm asked to do this whenever I'm available, but for some reason, another developer did it this time on March 13th — that is, two days ago — and I didn't find it out until today.

This minor thing reminded me of a substantial problem with campaign music in Wesnoth in regards to originality, which I've been perceiving for a few months already.

The last post (minus the hard disk driver crash during preparation for S3 issue) turns out to be a rather embarrassing case of not doing the research, so I deserve a punch from a drm/radeon dev for that.

What I should've done if I were connected to the Internet at the time I decided to start to experiment with KMS again, is reading the instructions from the X.org wiki for properly setting up the drivers with KMS support. It wasn't as trivial as I had expected because of the missing firmware blobs I mentioned the last time.

I was hoping to not have any more problems with Linux and my HP Pavilion dv5-1132la laptop since the last installment of this series of posts. However, with the release of Linux 2.6.33 and the improvements on KMS and general direct rendering support for the ATI R6xx and R7xx chipsets, I wanted to give KMS a try — this kernel version also improves support for some HDMI thing that I don't understand exactly what it is about, apparently related to digital audio devices or something.

(In other words, I shouldn't be complaining. Most of what comes in this post is the result of a voluntary experiment, and not some unfortunate incompatibility...probably.)

Since I'm fed up with libpng's complexity (which borders on obscurity), I asked for recommendations on libraries for reading, editing and writing PNG files in #defocus, freenode's social channel. Someone mentioned gd, which I mostly knew for being a dependency of PHP-based software that deals with graphics, so I decided to give it a try.

It turns out to be incredibly easy to learn and use — I quickly started the gd_writer branch for the Wesnoth Team Colorizer tool and nuked hundreds of convoluted lines of libpng calls to replace them with short, concise and clean code.

However, I then noticed something in the (thorough and easy to read) documentation that spelled trouble for my usage:

gd retains only 8 bits of resolution for each of the red, green and blue channels, and only 7 bits of resolution for the alpha channel. The former restriction affects only a handful of very rare 48-bit color and 16-bit grayscale PNG images. The second restriction affects all semitransparent PNG images, but the difference is essentially invisible to the eye. 7 bits of alpha channel resolution is, in practice, quite a lot.

No justification is provided in the documentation as far as I could find. However, the main header file (gd.h, which is perfectly readable unlike libpng's png.h) has this precious missing bit explaining the reasoning here:

If 'truecolor' [in a gdImage struct] is set true, the image is truecolor; pixels are represented by integers, which must be 32 bits wide or more.

True colors are repsented [sic] as follows:

ARGB

Where 'A' (alpha channel) occupies only the LOWER 7 BITS of the MSB. This very small loss of alpha channel resolution allows gd 2.x to keep backwards compatibility by allowing signed integers to be used to represent colors, and negative numbers to represent special cases, just as in gd 1.x.

Ah, good old backwards compatibility, always biting the programmer in the ass after a while — Windows 9x being a major commercial example of backwards compatibility gone mad.

While I agree that the difference might not be noticeable for the human eye, this still means that wesnoth-tc/gd produces output that differs from the original images in more than just team colors. It'd be often silly to use team-colored semi-transparent in Wesnoth unit sprites, but team color and palette switches can, thanks to the flexibility of the image path functions mechanism, be applied on virtually any kind of image that SDL_image can read into a surface pixmap, including things such as transparent haloes for visual effects.

Experimenting with wesnoth-tc/gd, a linear sequence of pixels with these alpha channel values:

255, 254, 253, 252, 251, 250, 249, 258 [...]

...turns into this in the application's output:

255, 255, 253, 253, 251, 251, 249, 249 [...]

What happens here is that values below 0x7F — which is 127 in base 10 and 1111111 in base 2, and as you can see, the maximum positive integer that can be represented with 7 bits — can only be even, and values above are odd. There's no 0x7F.

I could just ignore this issue and merge the gd_writer branch, but no, I think I'll just try yet another library. It's a pity because this experiment had created the possibility of making a web interface for recoloring/team-coloring Wesnoth artwork, but I believe this kind of loss of information isn't any good for our purposes, even if it should be virtually unnoticeable at the standard 72x72 sprite scale. At least unit shadows shouldn't be affected by this limitation since their alpha value is supposed to be 153, which would remain unchanged under this scheme.

And so, Wesnoth-TC 1.5 stays in Development Hell for now, and the search for a simple enough libpng wrapper continues.

I've been working on cleaning up the code for Wesnoth-TC's PNG reader/writer functionality and so far I've only partially succeeded. I cleaned the code up, sure, but I also messed up somewhere and mixed up something breaking Wesnoth-TC.

The new_writer branch is already in the Git repository but users will definitely want to use master instead. I haven't yet decided if new_writer will ever be merged back into master or I'll just throw it away.

But at this point I'm tempted to give up and try a different library for reading and writing PNG files — if I keep trying to understand how to use libpng correctly (if that's even possible), Wesnoth-TC 1.5 will never be finished.

It's been almost 3 days since the earthquake in Chile which affected many regions of the central-south areas, including the region of Santiago. While things are mostly normal here — besides the many closed stores, the many damaged buildings, unusable highway infrastructure in some places, some fallen buildings, zones without electric power, unstable mobile phone networks and a bit more than 30 dead in the region, the disaster here pales in comparison to what can be seen following the coastal line to the south near the region of Maule and Bío-Bío. There's no way I could properly summarize what happened there, so you'd better use Google if you are curious. Many locations look like ruined, flooded, destroyed battlefields.

And yet, opportunists have pillaged around 16 supermarkets in our region so far; some succeeded, but most failed. There was not enough damage on this region to justify such vandalism. Naturally, Concepción and many other locations in the south are also being constantly vandalized, enough to justify passing the control of two regions over to the army to try to keep the order. They haven't completely succeeded, and the vandals have started fires, given false alarms of tsunamis and destroyed or damaged private and public property.

The aftershocks have been scarce today and they've mostly faded out from our point of view in the Santiago region — that is to say, they haven't stopped in Maule and Bío-Bío and it doesn't let those people rest or sleep for much time. A few stronger aftershocks have reached us as well. Fortunately, they are not anything like the original earthquake.

However, they seem not to stop for me. I've been feeling dizzy since the moment I got out of the house during the earthquake and, no matter if I'm on my bed or sitting on a chair or on the toilet, everything feels the same as in the aftershocks, like a big boat on the sea that never stays still. It's very annoying but I still work on stuff and chat on IRC despite of this.

As sad as this is, and as grim as everything may seem, Santiago is mostly alright and we must move on and continue our lives and help however it is possible. But an earthquake with epicenter near our city isn't a terribly far-fetched possibility since we live next to several (relatively inactive) volcanoes, so let's not relax too much either.

More than one year ago I commented on the consequences of interrupted commit transactions with the Subversion version control system. Back then, SVN was the only VCS I was familiarized with, but nowadays I also have a basic grasp of Git for local and remote repository management.

The thing is, SVN is pretty simple and easy to learn for novice users — which is one of the reasons I haven't decided yet, as the founder admin of the Wesnoth-UMC-Dev Project, to switch to Git. A distributed version control system such as Git or Mercurial are not “better” than SVN, just like Linux cannot be “better” than Windows — they are completely different models for both users and site admins, and switching your version control system isn't as easy as switching from KDE to GNOME as your desktop environment or buying a new printer, especially when you have lots of users and the model conversion isn't easily reversible.

But let's not forget that there's more to SVN, or any other revision tracking system than just the philosophy and the model behind. There is an official client which ships in major Linux distributions such as Debian GNU/Linux, Ubuntu and openSUSE, and which also has shared library code used by third-party GUI front-ends such as kdesvn, or other SVN clients such as the git-svn infrastructure.

I have not seen the code, and I believe I do not want to see it with my eyes, but SVN's network code seems to be crap.

The issue mentioned in my second blog post remains the same after several versions of the vanilla SVN client. Then there are other issues that have been here for a long time, and an issue I only discovered some days ago:

• It is possible in middle of a networked transaction (commit, update) for the Subversion client process to get stuck if a network error occurs. Subversion normally traps SIGTERMs (and apparently SIGHUPs and SIGINTs too) to perform cleanup routines after such interruptions. However, when it gets stuck, the SIGTERM handler becomes useless and the client ignores the signal forever. This means that SVN can get stuck and sit idle on the terminal (most likely waiting for data from the remote host) until it a SIGKILL is sent to force the destruction of its process. Since SIGHUPs are also trapped, killing the terminal leaves a hidden, waiting SVN process. In other words, you can get a dead SVN client running for months if your power source is stable enough. Wonderful.
• Clients that terminate abnormally (SIGKILL and such) may leave random crap hidden in your SVN checkout's control directories that are normally cleaned up by the SIGTERM handler. While this is often inoffensive and svn cleanup or svn cleanup .. can handle it all, there are times when this is not enough and SVN gets confused for missing/extra files or directory metadata and refuses to update or cleanup a path. In such cases, removing the path and its contents and re-checking it out with svn update (or, if it was the whole working dir, svn checkout) is necessary.
• There seems to be a lot of overhead in the SVN subprotocol on any transport class, be it http, https, svn or svn+ssh. Commits containing simple changes to file/dir object properties can take as long as a regular commit diff when they should probably contain less data (if they contain as much data, then...?!). This is very noticeable on low-bandwidth connections for me. In comparison, a SVN commit of about 20 property changes can take longer than a Git push through SSH of about 10 large commits introducing whole new files.

Then there are some odd things with the SVN client library (libsvn), specifically the Perl bindings, namely the issues I mentioned at the start of this month, that leave me very disappointed at this version control system, rumored to be better than CVS (which I haven't ever used...imagine!). The Debian version in Squeeze, and possibly Lenny or upstream too, has a really nasty bug which caused a massive memory leak with Wesnoth-UMC-Dev's umcpropfix tool when I ran it to set properties on a version of Extended Era for Wesnoth, manipulating over 1300 files on multiple dirs. The parent Perl interpreter process allocated over 2 GB of overall virtual memory, making Linux page most memory out to swap, thus hurting performance.

The cause? Pool management. libsvn's Perl bindings are supposed to do automatic memory management unless the client wants to do their own pool management with the library's facilities, but that somehow causes the aforementioned leak instead. The solution turns out to be doing custom pool management by allocating a new pool for every libsvn call, and forgetting the old one. r6567 in the Wesnoth-UMC-Dev repository applies this workaround for our SVN property setting tool.

Honestly, I'm tired of SVN. I use git-svn wherever I can but this isn't a magic solution for the crappy design of SVN's innards. git-svn can skip upstream commits if the connection interrupts during a fetch operation, and it forgets about local commits during pushes (git svn dcommit) after it has sent the first commit and fetched missing ones, which can cause loss of commits and history if the connection to the remote host breaks at that point or git-svn exits in any other fashion.

git-svn doesn't replace SVN's network code either (it uses it instead), so it's still subject to the perceived overhead, but at least it doesn't get stuck forever ignoring SIGTERMs.

What Wesnoth-UMC-Dev needs at the moment is a distributed (yes) version control system that's as more user-friendly as than SVN and has a nice, well documented Windows front-end that's easy to setup, learn and understand. If we can't find that, I'll continue complaining about SVN at any time and on every place where I see fit, here or in IRC.