My former ATI-based laptop, the Acer Aspire 5050, had a 3D-capable adapter that somehow didn't work with Vista at all (despite the computer shipped with it!), but worked well with the free radeon X.org module and the DRI-based 3D rendering code from the Mesa project for most purposes. This HP laptop brings to me an inversion of the dilemma: ATI's drivers work very, very well with Vista, and although radeon and radeonhd have DRI support already (thanks to the drm/radeon modules in Linux kernel 2.6.29 and later), there seems to be no hope of reliable 3D support from Mesa so far.
ATI provides a proprietary driver for current X.org and Linux kernels, fglrx, but the last time I tried it I only managed to crash the Linux kernel and the BIOS (on reboot) whenever I switched consoles or restarted X.
Still, without accelerated 3D rendering, it seems like a waste to always allocate 256 MB of system RAM and never put them to good use for anything.
Open-source radeonhd and drm/radeon
Until now I'd been using radeonhd, which Ivanovic, the Wesnoth project manager, recommended to me back when I was trying to use the laptop with Debian Lenny last January. Since then I've always used the latest revision from the upstream git repository. Combined with the drm/radeon kernel module from Linux 2.6.30 and 2.6.31 it provides a stable (ahem) and rather fast environment with XRender acceleration, using EXA; thus I can use some of KDE 4's eye candy. However, the odd still happens from time to time and what seems to be a race condition locks up the system when resuming from hibernation:
[drm] wait idle failed status : 0xA0003030 0x00000003
[drm] wait idle failed status : 0xA0003030 0x00000003
[drm] wait idle failed status : 0xA0003030 0x00000003
[drm] wait idle failed status : 0xA0003030 0x00000003
...
This is not specific to my current flavored kernel (now with TuxOnIce!) and I've stumbled upon this with vanilla kernels ever since drm/radeon supported this chipset (RS780M/RS780MN) in mainline stable releases, using uswsusp for hibernation.
radeonhd also has an unusual habit of shaking the screen vertically for a few millisecs whenever I start SDL-based applications or XVideo clients (think VLC or Kaffeine).
Trying the open-source radeon and drm/radeon
Espreon pointed out to me on IRC that radeon displayed better performance on his computer than radeonhd. Trying it, this seems to hold true for a few XRender effects and KDE 4, but it's otherwise the same as radeonhd, minus the flickering, and with an additional bug — it occasionally fails to reset the console when giving control back to fbcon, the kernel's graphics mode console driver, resulting in black screens. Due to this, I disabled graphics mode and reverted to good old text consoles for testing radeon.
This driver has an annoying misfeature. By default it will try to enforce subpixel rendering on laptop screens and make my fonts look slightly hideously ugly when antialiased, despite this is explicitly defined in my ~/.fonts.conf for fontconfig-based applications. The solution turns out to be manually editing the X.org configuration in /etc/X11/xorg.conf and adding an option to disable subpixel rendering at the server side:
Section "Device"
BoardName "ATI Radeon HD 3200 (RS780M/RS780MN)"
BusID "1:5:0"
Driver "radeon"
Identifier "Device[0]"
VendorName "ATI"
# radeon-specific options follow
Option "SubPixelOrder" "NONE"
EndSection
Otherwise I could see the colorful antialiasing on the screens ruining the rendering of Tahoma, 7 pt., or Consolas, 10 pt.
I have yet to try this one for enough time to check if it also causes the occasional race condition on drm/radeon that locks up the system with radeonhd.
Trying ATI fglrx
RUN AWAY!
Actually, I abandoned any attempts at using fglrx with Debian Lenny when I noticed that the kernel module wouldn't compile with newer kernels — and I didn't trust ATI enough to go and fetch a newer version of the driver from upstream.
However, now that I'm using Squeeze I thought that I may as well check it out again, since the distribution uses Linux 2.6.30 already... Of course I could compile it. Now the question is, has this thing improved at all since the last time I tried it?
Virtual terminal switching works now, at least with text-mode consoles. Unsurprisingly, the old bug in which restarting X via something like invoke-rc.d kdm restart from a text console may cause the fglrx kernel module to do something silly and lock the kernel up/crash the video BIOS the next time I switch to the X console (#7) is still there.
Despite these issues, the driver is mostly stable with this chipset now and does mostly everything radeon and radeonhd can, and 3D acceleration works well enough to enable OpenGL compositing with KDE 4 and play with the Desktop cube effect. 
Certain KWin effects such as Box switch and Taskbar thumbnails won't work in OpenGL mode with trilinear filtering. It could be a KWin bug rather than fglrx's fault, though.
Minimizing XVideo clients such as VLC media player seems to cause occasional glitches as well, e.g. video surface disappearing until playback is restarted.
Additionally, trying to use SuperTux 0.1.3 in OpenGL mode crashed the whole thing and went as far as to make the kernel write corrupted data into /var/log/kern.log before I rebooted it with a magic SysRq sequence. That's a bad sign, you know.
Now for a completely different issue. I trust the free drivers in their judgment that my screen dimensions are 331x207 millimeters and the resolution is 98 DPI (although I couldn't find the exact specs from HP either). ATI's driver claims that my screen is 332x212 millimeters and that it has a resolution of 98x96 DPI! Even if that was right, it should provide me a way to override this — it does not. Instead, I convinced the font rendering engine to ignore the server's specs and use 98x98 DPI for fonts:
echo 'Xft.dpi : 98' >> ~/.Xdefaults
But after a while I figured I could just override the screen autodetection altogether instead and write my own Monitor section in the xorg.conf with the dimensions reported by the open-source drivers:
Section "Monitor"
Identifier "Monitor[0]"
VendorName "fixme"
ModelName "fixme"
DisplaySize 331 207
EndSection
Section "Screen"
DefaultDepth 24
Device "Device[0]"
Identifier "Screen[0]"
Monitor "Monitor[0]"
EndSection
All this mess to make Tahoma 7 pt. look pretty and readable enough.
This thing interacts rather weirdly with TuxOnIce. While hibernating and resuming works fine at the end, TuxOnIce often has to try twice to create the hibernation image. Since TuxOnIce needs to switch to a text console before proceeding to prepare the hibernation image, I suspect it could be related to a silent issue taking place whenever I switch from the X terminal to a text console during normal operation:
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
Assertion failed in ../../../../../../../../drivers/2d/lnx/fgl/drm/kernel/hal_rs780.c at line: 53
[fglrx:fireglAsyncioIntEnableMsgHandler] *ERROR* IRQMGR returned error 1 when trying to enable interrupt source ff000034
Returning to the X console doesn't cause additional messages; it's just leaving X that causes problems. While nothing crashes immediately in either case, I suspect that it'd not be a good idea to try and re-enable graphic mode consoles and then run X with fglrx.
Nonetheless I'm back using radeon + drm/radeon for now because I hear a faint buzzing while using fglrx for some reason — it can't be a good sign.
Linux 2.6.24 - 2.6.31 and Debian GNU/Linux
And what about the rest of this ATI-based system?
Linux 2.6.24, which is the kernel used by an old Debian Testing DVD I used to perform the initial installation of Lenny back in December 2008, had a network driver that OOPSed a lot when probing the wired LAN interface of this laptop — fortunately I could finish the installation and upgrade to the current packages of Testing at the time, including Linux kernel 2.6.26.
Shortly after Debian Lenny became the new Stable distribution I settled with the Linux kernel 2.6.28, which worked pretty fine for most things except one: drm/radeon didn't support my graphics controller, which meant no EXA or XRender acceleration for radeonhd 
I was reluctant to switch to Linux 2.6.29 for a long while because it accentuated a bug in which, on random boots, the touchpad and keyboard become unusable until the next reboot; this also affected 2.6.28 but only if I pressed a key very early during the boot process. I never ran 2.6.29 for daily use on the laptop and I ended up trying an early RC of 2.6.30 instead... oops.
Eventually I just settled with the released 2.6.30 kernel and later with 2.6.31 — neither is immune to the aforementioned input layer bug, and they accentuate another bug that I could reproduce on 2.6.28 only once: at random times, after resuming from hibernation, the touchpad becomes unusable and just the caps/num/scroll lock keys do too, in such a way that pressing them locks up the entire system for a second or two; after the first two or three lock-ups in such a scenario, the kernel complains, only once:
ACPI: EC: non-query interrupt received, switching to interrupt mode
Additionally, so far all kernels that have run on this laptop are affected by a bug that gets the input layer stuck with the last key event for a second or two at random (notice a pattern?).
atkbd.c: Spurious NAK on isa0060/serio0. Some program might be trying access hardware directly.
It also happens that the native hibernation and suspend-to-RAM mechanisms fail to bring the keyboard, touchpad and screen back to life. I've not been able to use suspend-to-RAM here with any of these kernels and suspend-to-disk/hibernation works only if using uswsusp from Debian Lenny or TuxOnIce on Debian Squeeze.
Naturally, I have not found a solution for these problems yet.
There's one problem I could solve myself with some help from Google, though; the ACPI DSDT of this system has a rule that doesn't allow any operating system but Windows Vista to access the thermal zone status. Notable because the DSDT has other rules to allow Windows XP, 2003 and GNU/Linux to work fine with it, so whitelisting just Vista for temperature readings does not make sense to me.
I ended up patching the DSDT and instructing Linux to override it at boot time — yet another reason for using custom flavored kernels. It's really important to be able to see the system's estimated temperature when it often goes around 94°C when working with the damned thing in summer.
What's next?
I'm really fed up with these ATI-based laptops as I always have some kind of problem with the few drivers that support ATI graphics controllers. The next time I can buy a laptop I will choose one that a) is not manufactured by HP; and b) doesn't use a single piece of hardware from ATI. The problem is, what graphic adapters are well supported for Linux, and what laptop manufacturers use durable, reliable components and provide good technical support in Chile? 
(And yes, I know I've linked too much to this, but can you believe it? Shared libraries turning into character and block device nodes and FIFOs? I'm not making this up.)
