Shadowmaster’s Blog - IRC

Rei 2 Feeds: Behind the scenes

nospam@example.com (shadowmaster) — Thu, 21 Apr 2011 00:56:00 +0000

Just today I completed an initial, working implementation of an XML feeds reader in the Rei 2 IRC Bot framework. This implementation was pushed into master and I later tagged Rei 2 version 0.2.8, which will be eventually deployed on the shikadibot production instance, once this new shiny feature has seen enough testing and appears to be stable enough.

Thus my nearly 12 hours long coding spree culminates.

There were various design issues I faced when developing the feeds reader functionality. I already came across inefficiencies inherent to the legacy single-threaded model for SVN bot functionality found in Shikadibot 0314 and reimplemented in Rei 2; it’s not directly feasible to listen and respond to interactive events from an IRC connection socket while waiting for a reply that might never arrive from a SVN server. In the feeds reader case, more complications arise from the need to download and parse multiple documents from different remote servers in quick succession.

Rei 2 is still bound to the Irssi IRC client, and will remain so for a good while until the new S² state tracker is completed. The current situation is by no means bad as it allows me to concentrate my efforts on the bot’s outermost functionality layer, but the dependence on a non-native (i.e. C as opposed to Perl) backend places several constraints on what can be done in Rei 2.

Instead of resorting to threads — a possibility that I still don’t dare test with Irssi+Perl — I ideated a convoluted mechanism taking advantage of child processes and a simple caching strategy. Rei 2 forks its own helper daemon (rei2-feeds-worker.pl) when loaded (not necessarily always at Irssi’s startup!). This assistant simply fetches all XML feeds in the background every once in a while, and saves them to disk at a special location in the configuration dir — no more, no less. Meanwhile, Rei 2 sets up timed events to trap into the feeds processor and quickly work with what’s in the local on-disk cache.

Theoretically, this model could be extended to other Rei 2 modules, the SVN bot code being a particularly attractive candidate. It could be possible to either fork more unique helpers in the background, or somehow merge them all into another modular application. Either way, I’m not strongly interested on messing with the design right now other than researching more optimal mechanisms to monitor cache activity — inotify comes to mind.

• • •

For testing purposes, Rei2 on freenode already tracks the Wesnoth-UMC-Dev frontpage aggregator feed in #wesnoth-umc-dev and the Wesnoth.org’ Off-Topic forum posts feed in #wesnoth-offtopic. There’s also a duplicate feed of this very blog competing against one of my earliest private-use modules which has been reporting blog posts and Twitter updates in ##shadowm nearly since Rei 2’s inception. I’m meant to port the latter two to the new model soon-ish.

To celebrate this rather important milestone, I’ve updated the Rei 2 IRC Bot project page in this site as well with some new information.

Resolutions!

nospam@example.com (shadowmaster) — Sat, 01 Jan 2011 06:30:00 +0000

Don’t throw your shoes at me! I didn’t come to continue the overly long “display resolution” gag!

I have been thinking about some stuff to do during this year for a while, actually. It’s really hard to decide because I’m a person who runs into all sort of trouble while trying to get projects accomplished (including procrastination).

One thing I’m already doing is learning some Japanese, for no particular reason at all — although you’ve got to admit that having multiple languages in your curriculum is worth a bunch of coolness points. Espreon is helping me along the way with his own recently gained knowledge. It seems quite fun to learn a language in a non-Latin alphabet, if not a tad overwhelming at times, especially with kanji.

It’d be a good idea to lose some weight this year, too. My addiction to sugary stuff isn’t quite compatible with my heart condition! (Nor is coffee, but… meh.)

Then there’s Wesnoth. I intend to finish the Second Act™ of After the Storm Episode I as soon as I may, even through the means of placeholders — I’m willing to do anything to rescue AtS out of Development Hell before the end of 2011.

Wesnoth RCX’s development is halted right now due to lack of interest on my part to invest energy on writing the rest of the new functionality (i.e. definition of custom ranges and palettes), but I know that once I touch Qt Creator’s awesome interface I can’t stop working for a while — so I may eventually get some inspiration to redesign the main window, which should inevitably lead me to tinker around with the rest of the code.

C# was the first “major” programming language I learned, not counting Visual Basic. I have some fond memories of my first experiments with C#, but after I embarked upon learning and using C++ I kind of forgot about it. I have been considering the possibility of writing an IRC client of sorts using C# just for kicks, and to not forget this language in case I ever need it again. Why IRC? Because clients for this protocol are simple and challenging to implement, both at the same time!

I’ve already started to learn a bit of Lua for my work on the aforementioned Wesnoth campaign — in fact, there’s already some released code within it written in this language, particularly in scenario 5! I have plans to rewrite parts of Invasion from the Unknown in Lua to clean it up a little, thus paving the road for future maintenance work by me or other people (don’t forget that IftU is still abandoned!).

Another software project I intend to tackle in the short term is Rei 2. Sure, she’s doing well and her main command handlers are many and useful enough for channels such as ##shadowm and #wesnoth-umc-dev, but her dependence on Irssi’s core might well be a curse for one of our use cases: Shikadibot (the Second), which runs on a resource-limited VPS where every drop of RAM has got gold value. I’m already brainstorming a possible abstraction layer (codenamed “API 3”) which could allow the Irssi core to be swappable with a custom, native IRC client core (codenamed “Anya”). There’s really not much in the current Irssi-based implementation of the internal interfaces (“API 2”) that make a dependency switch unfeasible.

Finally, I’m not going to stop producing ~~useless~~ updates for my website! Dorset5 0001 is already a reality, although there’s still much I want to do before replacing the current layout. This time I have placed an emphasis on readability and elegance that I don’t think the previous revisions have lived up to so far.

• • •

All in all, I always have so many ideas floating in my mind that I rarely carry to realization, so this can’t be considered a definitive list. There are other possibilities I’m contemplating for the long term regarding my personal life, but that’s a much more volatile subject to discuss so I’m avoiding it for now.

Preparing for yet another year

nospam@example.com (shadowmaster) — Tue, 30 Nov 2010 02:11:00 +0000

It’s almost over. Time flies even faster as we get closer to the end of 2010, and apparently there’s a lot to summarize despite we’re not in the finish line yet!

This has been a particularly difficult year for me in a more personal sense, and I’ve faced some trials I won’t speak about and then some, but I’ve also learned new things in the road — things that may be of greater use to me in the future. There’s really a lot that could be said about this year but I’ll restrict it to computer stuff to ~~avoid boring the audience too much~~ bore the audience as much as possible.

Continue reading "Preparing for yet another year"

General update

nospam@example.com (shadowmaster) — Sun, 10 Oct 2010 01:37:44 +0000

At last, it’s the second weekend of October and here in Chile we’re already switching to DST-based schedules. But I didn’t come here to talk about how nice it is to have sunlight past 7 PM, or to announce any plans for summer.

Instead, I came here to announce the early arrival of codename “Dorset4”, the new revision of the site’s layout, now with more CSS 3! (Really.)

It’s not really as big an update as I expected because there’s really very little to fix overall — I feel quite happy with my work on Dorset3 and save for some minor things no big improvements appear to be necessary. Of course, I introduced a couple of bugs with this revision (notably, in the Projects section index), which will be fixed over the next days. The most notable changes are the introduction of a few icons in places where it made sense, and the tidying up of the sidebar and footer, all with the purpose of presenting the most important information more cleanly than ever.

But these aren’t all the changes. Besides the stylesheet revisions, there’s also a new project page for Rei 2 IRC bot and a new separate page with information on the website. The blog has also now acquired entry paging powers, which means that multi-page articles will be possible in the future — in fact, the forthcoming follow-up to Wesnoth Evolution: 0.1 may take advantage of this new feature.

I know people in the northern hemisphere are going to freeze very soon as winter approaches, but I know I’ll still envy them in a few weeks when local temperatures start going over 32°C.

Another report on Rei2's development

nospam@example.com (shadowmaster) — Thu, 07 Oct 2010 04:04:13 +0000

Shikadibot 0314’s replacement, codenamed “Rei II” is slowly approaching completion and now the various SVN commands offered by shikadibot (dir, file, log, repo, and a couple more) are implemented and working. Thanks to my newly acquired handle of class inheritance with Object-Oriented Perl, it’s now also possible to write Git equivalents with minimal hassle, which may become useful in the future, especially once I implement a way to query arbitrary named repositories.

This time I’m going to try to make an IRC bot that can be more easily adapted for other people’s needs, something that was a goal for Shikadibot’s development as well, but quickly went to hell as the code became the nightmarish abomination that it is nowadays.

Now codename “Rei II” has an official name and source repository: Rei 2 IRC Bot at Gitorious.org.

All progress has taken place in a private repository at AI’s server, but from now on I’m mirroring history at Gitorious for those who might actually want to take a look at the code, and — in the future — use or contribute to Rei 2’s development. The source license is and will continue to be the ISC license to allow easier integration of the code in other suites without the load of problems created by the GNU General Public License.

A project page on this very site will be coming soon.

Building bots, introducing Rei2

nospam@example.com (shadowmaster) — Thu, 30 Sep 2010 06:51:47 +0000

Those who frequent my personal and Wesnoth-related IRC channels on freenode may have met Shikadibot before, but very few people have actually seen the mess that lies beneath its friendly user interface.

Shikadibot is, simply put, a shitload of hacks put together by someone who didn’t know better at that time — that someone being me. “Inspired” by Rhonda’s Wesbot who watches over the main Wesnoth channels, I quickly wrote Shikadibot around the middle of 2008 with nary a clue of proper Perl coding. I learned as much Perl as I needed to implement the bot’s basic functionality, and certainly not in the most orderly fashion possible. This bot is basically unmaintained and feature-frozen nowadays because I can’t be bothered to try to make something out of that mess.

And yet I felt compelled to revisit this old project and create something new and better using the same basic ideas.

Some days ago, I started to work on a new Irssi-based IRC bot under the codename “Rei II” (Rei2.pl) and building its components from scratch while trying to keep my Perl as elegant as possible while still taking advantage of advances techniques that’d have made maintaining Shikadibot much easier if I actually learned them back then. Most of the knowledge I’m applying comes from different Perl programs I wrote after Shikadibot, such as umcreg and a few random scripts for the Wesnoth-UMC-Dev Project.

I’d dare say that this is coming out nicely so far since I could quickly implement a small framework for the bot and seamlessly wire it into Irssi, and now I’m busier working on the fun stuff: the actual bot commands and functionality.

I’m not revealing the source code yet (boo!), but I can say that so far I have implemented the following aspects, cleanly and successfully:

“Hi” command (well, duh).
“Help” command (stub only though).
Acronyms look-up inspired by NetBSD’s (et al) wtf(6) utility.
Manual page abstract look-up using whatis(1), also inspired by the above.
Mathematical expression evaluation using GNU bc(1).
Dice-rolling with “d<no. sides>”, suggested by SpookyMagician on IRC.

Due to my involvement in two projects that been trolled by FSF fanboys, I have decided to publish the sources of this bot framework under a BSD-style license instead of the usual deal, since the terms of the GNU General Public License seem to be designed to provide whiners with the fuel they need to keep software developers busy with legalese instead of matters lying within their actual area of interest (id est, programming). After hearing of the FSF’s extremist views on certain issues, I don’t think I’ll use the GNU GPL again in much stuff, if at all.

Throwaway code

nospam@example.com (shadowmaster) — Tue, 09 Feb 2010 02:03:09 +0000

It has become increasingly common for me to come up with a program for an amazing task one day, to rewrite it the next day.

umcdist, the Wesnoth-UMC-Dev Distribution Tool, has been in development hell for a year mostly for this reason; the other reason is that it seems like it will perform worse than build-external-archive.sh a.k.a. “Scrappy” due to an excessive usage of Perl's system function. I cannot make up my mind and choose between performance and maintainability; Espreon knows that build-external-archive.sh is broken, but I can't be bothered to try to understand that awful piece of Bash unholy abomination again to fix it.

Meanwhile, umcstat (Wesnoth-UMC-Dev Statistics Service) is still a work in progress, but with more emphasis in progress; there's actual code written already, and I've been using freenode's Eir bot framework to test it.

While Eir could possibly be a nice way to get rid of umcreg's Net::IRC dependency and code, it's actually a C++ program that can be compiled only with GCC 4.4 at minimum, due to at least one C++0x feature used throughout the code: auto. The target machine runs Debian lenny, unlike my laptop (Squeeze), and therefore doesn't have GCC 4.4!

Instead of sticking with Eir, I'm refactoring umcreg's IRC code into a custom Perl-only framework, umcbotd, and making creative use of eval writing an abomination code-named “Naia”, which I'm rewriting again because the first version I wrote, which worked, was very badly designed and ugly. I know it's a problem when I have three classes or modules all depending upon each other's internals.

The goal of umcbotd/Naia is producing a Net::IRC-based abstraction layer for our bots that treats them as “services” with multiple “modules” (not in the Perl sense, though) that can be easily inserted and removed from the system by adding/removing their files. umcreg already runs with a prototype implementation of this mechanism, but it needs to be generalized further before it can be usable with Naia.

Writing our bots could be this simple, if Naia gets completed:

#!/bin/true
CO_SERVICE_COMPONENT('umcreg');

sub ctcp_version_reply () { "Wesnoth-UMC-Dev Registry Service, using " . Naia::version_string() }

sub eh_ctcp_version
{
    my ($self, $event) = @_;
    $self->ctcp_reply($event->nick, 'VERSION ' . ctcp_version_reply());
}

# ...

my $bot = Naia::get_bot('umcreg');
my %eh = (
    'cversion'      " style="display: inline; vertical-align: bottom;" class="emoticon" /> \&eh_ctcp_version,
    'msg'           " style="display: inline; vertical-align: bottom;" class="emoticon" /> \&eh_msg_private,
    '330'           " style="display: inline; vertical-align: bottom;" class="emoticon" /> \&eh_whoisloggedin,
    '318'           " style="display: inline; vertical-align: bottom;" class="emoticon" /> \&eh_endofwhois,
    '331'           " style="display: inline; vertical-align: bottom;" class="emoticon" /> \&eh_notopic,
    '332'           " style="display: inline; vertical-align: bottom;" class="emoticon" /> \&eh_topic,
    '403'           " style="display: inline; vertical-align: bottom;" class="emoticon" /> \&eh_nosuchchannel,
);

$bot->connect();
$bot->register_event_handlers(%eh);

And their modules would be like this, more or less the same as umcreg's modules already are:

#!/bin/true
# token, subroutine, privilege level (A: admin, H: half-admin, U: user)
CO_MODULE('RAW', \&co_raw, 'A');

sub co_raw
{
    my ($parent, $nick, $hostmask, $svaname) = (shift, shift, shift, shift);

    if(!@_) {
        $parent->notice($nick, "Not enough arguments for \002RAW\002.");
        return;
    }

    $parent->sl(join(' ', @_));

    broadcast_to_log("RAW [$hostmask]");
}

... And of course it would still involve lots of ugly stuff under the hood (eval magic), but if done right I shouldn't have to touch it whenever I wanted to add or remove a feature from any of our two services.

umcreg

nospam@example.com (shadowmaster) — Sat, 30 Jan 2010 19:54:40 +0000

At last, umcreg, Wesnoth-UMC-Dev's Registry Service, is finished, deployed and announced in the forums, thus completing the first part of introducing the Registry system to the project.

There were some changes from its original incarnation, but everything turned out pretty well for a bot written from scratch in approximately 6 days by a Perl fanatic with no knowledge of object-oriented Perl — I actually learned some object-oriented Perl while at this and I feel like I can do anything with it now.

I was in a hurry to get this done right before freenode deployed ircd-seven today (yays!), for no particular reason; this resulted in a security feature not working with hyperion-ircd until I introduced a quickie hack that I'll be retiring later today.

We are using a private git repository for managing the source code, but since I wrote it with an open-source licensing model in mind as usual, here's umcreg version 0.1.1 for the curious, released under the terms of the GNU GPL version 3.

umcreg 0.1.1 (gzip tarball, 35.7 KiB; MD5 sum: 7877c82082d42c45d45f68647c223459)

I have already registered our current members using some basic, known information about them — even including their join date. Old projects' registration will be a little slow as I need to research the current structure of the repository, of which I lost track a year ago, and retrieve original timestamps. The registry's web interface, provided by the umcreg::Web and Thoria::Web packages, can be found here.

umcreg is already working at the project's admin channel on freenode too. It only obeys the project staff's orders, though, so there's no point in trying to send messages to it.

The next step in implementing the Registry model is writing the Statistics service, codenamed “Listra”, and most likely going to be named umcstat. That will definitively take much longer than umcreg's development. Meanwhile, Espreon is trying to convince me to take care of umcdist (codename “Blackmore”) first.

Robots!

nospam@example.com (shadowmaster) — Fri, 22 Jan 2010 00:38:18 +0000

It's year 2010, and, despite the promises, there are no flying cars, no jetpacks, people still die, there's no interplanetary travel, no apocalyptic event triggered by humans made Antarctica disappear on 2000, and most importantly, we are still awaiting the day we can have our own robot pals.

But here's IRC to solve that! No, really! What do you mean, it's not awesome!?

Okay, obscure references aside, the Wesnoth-UMC-Dev started as a fairly casual repository set up by me and Eric S. Raymond for managing a few add-ons that were, at that time (March 2008) candidates for being mainlined at some point of the future: Invasion from the Unknown, Legend of Wesmere and Delfador's Memoirs. Guess which of them has not made it to Wesnoth's mainline project yet.

The way we worked before was fairly different to what we do nowadays. Some time before AI0867 joined me in this quest, I decided to open the “UMC Sandbox” for projects not intended/expected to be merged into mainline. This had some unexpected success by the end of the year and we got a fair amount of add-ons to manage. As I write this, I'm wondering what the hell we are hosting anymore — I lost track!

At the beginning we also had a wiki page at Wesnoth.org with the list of projects and authors. AI0867 and I tried to keep it up to date at all times, but we inevitably forgot its existence after a while. The fact that Wesnoth.org crashed around July 2008 and stayed dead for nearly one month, and was resurrected from 2 months-old backups didn't help. When I remembered the page, I decided to wipe the rusty listings out; the only thing that survived was the admin list, which is now happily hosted by AI0867 on our project's website.

But it'd benefit everyone if we didn't need to look at SVN directory listings or history to check what add-ons we host, or who are in charge of them. To solve this problem, two new services are born: the Statistics service (umcstat, codename “Listra”) and the Registry system (umcreg, codename “Thoria”). I've been asked about their purposes and goals a few times lately, mainly because umcreg is finally reaching completion, so here it is; a thorough description of our new pals.

Continue reading "Robots!"

Taking a level in chanop

nospam@example.com (shadowmaster) — Tue, 08 Dec 2009 22:18:24 +0000

Freenode's staff and a group of users have been busy testing ircd-seven, the ircd which will eventually replace the aging collection of ugly hacks known as hyperion-ircd. So far it seems I've been the only one who's found and reported bugs in channel management code (account bans + forwarding, in particular), which is a pity; we need more testers!

Occasionally, the server instances get restarted and some people complain.

I think I cannot complain, because I just got my own 10 seconds of fame. It's a pity that I didn't have a better one-liner or a larger audience for the occasion!

<@shadowmaster> eh.

(And if you don't use freenode or IRC at all, you won't understand any of this.

atheme-info.pl - Yet another irssi script

nospam@example.com (shadowmaster) — Wed, 18 Nov 2009 02:23:37 +0000

<+shadowmaster> ... I have acl, cs_info, and ns_info aliases on irssi.
<+shadowmaster> for some reason I tend to type ns_info when I want cs_info (for /msg ChanServ INFO)
<+shadowmaster> maybe I should just write a script for the /info command that decides which one is the correct service for me.

So here's a quick 5-minutes hack to patch my incompetence. This little irssi script introduces the /AINFO command for querying freenode's services (which are from the Atheme package) for information on a nickname or channel. I don't need to check whether I am querying the correct service anymore. Yays.

atheme-info.pl (1.4 KB)

The installation procedure is the same as with any other irssi scripts.

Irssi script: window-cmd

nospam@example.com (shadowmaster) — Sat, 24 Oct 2009 20:47:00 +0000

Some time ago I was searching for a legendary irssi script that would allow me to use commands like /1, /2, etc. to switch to the respective window numbers without having to type the complete /window command. Although I didn't succeed in my search, one of the freenode staffers gave me a little command to populate the command aliases list with some adequate (99) entries.

That's very nice except that the config file becomes a bit larger.

I was bored last night so I decided to reinvent the wheel and write such a script that doesn't require any client configuration changes to bind the same aliases automagically on startup. I am pretty sure it already existed, but since I couldn't find it...

window-cmd.pl (1.4 KB)

The installation procedure is the same as with any other irssi scripts.