Whitebait, Kleftiko, Ekmek Special

Giving An FTDI Serial Port A Persistent Device Node

2012-01-04T17:17:00.000Z

There are plenty of perfectly good reasons why one might have several FTDI USB-to-serial adaptors attached to the same PC at the same time. Anyone who does, though, will notice that they’re a royal pain, because every time you unplug and replug one, it gets a different /dev/ttyUSBn device node.

Fortunately, there’s a way to use udev to set up unchanging aliases for those ever-changing device nodes. This is done by setting up a udev rule for each FTDI adaptor that you care about, that picks it out by serial number and gives it a custom symlink.

Some of the details are here, but just to sew it all together, here’s what you need to do.

First attach your FTDI adaptor, making sure all others are unplugged.
ls /dev/ttyUSB*
— only one should be listed.
udevadm info --name=/dev/ttyUSB0 --attribute-walk
— (using the ttyUSB number listed in the previous step) should dump a huge list of attributes
udevadm info --name=/dev/ttyUSB0 --attribute-walk | grep serial
— will list only a few lines, such as the following —
SUBSYSTEMS=="usb-serial" ATTRS{serial}=="XR00U1BU" ATTRS{serial}=="000000000000" ATTRS{serial}=="0000:00:1d.7"
The first ATTRS line is the one we’re interested in. That’s the USB serial number of the FTDI adaptor. (The subsequent lines are the USB serial number of an intermediate hub, and the PCI address of the USB controller.)
Now you need to go and find the udev rules directory: in a standard udev install, that’s /etc/udev/rules.d. In that directory, make a file called 10-local.rules (you’ll need to be root), and add a line like the following (and it must be all on one line; this blog might show it split) —

KERNEL=="ttyUSB?", ATTRS{serial}=="XR00U1BU", SYMLINK+="ftdiDUINO", MODE="0666"
Obviously the ATTRS clause must match the one your adaptor listed above. The SYMLINK clause can be anything (that doesn’t clash with any other device), so name it after the thing it’s plugged into at the other end.
Unplug and replug your FTDI adaptor. Whatever ttyUSB number it gets, the symlink /dev/ftdiDUINO will be pointing to it.
Oh yes, that udev rule also makes the device node world-writable. So you won’t need to be root to issue
miniterm.py /dev/ftdiDUINO 115200

The Failure Mode Of Agile Development

2011-04-09T22:54:00.002+01:00

Here’s the “traditional” waterfall model of software development:

+------------+------------+------------+
| Design     | Implement  | Test       |
+------------+------------+------------+
^ Start                                ^ Deadline

And here’s its almost inevitable failure mode: design takes too long, or implementation takes too long, but the deadline doesn’t move, so test is cut short and the product ships full of bugs:

+------------>>>+------------>>>+------+.....+
| Design        | Implement     | Test |
+------------>>>+------------>>>+------+.....+
^ Start                                ^ Deadline

Software development organisations have been wearily familiar with this outcome, whether in their own code or other people’s, since at least Brooks (1975). Both the cause and the effect of schedule crunch are widely-known and well-understood: often the effect is so well-known that it can be specifically detected by reviewers and end-users.

Here, by contrast, is the fashionable new agile model:

+------------+------------+------------+
| Red        | Green      | Refactor   |
+------------+------------+------------+
^ Start                                ^ Deadline

In this case, “red” means you write the unit-tests first, but of course they fail (red light), because you haven’t written the code yet. Then you write the code, so they pass (“green” light). But so far what you’ve done is “the simplest thing that could possibly work” – in other words, you’ve been deliberately closely-focussed, overtly short-termist, to get your tests to pass, and a refactoring stage is needed to reintegrate your new functionality with the bigger picture.

Agile, of course, involves many of these cycles between project start and project deadline, not just one. (Indeed, some say that each cycle should be as small as a small tomato. I find that the going rate is two small tomatoes to one cup of tea.) So the agile diagram isn’t drawn to quite the same scale as the waterfall one: still, though, developers acquire the same sense of schedule crunch, they skip on to the next task too soon, and the corresponding failure mode occurs:

+------------>>>+------------>>>+------+.....+
| Red           | Green         | Refac|
+------------>>>+------------>>>+------+.....+
^ Start                                ^ Deadline

The cause is the same, but what’s the effect? The code was complete, and – if not perfect – then at least unit-tested, at the end of the green phase. So the product as actually shipped works, which is more than can be said for its waterfallen equivalent.

All that’s missing, in fact, is some of the refactoring effort. Unfortunately, that’s the only place in agile development where any large-scale design work gets done: the design debt that an agile shop takes out by not doing Big Design Up-Front, is paid off only in these refactoring installments. This means that in fact the effect of schedule crunch on agile projects, is that the system ends up under-designed and directionless at any other than the lowest level.

And unlike the paper-bag obviousness of the waterfall model’s failure mode, the agile model’s failure mode is subtle and pernicious. Product 1.0 ships and works – because agile development “releases” every sprint, and thus is perfectly fitted for triaging features against time. But the system is a ball of mud. Feature development on Product 1.5 and Product 2.0 takes longer than expected – which agile development also helps to hide, given its stubborn reluctance to countenance long-term planning – because developers eventually spend all their time battling, not intrinsic problems of the target domain, but incidental problems caused by previous instances of themselves.

Only the most obsessed agiliste would claim that agile development doesn’t have a failure mode. But because agile development is new, its failure mode is unfamiliar to us; and because that failure mode is less visibly catastrophic than Brooks’s, it’s easier to overlook. It is, however, real; its very subtlety requires us to pay particular care to look out for it, and to get right on top of fixing it once we see it start to happen.

Conversely, the fact that there are problems that agile development can’t solve, isn’t a fatal blow. Inevitably such problems are the most visible ones – because all the problems which agile development does solve easily, come and go without anyone really noticing. And the failure mode of agile development – the system’s complexity spiralling out of control – can be fixed without doing too much damage to the theory.

And how to fix it? Schedule in some serious refactoring, one subsystem at a time. In his paper Checklist for Planning Software System Production, RW Bemer, writing in August 1966 (August 1966!) says:

Is periodic recoding recommended when a routine has lost a clean structural form?

Nearly 45 years later, that’s still effectively the best available advice. Whether you call it refactoring or periodic recoding, it of course takes advantage of all the unit tests that the ball of mud already contains. This time round, it also takes advantage of knowledge about how all the parts of the subsystem operate. That knowledge is unlikely to be acquired in a single two-week sprint, so unless you can put someone on the task who already knows the subsystem inside-out (and most of the time you’re in this state, there won’t even be such a person), you’ll find yourselves breaking some of the rules of agile development by formally or otherwise block-booking someone’s time for a larger period. (Agile development aims at having any team member able to take on any task in any sprint – but for that to be okay, there mustn’t be any software problems complex enough to require more than two weeks’ thought. Some software problems just are that big, and context-switching can break the developers’ train of thought.)

This is the answer to the sometimes-asked question, “If, in agile development, everyone does design [as often as once per small tomato], what’s the rôle of the architect?”. Agile development is, in a way, Christopher Alexander’s observation that most things can be made piecemeal. But simplicity cannot be made piecemeal. The contribution of the software architect is simplicity.

Is “Factory Method” An Anti-Pattern?

2011-01-22T22:45:00.009Z

Let’s take another look at this version of the “two implementations, one interface” code from that one about portability:

// Event.h v8
class Event {
public:
  virtual ~Event() {}
  virtual void Wake() = 0;
  ...
};

std::auto_ptr<Event> CreateEvent();

// Event.cpp
std::auto_ptr<Event> CreateEvent()
{
  ... return whichever derived class of Event is appropriate ...
}

What I didn’t say at the time is that this is sort-of the Factory Method pattern, though a strict following of that pattern would instead have us put the CreateEvent function inside the class as a static member, Event::Create(). And the pattern also includes designs where CreateEvent is a factory method on a different class from Event, but it‘s specifically “self-factory methods” such as Event::Create that I’m concerned with here.

(As an aside: the patterns literature comes in for a lot of criticism for being simplistic. Which it is: the GoF book could be a quarter the size if it engaged in less spoon-feeding and pedagoguery. (And by pedagoguery I mean pedagoguery.) But in a sense the simplicity and obviousness of the patterns they’re describing is the whole point: thanks to the patternistas (patternauts?), a lot of simple and obvious things that often come up in programmers’ natural discourse about code, and that didn’t previously have names, now do. Reading a patterns book might not make your code much better unless you’re a n00b, but that’s not what it’s for. It’s for making your discourse about code better. In any n>1 team, being able to discourse well about code is a vital skill.)

But what I also didn’t say at the time, is that whether CreateEvent is a member or not, so long as it’s in event.h, this code appears to have cyclic dependencies — to be a violation of the principle of “levelisability” set out in the Lakos book.

What’s going on, as you can see on the left, is that, although the source file dependencies themselves don’t exhibit any cycles, viewing, as Lakos does, each header and its corresponding .cpp file as forming a component — the three grey rectangles — produces a component dependency graph with cycles: win32/event ↔ event ↔ posix/event.

One way around that would be to move CreateEvent out into its own component — a freestanding event factory — as seen on the right. With this change, the design is fairly clearly levelisable at both the file level and the component level. This refactoring is an example of what Lakos (5.2) calles escalation: knowledge of the multifarious nature of events has been kicked upstairs to a higher-level class that, being higher-level, is allowed to know everything. (The file event.cpp now gets a question-mark because, as the implementation file for what may now be a completely abstract class, it might not exist at all — or it might exist and contain the unit tests.)

But is it worth it? We’ve arguably complicated the code — requiring users of events to know also about event factories — for what’s essentially the synthetic goal of levelisability: synthetic in the sense that it won’t be appearing in any user-story. Any subsequent programmer working on the code would be very tempted to fold the factory back into event.h under the banner of the Factory Method pattern.

Moreover, in this case the warning is basically a false positive: if Event is truly an abstract class (give or take its factory method), then the apparent coupling between, say, posix/event and event is not in fact present: posix/event can be unit-tested without linking against event.o nor win32/event.o. (Not that, in this particular example, posix/event and win32/event would both exist for the same target — but factory methods obviously also get used for cases where both potential concrete products exist in the same build.) Though conversely, if Event had any non-abstract methods — any with implementations in event.cpp — then it’d be a true positive not a false positive, as all the different event types would be undesirably link-time coupled together.

One reason that the refactoring is worth it, is the same sort of reason that fixing compiler warnings, i.e. altering the code so they don’t trigger, is worth it, even in instances when the warning doesn’t point out a bug: because if you let warnings proliferate, real ones will get lost in the noise, and ideally you aim for the zero-warnings state in order that the introduction of any new warning is an easily-visible alert telling you that there’s a new potential bug to check for. Steve Maguire is talking here about unit tests, but the same applies to compiler warnings: “[W]hen they corner a bug, they grab it by the antennae, drag it to the broadcast studio, and interrupt your regularly-scheduled program”.

Exactly like compiler warnings, cyclic-dependency warnings — which are really design warnings — are sometimes false positives, but likewise it’s worth aiming for the “zero design warnings” state, because it makes new design warnings stand out so. I ran a cycle-checker script (in the spirit of the ones in Lakos) over my own Chorale project, and the result was that it effectively shone a spotlight on all the parts of the code where the design was a bit icky. Every cycle it produced — one was dvb::Service ↔ dvb::Recording, another was all the parts of db::steam depending on each other in a big loop — was a place where I’d at some time thought, “Hmm, this isn’t quite right, but it works for now and I’ll come back and do it properly”. And of course without anything to remind me, I never had gone back and done it properly.

So it turns out that you can’t have both factory methods and levelisability. You have to pick one or the other. And levelisability is better.

Quirking a Mac Pro into AHCI mode

2010-10-06T20:53:00.001+01:00

The Apple Mac Pro (2006 model) has an Intel ESB2 south-bridge, which includes the SATA controller. Under MacOS, this SATA controller is driven in AHCI mode, which is a great improvement on the “traditional IDE” mode — for one thing, it enables “NCQ” SCSI-like command queueing. But the pseudo-BIOS that boots non-MacOS operating systems on Intel Macs, forces the chip into traditional-IDE mode. This is certainly a good idea for wide compatibility, but for an OS which has AHCI drivers — such as Linux — it’d be good to force it back again.

Instructions do exist for forcing it back again using a hacked Grub bootloader, but for one thing that’s a hack, and for another I don’t use Grub, I use Lilo. The “right” way (well, other than convincing Apple to put an AHCI mode into their pseudo-BIOS) is to use Linux’s “PCI quirk” infrastructure. This patch (against 2.6.35.6, but similar should apply almost anywhere) gives me toasty AHCI goodness under Linux:

--- drivers/pci/quirks.c~ 2010-09-27 01:19:16.000000000 +0100
+++ drivers/pci/quirks.c 2010-10-06 20:29:04.000000000 +0100
@@ -1044,6 +1044,15 @@ DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDO
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_HUDSON2_SATA_IDE, quirk_amd_ide_mode);
 DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_HUDSON2_SATA_IDE, quirk_amd_ide_mode);
 
+static void __devinit quirk_intel_esb2_ahci(struct pci_dev *pdev)
+{
+    pci_write_config_byte(pdev, 0x90, 0x40);
+    pdev->class = PCI_CLASS_STORAGE_SATA_AHCI;
+    dev_info(&pdev->dev, "Intel ESB2 AHCI enabled\n");
+}
+
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x2680, quirk_intel_esb2_ahci);
+
 /*
  * Serverworks CSB5 IDE does not fully support native mode
  */

And here it is in my dmesg:

pci 0000:00:1f.2: Intel ESB2 AHCI enabled
ahci 0000:00:1f.2: flags: 64bit ncq pm led slum part 
ata1.00: 488397168 sectors, multi 16: LBA48 NCQ (depth 31/32)

All six SATA ports (four for the drive carriers, two by the optical drives) are seen by the ahci driver (which I have built-in to the kernel); loading the ata_piix driver as a module also enables the PATA port by the optical drives. (It's probably not a good idea to compile both ahci and ata_piix into the kernel, in case they fight over the ESB2 controller.)

Job’s a good ’un.

Portability, Or Cut-And-Paste?

2010-05-31T01:22:00.000+01:00

Suppose you’ve got some functionality with two completely different implementations on different platforms:

// Event.h v1
class Event // Win32 waitable event
{

  void *m_hEvent;
public:
  Event() : m_hEvent(::CreateEvent()) {}

  void Wake();
  ...
};

versus

// Event.h v2
class Event // Posix waitable event
{
  int m_fd[2];
public:
  Event() { ::pipe(m_fd); } // or maybe eventfd on Linux
  void Wake();
  ...
};

And, by the way, if you’ve got a class where only some of the implementation is different on different platforms, you’ve first got an opportunity for refactoring to extract that part of the implementation as a self-contained class. Portability and maintainability are aided if each of your classes is either wholly portable, or wholly platform-specific. But as it is, you’ve got a more immediate problem, which is that there’s two classes both called Event, and any translation unit that includes both headers isn’t going to compile. Here’s one solution:

// Event.h v3
#ifdef WIN32
class Event
{
  ...
};
#else
class Event
{
  ...
};
#endif

This, plus its close relative

// Event.h v4
#ifdef WIN32
#include "win32/Event.h"
#else
#include "posix/Event.h"
#endif

could well be the most commonly-encountered solutions to this problem in real-world code. But they’re not without problems. One problem is with code analysis tools, including such things as production of automated documentation with Doxygen. Really you want such tools to analyse your whole codebase, and if they do they’ll get hopelessly confused with the two classes both called Event. (You can set up preprocessor defines in Doxygen so that it only sees one Event class -- but that only means the other doesn’t get documented, or that you have two entire sets of Doxygen output for the two platforms, neither of which sounds desirable.)

To solve this problem, you’ve going to have to give the classes different names:

// Event.h v5
class Win32Event
{
  ...
};

class PosixEvent
{
  ...
};

No, no, no, this is C++, there’s a built-in language facility especially for namespaces, you don’t have to reinvent it in the class names themselves:

// Event.h v6
namespace win32 {
class Event
{
  ...
};
} // namespace win32

namespace posix {
class Event
{
  ...
};
} // namespace posix

(Why the comments on the “}” end-of-namespace lines? Because the error messages you get from C++ when you accidentally miss out an end-of-namespace brace are often insane and impenetrable, especially if you do it in a header file, and it’s convenient to be able to grep namespace *.h *.cpp and check whether braces appear in matching pairs in the grep output.)

Notice that both declarations are parsed on every compilation on either platform. This might require you to move some method definitions (those calling such unportable APIs as ::CreateEvent and ::pipe) into a cpp file. But even compiling the other platform’s declarations helps ensure that changes on one platform won’t break the other platform without somebody noticing. And a Doxygen run, or other automated code analysis, will at least get to see both sets of declarations even if not both sets of definitions.

While going down the route of helping platform developers avoid breaking other platforms, it’s been conspicuous so far that there’s nothing in Event.h making sure that the Win32 and Posix implementations keep the same API. Without such enforced consistency, there’s a risk of client code unwittingly using non-portable APIs. The way to enforce the consistency, of course, is to derive from an abstract base class containing the API:

// Event.h v7
class EventAPI {
public:
  virtual ~EventAPI() {}
  virtual void Wake() = 0;
  ...
};

namespace win32 {
class Event: public EventAPI
{
  ...
};
} // namespace win32

namespace posix {
class Event: public EventAPI
{
  ...
};
} // namespace posix

This is, after all, exactly the sort of thing you’d do if you had two different sorts of Event that were both used in the same build of the program. Really the fact that no one individual binary will contain instances of both classes, doesn’t mean that the quest for well-factored design should be thrown out of the window and replaced with cut-and-paste.

Now, though, all the method calls have become virtual function calls. For most of any given codebase, that won’t matter, but there’ll always be hot paths and/or embedded systems where it does, and indeed an event class might quite plausibly be on such a critical path. And after all, compiling the client code itself provides a, potentially large, body of checks that the API has remained consistent over time. It’s reasonable to adopt the position that using design techniques to discourage unthinkingly changing core APIs in a non-portable way isn’t actually necessary, so long as the compilation-failure results of such a change are always speedily available from all target platforms, such as from an automated-build or continuous-integration system.

If, conversely, your class is only used in situations where nobody’s really counting individual machine cycles, you could hide the implementations altogether, at the cost of ruling out stack objects and member variables of type Event and requiring a heap (new) allocation every time one is created:

// Event.h v8
class Event {
public:
  virtual ~Event() {}
  virtual void Wake() = 0;
  ...
};

std::auto_ptr<Event> CreateEvent();

// Event.cpp
std::auto_ptr<Event> CreateEvent()
{
  ... return whichever derived class of Event is appropriate ...
}

But for now let’s assume you can’t really conceal the two platform-specific declarations, and go back to Event.h version 6 or version 7. Let’s look at how client code gets to pick which of the two implementations it uses. For a start, not like this:

class NominallyPortableDomainAbstraction
{
#ifdef WIN32
  win32::Event m_event;
#else
  posix::Event m_event;
#endif
  ...
  m_event.Wake();
  ...
};

Flouting our design rule that each class is either wholly portable or wholly platform-specific, this sort of thing couples portability decisions into all client code, which is very undesirable. This is much better:

// Event.h v9
... as before in v6 or v7

#ifdef WIN32
typedef win32::Event Event;
#else
typedef posix::Event Event;
#endif

// Client code
class NominallyPortableDomainAbstraction
{
  Event m_event;
  ...
  m_event.Wake();
  ...
};

It’s still not great, though: if there are lots of classes involved, lots of code ends up inside the #ifdefs. How can we minimise the amount of #ifdef’d code? Well, here’s one way:

// Event.h v10
... as before in v6 or v7

#ifdef WIN32
using namespace win32:
#else
using namespace posix;
#endif

But this, of course, introduces everything from those namespaces into the surrounding namespace. Not only is that namespace pollution, it’s potentially misleading for client code, as there may be Win32-specific classes or functions -- ones with no Posix equivalent, for use in Win32 situations only -- which are now accessible in portable code without the telltale win32:: prefix. (And, of course, vice versa for Posix-specific ones.) Really we want to explicitly enumerate which classes and APIs are intended to be portable. So we can do this:

// Event.h v11
... as before in v6 or v7

#ifdef WIN32
namespace platform = win32;
#else
namespace platform = posix;
#endif

using platform::Event;
using platform::PollThread;
...

The intention is that the using declarations list precisely those facilities which are available on all platforms, but with differing implementations on each. Non-portable classes or functions can stay in namespace win32 (or namespace posix) so that any use of them in client code immediately flags that code up as itself non-portable.

There is one problem with this neat solution, though: it doesn’t actually compile. Or rather, once you do the same thing in several different header files, you’ll find that multiple namespace X = Y; statements for the same namespace X aren’t allowed, even if Y is the same each time. So, sadly, you have to come up with a different name for “platform” each time (or centralise the using in one header file, causing potentially-undesirable widespread dependency on that file):

// Event.h v12
... as before in v6 or v7

#ifdef WIN32
namespace eventimpl = win32;
#else
namespace eventimpl = posix;
#endif

using eventimpl::Event;
using eventimpl::PollThread;
...

There’s only one remaining wrinkle, which is that you can’t forward-declare a class-name if that name only exists due to using. So if there are header files that traffic in Event* or Event& and could otherwise be satisfied with a forward declaration class Event; and avoid including Event.h, then you can’t use Event.h version 12. (Version 9 is also out, as you can’t forward-declare typedefs either.) The best you can do is probably this:

// Event.h v13
... as before in v6 or v7

#ifdef WIN32
namespace eventimpl = win32;
#else
namespace eventimpl = posix;
#endif

class Event: public eventimpl::Event {};
class PollThread: public eventimpl::PollThread {};
...

although naturally that only works as-written if the eventimpl base classes don’t have constructors other than the default constructor; if they did, you’d have to write forwarding constructors in each derived class, making the code a lot less neat.

Think Same: Mapping An Apple Keyboard Like It’s A Normal One

2010-05-26T20:25:00.003+01:00

The aluminium “laptop-style” Apple keyboards are really nice: quick and easy to type on. I’ve got one on my main Linux machine. But they’re mapped oddly (or at least the UK one is): backtick is next to Z, and backslash next to Enter. As I’m usually watching the screen and not the keyboard while typing, the fact that these think-different mappings are borne out by the printing on the keycaps doesn’t help.

So I needed to swap them back again. And, X being the stovepipe it is, I needed to do it twice, once for inside X (including Gnome and KDE) and once for the console.

X was actually easier: I added this to ~/.xinitrc:

# Apple silver keyboard has these keycodes swapped xmodmap -e "keycode 94 = grave notsign grave notsign bar bar" xmodmap -e "keycode 49 = backslash bar backslash bar bar brokenbar"

For console use there doesn’t seem to be a similar way of modifying just certain keycodes, so you end up needing to make a whole new keymap:

dumpkeys > keymap.txt
(edit keymap.txt)
loadkeys < keymap.txt

Here’s the diff I had to apply to the standard UK map to get the Apple keyboard working the way my fingers expect:

--- std_keymap  2010-03-12 13:49:34.000000000 +0000
+++ silverapple.keymap  2010-03-12 13:50:39.000000000 +0000
@@ -93,9 +93,9 @@
        shift   control keycode  40 = nul             
        alt     keycode  40 = Meta_apostrophe 
        shift   alt     keycode  40 = Meta_at         
-keycode  41 = grave            notsign          bar              nul             
-       alt     keycode  41 = Meta_grave      
-       control alt     keycode  41 = Meta_nul        
+keycode  86 = grave            notsign          bar              nul             
+       alt     keycode  86 = Meta_grave      
+       control alt     keycode  86 = Meta_nul        
 keycode  42 = Shift           
 keycode  43 = numbersign       asciitilde      
        control keycode  43 = Control_backslash
@@ -201,10 +201,10 @@
        control alt     keycode  83 = Boot            
 keycode  84 = Last_Console    
 keycode  85 =
-keycode  86 = backslash        bar              bar              Control_backslash
-       alt     keycode  86 = Meta_backslash  
-       shift   alt     keycode  86 = Meta_bar        
-       control alt     keycode  86 = Meta_Control_backslash
+keycode  41 = backslash        bar              bar              Control_backslash
+       alt     keycode  41 = Meta_backslash  
+       shift   alt     keycode  41 = Meta_bar        
+       control alt     keycode  41 = Meta_Control_backslash
 keycode  87 = F11              F23              Console_23       F35             
        alt     keycode  87 = Console_11      
        control alt     keycode  87 = Console_11

And to get this loaded on every boot, I stuck it in /etc/rc.local:

# Apple keyboards start up in "F-keys are magic" mode; this puts them in
# "F-keys are F-keys" mode
echo 2 > /sys/module/hid_apple/parameters/fnmode

# Also they have the backslash and backtick keys swapped
loadkeys < /etc/silverapple.keymap

Oh, right, yes, the function keys: they think different too, or at least they default to think-different mode on power-up. To get them thinking same, you need to have the Linux hid_apple module loaded, and to set its “fnmode” parameter as above. (Other systems may use other ways of setting module parameters.)

If you’ve found this and want to check that your fingers are mapped the same way mine are, this is the list of keys that, with the setup above, do not generate the symbol printed on the keycap:

labelled	generates
§	` (backtick)
± (shift-§)	¬ (notsign)
@ (shift-2)	"
" (shift-')	@
\	#
\| (shift-\)	~
` (backtick)	\
~ (shift-`)	\|

Overall you lose the ability to type “§” and “±”, and gain “¬” (meh) and, rather surprisingly, the hash character “#”, which isn’t marked anywhere on the Apple keyboard.

How Facebook Wasted An Hour And A Half Of My Life

2010-01-31T21:24:00.000Z

Now it should be admitted up-front that in fact, taken as a whole, Facebook has in fact wasted far more than an hour and a half of my life. But all the other hours and halves, I at least felt it was my choice to waste them. This time, however, it was definitely Facebook’s choice instead — or Apple’s.

The problem started when Facebook for Iphone refused to do anything, showing only the message “This version of the Facebook application is no longer supported. Please upgrade to version 3.0.”

Despite having always thought that the whole point of web applications, or websites in general, is that you design them carefully-enough to start with that you can support old URLs or clients in the field indefinitely — you can’t, after all, reach out and rewrite everyone in the world’s bookmark files if you want to change a URL — I consoled myself that at least updates to Facebook for Iphone are free, so went off to the Iphone App Store to download the newer version. But that failed too: apparently version 3.0 of Facebook for Iphone required at least Iphone OS 3.0, and mine was still on 2.2.

Upgrading the OS on a jailbroken Iphone can be stressful, but I thought that, as Facebook was pretty much the only Iphone application I actually used, even the worse-case scenario of bricking the phone and going back to the Nokia 8890 wouldn’t lose me that much. It turns out that the procedure for upgrading a jailbroken Iphone to OS 3.1.2 is as follows:

Find out how. I’m not even including the finding in the hour and a half.
Reboot, because it only works from MacOS.
Download Pwnagetool, BL-3.9, and the IPSW. (I always think that the file icon for IPSW files ought to be a blue scarf, but it isn’t.)
Try and download BL-4.6, but the link doesn’t work.
Go and fetch BL-4.6 on a USB stick from the Windows laptop you last did an Iphone upgrade from. (This is actually an improvement over the previous process, which only lets you know you’ll need the BL files halfway through the upgrade itself.)
Run Pwnagetool, which immediately stops because you need at least Itunes 8 for the “restore from specific IPSW” feature, and you only have Itunes 7.
In Itunes, run “Check for updates”, which runs the system Software Updater program, which offers loads of nonsense. Untick everything except Itunes, and download and install it.
Run Pwnagetool again, which now does everything it needs to and creates the custom IPSW.
Put the Iphone into recovery mode.
Run Itunes, which refuses to start because it needs Quicktime 7.5.5 or later.
Go and find Software Updater (it’s in System Preferences) and run it again. Untick everything except Quicktime, and download and install it.
Reboot, because the Quicktime installer demands it.
Put the Iphone into recovery mode again because it’s fallen out of it during the reboot.
Re-run Itunes, and have it tell you “Itunes could not contact the Iphone software update server because you are not connected to the internet.”
Click on “More information” about this “no internet” error, and watch it open a web page in Firefox to tell you about it.
Start to wonder whether Apple are just deliberately messing with your head in order to scold you for jailbreaking their phone.
Unplug the network cable from the Mac and plug it directly into the router, bypassing the HTTP proxy.
Reconfigure both MacOS and Firefox not to use the proxy.
Quit and re-run Itunes, because there’s no obvious way of telling it to rescan for devices.
Actually install the upgrade, which takes ages to “prepare”, much of which time you spend eyeing the old Nokia 8890 which never caused you any of these problems.
Once the Iphone restarts, observe with relief the little magnifying-glass icon that 2.2 didn’t have, thus providing evidence that the upgrade actually did something. Also notice that it’s rearranged all your icons; arrange them back the way you like.
Install the Facebook application from the Iphone App Store.
Open “Contacts” and realise that in fact the upgrade has wiped all your user data.
Notice with relief that Itunes is now asking you whether to restore the Iphone from a backup.
Notice with alarm that the backup in question is from 2008.
Restore from the backup anyway because it’s got to be better than nothing.
Rearrange all your icons back the way you like them again.
Reinstall the Facebook application from the Iphone App Store, because the restore restored the old version.
Unplug the network cable again and plug the Mac in back behind the firewall where it belongs.
Reboot back into Linux.
Start wondering what important phone numbers you’ve obtained since 2008.

This Code Is Completely Bogus

2009-11-16T15:42:00.001Z

Here’s some rubbish code I wrote:

class Connection: public util::Stream
{
    ...

public:
    class Observer
    {
    public:
 virtual ~Observer() {}

 virtual unsigned OnHttpHeader(const std::string& key,
          const std::string& value) = 0;
 virtual unsigned OnHttpData() = 0;
 virtual void OnHttpDone(unsigned int error_code) = 0;
    };

private:
    explicit Connection(Observer*);
    friend class Client;

public:
    ~Connection();

    unsigned int OnActivity();

    // Being a Stream
    unsigned Read(void *buffer, size_t len, size_t *pread);
    unsigned Write(const void *, size_t, size_t*) { return EINVAL; }
};

typedef boost::intrusive_ptr<Connection> ConnectionPtr;

class Client
{
public:
    Client();

    /** Passing a NULL verb means POST (if !body.empty()) or GET (otherwise).
     */
    ConnectionPtr Connect(util::PollerInterface *poller,
     Connection::Observer *obs,
     const std::string& url,
     const std::string& extra_headers = std::string(),
     const std::string& body = std::string(),
     const char *verb = NULL);
};

It’s meant to be an HTTP client implementation, where the Connect() call returns a smart pointer to a connection object that manages its own lifetime. The Connect() call sets it all off, and attaches it to the PollerInterface object, which calls back into the Connection object (the OnActivity method) whenever the socket becomes writable (or readable, depending on where we are in the http transaction).

I mean, just look at it: it’s obviously completely bogus. [Short pause whilst you just look at it.]

Actually, no. It wasn’t at all obvious to me just from looking at it, that it was completely bogus. The only point, in fact, at which it became obvious that it was completely bogus, was when it started causing bugs.

And the bugs it caused were quite awkward ones: crashes (and Valgrind violations), obviously timing-related, to do somehow with Connection pointers still being used after the object had gone away — which should really have been disallowed by the smart-pointer class.

It turned out that the problem was if the transaction completed too quickly. At the time the code was written, the PollerInterface object didn’t own (smart pointers to) the pollable items hung off it. So, in order to stop the Connection object disappearing, due to the final reference going away, during a call to OnActivity, OnActivity itself creates and holds an additional reference to the Connection object during its execution. But if the transaction got started quickly enough, the first call to OnActivity would happen before the constructor returned — in other words, before the pointer had been assigned to the smart-pointer that’s the result of Connect(). So the “additional” reference held inside OnActivity would be the only reference — and when it went away at the end of the function, there’d be no outstanding references and the object would be deleted.

The effect was as if the constructor had included “delete this” — resulting in the new call in “p = new Connection” returning a pointer that was already dead and dangling before even being assigned to p.

Completely bogus. And worse, a completely bogus design; given the constraints, there was nothing that could possibly be done in the methods of the classes Client and Connection that would correctly implement that interface. The interface itself needed to change. Fortunately, I decided it didn’t need to change much:

class Connection
{
    ...
public:

    /** Start the HTTP transaction.
     *
     * Immediate errors (failure to parse host, failure of connect()
     * call) are returned here; errors happening any later come back
     * through Observer::OnHttpDone. In fact, OnHttpDone may be called
     * before Init() returns, i.e. you need to be ready for OnHttpDone
     * calls before you call Init(). If Init() returns a failure,
     * OnHttpDone has not been called, and is guaranteed not to be
     * called afterwards. Otherwise, it's guaranteed it WILL be
     * called.
     */
    unsigned int Init();
    ...
};

So you get a completely inert ConnectionPtr back from Client::Connect, and only then — once you’ve safely squirreled-away the smart-pointer — do you light the blue touch-paper by calling Connection::Init().

But although this version at least makes it possible to use the functionality correctly, it doesn’t really make it easy — and that should be the real goal of any act of API design. There’s that uneasy division of labour between Connect() and Init() — methods on two different classes — and there’s a whole paragraph of complex object-lifetime issues to read and understand (or, as they’re otherwise known, bugs waiting to happen) for users of Init().

This, and particularly the lifetime of the Connection::Observer object, which usually ends up having to call “delete this” in its OnHttpDone() method, left me with one of those code itches that tells me that I (and, the largely theoretical, other users of this API) am writing more complex and icky client code than I should be.

Neatening this up required making the object-lifetime issues more sane, which in turn involved greater use of smart pointers. (Not quite Wheeler’s Law, because there was already a use of indirection, and the change involved only a strengthening from reference to ownership.) In the next release of Chorale, the PollerInterface has been replaced by a Scheduler, which keeps smart pointers to its pollable items, allowing the HTTP client API to be simplified to this:

class Connection: public util::Stream
{
public:
    virtual ~Connection() {}

    /** Called with data from the returned HTTP body (if transaction
     * succeeds).
     */
    virtual unsigned Write(const void *buffer, size_t len, size_t *pwrote);

    /** Called with each incoming HTTP header (if connection succeeds).
     */
    virtual void OnHeader(const std::string& /*key*/, 
     const std::string& /*value*/) {}

    /** Called with the overall result of the connection/transaction attempt.
     */
    virtual void OnDone(unsigned int error_code) = 0;
};

typedef util::CountedPointer<Connection> ConnectionPtr;

class Client
{
    class Task;

public:
    Client();

    /** Passing a NULL verb means POST (if body != NULL) or GET (otherwise).
     */
    unsigned int Connect(util::Scheduler *poller,
    ConnectionPtr target,
    const std::string& url,
    const std::string& extra_headers = std::string(),
    const std::string& body = std::string(),
    const char *verb = NULL);
};

So the Scheduler owns the (unseen to the library user) connection Task objects, and the connection Task objects own the Connection stream target objects. Connect() can return immediate errors from the socket ::connect call or from URL parsing, while deferring any other errors to come back through a later OnDone callback — all without there being any ambiguity of the lifetime of the streams or their observers.

(There would be a problem if the Connection object also had a smart pointer to the Task object, as then neither would ever be deleted and both would become memory leaks. But, because the data is pushed from the Task to the Connection, the Connection never needs to see the Task object — and indeed can’t, because Tasks live and die entirely inside library code, and users of the library can’t even obtain pointers to them.)

Simplicity does not precede complexity, but follows it

—attributed to Alan Perlis

So here’s the thing. The third design is clearly better than the second. Well, it’s clearly better than the first, too, but mainly because the first doesn’t work, which is a boring and trivial way for one design to be better than another. The interesting thing is that it’s better than the second.

And better it certainly is — making this change halved the size of the code that uses the library, as well as making it more intentional and less fragile and stylised.

So why, having got to the second design, was I not satisfied? Why did I carry on thinking about it, waiting for the inspiration of the third design to strike? And why, having come up with the third design, was there a feeling of happiness that wasn’t present when writing the the second one, even when it passed all the unit tests the first one failed?

The only answer I can come up with is to theorise the existence of an almost aesthetic sense of code quality — which is worrying in a couple of ways. Firstly, because what is instinctive is rarely communicable, and what is not communicable is soon lost: a software engineer’s fate is that of the wheelwright in the old story of Duke Huan.

But worse than that: if code quality is in fact an aesthetic, and thus extrarational, experience, then it raises the prospect that others, even other good software engineers, could have a different sense of aesthetics, ultimately resulting in a point where you and they are pulling the same code in opposite directions. (I heard recently of a software organisation, believers in the currently-fashionable “agile”, “refactor-mercilessly” style of development, in which two otherwise talented engineers spent all their time rewriting each others’ code rather than pushing things forward — as their aesthetic senses, and frankly their assumptions about who was “in charge” in the deliberately un-micro-managed environment, butted heads.)

No aesthete could get away with “correcting” the second design above into the first: the failing unit tests would prevent that. But are there those who would correct the third design into the second, in the opposite direction to me? If so, why? And, even more importantly, if not, why not?

Productivity Gains With KDE4

2009-10-14T20:24:00.002+01:00

Honesty’s a good thing, usually. In particular, it’s usually a good thing in software engineering, in which most of what we do is digital and repeatable, is either one thing or the other; this fosters a culture of honesty in the same way that it does (or should do) in science, as Feynman points out in Cargo Cult Science. Unlike in a courtroom, or even in a courtroom drama, software engineering rarely comes down to one person’s word against another. (Well, unless your co-workers are sociopaths.)

But there are situations where more honesty isn’t a Good Thing. One example, in fact, is a courtroom drama: if you’re writing such a thing, the principle of pure honesty would have you title it something like “Not Guilty of Murder” — whereas, in fact, letting people know the verdict before they’ve seen the piece robs it of its whole point.

Now KDE3 came with a desktop toy called KPat, a patience game which (among others) includes an implementation of the same “Klondike” patience found in Windows Solitaire. The KDE3 (3.5.10) version of KPat had a feature where, if it detected you’d got into a situation where it was impossible to complete the hand, it would stop the game and tell you so. Naturally, such a feature has to be extremely conservative: it must stop the game only when it can be proved that forward progress is impossible. And in fact the algorithm in KDE3 KPat was very conservative indeed: it kicked-in so rarely that seeing the message always came as a surprise, even on games you already knew you’d lost.

Occasionally I’d idly wonder whether the lost-game detection could be improved — but then I realised that actually, if you improved it enough, you’d eventually get to a situation where the game detects, and tells you, the moment you’ve made a move that leads only to dead-ends. “OK, you’re an idiot, bye, next.” There’d be no point playing the game at all.

And yet, in the KDE4 (4.3.2) version of KPat, that’s exactly what’s been implemented. In the status bar, the whole time, is one of two messages — either “Solver: This game is winnable” or “Solver: This game is not winnable in its current state”. Any time you make a move that makes the first message change to the second, you soon reconsider!

And so, alongside the huge amount of work on the graphics (the original neat bitmaps have become huge and flouncy SVG images; the codebase diff is huge even ignoring the Solver; the whole thing has unexpectedly acquired an Ancient Egyptian feel) the developers have completely ruined the actual game. All the time you’re playing, it’s as if a stern examiner is watching over your shoulder, always ready to lean forwards and intone “Now I don’t believe you wanted to do that”. Worse, unlike the KDE3 algorithm, which just summarised information you could already see, the examiner can see the cards that you can’t, making it an eerily omniscient guide, not to mention a shocking cheat.

Solving patience is a great technical achievement. And it’s certainly scrupulously honest to tell the player exactly what the prospects of success are. But it’s a technical achievement that shouldn’t have been achieved (or shouldn’t be present in the game itself, even though it can be turned off), and a case where honesty is definitely not the best policy. And one of the best little time-wasters in KDE has, effectively, been eliminated.

Recursive Make Is A Time-Waster

2009-09-12T17:48:00.010+01:00

Now you’ve arranged to divide your project into libraries, how do you go about compiling them?

Clearly you want to be able to do the whole thing in one go: to do otherwise would be to fail the Joel test. And you want all the dependencies to be checked every time: there’s no bug harder to find than one that no longer actually exists in the source you’re looking at. (And stale objects, not updated following a header change, can lead to ODR violations: in other words, to random spookiness and borkage that the compiler and linker aren’t required to warn you about, and in many cases can’t even theoretically do so.)

GNU Automake is one answer to the problem, and makefiles in the same spirit as the ones it generates are popular even in projects that don’t use Automake. Such makefiles express inter-module dependencies, such as that of an application on a library, by recursively invoking make in the library directory before invoking it again in the application directory.

For a variety of reasons, documented once and for all in the famous paper Recursive make Considered Harmful, this is a bad idea. The paper suggests that for some systems, it’s even a bad idea for correctness reasons — that incorrect builds with inadequately-followed dependencies can result. But the example given is that of a system whose module dependencies can’t be serialised; this naturally means Automake’s sequential run over the sub-Makefiles can’t do the Right Thing. However, if the module dependencies can’t be serialised, that means there’s a cycle and they don’t form a true hierarchy; that’s a bad situation for more reasons than just Automake’s — bad for reusability, bad for unit-testability, and bad because it reveals that the system designers haven’t really thought through what the rôles of the modules are.

So if that doesn’t apply to you, if your modules are properly factored into a neat hierarchy, does that mean there’s less incentive to ditch a recursive-make scheme and take the time to write a non-recursive one? Less, perhaps, but decidedly not none — because there are substantial performance benefits from whole-project makefiles on modern systems.

This effect is, to be fair, mentioned in Recursive make Considered Harmful (7.1), but the author didn’t draw it out on pretty graphs, nor quantify the improvements, so there is scope for it to be better-documented.

Suppose your project has ten components (nine libraries and the application, say), each with between four and twenty source files. These source files won’t, naturally, all take exactly the same length of time to compile — in fact, typically there’ll be quite a wide range of compilation times, especially if only certain files include certain rather profligate library headers. And further suppose that the machine you’re compiling on has eight CPU cores or threads (such machines are desktop-class these days), so you use make -j8.

If your makefiles are like Automake’s, what happens is that make will make the first library using up to eight CPUs, then, once that’s done, make the second library using up to eight CPUs, and so on until it’s done all of them in the serialised order you used in the top-level Makefile. Which is correct, but it’s not efficient, because make needlessly serialises the entire of each library build against the entire of the next one, when typically a library’s components will each depend on only some of the previous one’s targets.

In graphical form, your CPU utilisation would look like the following diagram, where time runs from left to right, each horizontal row represents a different CPU, and blocks of the same colour belong to the same library; the top half of the diagram shows CPU utilisation during a recursive-make-driven recompilation, and the bottom half the same recompilation done with a whole-project Makefile.

click for full-size

Using one Makefile has allowed make -j8 to keep all eight CPUs busy the whole time — as, whenever a CPU comes free, any pending work from anywhere else in the project can be scheduled on it. By contrast, the top half of the diagram has lots of whitespace, where all the CPUs must wait each time until the last one in that library has finished.

In this example, the build takes scarcely 70% of the time a recursive build would do. If you have more cores, perhaps in a multi-machine build farm, the benefit goes up: building on four cores sees a reeduction only to 88%, but 12 cores sees the whole-project build take just 59% as long as the recursive version, and on 16 cores the time is better than halved. If, in the limit, you work for Dr. Evil, your entire company is basically one big compute surface and you can use “make -j one-meelion”, then the improvement factor is essentially the same as the number of libraries — in this case, 10x, though in practice if you went down that route you’d find communication overheads starting to bite into your scalability.

Four Quarters Of The Codebase

2009-09-12T16:56:00.006+01:00

As a software project grows, eventually it becomes unwieldy to keep the entire source in one single gigantic folder: you’ll need to split it up. Canonically, and indeed quite correctly, the first idea most architects then have is to split the source into a main program plus one or more libraries that provide ancillary services.

Once that’s done, everyone on the project who sits down to write some new code, will have to think at least for a moment about where to put the source file. The answer, of course, is that the new code should go in the main program if it’s fairly specific to the current project (or if it depends on code that’s already in there), and into the library otherwise. The “force of gravity” — developers’ sense of where code should go unless there’s good reason otherwise — should be in the direction of pulling functionality over into the generic code.

Sometimes this decision seems harder than it should be: if that’s the case, what’s often going on is that the functionality in question should really be split up — with a generic part in the library, and the specific use this project makes of it in the main program. In this way, source code organisation can act as a forcing function for good design.

If that’s so useful, can we get extra design-forcing out of how we organise the source? One issue that often throws a spanner in the works of software designs whose creators didn’t originally bear it in mind, is portability between software platforms. It would be decidedly helpful if the source code organisation, and/or the system architecture it (one hopes) reflects, could help prevent platform specialists from unthinkingly tying code to a particular platform when in fact it’ll be needed more widely in future.

So instead of just application versus library, it’s helpful to organise the codebase so that a developer sitting down to write new code, will have to choose between four different places to put it:

Application-specific, platform-specific	Library code, platform-specific
Application-specific, platform-agnostic	Library code, platform-agnostic

This means that, if you’re writing an application that you’re targetting at, say, both Iphone and Windows, or both desktop and embedded, then the actual application logic needs to go in the south-west corner, where implementations on any platform can use it; user-interface code mostly goes in the north-east, where other applications wanting similar UI components can use it; and only the minimum of code necessary to wire the two together goes in the north-west.

The force of gravity in this diagram acts towards the south and the east: if you can turn code from a Win32 application into generic Win32 components, or into a portable implementation of the logic, that’s a design improvement. If you can split the code up, even split a single class up, along those lines, that’s an improvement too. And by forcing developers to make the four-way choice before they write a source file, you’re helping everyone in the team to think about these issues and to factor the code in the best way as it’s first written. Fostering a feeling of “design guilt”, in yourself or others, when placing functionality further north or west than it belongs, gives everyone an easily-grasped metric for well-factored designs.

Really this just another way of thinking about the model-view-controller pattern; the mapping is straightforward, except that MVC doesn’t have a word for the fourth quarter, libraries which are assumed to be at a lower level than the components MVC deals with:

Controllers	Views
Models	(Libraries)

but, even so, embodying MVC in the source layout itself means that its issues get thought about earlier in the software process.

Now naturally, any real codebase above a certain size won’t consist of just “the” application and “the” library. There’ll be a whole collection of libraries, and perhaps of applications too, arranged in a dependency graph — hopefully, for reasons Lakos expounds, a graph which forms a non-cyclic hierarchy. But it should be possible to partition this graph into (up to) four parts, matching the four quarters of the codebase, and with all dependency arrows pointing southwards, eastwards, or south-eastwards. No view or model should depend on a controller; no platform-agnostic library should depend on any view, model or controller. Again, these are truisms of MVC design, but thinking about them in their four quarters gives an easy visual way of thinking about and discussing the issues.

Incidentally, my own Chorale codebase currently (as of v0.15) fails this test, or at least gets a technical pass only: it doesn’t have any platform-specific library code or directories, as all that code is rather bogusly stuffed in the application-specific, platform-specific directories. (Some classes are, in fact, platform-specific library classes, but the split isn’t enforced in the source organisation, either by libraries or by directories.)

Little Libraries

2009-07-16T20:19:00.000+01:00

How small can you make a library?

Of course, one way to make it smaller is to make each individual file smaller: by using an optimising compiler, by turning off exceptions, by removing debug information (in release builds). But even once you’ve done that, there’s extra overhead involved in just being a library: how can that be minimised?

One choice that affects the answer, is whether you want a shared or a static library. Sometimes other considerations force the answer, but if you have the luxury of being able to choose either type, which do you choose for best compactness?

Both types have their advantages; static libraries are useful when the client, or most clients, use only a fraction of the library facilities: unused objects from the archive simply aren’t linked. By contrast, client code that uses any one facility from a shared library must link all of it. And the position-independent code (PIC) techniques needed to build a shared library, may be more expensive than normal code on some architectures. On the other hand, shared libraries offer control over symbol visibility, so an internally-complex library with a simple interface, can end up simple rather than complex at link-time. And, because a shared library is all-or-nothing anyway, it can be built as a single translation unit with no loss of generality — enabling better compiler optimisation.

So I tried it. I took libupnpd from Chorale, which is a small-to-medium sized library although with very few entry points, and tried to make it as small as possible while retaining its identity as a separate library. Here are the sizes it came out as, in the various attempts, as reported by size(1):

	text	data	bss	total	relative		text	data	bss	total	relative
	amd64-linux						arm-linux
Static library	161443	8	94	161545	0.0%	-23.9%	140176	4	872	141052	0.0%	-24.1%
Shared library	205846	6216	112	212174	+31.3%	0.0%	177469	7968	412	185849	+31.8%	0.0%
Shared library, sec	205844	6216	80	212140	+31.3%	0.0%	177421	7960	248	185629	+31.6%	- 0.1%
Shared library, vis	144726	5328	112	150166	- 7.0%	-29.2%	120012	7472	412	127896	- 9.3%	-31.2%
Shared library, sec vis	144078	5328	80	149486	- 7.5%	-29.5%	119452	7472	248	127172	- 9.8%	-31.6%
Shared library, whole sec vis	133582	5312	80	138974	-14.0%	-34.5%	107623	7468	412	115503	-18.1%	-37.9%
Single object, whole	101011	3112	94	104217	-35.5%	-50.9%	85112	6688	380	92180	-34.6%	-50.4%
Single object, whole sec	100638	3112	94	103844	-35.7%	-51.1%	85112	6688	380	92180	-34.6%	-50.4%
Single object, whole sec vis	99775	3112	94	102981	-36.2%	-51.5%	85120	6688	380	92188	-34.6%	-50.4%
Single object, whole sec vis wp	99287	3112	94	102493	-36.6%	-51.7%	85028	6688	380	92096	-34.7%	-50.5%

Some explanation is, of course, in order.

1. Static library

Each file compiled separately, archived with ar.

2. Shared library

Standard ELF shared library: each file compiled separately with -fPIC, linked with gcc -shared. (Or, in fact, using libtool for simplicity. But that’s what libtool was doing behind the scenes.)

3. Shared library, sec

As (2), but compiled also with -ffunction-sections -fdata-sections, and linked with --gc-sections. This made approximately one gnat’s crotchet of difference.

4. Shared library, vis

As (2), but compiled also with -fvisibility=hidden -fvisibility-inlines-hidden, and with the small number of entry points labelled __attribute__((visibility("default"))). This should indeed have been much better than (2) or (3), but it came as a surprise to me that it’s also better than (1).

5. Shared library, sec vis

As (2) but with both the (3) and (4) optimisations applied. Again, --gc-sections makes a non-zero but insignificant difference.

6. Shared library, whole sec vis

OK, now we’re getting somewhere. This is as (5), but with the whole library built as a single translation unit. It’s the moral equivalent of GCC’s -combine, but as that only applies to C and this is C++, it’s done by writing a small .cpp file that does nothing but #include all the other .cpp files. (Doing this requires a certain discipline in the files in question, so as not to queer the pitch for subsequent code. But it’s certainly not an intolerable imposition.)

In the interests of scrupulous accuracy, I should point out that in fact not the whole of the library is compiled in the one file. One of the component parts needs special compiler options, so that one’s still compiled separately.

These settings correspond to what you get if you configure and build KDE3 with --enable-final.

7. Single object, whole

This is like the non-shared-library version of (6); the whole library is compiled in a single translation unit, but this time into an ordinary, non-PIC, object file, which is then put in a library by itself.

8. Single object, whole sec

As (7), but in little sections as per (3). This doesn’t make much difference on amd64-linux, and none at all on arm-linux.

9. Single object, whole sec vis

As (8) but with the visibility settings too. As those settings are only meant to apply to shared libraries, this shouldn’t have made a difference. But for some reason, it did, albeit a tiny one.

10. Single object, whole sec vis wp

As (9) but with the single library translation unit compiled with -fwhole-program, and the exported functions labelled __attribute__((externally_visible)).

Semantically and philosophically, this is saying much the same thing as the visibility settings. But there must be some extra little bit of optimisation the compiler can do in this situation. There isn’t a corresponding statistic for a shared library with -fwhole-program, as using that option disables PIC.

The effects of these various optimisations are remarkably similar, in relative terms, across both architectures.

Why size(1) Isn’t The Full Story

Especially considering the library is C++, some of the numbers above need to be taken with a slight pinch of salt. Most C++ programs have a lot of functions declared in header files; functions which are semantically inline but which, for various reasons (such as being listed in a vtable) aren’t actually purely inlined in practice. Such functions are emitted in every object file, in “link-once” sections, which are deduplicated by the final linker and emitted only once each.

The size(1) command counts these link-once sections under “text”, and doesn’t do any deduplication. So a static library made of many object files will often have a larger size (as quoted by size(1)) than it will actually add to the final binary — because of the multiple copies of all the link-once sections. So row (1) of the table is somewhat inflated — and in fact the “single object” rows are very slightly inflated, too, as although they only contain one each of the “link-once” sections, those will still get deduplicated again against any that are also used elsewhere in the final binary. In shared libraries, on the other hand, link-once sections are deduplicated when the shared library itself is linked, but can’t be deduplicated again at dynamic link time. So those numbers really do represent the amount that gets added to the final application footprint.

Interestingly (and fortunately, seeing as libupnpd isn’t actually the whole program), using -fwhole-program still emits the link-once sections, so they do get deduplicated against the final binary.

Why, If You Really Want Small, You Should Ignore All This

As the massive leap in density between the shared-library and single-object numbers in the graph show, the compiler can do a lot better the more of your program it sees at once. So for the very smallest binaries, take this principle and turn it up to eleven: abandon (for release builds) the idea of using separate libraries at all, and compile your entire program and all its “library” code as a single translation unit.

Not that that’s a good idea if the library code itself is part of the product: if you don’t install the library, no other clients can link against it, and if you install the library but your program doesn’t use it, that’s wasted RAM when other processes load the library copies of all that code. But for “libraries” tightly wedded to a single binary, and especially for embedded systems (where typically there are no other processes), it does get you better code size than even the best options above.

Type Erasure, boost::mutex, And The617

2009-06-17T15:32:00.004+01:00

The boost::mutex class is certainly not considered harmful. It’s jolly good, especially the way it, and its companion boost::condition, work properly on Win32. But it doesn’t half include a lot of header files to obtain all that jolly goodness.

Let’s get GCC to tell us what header files it includes. To do this, we need GCC’s dependency generation features, the ones normally used for generating Makefile fragments listing all the files a translation unit depends on.

$ echo "#include <boost/thread/mutex.hpp>" \
      | g++ -xc++ -E - -MP -M | grep ^/usr | sort
...long list...

OK, that’s indeed quite a lot of headers. How many exactly? (using GCC 4.3.3, amd64-linux, Boost 1.39.0)

$ echo "#include <boost/thread/mutex.hpp>" \
      | g++ -xc++ -E - -MP -M | grep ^/usr | sort | wc -l
618

Yes, a source file that includes nothing but <boost/thread/mutex.hpp> has actually also read six hundred and seventeen other headers by the time it’s done.

Now I’d hate to be one of those bloggers who does nothing but sit around and grouse about the people who are actually doing useful work. But I humbly submit that having to include 618 C++ headers just to get hold of a mutex class, is, in an objective sense, rather a lot.

Really what’s wanted is one of the “complete insulation” or type-erasure techniques from Lakos section 6.4, so that instead of the situation on the left — where every file in the project which needs a mutex, includes the Boost header and its 617 dependencies — we get the situation on the right, where only the encapsulating file needs to know the details, and everyone else can just include the simple leaf header my_mutex.h.

The problem is, every encapsulation technique discussed by Lakos has an inevitable cost in efficiency. Whichever way you slice it — whether you use a mutex protocol class plus a factory, or whether you use the handle/body or pimpl pattern — you tend to end up needing to call new and delete, not just every time you create or destroy a mutex, but every time you create or destroy a mutex scoped-lock object. Especially for rarely-contended locks, that overhead can easily swamp the actual implementation time.

The issue is, that to end up with a boost::mutex, client code must either construct one itself — depending on boost::mutex “in size” is Lakos’s term — or call other code which hands one back, which latter scheme necessarily requires an allocation.

So how can we solve this impasse? How can we depend on boost::mutex “in size”, without depending on all its headers?

Well, we can cheat. We can populate my_mutex.h with classes whose objects are the same size as boost::mutexes — and which, within the file my_mutex.cpp, are boost::mutexes — but whose actual type is invisible to other clients of my_mutex.h. The idea is to behave a bit like this:

class Mutex
{
    char m_data[sizeof(boost::mutex)];

public:
    Mutex()
    {
        new(m_data) boost::mutex;
    }
    ~Mutex()
    {
        ((boost::mutex*)m_data)->~mutex();
    }
};

though of course that doesn’t yet help as-is, as you still need to include the 617 to get sizeof(boost::mutex) to compile.

To eliminate the dependencies, you need to know sizeof(boost::mutex) ahead of time. Probably the best way to do this, is to get your configury to work it out, by adding lines like these to your configure.ac:

AC_LANG_PUSH([C++]) AC_CHECK_SIZEOF([boost::mutex],, [[#include <boost/thread/mutex.hpp>]]) AC_CHECK_SIZEOF([boost::condition],, [[#include <boost/thread/condition.hpp>]]) AC_CHECK_SIZEOF([boost::mutex::scoped_lock],, [[#include <boost/thread/mutex.hpp>]]) AC_LANG_POP

which will leave you with lines like this in your config.h:

#define SIZEOF_BOOST__CONDITION 88 #define SIZEOF_BOOST__MUTEX 40 #define SIZEOF_BOOST__MUTEX__SCOPED_LOCK 16

giving you exactly what you need to write my_mutex.h. (One day when you want to see something deliciously evil, go and look at the way Autoconf determines such size information when cross-compiling, without depending on any particular compiler or linker.) The result looks like this:

/* my_mutex.h */
#include "config.h"

class Mutex
{
    char m_data[SIZEOF_BOOST__MUTEX];

public:
    Mutex();
    ~Mutex();
};


/* my_mutex.cpp */
#include "my_mutex.h"
#include <boost/thread/mutex.hpp>

Mutex::Mutex()
{
    new(m_data) boost::mutex;
}

Mutex::~Mutex()
{
    ((boost::mutex*)m_data)->~mutex();
}

which is indeed good enough to free client code from the burden of the 617.

But if you’re doing this wrapping for more than one class — Chorale wanted at least boost::mutex, boost::mutex::scoped_lock, and boost::condition — you start to realise you’re writing the same thing many times over, and that it really ought to be wrapped up in a template. (An attempt is also made to avoid alignment issues, a worry which might already have been nagging at you after the above snippets; an assertion checks that the configury got the right answer for the size.) Here it is:

template <class T>
struct WrappedType;

/** Wrap up a type so that even clients who depend on it in size,
 * don't have to see its declaration.
 */
template <class T, unsigned int sz>
class Wrapper
{
    union {
        char m_data[sz];
        void *m_align;
    };

    /** Use a nested class, rather than using WrappedType<T>::type
     * directly, so that we can be sure that its destructor is called
     * "~Wrapped" -- if T1 is a typedef-name, its destructor won't be
     * called "~T1".
     */
    class Wrapped: public WrappedType<T>::type
    {
    public:
        Wrapped() {}

        template <class Arg>
        explicit Wrapped(Arg& arg) : WrappedType<T>::type(arg) {}
    };

public:
    Wrapper()
    {
        BOOST_STATIC_ASSERT(sizeof(Wrapped) == sz);
        new (m_data) Wrapped;
    }

    template <class Arg>
    explicit Wrapper(Arg& arg)
    {
        BOOST_STATIC_ASSERT(sizeof(Wrapped) == sz);
        new (m_data) Wrapped(arg);
    }

    ~Wrapper()
    {
        ((Wrapped*)m_data)->~Wrapped();
    }

    /** Calls to Unwrap() will only compile following a definition of
     * WrappedType<T> -- not in client code.
     */
    Wrapped& Unwrap() { return *(Wrapped*)m_data; }
};

Armed with this, my_mutex.h can be very straightforward:

#include "config.h"
#include "wrapper.h"

/** Wrap a boost::mutex so we don't include so very many headers.
 */
class Mutex: public Wrapper<Mutex, SIZEOF_BOOST__MUTEX>
{
public:
    Mutex();
    ~Mutex();

    class Lock: public Wrapper<Lock, SIZEOF_BOOST__MUTEX__SCOPED_LOCK>
    {
    public:
        Lock(Mutex&);
        ~Lock();
    };
};

class Condition: public Wrapper<Condition, SIZEOF_BOOST__CONDITION>
{
public:
    Condition();
    ~Condition();

    bool Wait(Mutex::Lock&, unsigned int sec);

    void NotifyOne();
    void NotifyAll();
};

and my_mutex.cpp not much less straightforward; note that WrappedType<T> is used like a traits class, in that the intended wrapped type is “revealed” by specialising WrappedType for the particular wrapper type — it’s only following such a specialisation, that the constructor, destructor, or Unwrap() calls will compile:

#include "my_mutex.h"
#include <boost/thread/mutex.hpp>
#include <boost/thread/condition.hpp>

template<>
struct WrappedType<Mutex>
{
    typedef boost::mutex type;
};

Mutex::Mutex()
{
}

Mutex::~Mutex()
{
}

template<>
struct WrappedType<Mutex::Lock>
{
    typedef boost::mutex::scoped_lock type;
};

Mutex::Lock::Lock(Mutex& mutex)
    : Wrapper<Lock,SIZEOF_BOOST__MUTEX__SCOPED_LOCK>(mutex.Unwrap())
{
}

Mutex::Lock::~Lock()
{
}

template<>
struct WrappedType<Condition>
{
    typedef boost::condition type;
};

Condition::Condition()
{
}

Condition::~Condition()
{
}

bool Condition::Wait(Mutex::Lock& lock, unsigned int sec)
{
    return Unwrap().timed_wait(lock.Unwrap(), boost::posix_time::seconds(sec));
}

void Condition::NotifyAll()
{
    Unwrap().notify_all();
}

void Condition::NotifyOne()
{
    Unwrap().notify_one();
}

So this technique is neat if a bit icky (with the casting and the explicit size-checking). Is it worthwhile? For Chorale, the answer was certainly “yes”. Chorale is vigorously multi-threaded, and many parts of the system use mutexes. Using a script a bit like the GCC-based dependency counter from the very top of this post, it turns out that Chorale’s 250ish source files and 250ish headers, had between them nearly 240,000 dependencies. Wrapping up just boost::mutex and boost::condition reduced that number to 115,000 — meaning that more than half of all header files compiled while compile the whole of Chorale, a fairly large and complex program, get compiled solely to satisfy the dependencies of boost::mutex. This startling figure is also borne out by the total time taken to compile (with a cold pagecache):

Before wrapping		After wrapping
real 4m0.304s user 10m33.296s sys 1m38.870s		real 2m39.516s user 6m10.523s sys 1m9.452s

...a 42% improvement. It is just possible that there’s a run-time efficiency impact of this wrapping, as the mutex operations now all get inlined into my_mutex.cpp and not directly into the client code. However, as is the way with these things, it’s really just as likely that this gives a performance improvement — some of these Boost calls inline a lot of code, especially under Win32.

How Not To Develop On Windows

2009-05-30T23:44:00.001+01:00

This is a HOW-TO on writing Windows software, including GUI software, without using a Windows box at all except for final testing. Among other benefits, this lets you develop in the simple, testable, scriptable Linux environment; it means you don’t have to forever check things in and out of source-control to test whether you’ve broken the other platform’s builds; and it’s also handy if you’ve got a powerful Linux box but only a feeble Windows box.

Well, actually, it isn’t really a HOW-TO, in the grand tradition; it’s more of a “WHETHER-TO”. That’s because it doesn’t always go into enough detail to let you reconstruct everything described, but it at least tells you that it’s possible — so that if you’re wondering whether-to try and develop Windows software in this fashion, you can be reassured, before setting off in that direction, that the destination is attainable.

There are a lot of bits and pieces to set up a complete cross-development and cross-test environment. We should all be hugely grateful for the vast amount of development effort put in by the GCC, binutils, Mingw, Wine, Qt and other projects to enable the setup described in this one small blog post.

1. A cross-compiler toolchain targetting Mingw32

apt-get install mingw32-binutils mingw32-runtime mingw32

Alternatively, configure and install GNU binutils for target “i586-mingw32”; install the headers from the mingw32-runtime package (which you can’t build yet); configure and install a Stage 1 cross-GCC with

--target=i586-mingw32 --enable-languages="c" --enable-threads=win32 --disable-libmudflap --disable-libssp --enable-__cxa_atexit --enable-sjlj-exceptions --disable-win32-registry --with-gnu-as --with-gnu-ld

as extra configure arguments; configure and install the w32api package; configure and install the mingw32-runtime package; and finally configure and install a Stage 2 fully-working cross-GCC with

--target=i586-mingw32 --enable-languages="c,c++" --enable-threads=win32 --disable-libmudflap --disable-libssp --enable-__cxa_atexit --enable-sjlj-exceptions --disable-win32-registry --with-gnu-as --with-gnu-ld --disable-libstdcxx-pch --enable-libstdcxx-allocator=new

as configure arguments. Note that, supposing you want everything to live in /usr/local/i586-mingw32, you need to give GCC and binutils “--prefix=/usr/local”, and everything else “--prefix=/usr/local/i586-mingw32”.

Except for using w32api and mingw-runtime instead of glibc, this isn’t that different from how to build a cross-compiler for any other target.

If you want to use the exact same versions of everything I did, it’s binutils 2.19.51.0.2, mingw-runtime 3.14, w32api 3.11, and GCC 4.3.3.

A recently-invented third alternative, which I haven’t tried, is the “official” cross-hosted Mingw build tool scripts, which are available on the Mingw Sourceforge page.

2. A native pkgconfig for your cross-compiled libraries

Cross-compiled libraries for Mingw32 will put their pkgconfig “.pc” files in /usr/local/i586-mingw32/lib/pkgconfig. In order for configure scripts targetting Mingw32 to find them, you’ll need a “cross-pkgconfig” — but one which, like a cross-compiler, is built for the build platform, not the target platform. If it’s named using the target prefix, as if it were part of the cross-compiler — i.e., in our case, i586-mingw32-pkgconfig — configure scripts will use it to determine which cross-compiled libraries are present.

Configure pkgconfig 0.23 with:

--with-pc-path=/usr/local/i586-mingw32/lib/pkgconfig --program-prefix=i586-mingw32-

(yes, that ends with a hyphen).

3. Cross-compiled versions of all the libraries you use

How hard it is to arrange for these, depends a lot on each individual library. In theory all you should need to do is configure the library with “--prefix=/usr/local/i586-mingw32 --host=i586-mingw32”, but in practice very few libraries do the Right Thing with that alone. (Honourable mentions here go to libxml2 and taglib.)

Other things you might have to do to more recalcitrant libraries include: setting CC=i586-mingw32-gcc (and sometimes CXX, AR and/or RANLIB similarly); disabling parts of the library (for libcdio use

--disable-joliet --disable-example-progs --without-iso-info --without-iso-read --without-cd-info --without-cd-drive --without-cd-read

to disable all the example programs) — or, if the worst comes to the worst, actually patching out bits of the library. I had to do that to make taglib compile as a static library.

Boost, as usual, presents the most challenging fight you’ll have with a build system. Here, without further commentary, is the Makefile snippet needed to cross-compile Boost for Mingw32; $(BUILD) is the build directory and $(PREFIX) is where to install the result — /usr/local would match a toolchain built as described above:

cross-mingw32-boost:
        mkdir -p $(BUILD)/cross-mingw32-boost
        tar xjf boost-*bz2 -C $(BUILD)/cross-mingw32-boost
        cd $(BUILD)/cross-mingw32-boost/* \
                && ./bootstrap.sh --prefix=$(PREFIX)/i586-mingw32 \
                        --libdir=$(PREFIX)/i586-mingw32/lib \
                        --includedir=$(PREFIX)/i586-mingw32/include \
                && echo \
"using gcc : : i586-mingw32-g++ : <compileflags>-mthreads <linkflags>-mthreads ;" \
              > jtl-config.jam \
                && ./bjam -q --layout=system variant=release \
                        link=static threading=multi --without-iostreams \
                        -sGXX=i586-mingw32-g++ --without-python \
                        threadapi=win32 --user-config=jtl-config.jam \
                && sudo rm -rf $(PREFIX)/i586-mingw32/include/boost \
                && sudo ./bjam -q --layout=system variant=release \
                        link=static threading=multi --without-iostreams \
                        -sGXX=i586-mingw32-g++ --without-python \
                        threadapi=win32 --user-config=jtl-config.jam install
        for i in $(PREFIX)/i586-mingw32/lib/libboost_*.a ; do \
   sudo i586-mingw32-ranlib $$i ; \
 done
        rm -rf $(BUILD)/cross-mingw32-boost

Again, if you’d like a checklist of successes I’ve had here, then with greater or lesser effort it’s proved possible to make cross-compiled versions of zlib 1.2.3, Boost 1.39.0, libcdio 0.80, taglib 1.5, and libxml2 2.6.30.

4. Wine Is Not an Emulator

Configure and install Wine 1.0.1. Admirably, this just works out of the box, though if your Linux box is 64-bit, you’ll need the 32-bit versions of its dependent libraries installed.

Having got this far, you should have all you need to compile, link, and test Windows programs. Of course, they do have to be Windows programs; Mingw is not Cygwin, and your program needs to be compatible with real, proper Windows including <windows.h>, WSAEventSelect, CreateWindowEx and all that jazz — plus, of course, the Windows text-encoding and file-naming rules.

Indeed, depending on how your project’s unit-tests are set up, you can probably run most of them, too, under Wine. Just arrange for them to be invoked using Wine: instead of “run-test various-args”, execute instead “wine run-test various-args”. In some situations, this alone would justify the effort of setting up a cross-development environment: the ability to know, before checking in code on Linux, that it passes all its tests both on Linux and Windows.

5. Qt

Trolltech’s, now Nokia’s, Qt framework has for a while been a really good way of writing open-source Linux GUI applications without getting bogged down in X11 or other unhelpful toolkits. Originally Qt was only available for free on the X11 platform, but subsequently the Windows (and even MacOS) versions were also made available to the free software community, and more recently still have been relicensed under the GNU LGPL. This makes it not only a good way of writing open-source Linux applications, but both open-source and proprietary applications on Linux and Windows (and, again, MacOS too).

So it would be handy if a cross-compiled version of Qt could be used to write and test Windows versions of Linux Qt applications using Wine. The problem is, Qt’s sources are huge — twice the size of KOffice, three times the size of Firefox, five times the size of GLib+GTK put together — which is enough to put a fellow off trying to cross-compile it.

But fortunately, Trolltech supply a binary installer for the Windows development libraries — an installer which works under Wine. So, download qt-win-opensource-x.y.z.exe and run it under Wine. Pick an installation directory (for instance, /usr/local/i586-mingw32/qt — or, in Mingw-speak, Z:\usr\local\i586-mingw32\qt), and let it install. When it asks for a Mingw installation, give it your cross-compiler’s prefix directory (e.g. /usr/local/i586-mingw32); it’ll moan, but let you ignore the moaning and install anyway (do so).

You then need to arrange for Qt’s pkgconfig files to be available to the cross-compiler. The Win32 installation of Qt doesn’t have pkgconfig files, but you can modify the ones from a native Linux installation of the same version of Qt. To do this, issue the following commands (as root):

# cd /usr/lib/pkgconfig     (Or wherever your existing QtCore.pc is)
# for i in Qt*.pc ; do sed \
   -e 's,^prefix=.*,prefix=/usr/local/i586-mingw32/qt,' \
   -e 's,-I.*/include,-I/usr/local/i586-mingw32/qt/include,' \
   -e 's,-l[^ ]*,&4,' \
 < $i > /usr/local/i586-mingw32/lib/pkgconfig/$i ; done

The three sed replacements fix up the prefix= lines in the .pc files, then fix up stray -I directives in the CFLAGS lines that don’t use the defined prefix, then finally take account of the extra versioning present in the Windows filenames (instead of QtCore.lib, Windows has QtCore4.lib, and similarly across the whole framework).

(It was getting a development version of Chorale’s Qt GUI more-or-less up under Wine, and thus bringing Win32 into its sights for the first time, that prompted the writing of this blog post.)

6. The Promised Land

So there you (hopefully, by now) have it. A well-behaved program, such as Chorale, should mostly configure, build, unit-test, and run its Windows version straightforwardly on a Linux box.

Naturally, you still need to use a real Windows box for final testing — not everything that can happen on a real Windows box can be modelled inside Wine, and nor is everything necessarily as compatible between different Windows versions as you’d hope. But by marginalising Windows out of its own development process until as late as possible, the rest of the development can be eased, indeed accelerated, by only having to develop on a single platform.

PathCanonicalize Versus What It Says On The Tin

2009-05-26T22:30:00.000+01:00

This post contains MathML mark-up, which may not display correctly on some browsers. (And if you’re wondering how to do MathML in Blogger, see here.)

Here’s what canonicalisation is. You’ve got a set of items, and you can test those items for equality, $x = y$ , but what you actually want to do is test for equivalence, $x ≍ y$ — where equality implies equivalence, but equivalence doesn’t imply equality. What’s needed is a canonicalisation function which maps the equivalence relation onto the equality relation, by mapping all (of each subset of) equivalent items onto a single representative item; a function $f$ such that $f (x) = f (y) \Leftrightarrow x ≍ y$ (plus you want canonicalisation to be idempotent: $f (f (x)) \equiv f (x)$ ).

More concretely, suppose you’re keeping a database of disk files, perhaps to enable searching or browsing. The question is, when do two filenames refer to the same file? You can’t just test them for string equality, as two distinct names might refer to the same file: on Unix, /home/peter/foo and /home/peter/src/../foo are the same file. Plus, symbolic links can be used to make even unrelated-looking names refer to the same file. If your database lists the file by one name, and someone does a query looking for another name, it won’t be found — and worse, the file could get into the database several times under different names, perhaps with conflicting or stale information.

But fortunately, “..”, “.”, and symlinks between them are about the size of it for Unix ways of obscuring the naming of a file, and the standard library comes with a suitable canonicalisation function that will reduce elaborated forms into the single unambiguous original pathname. (In fact, the GNU C library comes with two such functions, as the more-portable one, realpath(), needs a little care in use in order to avoid buffer-overrun attacks; the GNU one, canonicalize_file_name(), does not.) So you canonicalise all filenames as you store them into your database, and make sure to canonicalise all filenames in queries before you look them up, and you’ll get the right matches.

And then eventually you’re going to want to port that software to Windows — whereupon you’ve got a problem.

Indeed, you’ve got a whole little family of problems, because even once you’ve navigated the treacherous waters of the textual encoding of filenames under Win32, there still remain a bewildering variety of ways to refer to the same file:

`D:\Music\Fools Gold.flac`	— Probably canonical
`D:/Music/Fools Gold.flac`	— Slash versus backslash
`D:\MUSIC\Fools Gold.flac`	— Case-insensitive per locale
`D:\Music\FOOLSG~1.FLA`	— MS-DOS 8.3
`M:\Fools Gold.flac`	— After “subst M: D:\Music”
`\Device\HarddiskVolume2\Music\Fools Gold.flac`	— If D: is local
`\\server\share\Music\Fools Gold.flac`	— If D: is a network drive
`\\?\UNC\server\share\Music\Fools Gold.flac`	— Or like this
`\\?\D:\Music\Fools Gold.flac`	— Ultra-long-filenames mode
`\\.\D:\Music\Fools Gold.flac`	— Device namespace
`\\?\UNC\D:\Music\Fools Gold.flac`	— Allegedly
`\\?\Volume{GUID}\Music\Fools Gold.flac`	— Crikey

This whole dismal farrago really calls for a path canonicalisation function. Which is why it’s unfortunate that there isn’t one, and doubly unfortunate that there’s a function called PathCanonicalize() that particularly isn’t one, and not just because it’s spelled with a “Z”. All that PathCanonicalize() does is remove “/../” and “/./” substrings — it’s a purely textual transformation and doesn’t even touch the filesystem. It certainly doesn’t satisfy the “canonicaliser condition”:

$f (x) = f (y) \Leftrightarrow x$ and $y$ are the same file

No, there’s no shortcut for doing it laboriously and textually (nor for having lots of unit tests to cover all those ridiculous cases). The plan is, use GetFullPathName() to turn relative paths into absolute, then repeatedly call QueryDosDevice() to unwind subst’d drive letters, then call GetLongPathName() to get rid of 8.3-ness and canonicalise case, and then finally, if GetDriveType() says it’s remote, use WNetGetUniversalName() to convert the remaining drive letter into a UNC path.

std::string Canonicalise(const std::string& path)
{
    std::wstring utf16 = UTF8ToUTF16(path);

    wchar_t canon[MAX_PATH];

    /** Note that PathCanonicalize does NOT do what we want here -- it's a
     * purely textual operation that eliminates /./ and /../ only.
     */
    DWORD rc = ::GetFullPathNameW(utf16.c_str(), MAX_PATH, canon, NULL);
    if (!rc)
        return path;

    utf16 = canon;

    if (utf16.length() >= 6)
    {
        /** Get rid of \\?\ and \\.\ prefixes on drive-letter paths */
        if (!wcsncmp(utf16.c_str(), L"\\\\?\\", 4) && utf16[5] == L':')
            utf16.erase(0,4);
        else if (!wcsncmp(utf16.c_str(), L"\\\\.\\", 4) && utf16[5] == L':')
            utf16.erase(0,4);
    }

    if (utf16.length() >= 10)
    {
        /** Get rid of \\?\UNC on drive-letter and UNC paths */
        if (!wcsncmp(utf16.c_str(), L"\\\\?\\UNC\\", 8))
        {
            if (utf16[9] == L':' && utf16[10] == L'\\')
                utf16.erase(0,8);
            else
            {
                utf16.erase(0,7);
                utf16 = L"\\" + utf16;
            }
        }
    }

    /** Anything other than UNC and drive-letter is something we don't
     * understand
     */
    if (utf16[0] == L'\\' && utf16[1] == L'\\')
    {
        if (utf16[2] == '?' || utf16[2] == '.')
            return path; // Not understood

        /** OK -- UNC */
    }
    else if (((utf16[0] >= 'A' && utf16[0] <= 'Z')
              || (utf16[0] >= 'a' && utf16[0] <= 'z'))
             && utf16[1] == ':')
    {
        /** OK -- drive letter -- unwind subst'ing */
        for (;;)
        {
            wchar_t drive[3];
            drive[0] = (wchar_t)toupper(utf16[0]);
            drive[1] = L':';
            drive[2] = L'\0';
            canon[0] = L'\0';
            rc = ::QueryDosDeviceW(drive, canon, MAX_PATH);
            if (!rc)
                break;
            if (!wcsncmp(canon, L"\\??\\", 4))
            {
                utf16 = std::wstring(canon+4) + std::wstring(utf16, 2);
            }
            else // Not subst'd
                break;
        }

        wchar_t drive[4];
        drive[0] = (wchar_t)toupper(utf16[0]);
        drive[1] = ':';
        drive[2] = '\\';
        drive[3] = '\0';

        rc = ::GetDriveTypeW(drive);

        if (rc == DRIVE_REMOTE)
        {
            DWORD bufsize = MAX_PATH;

            /* QueryDosDevice and WNetGetConnection FORBID the
             * trailing slash; GetDriveType REQUIRES it.
             */
            drive[2] = '\0';

            rc = ::WNetGetConnectionW(drive, canon, &bufsize);
            if (rc == NO_ERROR)
                utf16 = std::wstring(canon) + std::wstring(utf16, 2);
        }
    }
    else
    {
        // Not understood
        return path;
    }

    /** Canonicalise case and 8.3-ness */
    rc = ::GetLongPathNameW(utf16.c_str(), canon, MAX_PATH);
    if (!rc)
        return path;

    std::string utf8 = UTF16ToUTF8(canon);
    std::replace(utf8.begin(), utf8.end(), '\\', '/');
    return utf8;
}

There are still ways to fool this function: for instance, by exporting a directory as \\server\share1 and a subdirectory of it as \\server\share2 — the client has no way of matching them up. But that’s a pretty pathological case, and it could be easily argued that it’s something you’d never do unless you wanted the shares to appear to be distinct. More seriously, the server for a network drive can be specified by WINS name, by FQDN or by IP address; neither canonicalise-to-IP nor canonicalise-to-FQDN is the Right Thing in all cases. For now I’m sweeping that issue under the carpet.

The one remaining wrinkle is that, unlike earlier versions of Windows, Vista allows “genuine” Unix-like symbolic links. Without doing real testing on Vista, it’s hard to make out from the documentation how the APIs used above behave when faced with such symbolic links. It’s even possible that the new-in-Vista GetFinalPathNameByHandle() call is the answer to all these problems; in which case, this code gets demoted to merely the way to do it on pre-Vista versions.

Unicode Is The One True God, And UTF-8 Is His Prophet

2009-05-06T23:21:00.000+01:00

One common anti-pattern in the Win32 API is the one whereby something is done wrong the first time, then later re-implemented properly — but, for compatibility reasons, the older, broken implementation can never actually be removed or fixed. (The uncharitable might say that another common anti-pattern is the same thing, but where the re-implementation doesn’t work either.)

An example of one or both of these anti-patterns is encoding support. The Win32 API’s Unicode support was invented before UTF-8 was, so Microsoft can’t exactly be blamed for not using UTF-8. Instead they used UCS-2 (later retconned to UTF-16), but, with 16-bit characters, this was grievously incompatible with Win16 — and, indeed, with all other C source in the world at that time. So they kept an 8-bit-character “ANSI” API around as well, compatible with the 8-bit code-pages of Win16.

Without the same compatibility millstone around its neck, Linux (eventually) embraced UTF-8 into its locale system, to the extent that nowadays it’s reasonable to describe all non-UTF-8 locales as “legacy”. When writing portable software (location-portable as well as system-portable) for Linux, it’s quite the natural thing to do to store all text in UTF-8, and, if happening to find oneself running in a legacy locale, to translate to UTF-8 on the way in and from UTF-8 on the way out. That way, almost all the code can just deal with normal char* and assume it’s encoded in UTF-8.

And then eventually you’re going to want to port that software to Windows — whereupon you’ve got a problem.

Indeed, you have a whole little family of problems, and they’ve got names such as fopen, rename, unlink, and more. All of these calls — and those are just the ISO C ones; Posix and C++ add more — take 8-bit strings as filename arguments, and, in Windows, there are some files that this just won’t let you see. Files in Windows filesystems have UTF-16 names, which, when you use 8-bit calls, are mapped down into the characters available in the current ANSI code-page. (Strictly speaking, the code-page set as the “Code-page for non-Unicode applications” in Control Panel.)

So suppose you have your code-page for non-Unicode applications set to CP1252 for generic Western European, and your Greek friend, who has his code-page for non-Unicode applications set to CP1253 for Greek, writes two files with Greek names onto a USB stick and hands it to you. The filenames will be stored as UTF-16, featuring characters which do not exist in your code-page. And if you have two files whose names are only distinguishable by characters not in your code-page, you can’t distinguish them — they’ll both come back as, say, “????.txt”, and, while there’s enough grody hacks in Windows that fopen will actually open “????.txt”, that still only gets you one of the two files. There’s no way of opening the other one using 8-bit calls.

There’s only a few things you can do about this. The easiest, certainly, is to ignore it completely — after all, most people only create files named in their own language. This is the position taken by Cygwin (before 1.7, anyway) and thus, implicitly, by all Unix software which has been ported to Windows by simply recompiling it against Cygwin mentioning no names. It is, of course, very unsatisfactory — for instance, among non-Czechs partial to MP3s of music composed by Antonín Dvořák.

Alternatively, you could go through all your code replacing all the strings used to keep filenames in with wide-strings, all the calls to fopen with _wfopen, and so on — or use all the ugly macros such as _tfopen (which was the Microsoft advice even for Windows-only programs, back in the day when Windows 95 was Win32 but not Unicode, so it was common to make two binary builds of the same sources). But this is a lot of work — and error-prone, particularly if you rely on other libraries that have been ported to Windows using the ignore-the-problem technique, and you thus have to go and update them too.

Fortunately, there’s another way of solving the problem — at least, if you’re targetting Windows using Mingw and are thus using the GNU binutils linker. If you add

-Wl,--wrap,fopen -Wl,--wrap,rename -Wl,--wrap,unlink … etc.

to the linker command-line, all calls to (for instance) fopen will be converted into calls to __wrap_fopen, which you can implement in your own code. In particular, you can make it do something like this:

extern "C" FILE* __cdecl __wrap_fopen(const char *path, const char *mode)
{
    std::wstring wpath, wmode;
    util::UTF8ToWide(path, &wpath);
    util::UTF8ToWide(mode, &wmode);
    return _wfopen(wpath.c_str(), wmode.c_str());
}

extern "C" int __cdecl __wrap_rename(const char *oldname, const char *newname)
{
    std::wstring woldname, wnewname;
    util::UTF8ToWide(oldname, &woldname);
    util::UTF8ToWide(newname, &wnewname);
    return _wrename(woldname.c_str(), wnewname.c_str());
}

(You don’t, of course, get prototypes for __wrap_fopen or __wrap_rename, so you must carefully ensure to give them the same signatures as the genuine functions.) This way, all your code can carry on using UTF-8 filenames everywhere, passing them to system calls such as fopen, and even passing them to external libraries, so long as you’ve wrapped the calls the libraries themselves make. (Which you can check by asking the linker for a map file: -Wl,-Map,map.txt.)

Naturally, there’s no point using ANSI Win32 APIs in Windows-specific parts of your code, only to then wrap them — it only makes sense there to go straight for the UTF-16 ones (with the W on the end, such as RegQueryValueExW) to start with. This wrapping technique is a way of avoiding rewriting the existing portable parts of your code.

It’s like having an ANSI code-page of UTF-8 all to your very own.

One Template To Rule Them All

2009-04-23T18:58:00.000+01:00

How do you write a template that can be instantiated for any type? Here’s a header file which purports to implement a tracing (logging) mechanism...

#include <stdio.h>

class Tracer {};

inline const Tracer& operator<<(const Tracer& r, int i)
{ printf("%d", i); return r; }

inline const Tracer& operator<<(const Tracer& r, const char *s)
{ printf("%s", s); return r; }

template <class T>
inline const Tracer& operator<<(const Tracer& r, T *p)
{ printf("%p", p); return r; }

class NullTracer {};

template <typename T>
inline const NullTracer& operator<<(const NullTracer& n, T) { return n; }

#ifdef DEBUG
#define TRACE Tracer()
#else
#define TRACE NullTracer()
#endif

...and here’s some example client code:

class X
{
    enum { A, B } m_state;

    void Foo() { TRACE << "x" << this << " is in state " << m_state << "\n"; }

public:
    X();
};

(I used this rather nifty script to do the syntax-colouring.)

What’s going on here is that, in debug builds (with -DDEBUG on the compiler command-line), TRACE expands to an expression that creates an object that acts a bit like std::cout, in that various different types of object can be sent to it and each printed in a sensible and type-safe manner. Just as with std::cout, the precedence and associativity of the << operator get everything printed in the right order.

In release builds, meanwhile, TRACE expands to an expression that creates an object of a different class, whose operator<< ensures that whatever type is sent to it, nothing is printed — and, indeed, the entire expression optimises away to nothing.

Except there’s a problem. The code above doesn’t actually compile in release builds (without -DDEBUG). Here’s what GCC 4.3.3 says about it:

wtf.cpp: In member function ‘void X::Foo()’: wtf.cpp:30: error: no match for ‘operator<<’ in ‘operator<< [with T = const char*](((const NullTracer&)((const NullTracer*)operator<< [with T = X*](((const NullTracer&)((const NullTracer*)operator<< [with T = const char*](((const NullTracer&)((const NullTracer*)(& NullTracer()))), ((const char*)"x")))), this))), ((const char*)" is in state ")) << ((X*)this)->X::m_state’ wtf.cpp:10: note: candidates are: const Tracer& operator<<(const Tracer&, int) wtf.cpp:13: note: const Tracer& operator<<(const Tracer&, const char*)

So naturally I knew at once what the problem was. Not. The error suggests that it can’t find an operator<< to do the job — but surely there’s one right there that can do any such job. Yet it isn’t even listed as a candidate.

While composing my GCC Bugzilla entry in my head, I tried to whittle down the problem. To my surprise, it was the enum which was causing the problem: trace statements with only strings and pointers compiled just fine. (Those experienced with what Abrahams and Gurtovoy call “debugging the error novel” might just about be able to make that out from the error message.)

So what’s up with m_state? What’s so very special about its type that it isn’t matched by template<typename T>? Well, there is one special thing about its type. It’s unnamed. When I wrote m_state into that code, neither its declaration, nor any of its uses, actually needed to name its type — so I applied Occam’s Razor and didn’t give it a name. And it turns out that that’s the problem: you can’t instantiate a template with an unnamed type as a template-argument. I can’t find a statement to that effect in Stroustrup-3, but it’s right there in the ISO C++03 standard ISO/IEC 14882:2003 at 14.3.1 [temp.arg.type]:

A local type, a type with no linkage, an unnamed type or a type compounded from any of these types shall not be used as a template-argument for a template type-parameter.

This does make perfect sense when you think about it a bit more: if templates could be instantiated with unnamed types, how would their names be mangled? Or, in other words, how could the compiler and linker conspire to arrange that multiple instantiations of NullTracer<unnamed-enum-type>, in multiple translation units, ended up as one instantiation in the final program? Without name matching, there’d be no way to do that — at least within the constraints that the C++ committee set for themselves, specifically that C++ compilers be usable with C-era linkers by dint of name mangling. The same mangling considerations are probably also the reasoning behind the ban on string literals as template arguments.

And the reason it works in non-debug builds, with the real Tracer, is that in that case no template is instantiated with m_state’s actual type: integral promotion makes it an int, and the overload for int is called. Nor can you fix up NullTracer by adding a template specialisation for int: the template specialisation rules aren’t quite the same as the overload resolution rules, and the compiler is doomed to pick the generic operator<<, and have it fail, before it tries the integral promotions. (SFINAE doesn’t help us, as again it is applied only with the enum type, not the promoted type.)

So what's needed is another overload, one that catches unnamed enum types:

inline const NullTracer& operator<<(const NullTracer& n, int) { return n; }

And now everything works as expected: unnamed enums (and, in fact, ints too, not that it matters) are passed via the non-templated overload, and everything else goes via the templated overload.

Debugging The Invisible

2009-04-15T16:56:00.000+01:00

The Empeg codebase, and the Empeg coding style for embedded C++, don’t use C++ exceptions. The original reason was that ARM support in GCC was still a bit sketchy when Empeg started, and exceptions plain didn’t work.

But I think that even with a toolchain in which C++ exceptions work properly, they should still be avoided, and not just in embedded systems either. Here’s why.

Exception safety

You need to write your code in exception-safe style: a style which neither the compiler, nor other tools, can enforce. Some would say that “write exception-safe code” is just another way of saying “don’t write buggy code”, or “don’t be a crap programmer”, but many other ways of saying that — “write const-correct code”, “don’t leak memory” — can be automatically checked, and thus a unit-test suite (perhaps with Valgrind) can give you great confidence that your code fulfils those criteria, and remains in fulfilment under subsequent maintenance. You can’t easily unit-test for exception safety.

Another consideration is that many, perhaps most, C++ developers used to be C developers. Most things these developers need to learn when switching to C++, are completely new things. But exception-safety is something they need to learn which changes the way they’d write perfectly reasonable straight-line C code, so that’s always going to be trickier for everyone.

Scott Meyers didn’t even bother writing exception-safe code until the third edition of Effective C++ (search for “dfc”).

Invisible control flow

The Empeg C++ coding style, like others, forbade non-const reference parameters. The reason is, that doing so forces by-reference objects which may be modified to go via pointers instead. This, in turn, forces call sites to use “&” when a function may modify the value, and no “&” when it doesn’t. This puts this essential information about the data flow at the call site, not just at the declaration — and, when debugging, it’s usually the call site and not the declaration that you’re looking at. So banning non-const references improves the understandability of the data flow by making it visible.

Exceptions are, in a way, even worse than non-const references: they bestow on your code unasked, not just data flow you can’t see, but control flow you can’t see. Consider, in particular, a function that doesn’t contain a try or a throw, but calls functions which do (or may do). There’s a second control flow through all that code, which you didn’t write — the one where something below it in the call-chain throws, and something above it catches — but which has a crucial effect on whether your code is buggy or not.

And again, you can’t unit-test for it: even if you use gcov to test that your tests cover the normal control flow of your code, it can’t tell you whether you’ve also exercised all the invisible, secondary control flows. (In theory, a much smarter gcov-like tool could, though.)

Joel agrees.

Speed, size, exceptions: pick two

In D&E, Stroustrup that claims a design goal for C++ exceptions was “No added cost (in time or space) to code that does not throw an exception” (16.2), but later on in 16.9.1 he admits that people have figured out how to avoid cost in time in the no-exception case, but not cost in space.

A medium-sized C++ program I have here (amd64-linux, GCC 4.3.2) has a text section of 548K and exception-handling tables of 127K. But even that 23% increase isn’t the whole story, as is revealed by disassembly: that try-less, throw-less, but non-leaf function — if it has local variables with destructors — also gains snippets of unwinding code for each such variable. These snippets appear in the text section, so can’t easily be totalled-up, but an unscientific test on one source file suggests that they make up about 7% of the text section.

These exception-handling tables do affect embedded systems more than desktops or servers — on which the exception-handling tables are never demand-paged in unless they’re used — but the unwinding snippets are still there affecting your paging bandwidth and filling your instruction cache.

But you don’t get to choose

In a sense, choosing which of the two distinct languages — C++-with-exceptions or C++-without-exceptions — to use to program your embedded system, is a choice you only get to make when starting the system from scratch. Or, at least, converting a codebase from C++-without-exceptions to C++-with-exceptions is a herculean and error-prone task, and a half-converted one is a bug tarpit, because of the exception-safety issues. If the actual goal is to use a library written in C++-with-exceptions with a main program written in C++-without-exceptions, the best way is to write wrappers for each library function, that catch(...) and report errors through the existing non-exception error handling mechanism. With a bit of luck you can then continue to compile the main program with -fno-exceptions (or similar) and avoid the space overhead.

Even this doesn’t really work when the library is something like Boost, which is chock-full of throws in headers that really you need to be including in any code that uses the library. (Including for things like locking a non-recursive mutex that’s already locked, which in my book is more like a programming error than an exceptional condition, and would be better as an assertion. But that’s a different rant.)

Autoconf, #ifdef, and #if

2009-04-12T20:24:00.000+01:00

The standard macros available in GNU Autoconf for “checking for...” C/C++ functions and header files, AC_CHECK_FUNCS and AC_CHECK_HEADERS, have a couple of things going for them: their syntax and semantics essentially haven’t changed for years, perhaps ever, and everyone — well, everyone who writes configure scripts, which if it isn’t you should be a big hint as to how much you’ll get from this blog — already knows how they work. Which is to say, they end up embedding in your generating header, typically config.h, lines which look like this:

/* Define to 1 if you have the <linux/inotify.h> header file. */
/* #undef HAVE_LINUX_INOTIFY_H */

/* Define to 1 if you have the `localtime_r' function. */
#define HAVE_LOCALTIME_R 1

That’s all you need to start making your code conditional on which facilities are available:

#include "config.h"
    [...]
#ifdef HAVE_LOCALTIME_R
    localtime_r(&start, &stm);
#else
    // This code isn't thread-safe and could
    // get the wrong time -- but the result
    // would just be an odd-looking filename, not 
    // incorrect operation, so we live with it
    stm = *localtime(&start);
#endif

But there’s a problem. In fact, there’re two problems. One of the best features of C and C++, one of the few things that lets the old guard lord it over today’s arriviste scripting languages, the pythons and the rubies, is that almost everything is checked at compile-time. If you mistype an identifier, say localtime_r, as localtimer or whatever else, your code won’t compile, even if it’s on a rarely- or never-used path through the code. Unless you’re unlucky enough to hit the exact name of another existing identifier (and even then the type system may save you, unless you’re unlucky again), your mistyping is caught right there as you’re writing the code, not in QA — or, horrors, in production.

The same can’t be said, though, of identifiers in #ifdef. It’s perfectly valid to use a name there that doesn’t exist — nor could it be otherwise, as there’s nothing else it does except distinguish existent from non-existent names. So if you fat-finger it — HAVE_LOCALTIMER, maybe — the code compiles without even a warning, and just does the wrong thing, or at least the less-optimal thing. Even into production.

The second problem is config.h itself: if you forget to #include it, again your code compiles without a warning, but all your features get turned off! There are ways around this, such as by having a little script that checks your sources using grep, or by using GCC’s -include command-line option to force the inclusion, but none is ideal. (For instance, the latter gets dependencies wrong if config.h changes.)

Really what you want is warnings at compile-time if you mistype the identifier, or forget the header. In order to do this, you’re going to need to use #if rather than #ifdef; in other words, you’re going to need a config.h with lines like this:

/* Define to 1 if you have <linux/inotify.h>, or to 0 if you don't. */
#define HAVE_LINUX_INOTIFY_H 0

/* Define to 1 if you have localtime_r(), or to 0 if you don't. */
#define HAVE_LOCALTIME_R 1

And to do that, you’re going to have to use custom Autoconf macros instead of the standard AC_CHECK_ ones. Which is the point at which most people would give up, as Autoconf macros expand into shell code but are themselves written in the macro language m4, whose syntax can be charitably described as quixotic, and is regularly described using rather shorter and more Anglo-Saxon words than that. So here, I’ve done it for you, have these ones:

AC_DEFUN([PDH_CHECK_HEADER], [
AC_MSG_CHECKING([for <$1>])
AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <$1>
]])],
          [AC_MSG_RESULT([yes]); pdh_haveit=1; pdh_yesno="yes"],
          [AC_MSG_RESULT([no]);  pdh_haveit=0; pdh_yesno="no"])
AC_DEFINE_UNQUOTED(AS_TR_CPP([HAVE_$1]), $pdh_haveit,
          [Define to 1 if you have <$1>, or to 0 if you don't.])
AS_TR_SH([ac_cv_header_$1])=$pdh_yesno
])

AC_DEFUN([PDH_CHECK_HEADERS], [
m4_foreach_w([pdh_header], $1, [PDH_CHECK_HEADER(pdh_header)])
])

AC_DEFUN([PDH_CHECK_FUNC], [
AC_MSG_CHECKING([for $1()])
AC_LINK_IFELSE([AC_LANG_PROGRAM([], [[(void)$1();]])],
        [AC_MSG_RESULT([yes]); pdh_haveit=1; pdh_yesno="yes"],
        [AC_MSG_RESULT([no]);  pdh_haveit=0; pdh_yesno="no"])
AC_DEFINE_UNQUOTED(AS_TR_CPP([HAVE_$1]), $pdh_haveit,
        [Define to 1 if you have $1(), or to 0 if you don't.])
AS_TR_SH([ac_cv_func_$1])=$pdh_yesno
])

AC_DEFUN([PDH_CHECK_FUNCS], [
m4_foreach_w([pdh_func], $1, [PDH_CHECK_FUNC(pdh_func)])
])

That bracketfest defines macros PDH_CHECK_HEADERS and PDH_CHECK_FUNCS which do the same job as the standard AC_CHECK_HEADERS and AC_CHECK_FUNCS...

PDH_CHECK_HEADERS([io.h poll.h errno.h fcntl.h sched.h net/if.h stdint.h stdlib.h unistd.h shlwapi.h pthread.h ws2tcpip.h sys/poll.h sys/time.h sys/types.h sys/socket.h sys/syslog.h netinet/in.h netinet/tcp.h sys/inotify.h sys/resource.h linux/unistd.h linux/inotify.h linux/dvb/dmx.h linux/dvb/frontend.h])
PDH_CHECK_FUNCS([mkstemp localtime_r setpriority getaddrinfo inotify_init gettimeofday gethostbyname_r sync_file_range sched_setscheduler])

...but leave you with identifiers in config.h that you can use #if on rather than #ifdef, so you can enjoy compile-time warnings if you get any of them wrong.

Ah, except you can’t just yet, because there’s one final wrinkle. The C and C++ standards allow undefined identifiers in #if just as in #ifdef — they don’t expand to nothing, giving the syntax errors you’d initially hope for, but are defined as evaluating to zero. So, back to square one. Except, fortunately, the authors of GCC appear to be as sceptical about the usefulness of that preprocessor feature as I am, and they provide an optional warning which can be emitted whenever the preprocessor is obliged to evaluate an undefined identifier as zero: -Wundef. So, if you use the configury macros above, and add -Wundef to your GCC command-lines, you can at last get the compiler to point it out to you when you mistype your configure macros.

When I wrote these new macros, I did have a nagging feeling that it was all just NIH syndrome, and that it was silly makework to be rewriting core bits of Autoconf for this marginal benefit. But once I’d done it, re-compiling Chorale showed up two warnings, both places where a mistyped identifier was causing real and otherwise hard-to-find problems. So I felt retrospectively justified in my tinkering. The moral of the story: if you’ve fixed a bug, you’ve done a day’s work; but if you’ve fixed a bug by ensuring that it, and its whole class of fellow bugs, can never happen undetected again — then, you’ve done a good day’s work.