How Not To Develop On Windows

This is a HOW-TO on writing Windows software, including GUI software, without using a Windows box at all except for final testing. Among other benefits, this lets you develop in the simple, testable, scriptable Linux environment; it means you don’t have to forever check things in and out of source-control to test whether you’ve broken the other platform’s builds; and it’s also handy if you’ve got a powerful Linux box but only a feeble Windows box.

Well, actually, it isn’t really a HOW-TO, in the grand tradition; it’s more of a “WHETHER-TO”. That’s because it doesn’t always go into enough detail to let you reconstruct everything described, but it at least tells you that it’s possible — so that if you’re wondering whether-to try and develop Windows software in this fashion, you can be reassured, before setting off in that direction, that the destination is attainable.

There are a lot of bits and pieces to set up a complete cross-development and cross-test environment. We should all be hugely grateful for the vast amount of development effort put in by the GCC, binutils, Mingw, Wine, Qt and other projects to enable the setup described in this one small blog post.

1. A cross-compiler toolchain targetting Mingw32

apt-get install mingw32-binutils mingw32-runtime mingw32

Alternatively, configure and install GNU binutils for target “i586-mingw32”; install the headers from the mingw32-runtime package (which you can’t build yet); configure and install a Stage 1 cross-GCC with

--target=i586-mingw32 --enable-languages="c" --enable-threads=win32 --disable-libmudflap --disable-libssp --enable-__cxa_atexit --enable-sjlj-exceptions --disable-win32-registry --with-gnu-as --with-gnu-ld

as extra configure arguments; configure and install the w32api package; configure and install the mingw32-runtime package; and finally configure and install a Stage 2 fully-working cross-GCC with

--target=i586-mingw32 --enable-languages="c,c++" --enable-threads=win32 --disable-libmudflap --disable-libssp --enable-__cxa_atexit --enable-sjlj-exceptions --disable-win32-registry --with-gnu-as --with-gnu-ld --disable-libstdcxx-pch --enable-libstdcxx-allocator=new

as configure arguments. Note that, supposing you want everything to live in /usr/local/i586-mingw32, you need to give GCC and binutils “--prefix=/usr/local”, and everything else “--prefix=/usr/local/i586-mingw32”.

Except for using w32api and mingw-runtime instead of glibc, this isn’t that different from how to build a cross-compiler for any other target.

If you want to use the exact same versions of everything I did, it’s binutils 2.19.51.0.2, mingw-runtime 3.14, w32api 3.11, and GCC 4.3.3.

A recently-invented third alternative, which I haven’t tried, is the “official” cross-hosted Mingw build tool scripts, which are available on the Mingw Sourceforge page.

2. A native pkgconfig for your cross-compiled libraries

Cross-compiled libraries for Mingw32 will put their pkgconfig “.pc” files in /usr/local/i586-mingw32/lib/pkgconfig. In order for configure scripts targetting Mingw32 to find them, you’ll need a “cross-pkgconfig” — but one which, like a cross-compiler, is built for the build platform, not the target platform. If it’s named using the target prefix, as if it were part of the cross-compiler — i.e., in our case, i586-mingw32-pkgconfig — configure scripts will use it to determine which cross-compiled libraries are present.

Configure pkgconfig 0.23 with:

--with-pc-path=/usr/local/i586-mingw32/lib/pkgconfig --program-prefix=i586-mingw32-

(yes, that ends with a hyphen).

3. Cross-compiled versions of all the libraries you use

How hard it is to arrange for these, depends a lot on each individual library. In theory all you should need to do is configure the library with “--prefix=/usr/local/i586-mingw32 --host=i586-mingw32”, but in practice very few libraries do the Right Thing with that alone. (Honourable mentions here go to libxml2 and taglib.)

Other things you might have to do to more recalcitrant libraries include: setting CC=i586-mingw32-gcc (and sometimes CXX, AR and/or RANLIB similarly); disabling parts of the library (for libcdio use

--disable-joliet --disable-example-progs --without-iso-info --without-iso-read --without-cd-info --without-cd-drive --without-cd-read

to disable all the example programs) — or, if the worst comes to the worst, actually patching out bits of the library. I had to do that to make taglib compile as a static library.

Boost, as usual, presents the most challenging fight you’ll have with a build system. Here, without further commentary, is the Makefile snippet needed to cross-compile Boost for Mingw32; $(BUILD) is the build directory and $(PREFIX) is where to install the result — /usr/local would match a toolchain built as described above:

cross-mingw32-boost:
        mkdir -p $(BUILD)/cross-mingw32-boost
        tar xjf boost-*bz2 -C $(BUILD)/cross-mingw32-boost
        cd $(BUILD)/cross-mingw32-boost/* \
                && ./bootstrap.sh --prefix=$(PREFIX)/i586-mingw32 \
                        --libdir=$(PREFIX)/i586-mingw32/lib \
                        --includedir=$(PREFIX)/i586-mingw32/include \
                && echo \
"using gcc : : i586-mingw32-g++ : <compileflags>-mthreads <linkflags>-mthreads ;" \
              > jtl-config.jam \
                && ./bjam -q --layout=system variant=release \
                        link=static threading=multi --without-iostreams \
                        -sGXX=i586-mingw32-g++ --without-python \
                        threadapi=win32 --user-config=jtl-config.jam \
                && sudo rm -rf $(PREFIX)/i586-mingw32/include/boost \
                && sudo ./bjam -q --layout=system variant=release \
                        link=static threading=multi --without-iostreams \
                        -sGXX=i586-mingw32-g++ --without-python \
                        threadapi=win32 --user-config=jtl-config.jam install
        for i in $(PREFIX)/i586-mingw32/lib/libboost_*.a ; do \
   sudo i586-mingw32-ranlib $$i ; \
 done
        rm -rf $(BUILD)/cross-mingw32-boost

Again, if you’d like a checklist of successes I’ve had here, then with greater or lesser effort it’s proved possible to make cross-compiled versions of zlib 1.2.3, Boost 1.39.0, libcdio 0.80, taglib 1.5, and libxml2 2.6.30.

4. Wine Is Not an Emulator

Configure and install Wine 1.0.1. Admirably, this just works out of the box, though if your Linux box is 64-bit, you’ll need the 32-bit versions of its dependent libraries installed.

Having got this far, you should have all you need to compile, link, and test Windows programs. Of course, they do have to be Windows programs; Mingw is not Cygwin, and your program needs to be compatible with real, proper Windows including <windows.h>, WSAEventSelect, CreateWindowEx and all that jazz — plus, of course, the Windows text-encoding and file-naming rules.

Indeed, depending on how your project’s unit-tests are set up, you can probably run most of them, too, under Wine. Just arrange for them to be invoked using Wine: instead of “run-test various-args”, execute instead “wine run-test various-args”. In some situations, this alone would justify the effort of setting up a cross-development environment: the ability to know, before checking in code on Linux, that it passes all its tests both on Linux and Windows.

5. Qt

Trolltech’s, now Nokia’s, Qt framework has for a while been a really good way of writing open-source Linux GUI applications without getting bogged down in X11 or other unhelpful toolkits. Originally Qt was only available for free on the X11 platform, but subsequently the Windows (and even MacOS) versions were also made available to the free software community, and more recently still have been relicensed under the GNU LGPL. This makes it not only a good way of writing open-source Linux applications, but both open-source and proprietary applications on Linux and Windows (and, again, MacOS too).

So it would be handy if a cross-compiled version of Qt could be used to write and test Windows versions of Linux Qt applications using Wine. The problem is, Qt’s sources are huge — twice the size of KOffice, three times the size of Firefox, five times the size of GLib+GTK put together — which is enough to put a fellow off trying to cross-compile it.

But fortunately, Trolltech supply a binary installer for the Windows development libraries — an installer which works under Wine. So, download qt-win-opensource-x.y.z.exe and run it under Wine. Pick an installation directory (for instance, /usr/local/i586-mingw32/qt — or, in Mingw-speak, Z:\usr\local\i586-mingw32\qt), and let it install. When it asks for a Mingw installation, give it your cross-compiler’s prefix directory (e.g. /usr/local/i586-mingw32); it’ll moan, but let you ignore the moaning and install anyway (do so).

You then need to arrange for Qt’s pkgconfig files to be available to the cross-compiler. The Win32 installation of Qt doesn’t have pkgconfig files, but you can modify the ones from a native Linux installation of the same version of Qt. To do this, issue the following commands (as root):

# cd /usr/lib/pkgconfig     (Or wherever your existing QtCore.pc is)
# for i in Qt*.pc ; do sed \
   -e 's,^prefix=.*,prefix=/usr/local/i586-mingw32/qt,' \
   -e 's,-I.*/include,-I/usr/local/i586-mingw32/qt/include,' \
   -e 's,-l[^ ]*,&4,' \
 < $i > /usr/local/i586-mingw32/lib/pkgconfig/$i ; done

The three sed replacements fix up the prefix= lines in the .pc files, then fix up stray -I directives in the CFLAGS lines that don’t use the defined prefix, then finally take account of the extra versioning present in the Windows filenames (instead of QtCore.lib, Windows has QtCore4.lib, and similarly across the whole framework).

(It was getting a development version of Chorale’s Qt GUI more-or-less up under Wine, and thus bringing Win32 into its sights for the first time, that prompted the writing of this blog post.)

6. The Promised Land

So there you (hopefully, by now) have it. A well-behaved program, such as Chorale, should mostly configure, build, unit-test, and run its Windows version straightforwardly on a Linux box.

Naturally, you still need to use a real Windows box for final testing — not everything that can happen on a real Windows box can be modelled inside Wine, and nor is everything necessarily as compatible between different Windows versions as you’d hope. But by marginalising Windows out of its own development process until as late as possible, the rest of the development can be eased, indeed accelerated, by only having to develop on a single platform.

PathCanonicalize Versus What It Says On The Tin

This post contains MathML mark-up, which may not display correctly on some browsers. (And if you’re wondering how to do MathML in Blogger, see here.)

Here’s what canonicalisation is. You’ve got a set of items, and you can test those items for equality, $x=y$, but what you actually want to do is test for equivalence, $x\asymp y$ — where equality implies equivalence, but equivalence doesn’t imply equality. What’s needed is a canonicalisation function which maps the equivalence relation onto the equality relation, by mapping all (of each subset of) equivalent items onto a single representative item; a function $f$ such that $$f\left(x\right)=f\left(y\right)\iff x\asymp y$$ (plus you want canonicalisation to be idempotent: $f\left(f\left(x\right)\right)\equiv f\left(x\right)$ ).

More concretely, suppose you’re keeping a database of disk files, perhaps to enable searching or browsing. The question is, when do two filenames refer to the same file? You can’t just test them for string equality, as two distinct names might refer to the same file: on Unix, /home/peter/foo and /home/peter/src/../foo are the same file. Plus, symbolic links can be used to make even unrelated-looking names refer to the same file. If your database lists the file by one name, and someone does a query looking for another name, it won’t be found — and worse, the file could get into the database several times under different names, perhaps with conflicting or stale information.

But fortunately, “..”, “.”, and symlinks between them are about the size of it for Unix ways of obscuring the naming of a file, and the standard library comes with a suitable canonicalisation function that will reduce elaborated forms into the single unambiguous original pathname. (In fact, the GNU C library comes with two such functions, as the more-portable one, realpath(), needs a little care in use in order to avoid buffer-overrun attacks; the GNU one, canonicalize_file_name(), does not.) So you canonicalise all filenames as you store them into your database, and make sure to canonicalise all filenames in queries before you look them up, and you’ll get the right matches.

And then eventually you’re going to want to port that software to Windows — whereupon you’ve got a problem.

Indeed, you’ve got a whole little family of problems, because even once you’ve navigated the treacherous waters of the textual encoding of filenames under Win32, there still remain a bewildering variety of ways to refer to the same file:

D:\Music\Fools Gold.flac	— Probably canonical
D:/Music/Fools Gold.flac	— Slash versus backslash
D:\MUSIC\Fools Gold.flac	— Case-insensitive per locale
D:\Music\FOOLSG~1.FLA	— MS-DOS 8.3
M:\Fools Gold.flac	— After “subst M: D:\Music”
\Device\HarddiskVolume2\Music\Fools Gold.flac	— If D: is local
\\server\share\Music\Fools Gold.flac	— If D: is a network drive
\\?\UNC\server\share\Music\Fools Gold.flac	— Or like this
\\?\D:\Music\Fools Gold.flac	— Ultra-long-filenames mode
\\.\D:\Music\Fools Gold.flac	— Device namespace
\\?\UNC\D:\Music\Fools Gold.flac	— Allegedly
\\?\Volume{GUID}\Music\Fools Gold.flac	— Crikey

This whole dismal farrago really calls for a path canonicalisation function. Which is why it’s unfortunate that there isn’t one, and doubly unfortunate that there’s a function called PathCanonicalize() that particularly isn’t one, and not just because it’s spelled with a “Z”. All that PathCanonicalize() does is remove “/../” and “/./” substrings — it’s a purely textual transformation and doesn’t even touch the filesystem. It certainly doesn’t satisfy the “canonicaliser condition”:

$f\left(x\right)=f\left(y\right)\iff x$ and $y$ are the same file

No, there’s no shortcut for doing it laboriously and textually (nor for having lots of unit tests to cover all those ridiculous cases). The plan is, use GetFullPathName() to turn relative paths into absolute, then repeatedly call QueryDosDevice() to unwind subst’d drive letters, then call GetLongPathName() to get rid of 8.3-ness and canonicalise case, and then finally, if GetDriveType() says it’s remote, use WNetGetUniversalName() to convert the remaining drive letter into a UNC path.

std::string Canonicalise(const std::string& path)
{
    std::wstring utf16 = UTF8ToUTF16(path);

    wchar_t canon[MAX_PATH];

    /** Note that PathCanonicalize does NOT do what we want here -- it's a
     * purely textual operation that eliminates /./ and /../ only.
     */
    DWORD rc = ::GetFullPathNameW(utf16.c_str(), MAX_PATH, canon, NULL);
    if (!rc)
        return path;

    utf16 = canon;

    if (utf16.length() >= 6)
    {
        /** Get rid of \\?\ and \\.\ prefixes on drive-letter paths */
        if (!wcsncmp(utf16.c_str(), L"\\\\?\\", 4) && utf16[5] == L':')
            utf16.erase(0,4);
        else if (!wcsncmp(utf16.c_str(), L"\\\\.\\", 4) && utf16[5] == L':')
            utf16.erase(0,4);
    }

    if (utf16.length() >= 10)
    {
        /** Get rid of \\?\UNC on drive-letter and UNC paths */
        if (!wcsncmp(utf16.c_str(), L"\\\\?\\UNC\\", 8))
        {
            if (utf16[9] == L':' && utf16[10] == L'\\')
                utf16.erase(0,8);
            else
            {
                utf16.erase(0,7);
                utf16 = L"\\" + utf16;
            }
        }
    }

    /** Anything other than UNC and drive-letter is something we don't
     * understand
     */
    if (utf16[0] == L'\\' && utf16[1] == L'\\')
    {
        if (utf16[2] == '?' || utf16[2] == '.')
            return path; // Not understood

        /** OK -- UNC */
    }
    else if (((utf16[0] >= 'A' && utf16[0] <= 'Z')
              || (utf16[0] >= 'a' && utf16[0] <= 'z'))
             && utf16[1] == ':')
    {
        /** OK -- drive letter -- unwind subst'ing */
        for (;;)
        {
            wchar_t drive[3];
            drive[0] = (wchar_t)toupper(utf16[0]);
            drive[1] = L':';
            drive[2] = L'\0';
            canon[0] = L'\0';
            rc = ::QueryDosDeviceW(drive, canon, MAX_PATH);
            if (!rc)
                break;
            if (!wcsncmp(canon, L"\\??\\", 4))
            {
                utf16 = std::wstring(canon+4) + std::wstring(utf16, 2);
            }
            else // Not subst'd
                break;
        }

        wchar_t drive[4];
        drive[0] = (wchar_t)toupper(utf16[0]);
        drive[1] = ':';
        drive[2] = '\\';
        drive[3] = '\0';

        rc = ::GetDriveTypeW(drive);

        if (rc == DRIVE_REMOTE)
        {
            DWORD bufsize = MAX_PATH;

            /* QueryDosDevice and WNetGetConnection FORBID the
             * trailing slash; GetDriveType REQUIRES it.
             */
            drive[2] = '\0';

            rc = ::WNetGetConnectionW(drive, canon, &bufsize);
            if (rc == NO_ERROR)
                utf16 = std::wstring(canon) + std::wstring(utf16, 2);
        }
    }
    else
    {
        // Not understood
        return path;
    }

    /** Canonicalise case and 8.3-ness */
    rc = ::GetLongPathNameW(utf16.c_str(), canon, MAX_PATH);
    if (!rc)
        return path;

    std::string utf8 = UTF16ToUTF8(canon);
    std::replace(utf8.begin(), utf8.end(), '\\', '/');
    return utf8;
}

There are still ways to fool this function: for instance, by exporting a directory as \\server\share1 and a subdirectory of it as \\server\share2 — the client has no way of matching them up. But that’s a pretty pathological case, and it could be easily argued that it’s something you’d never do unless you wanted the shares to appear to be distinct. More seriously, the server for a network drive can be specified by WINS name, by FQDN or by IP address; neither canonicalise-to-IP nor canonicalise-to-FQDN is the Right Thing in all cases. For now I’m sweeping that issue under the carpet.

The one remaining wrinkle is that, unlike earlier versions of Windows, Vista allows “genuine” Unix-like symbolic links. Without doing real testing on Vista, it’s hard to make out from the documentation how the APIs used above behave when faced with such symbolic links. It’s even possible that the new-in-Vista GetFinalPathNameByHandle() call is the answer to all these problems; in which case, this code gets demoted to merely the way to do it on pre-Vista versions.

Unicode Is The One True God, And UTF-8 Is His Prophet

One common anti-pattern in the Win32 API is the one whereby something is done wrong the first time, then later re-implemented properly — but, for compatibility reasons, the older, broken implementation can never actually be removed or fixed. (The uncharitable might say that another common anti-pattern is the same thing, but where the re-implementation doesn’t work either.)

An example of one or both of these anti-patterns is encoding support. The Win32 API’s Unicode support was invented before UTF-8 was, so Microsoft can’t exactly be blamed for not using UTF-8. Instead they used UCS-2 (later retconned to UTF-16), but, with 16-bit characters, this was grievously incompatible with Win16 — and, indeed, with all other C source in the world at that time. So they kept an 8-bit-character “ANSI” API around as well, compatible with the 8-bit code-pages of Win16.

Without the same compatibility millstone around its neck, Linux (eventually) embraced UTF-8 into its locale system, to the extent that nowadays it’s reasonable to describe all non-UTF-8 locales as “legacy”. When writing portable software (location-portable as well as system-portable) for Linux, it’s quite the natural thing to do to store all text in UTF-8, and, if happening to find oneself running in a legacy locale, to translate to UTF-8 on the way in and from UTF-8 on the way out. That way, almost all the code can just deal with normal char* and assume it’s encoded in UTF-8.

And then eventually you’re going to want to port that software to Windows — whereupon you’ve got a problem.

Indeed, you have a whole little family of problems, and they’ve got names such as fopen, rename, unlink, and more. All of these calls — and those are just the ISO C ones; Posix and C++ add more — take 8-bit strings as filename arguments, and, in Windows, there are some files that this just won’t let you see. Files in Windows filesystems have UTF-16 names, which, when you use 8-bit calls, are mapped down into the characters available in the current ANSI code-page. (Strictly speaking, the code-page set as the “Code-page for non-Unicode applications” in Control Panel.)

So suppose you have your code-page for non-Unicode applications set to CP1252 for generic Western European, and your Greek friend, who has his code-page for non-Unicode applications set to CP1253 for Greek, writes two files with Greek names onto a USB stick and hands it to you. The filenames will be stored as UTF-16, featuring characters which do not exist in your code-page. And if you have two files whose names are only distinguishable by characters not in your code-page, you can’t distinguish them — they’ll both come back as, say, “????.txt”, and, while there’s enough grody hacks in Windows that fopen will actually open “????.txt”, that still only gets you one of the two files. There’s no way of opening the other one using 8-bit calls.

There’s only a few things you can do about this. The easiest, certainly, is to ignore it completely — after all, most people only create files named in their own language. This is the position taken by Cygwin (before 1.7, anyway) and thus, implicitly, by all Unix software which has been ported to Windows by simply recompiling it against Cygwin mentioning no names. It is, of course, very unsatisfactory — for instance, among non-Czechs partial to MP3s of music composed by Antonín Dvořák.

Alternatively, you could go through all your code replacing all the strings used to keep filenames in with wide-strings, all the calls to fopen with _wfopen, and so on — or use all the ugly macros such as _tfopen (which was the Microsoft advice even for Windows-only programs, back in the day when Windows 95 was Win32 but not Unicode, so it was common to make two binary builds of the same sources). But this is a lot of work — and error-prone, particularly if you rely on other libraries that have been ported to Windows using the ignore-the-problem technique, and you thus have to go and update them too.

Fortunately, there’s another way of solving the problem — at least, if you’re targetting Windows using Mingw and are thus using the GNU binutils linker. If you add

-Wl,--wrap,fopen -Wl,--wrap,rename -Wl,--wrap,unlink … etc.

to the linker command-line, all calls to (for instance) fopen will be converted into calls to __wrap_fopen, which you can implement in your own code. In particular, you can make it do something like this:

extern "C" FILE* __cdecl __wrap_fopen(const char *path, const char *mode)
{
    std::wstring wpath, wmode;
    util::UTF8ToWide(path, &wpath);
    util::UTF8ToWide(mode, &wmode);
    return _wfopen(wpath.c_str(), wmode.c_str());
}

extern "C" int __cdecl __wrap_rename(const char *oldname, const char *newname)
{
    std::wstring woldname, wnewname;
    util::UTF8ToWide(oldname, &woldname);
    util::UTF8ToWide(newname, &wnewname);
    return _wrename(woldname.c_str(), wnewname.c_str());
}

(You don’t, of course, get prototypes for __wrap_fopen or __wrap_rename, so you must carefully ensure to give them the same signatures as the genuine functions.) This way, all your code can carry on using UTF-8 filenames everywhere, passing them to system calls such as fopen, and even passing them to external libraries, so long as you’ve wrapped the calls the libraries themselves make. (Which you can check by asking the linker for a map file: -Wl,-Map,map.txt.)

Naturally, there’s no point using ANSI Win32 APIs in Windows-specific parts of your code, only to then wrap them — it only makes sense there to go straight for the UTF-16 ones (with the W on the end, such as RegQueryValueExW) to start with. This wrapping technique is a way of avoiding rewriting the existing portable parts of your code.

It’s like having an ANSI code-page of UTF-8 all to your very own.

One Template To Rule Them All

How do you write a template that can be instantiated for any type? Here’s a header file which purports to implement a tracing (logging) mechanism...

#include <stdio.h>

class Tracer {};

inline const Tracer& operator<<(const Tracer& r, int i)
{ printf("%d", i); return r; }

inline const Tracer& operator<<(const Tracer& r, const char *s)
{ printf("%s", s); return r; }

template <class T>
inline const Tracer& operator<<(const Tracer& r, T *p)
{ printf("%p", p); return r; }

class NullTracer {};

template <typename T>
inline const NullTracer& operator<<(const NullTracer& n, T) { return n; }

#ifdef DEBUG
#define TRACE Tracer()
#else
#define TRACE NullTracer()
#endif

...and here’s some example client code:

class X
{
    enum { A, B } m_state;

    void Foo() { TRACE << "x" << this << " is in state " << m_state << "\n"; }

public:
    X();
};

(I used this rather nifty script to do the syntax-colouring.)

What’s going on here is that, in debug builds (with -DDEBUG on the compiler command-line), TRACE expands to an expression that creates an object that acts a bit like std::cout, in that various different types of object can be sent to it and each printed in a sensible and type-safe manner. Just as with std::cout, the precedence and associativity of the << operator get everything printed in the right order.

In release builds, meanwhile, TRACE expands to an expression that creates an object of a different class, whose operator<< ensures that whatever type is sent to it, nothing is printed — and, indeed, the entire expression optimises away to nothing.

Except there’s a problem. The code above doesn’t actually compile in release builds (without -DDEBUG). Here’s what GCC 4.3.3 says about it:

wtf.cpp: In member function ‘void X::Foo()’:
wtf.cpp:30: error: no match for ‘operator<<’ in ‘operator<< [with T = const char*](((const NullTracer&)((const NullTracer*)operator<< [with T = X*](((const NullTracer&)((const NullTracer*)operator<< [with T = const char*](((const NullTracer&)((const NullTracer*)(& NullTracer()))), ((const char*)"x")))), this))), ((const char*)" is in state ")) << ((X*)this)->X::m_state’
wtf.cpp:10: note: candidates are: const Tracer& operator<<(const Tracer&, int)
wtf.cpp:13: note: const Tracer& operator<<(const Tracer&, const char*)

So naturally I knew at once what the problem was. Not. The error suggests that it can’t find an operator<< to do the job — but surely there’s one right there that can do any such job. Yet it isn’t even listed as a candidate.

While composing my GCC Bugzilla entry in my head, I tried to whittle down the problem. To my surprise, it was the enum which was causing the problem: trace statements with only strings and pointers compiled just fine. (Those experienced with what Abrahams and Gurtovoy call “debugging the error novel” might just about be able to make that out from the error message.)

So what’s up with m_state? What’s so very special about its type that it isn’t matched by template<typename T>? Well, there is one special thing about its type. It’s unnamed. When I wrote m_state into that code, neither its declaration, nor any of its uses, actually needed to name its type — so I applied Occam’s Razor and didn’t give it a name. And it turns out that that’s the problem: you can’t instantiate a template with an unnamed type as a template-argument. I can’t find a statement to that effect in Stroustrup-3, but it’s right there in the ISO C++03 standard ISO/IEC 14882:2003 at 14.3.1 [temp.arg.type]:

A local type, a type with no linkage, an unnamed type or a type compounded from any of these types shall not be used as a template-argument for a template type-parameter.

This does make perfect sense when you think about it a bit more: if templates could be instantiated with unnamed types, how would their names be mangled? Or, in other words, how could the compiler and linker conspire to arrange that multiple instantiations of NullTracer<unnamed-enum-type>, in multiple translation units, ended up as one instantiation in the final program? Without name matching, there’d be no way to do that — at least within the constraints that the C++ committee set for themselves, specifically that C++ compilers be usable with C-era linkers by dint of name mangling. The same mangling considerations are probably also the reasoning behind the ban on string literals as template arguments.

And the reason it works in non-debug builds, with the real Tracer, is that in that case no template is instantiated with m_state’s actual type: integral promotion makes it an int, and the overload for int is called. Nor can you fix up NullTracer by adding a template specialisation for int: the template specialisation rules aren’t quite the same as the overload resolution rules, and the compiler is doomed to pick the generic operator<<, and have it fail, before it tries the integral promotions. (SFINAE doesn’t help us, as again it is applied only with the enum type, not the promoted type.)

So what's needed is another overload, one that catches unnamed enum types:

inline const NullTracer& operator<<(const NullTracer& n, int) { return n; }

And now everything works as expected: unnamed enums (and, in fact, ints too, not that it matters) are passed via the non-templated overload, and everything else goes via the templated overload.

Debugging The Invisible

The Empeg codebase, and the Empeg coding style for embedded C++, don’t use C++ exceptions. The original reason was that ARM support in GCC was still a bit sketchy when Empeg started, and exceptions plain didn’t work.

But I think that even with a toolchain in which C++ exceptions work properly, they should still be avoided, and not just in embedded systems either. Here’s why.

Exception safety

You need to write your code in exception-safe style: a style which neither the compiler, nor other tools, can enforce. Some would say that “write exception-safe code” is just another way of saying “don’t write buggy code”, or “don’t be a crap programmer”, but many other ways of saying that — “write const-correct code”, “don’t leak memory” — can be automatically checked, and thus a unit-test suite (perhaps with Valgrind) can give you great confidence that your code fulfils those criteria, and remains in fulfilment under subsequent maintenance. You can’t easily unit-test for exception safety.

Another consideration is that many, perhaps most, C++ developers used to be C developers. Most things these developers need to learn when switching to C++, are completely new things. But exception-safety is something they need to learn which changes the way they’d write perfectly reasonable straight-line C code, so that’s always going to be trickier for everyone.

Scott Meyers didn’t even bother writing exception-safe code until the third edition of Effective C++ (search for “dfc”).

Invisible control flow

The Empeg C++ coding style, like others, forbade non-const reference parameters. The reason is, that doing so forces by-reference objects which may be modified to go via pointers instead. This, in turn, forces call sites to use “&” when a function may modify the value, and no “&” when it doesn’t. This puts this essential information about the data flow at the call site, not just at the declaration — and, when debugging, it’s usually the call site and not the declaration that you’re looking at. So banning non-const references improves the understandability of the data flow by making it visible.

Exceptions are, in a way, even worse than non-const references: they bestow on your code unasked, not just data flow you can’t see, but control flow you can’t see. Consider, in particular, a function that doesn’t contain a try or a throw, but calls functions which do (or may do). There’s a second control flow through all that code, which you didn’t write — the one where something below it in the call-chain throws, and something above it catches — but which has a crucial effect on whether your code is buggy or not.

And again, you can’t unit-test for it: even if you use gcov to test that your tests cover the normal control flow of your code, it can’t tell you whether you’ve also exercised all the invisible, secondary control flows. (In theory, a much smarter gcov-like tool could, though.)

Joel agrees.

Speed, size, exceptions: pick two

In D&E, Stroustrup that claims a design goal for C++ exceptions was “No added cost (in time or space) to code that does not throw an exception” (16.2), but later on in 16.9.1 he admits that people have figured out how to avoid cost in time in the no-exception case, but not cost in space.

A medium-sized C++ program I have here (amd64-linux, GCC 4.3.2) has a text section of 548K and exception-handling tables of 127K. But even that 23% increase isn’t the whole story, as is revealed by disassembly: that try-less, throw-less, but non-leaf function — if it has local variables with destructors — also gains snippets of unwinding code for each such variable. These snippets appear in the text section, so can’t easily be totalled-up, but an unscientific test on one source file suggests that they make up about 7% of the text section.

These exception-handling tables do affect embedded systems more than desktops or servers — on which the exception-handling tables are never demand-paged in unless they’re used — but the unwinding snippets are still there affecting your paging bandwidth and filling your instruction cache.

But you don’t get to choose

In a sense, choosing which of the two distinct languages — C++-with-exceptions or C++-without-exceptions — to use to program your embedded system, is a choice you only get to make when starting the system from scratch. Or, at least, converting a codebase from C++-without-exceptions to C++-with-exceptions is a herculean and error-prone task, and a half-converted one is a bug tarpit, because of the exception-safety issues. If the actual goal is to use a library written in C++-with-exceptions with a main program written in C++-without-exceptions, the best way is to write wrappers for each library function, that catch(...) and report errors through the existing non-exception error handling mechanism. With a bit of luck you can then continue to compile the main program with -fno-exceptions (or similar) and avoid the space overhead.

Even this doesn’t really work when the library is something like Boost, which is chock-full of throws in headers that really you need to be including in any code that uses the library. (Including for things like locking a non-recursive mutex that’s already locked, which in my book is more like a programming error than an exceptional condition, and would be better as an assertion. But that’s a different rant.)

Autoconf, #ifdef, and #if

The standard macros available in GNU Autoconf for “checking for...” C/C++ functions and header files, AC_CHECK_FUNCS and AC_CHECK_HEADERS, have a couple of things going for them: their syntax and semantics essentially haven’t changed for years, perhaps ever, and everyone — well, everyone who writes configure scripts, which if it isn’t you should be a big hint as to how much you’ll get from this blog — already knows how they work. Which is to say, they end up embedding in your generating header, typically config.h, lines which look like this:

/* Define to 1 if you have the <linux/inotify.h> header file. */
/* #undef HAVE_LINUX_INOTIFY_H */

/* Define to 1 if you have the `localtime_r' function. */
#define HAVE_LOCALTIME_R 1

That’s all you need to start making your code conditional on which facilities are available:

#include "config.h"
    [...]
#ifdef HAVE_LOCALTIME_R
    localtime_r(&start, &stm);
#else
    // This code isn't thread-safe and could
    // get the wrong time -- but the result
    // would just be an odd-looking filename, not 
    // incorrect operation, so we live with it
    stm = *localtime(&start);
#endif

But there’s a problem. In fact, there’re two problems. One of the best features of C and C++, one of the few things that lets the old guard lord it over today’s arriviste scripting languages, the pythons and the rubies, is that almost everything is checked at compile-time. If you mistype an identifier, say localtime_r, as localtimer or whatever else, your code won’t compile, even if it’s on a rarely- or never-used path through the code. Unless you’re unlucky enough to hit the exact name of another existing identifier (and even then the type system may save you, unless you’re unlucky again), your mistyping is caught right there as you’re writing the code, not in QA — or, horrors, in production.

The same can’t be said, though, of identifiers in #ifdef. It’s perfectly valid to use a name there that doesn’t exist — nor could it be otherwise, as there’s nothing else it does except distinguish existent from non-existent names. So if you fat-finger it — HAVE_LOCALTIMER, maybe — the code compiles without even a warning, and just does the wrong thing, or at least the less-optimal thing. Even into production.

The second problem is config.h itself: if you forget to #include it, again your code compiles without a warning, but all your features get turned off! There are ways around this, such as by having a little script that checks your sources using grep, or by using GCC’s -include command-line option to force the inclusion, but none is ideal. (For instance, the latter gets dependencies wrong if config.h changes.)

Really what you want is warnings at compile-time if you mistype the identifier, or forget the header. In order to do this, you’re going to need to use #if rather than #ifdef; in other words, you’re going to need a config.h with lines like this:

/* Define to 1 if you have <linux/inotify.h>, or to 0 if you don't. */
#define HAVE_LINUX_INOTIFY_H 0

/* Define to 1 if you have localtime_r(), or to 0 if you don't. */
#define HAVE_LOCALTIME_R 1

And to do that, you’re going to have to use custom Autoconf macros instead of the standard AC_CHECK_ ones. Which is the point at which most people would give up, as Autoconf macros expand into shell code but are themselves written in the macro language m4, whose syntax can be charitably described as quixotic, and is regularly described using rather shorter and more Anglo-Saxon words than that. So here, I’ve done it for you, have these ones:

AC_DEFUN([PDH_CHECK_HEADER], [
AC_MSG_CHECKING([for <$1>])
AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <$1>
]])],
          [AC_MSG_RESULT([yes]); pdh_haveit=1; pdh_yesno="yes"],
          [AC_MSG_RESULT([no]);  pdh_haveit=0; pdh_yesno="no"])
AC_DEFINE_UNQUOTED(AS_TR_CPP([HAVE_$1]), $pdh_haveit,
          [Define to 1 if you have <$1>, or to 0 if you don't.])
AS_TR_SH([ac_cv_header_$1])=$pdh_yesno
])

AC_DEFUN([PDH_CHECK_HEADERS], [
m4_foreach_w([pdh_header], $1, [PDH_CHECK_HEADER(pdh_header)])
])

AC_DEFUN([PDH_CHECK_FUNC], [
AC_MSG_CHECKING([for $1()])
AC_LINK_IFELSE([AC_LANG_PROGRAM([], [[(void)$1();]])],
        [AC_MSG_RESULT([yes]); pdh_haveit=1; pdh_yesno="yes"],
        [AC_MSG_RESULT([no]);  pdh_haveit=0; pdh_yesno="no"])
AC_DEFINE_UNQUOTED(AS_TR_CPP([HAVE_$1]), $pdh_haveit,
        [Define to 1 if you have $1(), or to 0 if you don't.])
AS_TR_SH([ac_cv_func_$1])=$pdh_yesno
])

AC_DEFUN([PDH_CHECK_FUNCS], [
m4_foreach_w([pdh_func], $1, [PDH_CHECK_FUNC(pdh_func)])
])

That bracketfest defines macros PDH_CHECK_HEADERS and PDH_CHECK_FUNCS which do the same job as the standard AC_CHECK_HEADERS and AC_CHECK_FUNCS...

PDH_CHECK_HEADERS([io.h poll.h errno.h fcntl.h sched.h net/if.h stdint.h stdlib.h unistd.h shlwapi.h pthread.h ws2tcpip.h sys/poll.h sys/time.h sys/types.h sys/socket.h sys/syslog.h netinet/in.h netinet/tcp.h sys/inotify.h sys/resource.h linux/unistd.h linux/inotify.h linux/dvb/dmx.h linux/dvb/frontend.h])

PDH_CHECK_FUNCS([mkstemp localtime_r setpriority getaddrinfo inotify_init gettimeofday gethostbyname_r sync_file_range sched_setscheduler])

...but leave you with identifiers in config.h that you can use #if on rather than #ifdef, so you can enjoy compile-time warnings if you get any of them wrong.

Ah, except you can’t just yet, because there’s one final wrinkle. The C and C++ standards allow undefined identifiers in #if just as in #ifdef — they don’t expand to nothing, giving the syntax errors you’d initially hope for, but are defined as evaluating to zero. So, back to square one. Except, fortunately, the authors of GCC appear to be as sceptical about the usefulness of that preprocessor feature as I am, and they provide an optional warning which can be emitted whenever the preprocessor is obliged to evaluate an undefined identifier as zero: -Wundef. So, if you use the configury macros above, and add -Wundef to your GCC command-lines, you can at last get the compiler to point it out to you when you mistype your configure macros.

When I wrote these new macros, I did have a nagging feeling that it was all just NIH syndrome, and that it was silly makework to be rewriting core bits of Autoconf for this marginal benefit. But once I’d done it, re-compiling Chorale showed up two warnings, both places where a mistyped identifier was causing real and otherwise hard-to-find problems. So I felt retrospectively justified in my tinkering. The moral of the story: if you’ve fixed a bug, you’ve done a day’s work; but if you’ve fixed a bug by ensuring that it, and its whole class of fellow bugs, can never happen undetected again — then, you’ve done a good day’s work.