This week’s puzzle, Enigma number 1638, is in that category.  Here are the facts of the problem.

• I am thinking of three numbers.
• Each number is a perfect square.
• Each number has five digits.
• Five different digits occur in all three numbers.
• Each of the five digits occurs a different number of times.
• The five numbers of times is the same as the five digits.
• No digit occurs its own number of times.  (E.g., 4 is not used 4 times.)
• If you knew which digit occurs once, you could uniquely identify the numbers.

Your task is to write a program that finds and prints the three perfect squares I am thinking of.

I chose to use Python 3 to solve this problem.

The brute force solution is to enumerate all the perfect squares with five digits.  Then try them in all combinations taken three at a time.

from itertools import combinations, count, takewhile

squares = filter(lambda s: len(s) == 5,
                 takewhile(lambda s: len(s) <= 5,
                           (str(s**2) for s in count())))

for a, b, c in combinations(squares, 3):
    # ...

Each trio of numbers has to meet three criteria.

C1. five different digits occur in the trio.
C2. each digit occurs a different number of times (has a different frequency).
C3. the five frequencies are the same as the five digits.

A digit frequency histogram would help with all of those.  Let’s define a function to build it.

from collections import defaultdict

def histogram(s):
    d = defaultdict(int)
    for c in s:
        d[c] += 1
    return d

That returns a dictionary mapping each digit to the number of times it occurs.  For example, hist(’31415′) would return {1: 2, 3: 1, 5: 1}, meaning 1 occurs twice and 3 and 5 each occur once.

Now we can test for the three criteria like this.

    hist = histogram(a + b + c)
    digits = set(hist.keys())
    frequencies = set(hist.values())
    if   (len(digits) == 5 and                     # five different digits
          digits == frequencies and                # digits == frequencies
          not any(hist[k] == k for k in hist)):    # k does not occur k times.
        # then a, b, c meet the criteria.

We need to find all trios that meet the criteria, then find the one whose singleton digit is unique.  So we’ll build a map from singleton digit to the set of trios with that singleton.

matches = defaultdict(list)
for a, b, c in combinations(squares, 3):
    # ...
    if «criteria met»:
        inverse_hist = dict((hist[k], k) for k in hist)
        singleton = inverse_hist[1]
        matches[singleton].append((a, b, c))

Now we need to find the entry in matches that has length one.  We’ll do that by creating yet another map, this time from number of matches to the list of matches.  Then we can extract the answer directly.

match_counts = dict((len(ls), ls) for ls in matches.values())
print(*match_counts[1][0])

Here is the whole program.

#!/usr/bin/python3

from collections import defaultdict
from itertools import combinations, count, takewhile

def histogram(s):
    d = defaultdict(int)
    for c in s:
        d[int(c)] += 1
    return d

squares = filter(lambda s: len(s) == 5,
                 takewhile(lambda s: len(s) <= 5,
                 (str(s**2) for s in count())))

matches = defaultdict(list)
for a, b, c in combinations(squares, 3):
    hist = histogram(a + b + c)
    digits = set(hist.keys())
    frequencies = set(hist.values())
    if    (len(digits) == 5 and
           digits == frequencies and
           not any(hist[k] == k for k in hist)):
        inverse_hist = dict((hist[k], k) for k in hist)
        singleton = inverse_hist[1]
        matches[singleton].append((a, b, c))

match_counts = dict((len(ls), ls) for ls in matches.values())
print(*match_counts[1][0])

A note on efficiency.  This programs runs for about 30 seconds on my computer.  You can reason about the problem and reduce the amount of work it does by a factor of 10,000 or so.  I’ll leave that as an exercise. (-:

The focus has been Scheme macros and separate compilation.  I’m trying to understand how a production quality Scheme system “should” be organized.  “Should” is in scare quotes because every Scheme user has at least one opinion on the topic, and no two seem to agree.

So here are the papers I’ve looked at in the last few days.

You have to love a ‘blog called Axis of Eval.  This post is a semi-rant requesting that people tackle the hard problems of Lisp language design instead of creating toy subsets.  It includes a number of good pointers to other papers on macros, and this is where I found most of these links.  I only followed the links on macros; I hope the other links are just as good.

This 2002 paper explains why separation of compile-time and run-time phases is a good idea.  Then it describes PLT Scheme’s macro and module system.

Also from the Axis of Eval.  This is the simplest explanation of hygiene I’ve ever seen.  I didn’t find Part 2; maybe it hasn’t been written yet.

SRFI is pronounced surfy, but it stands for Scheme Request For Implementation. They’re a series of mini-standards proposals for new language features.  Similar to Python’s PEPs.

SRFI 72 is a macro system very similar to what ended up in R6RS.  In fact, the reference implementation (see below) has been renamed “R6RS macros”.  Defines a consistent set of hygiene rules and a set of primitives that are both useful by themselves and a sufficient basis to implement syntax-case, with-syntax, and syntax-rules.

Further links to van Tonder’s macros:

Those will take a long time to chew through.

If this article is any indication, I should read Simionato’s whole series.  Here, he talks about the Tower of Reflection, and he gives an example of how easy it is to write macros that fail by mixing phases.  (That is a common theme in these articles.  It’s very easy to screw up.)

Then he gives an example where an R6RS library needs to export names at a negative meta level.  Which is only surprising until you realize it’s a negative level relative to the library’s own level.

Lisp in Small Pieces has been a great text, explaining difficult concepts thoroughly.  But the macro chapter was a letdown.  I think the problem is that at the time of publication (1994), macros weren’t fully understood yet.  The Scheme community certainly hadn’t reached any kind of consensus on them.

An introduction to our favorite object file format.  I’ve been wondering whether it would be a suitable container for compiled Scheme bytecode files.  I haven’t read enough of it to have an opinion yet.  I’m also not entirely sure what a compiled Scheme object file needs to contain.

I haven’t read this one yet.  It’s old, written in 1988, but it’s cited a lot.  And the title is cool.

So.  What have I learned from all of this?  I believe I’ve been convinced that the separate phases model of compilation is the way to go.  I think that with careful study of SRFI 72 and its reference implementation, I can implement a useful macro expander.  I think that a bytecode object file format is attainable, though I don’t have a clear direction to follow yet.  And I’m still uncertain how to bootstrap the process.  How much of the system do you have to build in the bootstrap language, and how much can you write in Scheme?

C has a poverty-stricken exception mechanism called setjmp/longjmp.  setjmp saves the current machine registers in a structure called a jmp_buf.  Then it returns 0.  When longjmp is called, it restores the registers to their saved values, and returns whatever you passed as its second argument.  Except that because it restored the stack pointer, the place it returns to is setjmp‘s caller, not longjmp‘s.  So setjmp‘s caller can check the return value to know whether setjmp returned normally or longjmp has raised an exception.

(You see what they did there?  jmp_buf is spelled with a u.  So much for the theory that the guy who designed this stuff had no U key on his keyboard.  Hey, it was the 1970s.  Sideburns were long, and identifiers were short.  They would have needed $10,000 for an additional kilobyte of core memory if they used one vowel too many.)

Here is an example of setjmp and longjmp.

jmp_buf foo;

void f()
{
   if (the_moon_is_full())
    longjmp(foo, 1);
}

void main()
{
    if (setjmp(foo) == 0)
        printf("I just called setjmp\n");
    else
        printf("Got here from longjmp\n");
    f();
}

It doesn’t really work very well because compilers do far more optimizations than they did in 1977 when setjmp/longjmp were invented.  Here’s what the C99 standard says.

An invocation of the setjmp macro shall appear only in one of the following contexts:

• the entire controlling expression of a selection or iteration statement;
• one operand of a relational or equality operator with the other operand an integer constant expression, with the resulting expression being the entire controlling expression of a selection or iteration statement;
• the operand of a unary ! operator with the resulting expression being the entire controlling expression of a selection or iteration statement; or
• the entire expression of an expression statement (possibly cast to void).

If the invocation appears in any other context, the behavior is undefined.

In other words, you can use setjmp in one of these ways.  If you do anything else with it, your program may not work.

if (setjmp(...)) ...

while (setjmp(...)) ...

switch (setjmp(...)) ...

if/while (setjmp(...) == N) ...

if/while (!setjmp(...)) ...

(void)setjmp(...);

So that’s setjmp.  Now back to Schetoo.

Schetoo’s interpreter keeps its state in two registers between instructions: cont and values.  It’s not important to understand how the registers are used, just that each instruction modifies them, and instructions communicate primarily through these two registers.

We have a structure that holds the two registers, cv_t.

typedef struct cv {    /* cont and values */
    obj_t cv_cont;
    obj_t cv_values;
} cv_t;

And we have two static root pointers that contain the registers’ current values.  Note that these are only valid during a garbage collection.  Most of the time they hold garbage values.

obj_t cont_root   = UNINITIALIZED_OBJ;
obj_t values_root = UNINITIALIZED_OBJ;

Here is the main loop of the interpreter.  cont_proc() returns a pointer to the C function that implements the next instruction.  That function is called with the two registers as arguments, and it returns the two registers’ new values in a cv_t structure.  Then the registers are copied back, the instruction is committed (debug build only, see previous post), and we go on to the next instruction.

while (!is_null(cont)) {
    cv_t ret = cont_proc(cont)(cont, values);
    cont   = ret.cv_cont;
    values = ret.cv_values;
    COMMIT();
}

Now let’s look at the whole function that implements the main loop.  (I’ve simplified it for readability.)  It starts with handlers for two kinds of exceptions, then falls into the main loop.

Notice how the two registers are copied into the root pointers before GC, then copied back afterwards.

obj_t core_eval_cont(volatile obj_t cont, volatile obj_t values)
{
    switch (setjmp(eval_restart)) {

    case 0:
        break;

    case LT_THROWN:
        {
            cv_t ret = push_exception(cont, values);
            cont     = ret.cv_cont;
            values   = ret.cv_values;
        }
        break;

    case LT_HEAP_FULL:
        cont_root   = cont;
        values_root = values;
        collect_garbage();
        cont        = cont_root;
        values      = values_root;
        cont_root   = UNINITIALIZED_OBJ;
        values_root = UNINITIALIZED_OBJ;
        break;
    }

    while (!is_null(cont)) {
        cv_t ret = cont_proc(cont)(cont, values);
        cont   = ret.cv_cont;
        values = ret.cv_values;
        COMMIT();
    }
    return CAR(values);
}

So there it is.  The skeleton of an interpreter where any instruction can be failed or restarted, and the need for GC is treated as an exception.

What is the overhead?  Every instruction is invoked through an indirect function call.  The registers are copied three times for every instruction.  (Fortunately, there are only two registers.)  In exchange, there is no bookkeeping overhead for local root pointers, and each exception only requires a single test and branch in the case where it isn’t raised.

But more importantly, it’s easier to write correct code.

In the last post, I proposed Rules 1, 2 and 2a.

Rule 1.  Any instruction of the interpreter may be abandoned and restarted.

Rule 2.  Do everything that might fail before doing anything with side effects.

Rule 2a.  “Side effects” only means memory that was already reachable before the current instruction (and I/O).

Schetoo allocates memory sequentially.  The pointer next_alloc points to the beginning of free memory to allocate.  Here’s the allocator, somewhat simplified.

void *next_alloc, *alloc_end;

heap_object_t *mem_alloc_obj(size_t alloc_size)
{
    if (next_alloc + alloc_size > alloc_end)
        send_heap_full();      /* calls longjmp() */
    heap_object_t *fresh_object = next_alloc;
    next_alloc += alloc_size;
    return fresh_object;
}

That means that memory allocated by the current instruction is contiguous.  Before each instruction starts, the interpreter calls a macro called COMMIT(), which saves a copy of next_alloc in a variable called committed.  Any object whose address is between committed and next_alloc must have been freshly allocated in the current instruction.  As the name implies, COMMIT() commits the freshly created objects after the instruction successfully completes.

When memory allocation fails, we raise an “exception”.  C doesn’t have real exceptions, but the setjmp/longjmp facility is good enough.

An instruction has two phases: Can Fail and Can’t Fail.  In the former, the instruction can check arguments and allocate memory, and any of those operations can raise an exception that fails the instruction.  As soon as the instruction has a side effect, it moves into Can’t Fail phase.

Schetoo has a global flag, retry_okay, that records which phase the current instruction is in.  (The name is a little misleading.  Think of it as failure_okay, since some failures don’t lead to retry.)

Side effects occur in a small number of places.  Each of those places sets the current phase to Can’t Fail and records the current function and line number.  This is done through a macro, SIDE_EFFECT().

Object mutations use a higher-level macro, MUTATE(obj).  MUTATE calls SIDE_EFFECT, but only if the object is committed.

Failures can occur in an even smaller number of places.  One is the allocator.  Another is a macro called CHECK that is used for all argument checking.  Those call a macro called COULD_RETRY().  COULD_RETRY verifies that the instruction is in Can Fail phase, and if it isn’t, it raises an assertion error immediately, with the function and line number where the most recent side effect occurred. (Once again, the name isn’t right.  It should be called COULD_FAIL, I think.)

All of these checks happen in a debug build.  In a production build, the four macros, COMMIT, SIDE_EFFECT, MUTATE, and COULD_RETRY, are all redefined to do nothing, and no checking is done.  So there is no overhead at all in a production build.

I’ve mentioned that exceptions use longjmp().  In my next post, I’ll explain how the interpreter’s main loop catches them.

You can see them both on github.

https://github.com/kbob/kbscheme
https://github.com/kbob/schetoo

I undertook these projects to learn more about Scheme, interpreters, and garbage collection. Along the way, I came up with an interesting technique to make interpreter implementation less error-prone. That’s what I want to write about today.

Any interpreter with garbage collection needs some way to find all the live objects in the heap. That generally means they need to keep track of all the “root pointers”, pointers that live outside the heap but point into it. When you write C code, you generate a lot of root pointers.

For example, here is kbscheme’s implementation of string->list.  string->list is a primitive procedure that returns the characters in a string as a linked list.

DEFINE_PROC(L"string->list")
{
    AUTO_ROOT(str, pair_car(F_SUBJ));
    AUTO_ROOT(list, NIL);
    AUTO_ROOT(chr, NIL);
    size_t pos;
    for (pos = string_len(str); pos; --pos) {
        chr = make_character(string_value(str)[pos - 1]);
        list = make_pair(chr, list);
    }
    RETURN(list);
}

AUTO_ROOT is a macro that declares and initializes a local variable, then pushes it onto a global linked list of root pointers. RETURN pops all the roots allocated in the current function before returning.

That’s a lot of runtime overhead. That’s also a lot of noise in the source code. And root pointers were the source of several bugs where roots either were not recorded, contained invalid pointers at GC time, or were pushed onto the root stack more than once. Each bug was a heap corruption Heisenbug, of course, since memory is only corrupted if the garbage collector runs while the function is active, and it only shows up later when the object’s memory is reused. That is the very worst kind of problem to debug.

For those reasons, one of my goals for Schetoo was to eliminate manual tracking of root pointers. Here is the same procedure in Schetoo.

DEFINE_PROC(L"string->list", 1)(obj_t string)
{
    const char_t *chars = string_value(string);
    obj_t list = EMPTY_LIST;
    for (size_t pos = string_len(string); pos; --pos)
        list = make_pair(make_character(chars[pos - 1]), list);
    return list;
}

No root bookkeeping. No macros, aside from the weird function name.  A whole class of bugs eliminated. I’d like to say no runtime overhead, but that’s not quite true, as you’ll see.

Schetoo has a simple rule.

Rule 1. Any instruction of the interpreter may be abandoned and restarted.

In this example, if make_pair() triggers a garbage collection, it signals an exception using longjmp(), and string->list is restarted after the collection.

Schetoo uses the same mechanism for user exceptions. In this example, both string_value() and string_len() check that they are passed a string. If not, they raise an exception which also longjmps back to the interpreter’s main loop, so string->list doesn’t need any error checking logic.

Why is there no error checking the kbscheme example? In kbscheme, any error at all was fatal and terminated the whole interpreter. D’oh!

There’s more to it than that, of course. Now there is a new rule that every procedure has to follow:

Rule 2. Do everything that might fail before doing anything with side effects.

There are two kinds of side effects in Schetoo: modifying memory and I/O. We won’t talk about I/O here except to mention that we have to be careful to follow Rule 2 when we do it.

string->list doesn’t have any side effects, so it follows Rule 2 trivially.

What about all those calls to make_list()? Don’t they modify the heap? Aren’t they side effects? Yes, but they only modify freshly allocated memory. That doesn’t count, because with no root pointers, that memory will be instantly turned into garbage when the instruction is abandoned.

Rule 2a. “Side effects” only means memory that was already reachable before the current instruction (and I/O).

Is this an improvement? We have replaced kbscheme’s tricky rule, “Always track local variables that are root pointers”, with Rule 2, which also looks tricky.

Yes, this is better, because Schetoo can verify Rule 2 at runtime.  In my next post, I’ll explain how it works.

Here’s a cute hack to make variadic functions easier to write and a little harder to call incorrectly.

As a contrived example, let’s write a function that copies a tag and a sequence of floating-point arguments into a linked list allocated on the heap.  The first attempt might look like this.

#include <stdarg.h>

typedef struct node node;
    struct node {
    double datum;
    char   tag;
    node  *next;
};

node *link_list_A(char tag, int count, ...)
{
    node *head = NULL;
    node **patch = &head;
    int i;
    va_list ap;

    va_start(ap, count);
    for (i = 0; i < count; i++) {
        node *p  = malloc(sizeof *p);
        p->datum = va_arg(ap, double);
        p->tag   = tag;
        p->next  = NULL;
        *patch   = p;
        patch    = &p->next;
    }
    va_end(ap);
    return head;
}

You’d call it like this.

    node *p = link_list_A('A', 5, 1.0, 1.5, 2.0, 2.5, 3.0);

There’s just a little problem with that.  What happens if you go back later and change the arguments?  You have to remember to change the count.  Also, you might miscount and tell it it has 16 arguments when it really has 17.

Another approach is to use an end marker instead of a count.

# define ENDARG (-1.0)

node *link_list_B(char tag, ...)
{
    node *head = NULL, **patch = &head;
    double datum;
    va_list ap;

    va_start(ap, tag);
    while ((datum = va_arg(ap, double)) != ENDARG) {
        node *p  = malloc(sizeof *p);
        p->tag   = tag;
        p->datum = datum;
        p->next  = NULL;
        *patch   = p;
        patch    = &p->next;
    }
    va_end(ap);
    return head;
}

…

    node *p = link_list_B('B', 1.0, 1.5, 2.0, 2.5, 3.0, ENDARG);

That’s better.  But wouldn’t it be even nicer if we could eliminate ENDARG in the caller?  It’s a big wart on the readability of the program.

C99 has variadic macros as well as variadic functions.  By wrapping the function in a macro, the macro can supply the end marker instead of sprinkling it throughout the source code.

(Hey, C99 has it’s been ratified for eleven years.  Don’t look so scandalized.)

#define link_list_C(tag, ...) \
    (link_list_B(tag, __VA_ARGS__, ENDARG)) // link_list_B defined above

Now you call it like this.

    node *p = link_list_C('C', 1.0, 1.5, 2.0, 2.5, 3.0);

It’s not a big deal, but it makes the code a little more readable. And it’s an idiom I haven’t seen before.

I use the idiom several places in my Scheme interpreter, schetoo.  For example, Schetoo has a C macro, MAKE_LIST, which is analogous to Scheme’s list procedure.

Scheme:

(list 'a 'b (list 'c))
=> (a b (c))

C:

obj_t a = make_symbol_from_C_str(L"a");
obj_t b = make_symbol_from_C_str(L"b");
obj_t c = make_symbol_from_C_str(L"c");
MAKE_LIST(a, b, MAKE_LIST(c));
=> (a b (c))

It’s a pretty good analogue, I think.

The source for schetoo is on github.

A ‘blogger named Peter Michaux started a series in January called Scheme from Scratch. His goal was to write a very quick and dirty Scheme, just enough to bootstrap a Scheme compiler. I read his postings and decided to get started again.

Schetoo is similar in concept to my first Scheme, kbscheme. In fact, I reused a fair amount of code to get Schetoo up and running faster. But at the same time, I’m applying what I learned to try to get a better design.

I’ve published the code on GitHub.

http://github.com/kbob/schetoo

Goals

The primary goal for Schetoo is pedagogical. I want to understand the Scheme language and how to design language runtimes. I intend to focus on performance more than last time, because kbscheme was running for many seconds just initializing itself.

Another goal is to have no dynamic memory outside of the heap. No recursion so stack size is bounded. No mallocs. At present, it falls short of that in two ways. My I/O subsystem uses malloc for memory buffers, and the print routine is recursive. Both were thrown together early on to get something running, and both will eventually be rewritten. I’ll rewrite I/O in C and rewrite print in Scheme so it can remain recursive.

The third goal is to integrate macro translation into Schetoo. I’m not sure how that will work.

Status

Right now, Schetoo is usable for toy programs.

It has a heap with lots of primitive data types: bindings, bytevectors, Booleans, Unicode characters, continuations, fixnums, pairs, procedures, records, record type descriptors (RTDs), strings, symbols, and vectors. The interpreter also has a hierarchy of condition types constructed from records. (A Scheme condition is like a Python exception object.)

It has an assortment of primitives for operating on and converting among those types. I count 44 primitives at the moment.

The reader came from kbscheme, and just like kbscheme, it handles the whole R6RS Scheme language except for non-fixnum numbers.

It also has an evaluator with a nice extension mechanism for adding more procedures and special forms. Some of the more interesting forms I’ve implemented are apply, call/cc, define, eval, if, lambda, quote and set!.

There is an exception mechanism, and all the forms are wired to generate exceptions as needed. There is no way to set an exception handler yet — the default handler just prints the message and pops back to the REPL.

Finally, there’s a clever mechanism to track stack-based heap roots with zero overhead. That will be the subject of a future post.

I’m planning to implement dynamic-wind and multiple-value expressions someday. The evaluator was designed with them in mind, but they’re not implemented yet.

Design

Once again, the implementation language is C. The overall structure is a trampoline-style interpreter using explicit continuations instead of a call stack. This time, I’m planning to write less C and more Scheme. I’m almost at the point where the read-eval-print loop (REPL) can be written in Scheme.

I used the same basic heap and garbage collector design as last time. There are two significant differences, though.

First, small objects are stored as immediate pointers. Those include fixnums, (Unicode) characters, Booleans, the empty list, and some other constants. For example, the Boolean false is represented as a pointer with the binary value 0x0000000E. There’s nothing stored at address 0x0000000E, it’s just a bit pattern.

Second, I’ve implemented records. Records are Scheme’s objects. Each record has a class (called a record type descriptor or RTD), and classes have single inheritance. Those both went into the heap design fairly easily.

The evaluator is also similar at a high level, but completely different in the details. There is a virtual machine (VM), and the VM is a little better defined than last time. I think I’ll write a future post about the VM design, too.

And exceptions got designed in early. That’s an important part of building a usable interactive environment.

Next Steps

I’d like to get the REPL into Scheme. To do that, I need user exception handlers and I/O primitives.

A couple of weeks ago, an article on Anandtech explained that the XBMC media player now supports a new Broadcom video decoder. They also mentioned that it works on a Mac Mini.

We already had a Mac Mini under our TV. And we were already using XBMC. And we’ve got some high bitrate HD video that the Mini couldn’t decode in real time.

I opened up the Mini and confirmed that it really did have a mini PCIe slot. It does. (This must be the age of the minicomputer.) The slot was occupied by the Wi-Fi card. So I could have either high bitrate HD or Wi-Fi. The house is wired with Cat-5 and the Mini was already plugged in to a gigabit switch, so Wi-Fi isn’t needed.

So I ordered the video decoder. A Broadcom BCM970012 on a mini PCIe card from Logic Supply, as mentioned by Anandtech.

The video card arrived. I took the Mini apart. Slipped the new card in. Put it back together. Found out the audio board had come unplugged. Took it apart again. Plugged in the audio board. Put it back together again. Launched XBMC. Played some video. It works.

The whole thing was so easy it was quite anticlimactic.

I don’t have any idea how XBMC is talking to the video card. It’s not visible in Mac OS’ System Profile. It’s kind of weird to install hardware like this and not know how the operating system interfaces to it. Normal people cope with this kind of ambiguity, so I guess I can too.

Recommended.

The Original Goals

At the beginning of thie project, in September 2008, I wrote some goals.

I don’t know Scheme. I took a couple of classes that used Lisp way back when, but don’t really know Lisp or Scheme. There’s no better way to learn a language than to implement it. (-:

Goal met. I probably know more about the language’s nooks and crannies than the average practitioner.

I want to write something in C. My C is very rusty, since I’ve been using Python exclusively for the last year, and C++ for the last fifteen years.

Goal exceeded. Got to write lots of C. Also started using C at work again a few months later. Fully fluent in C again, though I’m much less tolerant of its limitations than I was, say, 20 years ago.

I want to write a garbage collector. They’ve always fascinated me, but I’ve never written one.

Goal met, just barely. I wrote a simple stop-and-copy GC. Copying collectors have always fascinated me and I wanted to see what limitations it would impose on the rest of the implementation. (See Roots below.)

I have some research ideas about garbage collection that I want to explore.

Not met. I wrote the simplest possible GC, and then went on to other parts of the system. I still have some untested ideas about GC.

In the 16 months since I wrote those goals, they’ve shifted somewhat. To the list, I’d add learning about Unicode, learning about macros, and understanding R6RS’s semantics (and Scheme semantics in general) in great detail.

What went well?

Memory Manager

I like the design of the memory subsystem a lot. It is easily extensible, fast enough, and it feels production quality.

Evaluator

The evaluator came together pretty well too. It took a while to get it right, but then it just worked while lots of things around it changed.

Reader

I’m very pleased with the reader. I should be — I rewrote it four or five times. The final result is a handcoded LL0 parser generator that builds the parsing tables at startup time. It exactly matches the language spec, because it uses the spec’s grammar as its input.

Scanner

The scanner is also very good, though it’s incomplete. It only recognizes a limited number syntax. I poked around a little with the idea of writing a regular expression compiler, sort of a handbuilt lex, but didn’t do much, because the existing scanner was fine.

What went badly?

R6RS

I started with the Revised^6 Report on Scheme, R6RS. That seemed like a good idea for several reasons: R6RS uses Unicode throughout, it has well defined exception semantics, it has a library system, it has fewer loose ends and places for the implementor to improvise (aka create incompatibility) than the other major contender, R5RS.

But R6RS is a very unpopular Scheme. Very few implementations are compliant, so it was hard to find good examples to study. I definitely would have made faster progress implementing R5RS, even with the need to improvise.

Exceptions

I never implemented any kind of exception handler. Any error, no matter how minor, crashes and core-dumps the interpreter. Now there’s a code base with several hundred error checks, with no differentiation between runtime errors, syntax errors, heap corruption, etc. That’s made debugging harder than it should have been. And the interpreter is much less useful than one that recovers from a missing parenthesis.

Roots

The way I handled root pointers in automatic C variables was error-prone and slow.

A root pointer is a pointer that is stored outside the heap and points into the heap. On garbage collection, all root pointers must be traced. Any memory allocation can trigger a GC. So any allocation might force a walk of all the roots, including those that are local variables in active C functions.

My solution was to keep a global (per-thread, actually, if threads ever got implemented) linked list of root pointers and define some macros to push automatic variables onto that list and pop them off. Those macros have to be called explicitly every time a root pointer goes into or out of scope.

I must have debugged a hundred occasions when a root pointer was incorrectly pushed or popped. It made me averse to writing code in C because it was so hard to keep the bookkeeping right. And continuously updating the linked list couldn’t have been good for performance.

Libraries

Originally, I tried to use the R6RS library mechanism as my primitive means for linking the interpreter together. That didn’t work so well. Then I tried linking everything in but implementing the libraries in C. That worked okay but was too rigid. At the end, I started on a redesign where all the primitives are in a global namespace, and a subset namespace is created for user code. Didn’t finish that.

Macros

I got bogged down in macro processing. I need to spend more time with a working macro implementation so I understand them better, I think.

Bootstrapping

This is the big one. My original ideas about how to bootstrap the interpreter were kind of naive. I thought I’d write the memory manager, some primitives, and an evaluator in C, then start writing more procedures in Scheme. Didn’t really know how macro expansion would fit into the process.

After building a lot of low-level code and getting to the point where my interpreter would read and evaluate code, I spent a long time trying to map out a path to implement everything in terms of slightly lower-level units. Made a lot of progress — there are over 100 primitives implemented. But I still didn’t find a path.

I probably invested a couple of hundred hours into reading and re-reading R6RS, R5RS, and Kent Dybvig’s papers on macro expansion looking for clues as to how the pieces fit together. There’s a lot of intro material on the web about building toy interpreters with no macros at all and a lot of research papers about the hard problems of building production-quality Scheme systems, but the middle ground is not well-covered.

So I’m taking a break. Maybe a long break.

I’ve pushed the current sources out to both github and gitorious. Download using either of these commands.

git clone git://github.com/kbob/kbscheme.git
git clone git://gitorious.org/kbscheme/mainline.git