Movitz: A Common Lisp OS development platform

$Id: movitz.html,v 1.8 2004/07/08 00:12:43 ffjeld Exp $

Files

The latest los0 kernel image and its associated change-log and copying license can be found here. And there is the current technical documentation.

I'd be interested in receiving reports of crashes, and preferably reproducible ones. I know there are issues with non-US AT keyboards. If you have keyboard trouble, you might be able to solve it by saying :mapkey code where code is the ASCII value of the character you want some key to be mapped to.

If the built-in bootloader of los0.img doesn't work (x86 bootloading is a complicated business, apparently), please try loading it with GRUB. I've prepared a GRUB bootloader image that will load a kernel that is appended to it. That is, you can prepare a bootable floppy something like this:

 cat grub-bootloader.img los0.img >/dev/fd0

The GRUB loader should also work on any other boot-devices it supports, such as a hard-drive, CD-ROM, or USB-pen. You probably need to look up the GRUB documentation for installation instructions. The file los0.img is a multiboot compliant kernel.

Introduction

The Movitz system aspires to be an implementation of ANSI Common Lisp that targets the ubiquitous x86 PC architecture "on the metal". That is, running without any operating system or other form of software environment. Movitz is a development platform for operating system kernels, embedded, and single-purpose applications. There can potentially be several completely different operating systems built using Movitz.

The Movitz system is two things:

A minimal run-time environment which is designed for the x86 architecture running in protected mode. It is minimal in the sense that all Movitz software can depend on them being present. This is intended to include the full ANSI Common Lisp specification, as well as the basic functionality required to perform the chores of typical operating systems. A particular OS system built with Movitz will presumably extend this minimal run-time with whatever services it finds appropriate. This part of Movitz is called Muerte.
A cross-compiler that targets the Muerte run-time. This cross-compiler is written in plain Common Lisp, and is currently being developed under Allegro CL, GNU Emacs, and FreeBSD, although (with minor exceptions) nothing in particular binds it to this platform.

The los0 kernel image

The los0 kernel image is the first instance of a Movitz kernel application, or MoKA, and contains the current development version of the Muerte. Los0 is a very minimal MoKA that is used to drive the development of Muerte. This system provides no run-time services beyond a typical lisp REPL listener, which may be used to interactively peek into your system; be it the lisp system per se or the hardware environment. Try apropos, for example. Some REPL top-level commands (mostly debugging-related) are :bt :pop :restart :trace :untrace :error :help :cpu-reset.

Hardware requirements are approximately: A 386 or better CPU with slightly more than 2 MB of RAM (the less you have, the sooner you run out..). The image file can be booted as a floppy image with a PC emulator like Bochs, VMware, or VirtualPC, as an actual floppy on a real machine (just cat the image file to the floppy device), or using GRUB or another Multiboot compliant bootloader.

Status

The compiler

The cross-compiler is fairly complete in terms of the basic mechanisms required by ANSI CL. However there are most likely bugs to be weeded out in the more exotic corners of it, like dynamic control-transfers and the like. In terms of code quality, in my opinion the compiler produces code that is reasonable yet far from "great". It is particularly lacking in the type-inference department. However, the machine-code produced tends to be shorter than any other x86 Common Lisp compilers I've compared it to.

ANSI Common Lisp

ANSI Common Lisp specifies a quite extensive set of functions, macros, and types, and these are not all implemented yet. But many are, and in general my mode of working is to write what to me is "natural" Common Lisp, and if some operator is not implemented in Muerte, I will implement it rather than avoid it. CLOS and quite a bit of MOP is supported in the cross-compiler. At run-time the support is weaker in the sense that there is for example no facilities for defining new classes or methods on-the fly. But the introspection is there, and make-instance and other generic functions will do what they are supposed to do.

Muerte

Aside from the ANSI Common Lisp library, Muerte consists of functions to manipulate the CPU and other hardware. There are primitive accessors like x86:io-port, x86:memref and x86:memref-int, x86:eflags, x86:control-register-hi20/lo12, x86:segment-register, x86:cpu-id, and x86:read-time-stamp-counter. CPU introspection is facilitated with x86:cpu-signature and x86:find-cpu-features. All in all, most basic functionality required to interface any hardware without resorting to inline assembly is either in place or can easily be added.

Garbage Collection

There is now preliminary support for GC in Muerte. This support is two things. Firstly, the object layouts, stack discipline, etc. are such that the system as such is amenable to scanning for GC purposes. Secondly, there are two functions in Muerte that are expectedly useful for implementing GC architectures:

map-heap-words maps a function over every potential pointer in a specified memory region, denoted by start and end memory locations. This can be used both to implement GC as such (if the function migrates objects according to some GC scheme), and as a help for GC debugging (if the function merely checks some invariant or prints information).
map-stack-words works similarly for a control stack. A stack is wrapped as a vector specialized to (unsigned-byte 32) in memory, so it will not be recognized as pointers by e.g. map-heap-words. Hence, this function must be used explicitly over each live control stack in order to capture all pointers in the system. Another reason why stacks are special, is that they contain untagged pointers to code-vectors, and this requires special treatment.

The los0 image currently uses these two functions to implement a rather simple Cheney-style stop-and-copy GC architecture. In short, two 256 KB buffers are set up, and the memory allocation primitives are changed to allocate from one of these buffers designated as "newspace". When newspace goes full, the roles of oldspace and newspace are switched, and the live objects in the (newly become) oldspace are evacuated into the (newly become) newspace. This evacuation is performed rather naively, namely by scanning the entire heap, which tends to be somewhere between one and two megabytes in los0. Note that under this simple scheme there is no way for objects to be promoted from the two 256 KB buffers, so you cannot have more than this amount of live, dynamically allocated data. You may trigger the GC process explicitly with (stop-and-copy). Note that there are still rough edges remaining this GC implementation. E.g. there is no support for code-vectors migrating yet, although this will only become an issue when new code-vectors are consed up (i.e. by incremental compilation of some sort).

About OS design in Common Lisp

Unlike the previous sections, this section is more or less loose thoughts about what will be rather than describing an existing system.

Layering

I expect Movitz software modules to be categorized broadly as follows:

Muerte level: This is lower-level functionality that depend solely on Muerte services. The idea at this level is to provide a very thin abstraction over the hardware, just enough to make it convenient to work with in lisp. This means a functional, stateless interface that makes as few assumptions about how it will be used as possible. At this level, "functional" (meaning free of side-effects) applies in a very strong sense, such that e.g. consing must be kept to an absolute minimum, and when consing can occur must be predictable. For example, make no assumptions about whether a hardware device will be used in interrupt-driven or polled mode. The idea is that modules at this level facilitate exploratory programming, and also it can be common to every MoKA.
MoKA level: This software utilizes the Muerte-level modules to build 3D desktop AI-enabled operating systems and the like.

Frode Vatvedt Fjeld