Reflection for the Masses

Programming languages provide more or less narrow models on how solutions should be represented and thought about, and enforce such models in more or less strict ways. However, it always happens that the offered models are not completely appropriate, but need to be adapted in some ways to better fit a concrete problem at hand. Design patterns and programming styles provide ways to deal with such situations: By applying some principles in your code, you can work around limitations of a programming language and/or benefit from properties that arise from such principles. For example, it is then possible to replace algorithms at runtime in an otherwise static language, take advantage of a "pure" functions in an otherwise imperative language, and so on.

Reflection is a more systematic way to open up a programming language and extend it beyond the designer's original intent. By exposing internal implementation details of the language to programmers, they can add and modify features of the language in a principled way. For example, metaobject protocols are prime examples of reflection in programming languages.

Reflection is one of the corner stones of programming languages: Many programming languages provide some form of reflection, and certainly almost all of the widely used ones. So there seem to be clear benefits from being able to inspect and extend languages from within themselves.

The first dedicated account of reflection was provided by Brian C. Smith at the beginning of the 1980's, and he used his own dialect called 3-Lisp as a way to illustrate the principles behind reflection, especially for procedural, but also for structural reflection. (Instead of "procedural reflection," we would nowadays say "behavioral reflection.") Unfortunately, Smith's papers and PhD thesis are very hard to follow and understand: Since he is primarily a philosopher, and not a computer scientist, he uses terminology borrowed from philosophy, and on top of that, takes concepts from Lisp dialects of his time for granted that even seasoned modern Lispers do not fully grasp anymore.

In a recent attempt to better understand the ideas and concepts behind Smith's account of reflection and 3-Lisp, Charlotte Herzeel and myself carefully studied Smith's papers and the follow-up literature that directly referred to Smith's work in detail. Eventually, Charlotte reimplemented 3-Lisp in Common Lisp, and we discussed several aspects of that implementation from various perspectives. In the end, we were both surprised how well thought out Smith's conceptualization is even with regard to lots of details you have to face when actually implementing reflection - details that many of the follow-up authors in their own accounts seemed to have missed. However, we are also convinced that Smith made some "mistakes" - especially, we are now convinced that the model of an "infinite reflective tower" is at best a neat theoretical setup, but not at all useful for practical purposes.

Our work culminated in a paper called "Reflection for the Masses," co-authored with Theo D'Hondt, which Charlotte presented at this year's Workshop on Self-sustaining Systems (S3) in Potsdam, Germany. The paper is now also available on my website. It discusses Charlotte's implementation of 3-Lisp in detail and explains the concepts and details of reflection as we see them. It also contains the full implementation of 3-Lisp in Common Lisp as an appendix.

We are very proud of that paper. We think that we achieved a major step forward in better explaining reflection to a more general audience. It is still a presentation that is probably a lot easier to understand for Lispers, and probably quite hard to follow for non-Lispers, but we removed a lot of Smith's obscurities in his original presentation and are convinced that especially Common Lispers should be able to easily understand and enjoy our version.