Chapter 1.19 --- Extra Credit

Domain-Specific Languages

"Most of us know the code is wrong; almost everybody breaks it. But we pay Danegeld by feeling guilty and giving lip service. Willy-nilly, the code rides us, dead and stinking, an albatross around the neck."

You've already seen a number of Domain-Specific Languages in Lisp---to recap, a DSL is a convenient, embedded syntax, purpose-designed to achieve natural expression of a problem domain that falls outside the realm of the host programming language and its native syntax. Common Lisp even comes with two notable DSLs baked into the core language specification---the Loop macro and Format strings. Love 'em or hate 'em, Loop and Format were purpose-designed, provide essential functionality, and fulfill their purposes with internally consistent syntaxes.

But you've also had a chance to play with DSLs for HTML/XML, CSS, and JavaScript. Lisp makes it surprisingly easy, compared to other programming languages, to roll-your-own DSL for whatever problem domain you have to deal with. This goes above and beyond writing a library of functions, macros, classes and methods. You write a Domain-Specific Language when even a full, native Lisp library won't solve your problem---because the problem requires a different way of thinking to express. The DSLs for HTML/XML, CSS, and JavaScript are good examples to illustrate this point---after all, if they weren't absolutely necessary to make the world-wide-web of today a reality, they wouldn't exist.

Structured Data is fundamental to the web; search engines rely on it to correctly parse and catalogue every website on the internet, so that you can get to the information you need when you need it. But XML can be a tedious language to work with, if you find yourself having to hand-edit it; this is why so many full-time XML developers avoid doing so---they prefer using expensive software to manage their XML code in a sensible way, visually. HTML is much the same---even though it is not as tedious to work with as the full XML spec, many professional web developers prefer using WYSIWYG editors to develop their web templates, never even looking at a single line of HTML if they can help it.

CSS is just as important---it is purpose-designed for dictating the rendering rules for structured data in HTML and XML, for multiple output formats. And then there's JavaScript---the little web scripting language that has taken the world by storm, despite its obvious limitations and logical problems.

The problem domains that these web languages handle are real---and obviously, they meet their need or people wouldn't use them. Now, I've said before that you can replace them all with Lisp, and as you've seen, it's much easier to manage a single Lisp code-base than having to constantly shift gears to switch between the distinct paradigms required for writing Markup, Styling, Scripting, and Logic code. But that being said, it's important to remember the distinct problem domains; when you approach a DSL, you have to make sure it solves the problem on its own terms and not the terms of the host language.

Now it's time to learn how to write your own DSL. It's a much bigger challenge than anything else we've covered so far---but the rewards are equal to the effort. Being able to write a clean, native Lisp interface to any imaginable problem-domain is one of the language's greatest strengths.

This chapter will contain exercises on:

• Lisp Macros and DSLs
• A brief tour of popular problem-domains
• Overview of S-SQL, a DSL for writing SQL queries with S-Expressions
• Write your own DSL!
  • Choosing a problem domain
  • Identify a domain's most natural expression
  • Write out how you would like the DSL to work
  • Implementing the syntax of your DSL

Exercise 1.19.1

Lisp Macros and DSLs:

When standard functions fail to meet your needs, using the macro definition syntax you learned in Chapter 1.10 you can manipulate Lisp syntax in order to create a DSL that's perfectly suited to your needs and a specialized problem domain. You have already seen several DSLs in action, two of which are built right into the Common Lisp standard: LOOP and FORMAT strings. They also represent two different approaches to DSLs---inline syntax and string-parsing.

One advantage to string-parsing is that you can write functions to parse your strings instead of writing macros. This is how FORMAT is implemented in Common Lisp.

(format nil "~A" 'format)

Of course with inline syntax, your DSLs expand to native Common Lisp code, instead of having to be parsed before you can build an abstract syntax tree; and while the LOOP macro doesn't use S-Expressions in its internal syntax, you can certainly take advantage of them in your DSL to get all the other benefits of Lisp you've learned so far.

(loop for x below 10 collect x)

Compare this again to an S-Expression syntax using Iterate:

(ql:quickload "iterate")
(use-package 'iterate)
(iterate (for x below 10)
         (collect x))

Note: If you get errors 'using' the iterate package, follow the debugger instructions to shadow the cl-user package functions with iterate.

These LOOP and Iterate examples are equivalent, and are both equally fast. The main advantage to Iterate over the built-in LOOP macro is that it uses S-Expressions, so you can more easily embed Iterate in other macros, and reason about Iterate code through code-walking. These are important points to consider when designing your own DSL.

Exercise 1.19.2

A brief tour of popular problem-domains:

• Logic Programming
• Graphics Programming
• Mark-up
• Assembly and Byte-Code

Exercise 1.19.3

Overview of S-SQL, a DSL for writing SQL queries in S-Expression syntax:

• Keyword Expressions
• SQL Queries
• Re-ordering of Expressions
• Handling results

Exercise 1.19.4

Write your own DSL! Overview of Methodology:

Exercise 1.19.5

Write your own DSL! Choosing a problem domain:

Exercise 1.19.6

Write your own DSL! Identify a domain's most natural expression:

Exercise 1.19.7

Write your own DSL! Write out how you would like the DSL to work:

Exercise 1.19.8

Write your own DSL! Implementing the syntax of your DSL: