Introduction to Metaprogramming

Learn how macros and abstract syntax trees enable metaprogramming.

We'll cover the following

This course will teach you how to write code that writes code with macros. Elixir macros enable metaprogramming and make it a breeze to write powerful programs.

Code that writes code might sound like a neat trick, but we’ll soon learn how it forms the basis of Elixir’s own construction.

We can extend the language with powerful first-class features, save time, and share functionality in fun and productive ways. Metaprogramming lets us create clear, concise programs that treat source code as building blocks instead of lines of rote instructions.

The world is your playground

Metaprogramming in Elixir is all about extensibility. Have you ever wished your favorite language would adopt that one neat feature? If you’re lucky, it might take years to happen. Often it never happens at all. In Elixir, we can introduce new first-class features at will.

Let’s consider the familiar while loop that we find in most languages. It’s missing from Elixir, but we can imagine writing one like this:

while Process.alive?(pid) do 
  send pid, {self, :ping} 
  receive do
    {^pid, :pong} -> IO.puts "Got pong" 
  after 2000 -> break
  end
end

In this course, we’ll make this while loop a reality. Moreover, We can define new languages with the Elixir language, to express all kinds of problems in a natural syntax. We can see a valid Elixir program below:

div do
  h1 class: "title" do
    text "Hello" end
  p do
    text "Metaprogramming Elixir"
  end 
end
#Output:
#<div><h1 class=\"title\">Hello</h1><p>Metaprogramming Elixir</p></div>"

Elixir makes things like HTML domain-specific language (DSL) possible, which we’ll also create in this course.

We will understand how these things work later in this course. For now, just remember that macros make all this possible. Elixir pushes this idea further than we’ve ever seen.

The two essential concepts

Let’s quickly review the two essential concepts of metaprogramming in the Elixir system and learn how they fit together.

  1. The Abstract Syntax Tree (AST)
  2. Macros

The Abstract Syntax Tree

To master metaprogramming, we first have to understand how Elixir code is represented internally by the Abstract Syntax Tree (AST). Most languages we’ve worked with have an AST, but you’re typically not aware of it.

When our programs are compiled or interpreted, their source is transformed into a tree structure before being turned into bytecode or machine code. This process is usually masked away, and we usually don’t think about it.

José Valim, the creator of Elixir, chose to do something very different. He exposed the AST in a form represented by Elixir’s own data structures and gave us a natural syntax to interact with it.

Having the AST accessible by normal Elixir code lets us do very powerful things because we can operate at the level typically reserved only for compilers and language designers. We interact with Elixir’s AST at every step of the metaprogramming process, so let’s jump in and find out what it’s all about.

Metaprogramming in Elixir revolves around manipulating and inspecting ASTs. We can access the AST representation of any Elixir expression by using the quote macro. Code generation relies heavily on quote, and we’ll be using it throughout the course to carry out our exercises.

Let’s use it to return the AST representation of a couple of basic expressions.

Execute the following commands in the terminal given below:

quote do: 1 + 2
quote do: div(10, 2)

Note: We can try more expressions on our own and explore their AST representations.

Terminal 1
Terminal
Loading...

We can see that the AST representation of 1 + 2 and div produced simple data structures in Elixir’s terms. Let that sink in for a moment. We can access the representation of any code We write as an Elixir data structure.

Quoting expressions gives us a unique feature in a language: the ability to peer into the internal representation of our code, within a data structure we already know and understand.

This lets us infer meaning, optimize performance, or extend functionality while staying within Elixir’s high-level syntax.

With full AST access, we can perform neat tricks during compilation. For example, the Logger module in Elixir’s standard library can optimize logging by completely removing the expressions from the AST.

Suppose we’re writing to a file and would like to print the file path in development but ignore the expression in production. We might write something like the following

def write(path, contents) do
  Logger.debug "Writing contents to file #{path}" 
  File.write!(path, contents)
end

In production, the Logger.debug expression would be completely removed from the program. This is because we can interact with the AST during compilation to skip this development-related call.

Most languages would have to invoke the debug function and waste CPU cycles checking for ignored log levels at runtime, because their source code cannot interact with the underlying AST.

How Logger.debug is able to perform this feat brings us to the next essential ingredient of the metaprogramming process: macros.

Macros

Macros are code that writes code. Their purpose is to interact with the AST using Elixir’s high-level syntax. This is how Logger.debug can perform its optimization tricks while appearing like normal Elixir code.

Macros are used for everything from building Elixir’s standard library to serving as a core infrastructure of a web framework. In either case, the same metaprogramming rules apply.

We don’t have to choose between complex, performant code or slower, elegant APIs. Elixir macros let us write simple code with high performance. They turn the programmer from language consumer to language creator. We are no longer merely a user of the language. We have access to all the tools and power that Elixir’s creator used to write the standard library. He opened the language up for our own extension.

We might think we’ve largely avoided macros until now, but they’ve been hiding in plain sight all along.

Consider this simple block of code:

defmodule Notifier do
  def ping(pid) do
    if Process.alive?(pid) do 
      Logger.debug "Sending ping!" 
      send pid, :ping
    end 
  end
end

It might look unremarkable, but we’re looking right at four macros. Internally, defmodule, def, if, and even Logger.debug are implemented as macros, like most of Elixir’s top-level constructs.

Looking up the documentation in iex provides an if macro. This macro expects the first argument to be a condition and the rest are keyword arguments.

iex> h if
        defmacro if(condition, clauses)

We might wonder about the advantage of Elixir using macros for its own constructs since we get by fine in most languages without this structure. The most powerful advantage is that macros allow us to extend the language with our own keywords while using existing macros as building blocks.

The best way to think about metaprogramming in Elixir is to throw away the notion of rigid keywords and opaque language internals. Elixir was designed with extension in mind. The language is open to our exploration and custom features. This is what makes metaprogramming in Elixir so pleasantly natural.