DOM Manipulation & D3 Fundamentals

Material based on the lecture by Carlos Scheidegger, the D3 Intro by Vadim Ogievetsky, Scott Murray’s Interactive Data Analysis for the Web, and the D3 website.

DOM Manipulation

After we covered the basics of JavaScript last lecture it’s now time to explore how javascript interacts with the DOM and thus makes all the interesting things in the browser happen. Like we’ve seen before, the HTML we write is represented as a tree inside a web browser. What we are going to turn to now are the JavaScript APIs that web browsers provide to let you edit the DOM dynamically, so that we can build our visualizations with code instead of text editors.

While we will manipulate the DOM mainly through libraries (we will use D3, but JQuery is the most popular DOM manipulation library) you can also manipulate the DOM directly through the standard DOM API through the document object. Check out the MDN reference on the document object.

Here are the very first steps:

document is a global javascript object that contains the DOM. If you log the document object you will see exactly the same as in the DOM view in the web inspector. It is this object that we are manipulating dynamically and that the browser then renders. The document object contains functions, like the getElementByID() method we’re using here to retreive a specific element, or the createElement(), which creates a new HTML element, or the createTextNode() functions used to create new text content. We’ve also used the appendChild() method (which is defined on a node) that appends the passed element to the end of the list of the node.

Now we can make use of the features we learned about last lecture to make more complicated programs that write websites:

We can also control the attributes of elements, e.g., for styling and positioning with CSS:

The coloring here is defined using the HSL color model.

Remember that in JavaScript we can attach new fields to existing objects. You can do this to DOM elements returned by the API, and that turns out to be very powerful. Note how in the snippet below, we are adding a new method update to the node returned by divWithText. When this method is called, we add the passed value to the current amount (stored at v), compute new positions from v, and update the text content of the node. By calling the update method we can get an animation:

Notice the tickForever function. We can’t use an infinte while loop here. If we did, the element attributes would be changed, but the user of the web browser does not get to see it, because the web browser does not ever get a chance to update the graphical representation of the DOM. The way to solve this problem is by using a special browser API called requestAnimationFrame. This API lets you tell a web browser that you’d like the opportunity to change something in the DOM. The next time the web browser is sitting idly, after having drawn all of its needed graphics, it will call the function passed as a parameter. Then, we just need to make sure that after updating the graphics, we call requestAnimationFrame again. The solution is a recursive version of the endless loop above (function f() { tick(); f(); }). The fundamental difference here is that instead of making the recursive call directly, we ask the browser to make the recursive call, after it has updated the graphics. This way there’s always a step in between every update where the web browser updates the UI and graphics, and you get nice animations as a result.

While we’ve worked with regular HTML elements here, the possibilities of demonstrated equally apply to SVG, of course. So in theory, you would have all the tools to create data driven visualizations. Of course, libraries such as D3 make our lives easier!

JavaScript Events

Up to this point we have only dealt with documents that are independent from user interaction. For data visualization, however, interaction is critical. Here we will introduce a couple of elementary concepts of how you can define and listen to events in JavaScript. We will be using DOM event handlers, as always, take a look at the MDN documentation.

Here is the most simple example possible for events, using two different methods:

The first button defines the function to be called - messageMe() directly in HTML. When the button is clicked, the corresponding function is called. The second button has an identifier. In the script we retrieve the button and add the function to be called dynamically.

We don’t have to use buttons to react to clicks though! Any DOM element can trigger an event. So we’ve also added a onClick event call to a <div>.

onclick is of course not the only event that we can listen to. Here is an example for how to react to changes in a drop-down box with the onchange events.

Here we also make use of the Event object that is passed along with the event call to identify the HTML element that triggered the event and retrieve its value.

Other useful events are onload, onmouseover, onmouseout, onresize, etc.

And finally, we can of course listen to events that are triggered by interacting with an SVG element:

So far so good - you can now write interactive code! If you know how to use these events you should be able to deal with all the interaction that you will encounter in the course of this class!

Running a web server

Before we start doing data driven visualization, we need to briefly talk about web servers. All of the code we talked about up to this point can simply be run by opening any of the html files on your local computer. As we go forward, however, this will no longer be possible. Many browsers have restrictions on loading local files via JavaScript, for security reasons. So, as soon as we start working with data stored in files we need a web server to test our code. There are many different servers and you’re welcome to use whichever one you like. I’ll briefly introduce two simple solutions:

The Python SimpleHTTPServer

If you have python installed you can run the built in simple server. To do this, cd to the directory where your code is located and run the following command:

1 # for python 2
2 python -m SimpleHTTPServer
3 # for python 3
4 python -m http.server

Open http://localhost:8000/ to get you to the server. You can only have one server at the same time (unless you specify ports explicitly). Hit Control-C to quit the server.

Running a Server and Debugging with WebStorm

As you know, we made educational licenses of WebStorm available for you in this class. If you develop your code in an IDE like WebStorm you get many powerful features, such as code completion, syntax highlighting, refactoring and interactive debugging. While you can use the chrome developer tools to debug your code, it’s more convenient to debug the code in the place you’re writing it, as you can immediately edit if you see something wrong. To debug code, you first have to run it on a web server. WebStorm comes with a built-in web server that you can activate via the Run menu. Here is an example configuration:

You should install the Chrome Plugin for Webstorm to get a fully-fledged debugging solution. Here I’ve set a breakpoint in the event handler function we just discussed:

Especially when you’re working on larger projects I recommend using a good IDE, as it will make you much more efficient, once you’ve learned to use it.

D3: Data Driven Documents

D3 is a javascript library for manipulating the DOM based on data. D3 was originally written by Mike Bostock, Vadim Ogievetsky, and Jeff Heer and published as a paper at InfoVis, the main information visualization conference; at this point it has a large number of contributors, yet Mike Bostock still is the core developer. It’s also one of the overall most popular projects on GitHub (!). D3 certainly owes some of its popularity to riding the everything-on-the-web wave. Nevertheless, the way in which you can express relationships between data and visual elements is fundamentally superior than any other library available, open source or not! It is nothing short of a breakthrough in the way we use code to express visual encodings.

D3 can be used to manipulate pure HTML, but most commonly it’s used in combination with SVG, i.e., we will be producing SVG charts using D3.

In addition to the introduction in Scott Murray’s book (the mandatory reading) you should work with the D3 API Reference to look up particulars of all the features of D3. And of course, you should be learning by examples. A great collection are Mike Bostock’s blocks which contain simple examples, such as a bar chart to complex examples such as this calendar view.

You can download the library to run locally on your computer from the D3 Website, or you can link directly to the latest release with this snippet:

You should also note that D3 has just undergone a major version change, from D3 3 to D3 4. Unfortuantely, most example you’ll find on the web will be using version 3. We’ll be using version 4 in this class though, so you can’t expect to just copy and paste code for version 4 and have it work. Check out the release notes to learn about changes in D3 v4.

Selections

See API Reference

Here is a minimal D3 example taken from the D3 website:

You can see that we achieve a similar result to the DOM manipulation examples we had before. We select an existing element in the DOM, here the first <p> element, and apply a style. However, you can also see differences: instead of having to use the API standard document.getElementsByTagName we use d3.select; and instead of using setAttribute("style", "color: steelblue;") we use D3’s style method.

d3.select selects the first element that matches a selector. Selectors can specify tags (p in our example above), classes, and IDs, all through the same interface:

Notice, however, that as mentioned previously, only the first element that matches is selected. Of course, it is more practical to select all elements of a certain type, which we can achieve with d3.selectAll

The last example illustrates the declarative approach of D3: we don’t have to iterate over a list of elements and apply the style. Instead we select a set of elements through rules and declare properties.

Once you have a selection, you can bulk-modify it’s content, not only in terms of style, but we can modify arbitrary properties using selection.property(name[, value]), the textual content of the elements with selection.text([value]), etc. We can also append elements:

Here is a code snippet that we will use in future examples, included as run.js:

1 var button = d3.select("body").append("button");
2 button.text("Run!");
3 button.on("click", execute);

Here we have three SVG rectangles and use selectAll to apply a new style to them. Notice two things:

1. the method chaining used to make this convenient to write,
2. the use of anonymous functions and the parameters d and i.

Data

THE key feature of D3 is mapping DOM elements with data. We can do this by calling the data() method on a slection:

D3 binds the data that is passed to a selection directly to the DOM elements. We can see this by logging and/or inspecting the selection.

Here we’ve also used data-driven styling for the first time! By setting the width of the rectangle dynamically to the size of the data item we get a data driven bar chart!

What happens if we have more data points than elements? Let’s try it:

There are still only three elements, it doesn’t matter how many data points we have - we can’t use more than there are elements to select. What to do? enter()! The enter selection holds placeholders for all the data elements that had no corresponding DOM element. We can use this selection to append new elements.

Progress - we have one element for each data item, but it doesn’t look good and it’s not drawn based on data. To fix this, we can apply data-driven attributes to the enter selection as well:

Now, that works! But we’re duplicating code. So instead we can do this shorter version:

So what if we don’t have initialized svg elements at all?

That’s great! D3 can also select things that aren’t there. Sounds strange, but that is very practical. We rarely start with existing DOM elements but mostly want to create them completely in code.

Now let’s explore what happens when we have fewer data items than DOM elements:

We have a similar problem as before. Here, the element that was not bound to data just remains as it was. Fortunately, there is a solution, which is the exit() selection.

We now have all the knowledge to select, bind data, react to changes in data. Next, we’ll look at how we can make state changes look great.

Transitions

One of the cool features of D3 is that transitions are baked in!

Cool, but we’re missing the enter again:

Here, we’ve handled the enter and chained two transitions.

Drawing Lines

Here’s one way to draw a line:

But we already know that. There must be something better, and there is!