Full tutorial coming soon! In the meantime, check the JSFiddle below.

This visualization shows the top 2011 NYC baby names by mother’s ethnicity.

Background from the D3 API reference and more:

• bubble chart as a flattened pack layout
• the pack layout is one of D3’s hierarchy layouts

This tutorial is based on our previous bar chart tutorial, Bostock’s bubble chart example, and InfoCaptor’s Bubble My Page service.

CSS

We set up a text size and style for our vis.

text {
 font: 10px sans-serif;
}

HTML

We include D3.js for the graphics, and Jquery for the rollovers.

<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
<script src="http://code.jquery.com/jquery-latest.min.js"></script>

Javascript

First we set up the global variables for our visualization size, type, colors, and bubbles.

var diameter = 700,
 format = d3.format(",d"),
 color = d3.scale.category20c();

var bubble = d3.layout.pack()
 .sort(null)
 .size([diameter, diameter])
 .padding(1.5);

var svg = d3.select("body").append("svg")
 .attr("width", diameter)
 .attr("height", diameter)
 .attr("class", "bubble");

We then query NYC Open Data for the baby name dataset, and throw an error if this query is not successful.

d3.csv("https://data.cityofnewyork.us/resource/25th-nujf.csv?$limit=10000", function (error, root) {
 if (error) throw error;

Within the d3.csv function, we set up sub-arrays, divided by ethnicity, to record baby names and related attributes for each ethnicity type (which are pre-determined in the data set). We also set up arrays called data and dobj in which we will concatenate all the sub-arrays.

//baby names
 var AsianNameList=[];
 var BlackNameList=[];
 var HispanicNameList=[];
 var WhiteNameList=[]; 
 
 //number of babies with name
 var AsianNameCount=[];
 var BlackNameCount=[];
 var HispanicNameCount=[];
 var WhiteNameCount=[]; 
 
 //ethnicity of mother
 var AsianEthnicity=[]; 
 var BlackEthnicity=[]; 
 var HispanicEthnicity=[]; 
 var WhiteEthnicity=[]; 
 
 var data=[]; //3D array of nameList, nameCount, ethnicity
 var dobj=[]; //array formated specifically for hierarchical processing

We now examine all the names in the dataset and bin them into the sub-arrays. There are duplicate records in the dataset, so we only add the name if it’s not already in the array. We use the “indexOf” function to check if the name is already in the array.

root.forEach(function (d) {
 if (+d["RNK"] <= 10) {
 
 //only add name if it's not uniquely in the array
 if ((d["ETHCTY"]==="ASIAN AND PACIFIC ISLANDER") && (AsianNameList.indexOf(d["NM"]) ===-1)) {
 AsianNameList.push(d["NM"]);
 AsianNameCount.push(+d["CNT"]); //force the string into an integer
 AsianEthnicity.push(d["ETHCTY"]);
 }
 else if ((d["ETHCTY"]==="BLACK NON HISPANIC") && (BlackNameList.indexOf(d["NM"]) ===-1)) {
 BlackNameList.push(d["NM"]);
 BlackNameCount.push(+d["CNT"]); //force the string into an integer
 BlackEthnicity.push(d["ETHCTY"]);
 }
 else if ((d["ETHCTY"]==="HISPANIC") && (HispanicNameList.indexOf(d["NM"]) ===-1)) {
 HispanicNameList.push(d["NM"]);
 HispanicNameCount.push(+d["CNT"]); //force the string into an integer
 HispanicEthnicity.push(d["ETHCTY"]);
 }
 else if ((d["ETHCTY"]==="WHITE NON HISPANIC") && (WhiteNameList.indexOf(d["NM"]) ===-1)) {
 WhiteNameList.push(d["NM"]);
 WhiteNameCount.push(+d["CNT"]); //force the string into an integer
 WhiteEthnicity.push(d["ETHCTY"]);
 }
 }
 });

We then concatenate all this data into one array called data. There is some simple test data currently commented out; this can be helpful for if you are having trouble managing your larger dataset.

data=[
 //test data is commented out:
 //["Tea","Coffee","Soda","Chips","Milk","Chocolate","Beer","Wine"],
 //[130,30,200,40,230,150,80,65]
 AsianNameList.concat(BlackNameList,WhiteNameList,HispanicNameList), 
 AsianNameCount.concat(BlackNameCount,WhiteNameCount,HispanicNameCount), 
 AsianEthnicity.concat(BlackEthnicity,WhiteEthnicity, HispanicEthnicity)
 ];

We prepare an array called dobj to record the index of each name. We also run the function called display_pack.

for (var di=0;di<data[0].length;di++) { 
   dobj.push({"key":di,"value":data[1][di]}); 
} 

display_pack({children: dobj});

The function display_pack consists of three parts. First, we set up SVG nodes (bubbles): here we set up bubble position, color, and mouseover text.

function display_pack(root) {
 var node = svg.selectAll(".node")
 .data(bubble.nodes(root)
 .filter(function(d) { return !d.children; }))
 .enter().append("g")
 .attr("class", "node")
 .attr("transform", function(d) { return "translate(" + d.x + "," + d.y + ")"; })
 .style("fill", function(d) { 
 //color is based on ethnicity
 return color(data[2][d.key]); })
 .on("mouseover", function(d) {
 d3.select(this).style("fill", "gold"); 
 showToolTip(" "+data[0][d.key]+"<br> "+data[2][d.key]+"<br>count: "+data[1][d.key]+" ",d.x+d3.mouse(this)[0]+50,d.y+d3.mouse(this)[1],true);
 //console.log(d3.mouse(this));
 })
 //.on("mousemove", function(d,i) {
 //tooltipDivID.css({top:d.y+d3.mouse(this)[1],left:d.x+d3.mouse(this)[0]+50});
 //}) 
 .on("mouseout", function() {
 d3.select(this).style("fill", function(d) { return color(data[2][d.key]); });
 showToolTip(" ",0,0,false);
 });

Then size our bubbles.

node.append("circle")
 .attr("r", function(d) { return d.r; });

Finally, we add text to our bubbles.

node.append("text")
 .attr("dy", ".3em")
 .style("text-anchor", "middle")
 .style("fill","black")
 .text(function(d) { return data[0][d.key].substring(0, d.r / 3); });

The function showToolTip is called from display_pack, so we need to write showToolTip. And that’s it!

function showToolTip(pMessage,pX,pY,pShow)
 {
 if (typeof(tooltipDivID)==="undefined") {
 tooltipDivID =$('<div id="messageToolTipDiv" style="position:absolute;display:block;z-index:10000;border:2px solid black;background-color:rgba(0,0,0,0.8);margin:auto;padding:3px 5px 3px 5px;color:white;font-size:12px;font-family:arial;border-radius: 5px;vertical-align: middle;text-align: center;min-width:50px;overflow:auto;"></div>');

 $('body').append(tooltipDivID);
 }
 if (!pShow) { tooltipDivID.hide(); return;}
 //MT.tooltipDivID.empty().append(pMessage);
 tooltipDivID.html(pMessage);
 tooltipDivID.css({top:pY,left:pX});
 tooltipDivID.show();
 }

Bar charts are some of the simplest forms of visualization and can be a good place to start when making your first vis. Inspired by some of the Socrata and D3 examples, I put together a simple chart showing the daily distribution of my dataset from the SODA tutorial (311 noise complaints, in zip code 11231, for September 2015). As you can see, most complaints occur on the weekend.

This chart is based on Mike Bostock’s simple bar chart tutorial, with modifications made to read the relevant NYC Open Data. Like with many of these types of visualizations, its code is written as CSS, HTML, and Javascript. This tutorial assumes proficiency with CSS and HTML, and a basic understanding of Javascript.

CSS

In the style section, we set up the fills, strokes, fonts, and shape rendering styles for our bars and axes. We also specify that there will be no x-axis line.

.bar {
 fill: orange;
}

.bar:hover {
 fill: lightpink;
}

.axis {
 font: 10px sans-serif;
}

.axis path,
.axis line {
 fill: none;
 stroke: #000;
 shape-rendering: crispEdges;
}

.x.axis path {
 display: none;
}

HTML

Our HTML body contains only two lines to include the Javascript libraries D3.js and Moment.js. And of course, it contains our custom JS functions, but we’ll discuss that below.

<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.10.6/moment.min.js"></script>

Javascript

First, we declare and initialize our spatial variables: the margins, width, and height of the vis; the x and y positioning variables, the x-axis and the y-axis, and the SVG (scalable vector graphic) into which we’ll draw our vis.

Note: I’ll add more detail later about the D3 functions used to initialize these variables, and about the D3 functions used throughout the script. In the meantime, you can start with this info, and you can also drill down into the D3 documentation from Mike Bostock’s simple bar chart tutorial.

var margin = {top: 20, right: 20, bottom: 30, left: 40},
 width = 960 - margin.left - margin.right,
 height = 500 - margin.top - margin.bottom;

var x = d3.scale.ordinal()
 .rangeRoundBands([0, width], .1);

var y = d3.scale.linear()
 .range([height, 0]);

var xAxis = d3.svg.axis()
 .scale(x)
 .orient("bottom");

var yAxis = d3.svg.axis()
 .scale(y)
 .orient("left")
 .ticks(10);

var svg = d3.select("body").append("svg")
 .attr("width", width + margin.left + margin.right)
 .attr("height", height + margin.top + margin.bottom)
 .append("g")
 .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

Our remaining bit of code inputs the NYC Open Data csv, parses it, processes it, and visualizes it.

Input it

Now we’ll call D3’s CSV function to input our NYC data. Notice that my query from Tutorial 1 has an addition filter: “&$select=created_date.” I added this filter to output ONLY the created_date for each record, because this is the only field that concerns me for this visualization. If there is an issue with the data being read, the “if (error) throw error” will halt the script with an error code.

d3.csv("https://data.cityofnewyork.us/resource/erm2-nwe9.csv?$where=starts_with(complaint_type,'Noise') AND created_date >='2015-08-01T00:00:00' AND incident_zip='11231' &$select=created_date", function (error, data) {
    if (error) throw error;

Parse and Process it

Now we’ll set up an array “week” to represent Sunday (represented as week[0]) through Saturday (represented as week[6]). We will iterate through our NYC dataset with a “forEach” loop, and use the array to count the occurrence of noise complaints for each day.

This section of code is where we parse each date using Moment.js. Because NYC Open Data is NOT outputting dates in ISO 8601 format, we use Moment to parse the date. We also use Moment’s “day()” function to interpret that date as a day of the week.

var week = [0, 0, 0, 0, 0, 0, 0]; 

data.forEach(function (d) {

if (moment(d["Created Date"], "M/DD/YYYY hh:mm:ss a").day() == 0) {
 week[0]++;
 } else if (moment(d["Created Date"], "M/DD/YYYY hh:mm:ss a").day() == 1) {
 week[1]++;
 } else if (moment(d["Created Date"], "M/DD/YYYY hh:mm:ss a").day() == 2) {
 week[2]++;
 } else if (moment(d["Created Date"], "M/DD/YYYY hh:mm:ss a").day() == 3) {
 week[3]++;
 } else if (moment(d["Created Date"], "M/DD/YYYY hh:mm:ss a").day() == 4) {
 week[4]++;
 } else if (moment(d["Created Date"], "M/DD/YYYY hh:mm:ss a").day() == 5) {
 week[5]++;
 } else if (moment(d["Created Date"], "M/DD/YYYY hh:mm:ss a").day() == 6) {
 week[6]++;
 }
 });

I threw in a few debug statements to check the contents of the final array, as well as check the day with the largest number of complaints. If you’re modifying this tutorial with your own data, these debug statements may be useful to you.

 console.log("week:");
 for (index = 0; index < week.length; index++) {
 console.log(week[index]);
 }
 console.log("most complaints in a day:", Math.max.apply(Math, week));

Note: If you’re using Chrome and running the code for this bar chart as a browser page, you can see the output for these statements in View > Developer > Javascript Console. Firefox and Safari have similar Javascript consoles.

Visualize it!

Now it’s time to visualize the array. First, we’ll set up our domains, or the array of values that our axes must span. Domain values will be displayed as labels on our axes.

• For the y-axis, the domain is from 0 to the maximum value of complaints aggregated from a given day of the week.
• For the x-axis, the domain is the days of the week.

y.domain([0, Math.max.apply(Math, week)]);
x.domain(["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"]);

Next we will draw the x-axis.

 svg.append("g")
 .attr("class", "x axis")
 .attr("transform", "translate(0," + height + ")")
 .call(xAxis);

Now we’ll draw the y-axis. We will label this axis “Number of Complaints.”

  svg.append("g")
 .attr("class", "y axis")
 .call(yAxis)
 .append("text")
 .attr("transform", "rotate(-90)")
 .attr("y", 6)
 .attr("dy", ".71em")
 .style("text-anchor", "end")
 .text("Number of Complaints");

Finally, we will draw a vertical bar for each day of the week. We iterate through the array week for this process.

• To determine the value of x, we use the index of week to look of the corresponding value of x.domain (which will be Sunday or another day of the week). The position of x is set by its “rangeBand,” which we set up early on in our variable declarations.
• The determine the value of y, we simply take the value of week‘s element. The position of y is set by the total height of the vis minus the value of y.

  svg.selectAll(".bar")
 .data(week)
 .enter().append("rect")
 .attr("class", "bar")
 .attr("x", function (d, i) {return x(x.domain()[i]);})
 .attr("width", x.rangeBand())
 .attr("y", y)
 .attr("height", function (d) {return height - y(d);});

Checking and Troubleshooting

You can fork and fiddle this vis using JSFiddle. Because I will require a standalone file from you for the midterm, it might be helpful to see the vis as a standalone HTML file. You can copy the file to your own server and modify the code with your own data. From there, you can debug and see console statements using Chrome > View > Developer > Javascript Console (if you’re using Chrome).

Background

What is Socrata?

Socrata is a Seattle-based company, originally founded as Blist in 2007, that has engineered the platform for a number of open government databases, including that of New York, Chicago, Baltimore, the White House, and a number of federal agencies. Socrata catalogs all its open datasets here.
It also hosts community hackathon sites here.
SODA = Socrata Open Data API
SoQL = Socrata Query Language

Why are we learning Socrata tools?

Socrata platforms are the gateway to a number of government datasets you may want to use. I also like that the platforms are well-structured, relatively well-documented, and offer an SQL-like query language that can be a good preview to using MySQL and other database packages.

Do I always have to use SoQL to get Socrata data?

Not at all! Sometimes a dataset is small enough that you can download it through the Socrata web UI. But in the case that you are dealing with massive datasets like the NYC 311 callbase, you will need to be able to use SoQL to get exactly what you want (unless you want to wait for hours and days to download massive datasets).

Getting Started: NYC Open Data

Please peruse the Socrata Open Data API Docs to understand the following NYC Open Data queries. I wanted to query this month’s 311 noise complaints in in my neighborhood (zip code 11231), so I obtained the dataset’s API endpoint, then I modified the format extension to output CSVs instead of JSON, and finally I added filter and query parameters to obtain this month’s noise complaints in 11231.

Try pasting the following queries as URLs in your browser; for each query, a small a CSV will download to your computer.

• First, try using a simple filter to get some 311 calls in 11231. The default number of records is 1000, and the default start date is in 2010.

https://data.cityofnewyork.us/resource/erm2-nwe9.csv?incident_zip=11231

• I want to start making more complex queries to obtain more recent records just from my neighborhood, so I changed the syntax to the SoQL format.

https://data.cityofnewyork.us/resource/erm2-nwe9.csv?$where=incident_zip='11231'

• Then I formed a query to increase the output to 10,000 records:

https://data.cityofnewyork.us/resource/erm2-nwe9.csv?$where=incident_zip='11231'&$limit=10000

• And a query for records created on or after September 1 2015:

https://data.cityofnewyork.us/resource/erm2-nwe9.csv?$where=created_date >='2015-09-01T00:00:00'

• And a query for noise complaints only:

https://data.cityofnewyork.us/resource/erm2-nwe9.csv?$where=starts_with(complaint_type,'Noise')

• Finally, I combined all the previous queries to get exactly what I wanted.

https://data.cityofnewyork.us/resource/erm2-nwe9.csv?$where=starts_with(complaint_type,'Noise') AND created_date >='2015-08-01T00:00:00' AND incident_zip='11231'

Now Try It!

Expand the NYC Open Data exercise by looking at December 2014, the month of protests around Eric Garner’s death. Form a query that outputs a CSV with the following attributes:

• data source: NYC Open Data
• time period: December 2014
• CSV size: smaller than 1 MB

Start Visualizing

If you have some visualization experience, you can use the Socrata examples to help you turn your new dataset into a visualization. This will be due in two weeks. If you aren’t there yet, give it a try. I’ll post a brief tutorial on visualizing your data next week.

And a Handy Thing

You will probably be using Excel, Google Sheets, Numbers, or another spreadsheet tool to view your data. It can be very handy–especially when you’re annotating your vis or writing a report on it–to know how to put together basic formulas using spreadsheet functions. Since we all have access to Google via our NYU addresses, here are some basic how-tos on putting together a formula using Google Sheet functions:

• Add formulas to Google Sheets
• Google Sheets function list