Make Your Data Work for You With CSV Parsing

d3-dsv is a JavaScript library for parsing and serializing CSV files. In the case of this library, "parsing" means "convert this CSV to JSON format" and serialize means "convert this JSON to CSV." In the case of this app, we are using the parsing portion to turn a CSV into JSON, where we can then transform and analyze the data further.

For the d3-dsv Starter App, we created a tool that will parse an input CSV (a file from the user's machine) and provide a few basic transforms/analyses. The output from the main d3-dsv parsing function is an array of JSON objects, each one representing a row in the CSV. From there, this app can do several things with the input data: it can count the rows in the CSV, it can list the fields in the CSV, and it can sort the resulting JSON by selected field.

In the next section, we describe how the code in this app works. Feeling comfortable with the app? Skip down to "" section!

How it works #

The code powering this app is in index.html, style.css, and script.js.

index.html contains the initial scaffolding of the app -- the main structure for the elements on the page. (Note: other page elements will be added later via JavaScript in script.js, but we can place the elements we know we want to be on the page when it loads directly into index.html.)

The section of index.html imports a few files that will be needed to make the app work: style.css for styling, script.js for the app functionality, and d3-dsv (d3-dsv.v1.min.js) for CSV parsing.

The index.html contains mostly-empty

elements. These are included for styling the data with CSS, which is contained in the style.css file (described below). The one additional thing we add in index.html is an input element for the CSV file upload -- this must be on the page at load time because adding a CSV file is what triggers the rest of the app!

style.css contains styling information for the page. In this app, the CSS includes styling for the JSON output (black background with white text), the "choose file" prompt, the buttons, and every other visible element you see on the page. The @media rule is a "media query" that collects some information and sets style rules based on that information. In this case, it changes the layout of the page based on how wide your browser window is so it looks good on both mobile or desktop.

The JavaScript in this app is -- when an fires on the page, it triggers a certain piece of code. When you check a box on a web page? A "click" event is fired. Pick from a dropdown menu? A "select" event is fired. In this app, where different things are happening to the data depending on what button is pressed, it makes sense to have individual functions that are triggered when an event happens on the page.

At the beginning of the file, we declare two variables: json and output. json does not get assigned a value and output is set to point to the json-output HTML element. These variables exist outside of the scope of of any function, so their values can be updated and/or shared across all of the functions. This is helpful for updating the json value when new files are added and for using output without typing document.getElementById('json-output') every time we want to access that element.

The rest of the file contains a set of functions that are triggered by page events or are called from within other functions. The following table lists the names of the functions in this app, what they do, and what behavior triggers or calls them.

In the very last line in the file, we add an event listener to the element -- on a "change" event, call loadFile(). The "change" event happens when the user selects a CSV file.

When the loadFile() function is called, the "change" event itself (represented by an object) is passed into the function as a parameter. This is especially useful for us, because information about the file that was loaded is included in the event object. Once we have the file, we can make sure it's actually a CSV.

Once the file is good to go, we use the FileReader to read the file in the browser. We define an to be called when our instance of FileReader reads the file, and inside the file we parse the CSV data to JSON with d3-dsv, call printToOutput with that parsed JSON, and call addButtons() to add additional operations to the page (count rows, get fields, sort by).

This function uses d3-dsv via d3.csvParse() to transform the CSV into JSON. d3.csvParse() takes two parameters: the CSV string to be transformed, and an optional function to do something special when transforming. In this case, we use as the second argument. This is a special function provided by d3-dsv that will try to automatically infer the value type for each field and coerce that value in each row. For example, if there is a Year value in the data with values "2017", "2018", and "2019", using d3.autoType will coerce those values to 2017, 2018, 2019.

For more detailed information on each of the callback functions, see the code comments in the project.

The Challenge #

Our challenge to you is to take the d3-dsv starter app and extend it to do a few extra things:

Add a toggle to sort the data by field ascending instead of descending (currently the code only sorts descending)
Add a function to "pivot" an input CSV when the rows and columns are flipped in the data (example "flipped" dataset) – be sure to add a button and an event handler that calls the function!
Update the app to accept kind of delimiter-separated values (semicolons, tabs, etc.) instead of just CSV
Update the app to pull data from an API in addition to adding a local file After you've made the app your own, submit it using the form below. Please send us a link even if you couldn't get it to work or it's not your favorite thing — we want to see it!

To recap:

We can't wait to see what you make. If you have any questions or need help, reach out to us on Twitter @glitch or post your question in the support forums.