Earthquake Project – Part 1

About 7 years ago I created a video that showed all of the world’s earthquakes from 1980 until the present time. I’ve received several requests for an up-to-date version but I’ve lost the source code I wrote to generate the video. So, I decided that I would redo it and post the process here.

The first thing we’ll need is the data. The USGS website has changed a bit since I last used it. There used to be a link to download all of the data in one file. Now you have to specify a range and the output is limited to 20,000 rows. After trying several combinations of date ranges, one month seems like the correct range to bulk download events. This will be a tedious process though to manually enter in the date ranges, download, then rename each file. Instead of doing it by hand, let’s write an application that will do it for us!

We’ll use NodeJS for this project. If you don’t already have Node installed, I’d highly recommend using nvm to install it. Read more about it here.

The API documentation that the USGS provides can be found here: https://earthquake.usgs.gov/fdsnws/event/1/

The specific API URL that we’ll interact with is: https://earthquake.usgs.gov/fdsnws/event/1/query?format=csv&orderby=time-asc&starttime=<start time>&endtime=<end time>

Go ahead and create a new node project by typing:

npm init

Once you have that started, we’ll need to install the “node-fetch” dependency by typing:

npm install --save node-fetch

Node-fetch will allow us to use the Fetch API within our Node application. The Fetch API makes data retrieval much easier.

Let’s create a file, “data-fetcher.js” and add the following contents:

const fetch = require('node-fetch');
const fs = require('fs');

const startRange = new Date("1980-01-01 00:00:00");
//const endRange = new Date("1982-01-01");
const endRange = new Date();

const frameSize = 1;	// In months

async function fetchData(startFrame, endFrame) {
	let url = `https://earthquake.usgs.gov/fdsnws/event/1/query?format=csv&orderby=time-asc&starttime=${startFrame.toISOString()}&endtime=${endFrame.toISOString()}`;

	console.log(`Fetching data from ${url}`);
	let response = await fetch(url);
	let data = await response.text();

	if (response.status != 200) {
		throw data;
	}

	return data;
}

async function fetchAllData() {
	let frameStartTime = new Date(startRange);
	let frameEndTime = new Date(startRange);
	frameEndTime.setMonth(frameEndTime.getMonth() + frameSize);

	let done = false;

	let stream = fs.createWriteStream('data.csv');
	let firstRequest = true;

	while (!done) {
		let data = await fetchData(frameStartTime, frameEndTime);
		if (!firstRequest) {
			data = data.substring(data.indexOf("\n")+1);
		} else {
			firstRequest = false;
		}
		stream.write(data);

		frameStartTime.setMonth(frameStartTime.getMonth() + frameSize);
		frameEndTime.setMonth(frameEndTime.getMonth() + frameSize);

		if (frameStartTime > endRange) {
			done = true;
		}
	}
}

fetchAllData();

At the top of this file, we define our total date range for all of the earthquakes we want. The first function, “fetchData” will issue a GET request out to the API URL we mentioned near the top of this article and download the data for the current time frame. If there are any errors with the request it will throw the response back to the calling function.

The “fetchAllData” function is our main function and will break our total range up into smaller time frames. The size of the time frame is specified by the global “dateRange” variable at the top. If the request to the USGS Web API is successful, we’ll write the data out to “data.csv” then advance the time frame until we’re done. If the request out to USGS fails for whatever reason we break out of the loop and render the response text. Typically, this will be a description of the error.

Running this took about 15 or 20 minutes on my laptop. It may take longer on your machine. The total output size as of writing this is around 500 MB. Unfortunately, Github will not allow files larger than 100MB to be uploaded to a repository so you will have to fetch the data from USGS if you want to follow along with this project series.

Source code to the entire series is available here: