blog.semmy.me

Web Applications with Express and Node.js

March 18, 2012

If you worked through the previous tutorials, I hope that you managed to create a basic node.js application, stream data from twitter, and, finally, store aggregate information about that data in a redis data store. To serve up this data in an application, we used the http library to serve up web pages. So if you also completed the suggested exercise in the previous tutorial, your code might look something like this:

Note that we used redis’s mget command to send in an array of keys and get back an array of values. This isn’t something that I described in my previous blog posts, but it is described in the redis command documentation. If you continue to use redis, it would be helpful to become familiar with some of the commands that redis offers.

Now suppose we wanted to actually style this code and include some javascript. Notice how quickly this can add complexity to our callback function:

This is definitely an unsustainable approach. In fact, if you’re coming from an apache or rails background, this likely seems a little ridiculous. Does this mean we need to hard-code static html files?

Well, the answer is yes and no. If you stick with the core node.js API, there is no built-in method for serving up static html files (at least not to my knowledge). There’s also no built-in method for serving up html templates and partial views like in rails.

That’s where the Express web framework comes in. Express is a web application framework that is inspired by the Sinatra framework for Ruby. It provides a layer on top of the http library that provides these features among others. In this post, I’ll describe how to install express and create an Express application. Then I’ll talk a little bit about how to edit and create routes in your application.

If you’re using vagrant, start by firing up your vagrant virtual machine and ssh’ing into it. Next, you’ll want to install the express binary globally using npm. We haven’t done this type of npm install before; the difference is that we have to use sudo (since this will install express into a system directory) and add the -g flag. Since this is a global operation, it doesn’t matter which directory you are in.

$ sudo npm install -g express

Now we can easily bootstrap an express application. Make sure you’re in the app directory of your virtual machine. You can verify this by typing pwd (print working directory) at the command line. Your output should look something like this:

$ pwd /home/vagrant/app

If you’re not in /home/vagrant/app, go ahead and use cd to get there. Now we’ll create a new application using the express binary.

$ express destination is not empty, continue? Y create : . create : ./package.json create : ./app.js create : ./public create : ./public/javascripts create : ./public/images create : ./public/stylesheets create : ./public/stylesheets/style.css create : ./routes create : ./routes/index.js create : ./views create : ./views/layout.jade create : ./views/index.jade dont forget to install dependencies: $ cd . && npm install

Note that express created a series of files, including a file of required dependencies file called package.json. Thanks to that file, we can install all of the new required dependencies by making sure we’re in the directory with package.json and using

npm install

Now you’ll have the Express file structure in your app directory, and your default express application should be in the file called app.js. You can run your express application by running the following command:

node app.js

Now pointing your browser to localhost:3000, you should see the ‘Welcome to Express’ message. Kill your server by typing CTRL-C, and let’s take a look at the directory structure that express created. Type ls at the command prompt and, in addition to server.js, twitter.js and credentials.js, you should see all of the following files/directories:

app.js: the auto-generated express application
node_modules: your application’s module dependencies
package.json: a file that specifies your application’s dependencies
public: your static client-side files (usually stylesheets and client-side JavaScript files, but it sometimes includes html files)
routes: callbacks for the various routes in your application
views: your html templates

For this blog post, we’re primarily interested in the app.js file and the routes directory. Open up the app.js file and take a look at the section that looks like

// Routes app.get(‘/’, routes.index);

Routes are the URL paths in your application. For example, let’s suppose you had a web application located at mygreatwebapp.com. If your application allowed for multiple users, you might want the URL for their user profile to appear at mygreatwebapp.com/[user]. Both github and twitter do this: visit github.com/semmypurewal or twitter.com/semmypurewal. In this code, we have a single route that is just the index of the page. The callback is defined in a file in the routes directory, but we’ll take a look at that in a moment.

Start up your application and visit localhost:3000. You should get the same ‘Welcome to Express’ message as before, because that’s the ‘/’ route that is defined above. Now try to visit localhost:3000/semmypurewal. You should get an error that says something like ‘Cannot GET /semmypurewal.’ So how do we add a new route to handle that type of request? Modify the code above to add a route like this one:

// Routes app.get(‘/’, routes.index); app.get(‘/semmypurewal’, function(req, res) { res.send(‘Welcome to the profile of Semmy Purewal’); });

Now restart your app, and point your browser to localhost:3000. You’ll see the same ‘Welcome to Express’ screen as before. But if you point your browser to localhost:3000/semmypurewal, you’ll now see the application handles the request as you describe in your callback function.

This is great, but obviously, if our application has over a million users it would be tedious to create a separate callback for each one. But we can easily generate an infinite number of routes by using variables. For example, try adding a route like this:

// Routes app.get(‘/’, routes.index); app.get(‘/semmypurewal’, function(req, res) { res.send(‘Welcome to the profile of Semmy Purewal’); }); app.get(‘/users/:user’, function(req, res) { res.send(‘Welcome to the profile of ’ + req.params.user + ‘!’); });

Now you can point your browser to localhost:3000/users/semmypurewal or localhost:3000/users/helloworld. In fact, it will respond to any user name. Typically, in the callback for something like this, we would check to see if the username exists and if it does, we would render the page, otherwise we would send a message like ‘that user was not found!’ For now, we’ll keep things simple, though.

But because the code in our callback has the propensity to callback become complicated it’s a good idea to move it into a different file. It’s best to leave express’s app.js file for specifying the configuration of the application along with its routes. Open up the routes/index.js and you’ll see something that looks like this:

exports.index = function(req, res){ res.render(‘index’, { title: ‘Express’ }) };

The index is rendering a view, which we’ll learn about in the next blog post. For now, let’s modify this file to include the user route like this:

exports.index = function(req, res){ res.render(‘index’, { title: ‘Express’ }) }; exports.user = function(req, res) { res.send(‘Welcome to the profile of ’ + req.params.user + ‘!’); }

and also change the route specification in app.js to look like this:

app.get(‘/’, routes.index); app.get(‘/users/:user’, routes.user);

Now starting up your app should allow for both routes with the callbacks specified in a separate file!

Obviously, we’re more interested in having routes for all of the words we’re tracking on twitter, so go ahead and create a route called ‘words’ that takes in a parameter. Specifically, we’d like the route ‘localhost:3000/words/awesome’ to show us the number of times awesome has appeared, while in the callback for a word we’re not tracking it should show us ‘word not found’. Remember that, in order to get this working, you’ll need to require(‘redis’) at the top of your express file, and create a client. If you can do this you’re doing great!

In the next blog post, we’ll learn how how to render views using embedded JavaScript (ejs). We’ll also learn a slightly better way of organizing route callbacks which will be more in-line with the Model-View-Controller architecture.

Persistence with Redis and Node.js

February 26, 2012

In my previous two blog posts, we set up a Node.js development environment using Vagrant and Chef, and then we used the ntwitter module to connect to Twitter’s streaming API. Next, we’d like to store some information on the server and then display this information to the user. In this post, we’re going to focus on the former: we’ll write some code that keeps tracks some aggregate information about the tweets using Redis. I’ll assume you’ve worked your way through the previous two tutorials.

Redis is a key-value store; some refer to it as a data-structure server. These are different from traditional databases in that they store information in volatile memory (keeping it very fast) and also allow you to organize data into traditional structures (hashes, lists, sets, etc). It is perfect for storing data that needs to be accessed quickly (like session information) or for caching to improve the response time of your applications.

Our goal is to keep track of the number of times each of the following words appear: ‘awesome,’ ‘cool,’ ‘rad,’ ‘gnarly,’ and ‘groovy.’ To do this, we’ll simply use a key for each of the words, and each value will be an integer representing the number of times the word appears.

To get a feeling for how redis works, let’s interact with it via the command line client. Fire up vagrant and ssh into the box

$ vagrant up
$ vagrant ssh

Now we should be logged into our virtual machine where redis is already installed and configured. The following command starts up the redis client and creates a key for ‘awesome’ and sets its value to 0.

$ redis-cli
redis 127.0.0.1:6379> set awesome 0

If all goes well, redis should respond with ‘OK.’ We can check the value of the key by using the get command, and we can increment it by using the incr command.

redis 127.0.0.1:6379> get awesome
"0"
redis 127.0.0.1:6379> incr awesome
(integer) 1
redis 127.0.0.1:6379> incr awesome
(integer) 2
redis 127.0.0.1:6379> get awesome
"2"

To exit out of the interactive redis client, you can type ‘exit’ or press CTRL-D to send an End-Of-File character to the process. If you want to practice and learn more about redis, try out this outstanding interactive redis tutorial.

So now we know how to create a key for each word and then increment the value interactively, but we’d like to be able to do it programmatically in Node. To do this, we’ll need to install the redis module via npm. To do this enter your app directory on your virtual machine and install redis.

$ cd app
$ npm install redis

Next, on your host machine, open the twitter.js file that we created in the last tutorial. It should be in the app directory, and you can open it in any text editor you’d like. Modify it so it looks like this:

var twitter = require('ntwitter');
var redis = require('redis');
var credentials = require('./credentials.js');

//create redis client                                                                                                                                                                                                                       
var client = redis.createClient();

//if the 'awesome' key doesn't exist, create it                                                                                                                                                                                             
client.exists('awesome', function(error, exists) {
    if(error) {
        console.log('ERROR: '+error);
    } else if(!exists) {
        client.set('awesome', 0); //create the awesome key
    };
});

var t = new twitter({
    consumer_key: credentials.consumer_key,
    consumer_secret: credentials.consumer_secret,
    access_token_key: credentials.access_token_key,
    access_token_secret: credentials.access_token_secret
});

t.stream(
    'statuses/filter',
    { track: ['awesome', 'cool', 'rad', 'gnarly', 'groovy'] },
    function(stream) {
        stream.on('data', function(tweet) {
            console.log(tweet.text);
            //if awesome is in the tweet text, increment the counter                                                                                                                                                                        
            if(tweet.text.match(/awesome/)) {
                client.incr('awesome');
            }
        });
    }
);

The new lines in this program illustrate the non-blocking, event-driven nature of Node.js. If you’re coming from a more traditional programming background, you might be more comfortable with this approach checking the existence of the awesome key:

var exists = client.exists('awesome'); //returns true if the key exists
if(!exists) {
    client.set('awesome', 0);
};

In the first line of this code, note that the program halts until the Redis (or other DB) client returns the result of the query. So in the second line (the if statement), we can use the boolean value.

Node’s approach is different. It queries Redis, and while it’s waiting for the response, it continues executing the remainder of the program. That’s why we send in an anonymous function as a callback to the query — this tells node what to do when Redis returns with an answer. The variables in our anonymous function take on the actual values when Redis returns; they act like the left hand side of the assignment operator in the blocking approach.

So the non-blocking approach ends up looking like:

//redis gives a value to error and exists                                                                                                                                                                                          
client.exists('awesome', function(error, exists) {
    //if error is defined, then there was probably some
    //problem connecting to redis
    if(error) {
        console.log('ERROR: '+error);
    }
    //otherwise exists will be available, and we can do something with it
    else if(!exists) {
        client.set('awesome', 0); //create the awesome key
    };
});

You can now modify this code so it keeps track of the counts for all of the words that the ntwitter module is watching. This program will be our background worker; my next blog post will show you how to use second Node.js program to create a web server that will allow us to share these counts on a web page. We’ll use the Express web framework to do this.

Streaming Twitter with nTwitter and Node.js

February 10, 2012

Accessing the Twitter streaming API with Node.js is a breeze with the nTwitter module. This post should show you how to set up a twitter application, install ntwitter, and start getting data from the streaming API. Before we begin, you’ll need to set up a Node.js development environment in Vagrant by following the instructions here.

If you’d like to create a new project in git (which I recommend), then you’ll want to delete the .git folder in the main directory of the node-dev-bootstrap repository.

$ rm -rf .git

Enter the app directory, and initialize a new git repository.

$ cd app
$ git init

Next, login to your vagrant box, enter the app directory and add the ntwitter module via npm

$ vagrant ssh
$ cd app
$ npm install ntwitter

To get set up, you’ll need to first set up an application by heading here and logging in with your twitter credentials. Once you fill out the form and accept the licensing terms, you’ll need to go to the bottom of the page and click “create my access token” button. This will generate all of the credentials that you need to use the streaming API.

Set up a file called credentials.js in your app directory. Since the app folder is shared between your vagrant VM and your local machine, you can use any text editor to set up this file in the app folder on your local machine. This file will be organized as a CommonJS module so you can include it via a require statement in your program All you need to do is replace the strings below with your actual credentials.

var credentials = {
    consumer_key: 'your consumer key here',
    consumer_secret: 'your consumer secret here',
    access_token_key: 'your access token key here',
    access_token_secret: 'your access token secret here'
};

module.exports = credentials;

Next, we’ll create a simple program in our app directory that prints out tweets that contain any of the following words: ‘awesome’, ‘rad’, ‘cool,’ ‘gnarly,’ and ‘groovy.’ Save the following code in a file called twitter.js.

var twitter = require('ntwitter');
var credentials = require('./credentials.js');

var t = new twitter({
    consumer_key: credentials.consumer_key,
    consumer_secret: credentials.consumer_secret,
    access_token_key: credentials.access_token_key,
    access_token_secret: credentials.access_token_secret
});

t.stream(
    'statuses/filter',
    { track: ['awesome', 'cool', 'rad', 'gnarly', 'groovy'] },
    function(stream) {
        stream.on('data', function(tweet) {
            console.log(tweet.text);
        });
    }
);

If everything is set up correctly, you can run your app from your vagrant VM. If you don’t have a terminal window open that you’ve already run vagrant ssh in, you’ll need to do that first and enter the app directory.

node twitter.js

and you should see tweets (assuming anyone is tweeting about awesome, rad or cool stuff, which is highly likely). To stop the stream, press CTRL-C to halt the node process.

Note that it uses the credentials file that you created in the previous step. Why should we keep our credentials in a separate file? It’s always a good idea to keep private information (like API keys and passwords) out of your git repository. How do you do that? You can create a .gitignore file that contains the name of all files that you’d like to ignore:

credentials.js

Now typing

$ git status

should show that the only file that hasn’t been added is twitter.js and the ntwitter module. The credentials.js file is ignored. So go ahead and add the file and make a commit:

$ git add .
$ git commit -m "initial commit"