April 9, 2013
Now we have an application that is made up of a twitter worker and an express application that serves up data. The next piece of the puzzle is deploying our application on the web. That used to require a lot of work, since we’d have to purchase server space, set up the server with required software, point a domain name to our application, etc. Times have changed a lot since then, and there are now services called “platforms-as-a-service” that take care of the hard work for you.
We’ll use CloudFoundry, the open-source platform as a service, but there are many other options including Heroku and Nodejitsu. For CloudFoundry, you’ll start by signing up for an account here.
Next, we’ll set up the command-line client for CloudFoundry. To do so, we’ll need to start by installing Ruby. This video walks you through installing Ruby (and the RedCar text editor)in both MacOS and Windows.
Once you have Ruby installed, open your ruby command prompt (which will just be your normal terminal window in MacOS, and in Windows you’ll have an application called “Open Command Prompt with Ruby”). Once there, install the vmc client using RubyGems:
$ sudo gem install vmc
Now, let’s create a very simple Node.js program and deploy it. Let’s use the following program:
var http = require("http");
var server = http.createServer(function (req, res) {
res.writeHeader({
"Content-Type":"text/html"
});
res.end("Hello from CloudFoundry!");
}).listen(3000);
Once we’re sure that our programming is correctly running locally, we’re ready to deploy it. To do this, we’ll open up our ruby command prompt and navigate to the directory storing the program. Once there, we’ll type a series of commands:
$ vmc target api.cloudfoundry.com
Setting target to https://api.cloudfoundry.com... OK
$ vmc login
target: https://api.cloudfoundry.com
Email> youremail@example.com
Password> *********
Authenticating... OK
Now we’ll deploy using the push command. This will enter us into a dialogue with vmc.
$ vmc push
Name> cloudfoundrytutorial
Instances> 1
1: node
2: other
Framework> node
1: node
2: node06
3: node08
4: other
Runtime> 1
1: 64M
2: 128M
3: 256M
4: 512M
5: 1G
Memory Limit> 64M
Creating cloudfoundrytutorial... OK
1: cloudfoundrytutorial.cloudfoundry.com
2: none
Domain> cloudfoundrytutorial.cloudfoundry.com
Updating cloudfoundrytutorial... OK
Create services for application?> n
Bind other services to application?> n
Save configuration?> n
Uploading cloudfoundrytutorial... OK
Starting cloudfoundrytutorial... OK
Checking cloudfoundrytutorial...
0/1 instances: 1 starting
1/1 instances: 1 running
OK
Your answers will be the same, with the exception of the name. There can only be one application with any given name and URL on CloudFoundry, so if there’s a collision you will be notified.
Once you’re successful, you can point your browser to http://[the name you entered above].cloudfoundry.com and you should see your app running.
It’s probably helpful to check out the vmc help by simply typing vmc in your ruby command prompt. You should get a list of all CloudFoundry commands. One command you’ll probably use pretty often is the apps command. This shows a list of all of your deployed apps and their current statuses.
Next, we’ll see how to connect to CloudFoundry’s Redis instance. once our app is deployed.
April 3, 2013
If you’ve followed the tutorials up to this point, we have two Node.js source files. We have app.js which is the Express server that returns the JSON object containing the counts of the words, and you have twitter.js which we can think of as a worker. The code for twitter.js looks something like this:
var twitter = require("ntwitter");
var redis = require("redis");
var credentials = require("./credentials.js");
//create redis client
var client = redis.createClient();
var t = new twitter({
consumer_key: credentials.consumer_key,
consumer_secret: credentials.consumer_secret,
access_token_key: credentials.access_token_key,
access_token_secret: credentials.access_token_secret
});
t.stream(
"statuses/filter",
{ track: ["awesome", "cool", "rad"] },
function(stream) {
stream.on("data", function(tweet) {
console.log(tweet.text);
//if awesome is in the tweet text, increment the counter
if(tweet.text.indexOf("awesome") > -1) {
client.incr('awesome');
}
});
}
);
http.createServer(function (req, res) {
client.get("awesome", function (error, awesomeCount) {
if (error !== null) {
//handle error here
console.log("error: " + error);
} else {
res.writeHead(200, {"Content-Type": "text/plain"});
res.end("The awesome count is " + awesomeCount);
}
});
}).listen(3000);
And the code for app.js might look something like this:
// We need to 'require' the
// following modules
var express = require("express"),
http = require("http"),
path = require("path"),
app = express();
// This is our basic configuration
app.configure(function () {
// Define our static file directory, it will be 'public'
app.use(express.static(path.join(__dirname, 'public')));
});
// Create the http server and get it to
// listen on the specified port 3000
http.createServer(app).listen(3000, function(){
console.log("Express server listening on port 3000");
});
// Create and return the word counts as a JSON object
app.get("/counts.json", function (req, res) {
redisClient.mget(["awesome", "cool", "rad"] , function (error, results) {
if (error !== null) {
// handle error here
console.log("ERROR: " + error);
} else {
var jsonObject = {
"awesome": results[0],
"cool": results[1],
"rad": results[2]
// ...etc
};
// use res.json to return JSON objects instead of strings
res.json(jsonObject);
}
});
});
Our current workflow is a bit unsustainable right now, since we have to run the twitter worker for a bit, then stop it, and then start our express server, to display our data to the user. We’d like to just start the application once and have it initialize the worker to run in the background.
We can do this by converting twitter.js into a module. We did something similar to have our twitter credentials file separated from our twitter.js file. A basic module looks something like this:
var ImportantFunction = function () {
//code for the module goes here
}
module.exports = ImportantFunction;
In the credentials.js example, we exported a JavaScript object literal, and that’s fine as well. In fact, your module can export multiple things if you include them as part of the “exports” object.
var ImportantFunction = function () {
//code for the function goes here
}
var objectLiteral = {
"hello":"world";
}
module.exports.ImportantFunction = ImportantFunction;
module.exports.objectLiteral = objectLiteral;
You can then access these from another file by using the require statement:
var myModule = require("./myModule.js");
//access ImportantFunction
var ImportantFunction = myModule.ImportantFunction;
var objectLiteral = myModule.objectLiteral;
For our app, we’ll simply wrap the twitter object in a function called worker, and export that directly. We’ll also remove the http server, since that will now be handled entirely by Express. This makes our code look something like this:
var worker = function () {
var twitter = require("ntwitter");
var redis = require("redis");
var credentials = require("./credentials.js");
//create redis client
var client = redis.createClient();
var t = new twitter({
consumer_key: credentials.consumer_key,
consumer_secret: credentials.consumer_secret,
access_token_key: credentials.access_token_key,
access_token_secret: credentials.access_token_secret
});
t.stream(
'statuses/filter',
{ track: ["awesome", "cool", "rad"] },
function(stream) {
stream.on("data", function(tweet) {
console.log(tweet.text);
//if awesome is in the tweet text, increment the counter
if(tweet.text.indexOf("awesome") > -1) {
client.incr("awesome");
}
});
}
);
};
module.exports = worker;
And now we can require it in our app.js program by modifying it to be the following:
// We need to 'require' the
// following modules
var express = require("express"),
http = require("http"),
path = require("path"),
app = express(),
twitterWorker = require("./twitter.js");
And then we can start the worker before we start the server by calling the function:
// Start our Twitter worker
twitterWorker();
// Create the http server and get it to
// listen on the specified port 3000
http.createServer(app).listen(3000, function(){
console.log("Express server listening on port 3000");
});
Now, running the Express server should launch the twitter worker. If you load localhost:3000, you should be able to reload the page and watch the numbers update.
March 27, 2013
If you worked through the previous tutorials, I hope that you managed to create a basic node.js development environment using Vagrant, stream data from twitter, and, finally, store aggregate information about that data in a Redis data store. To get data from the Redis data store to the web browser, we used the http library to return a simple text page. So if you also completed the suggested exercise in the previous tutorial, your code might look something like this:
var server = http.createServer(function (req, res) {
client.mget(["awesome","cool"], function(err, results) {
if (err !== null) {
//handle error here
console.log("error: " + err);
} else {
var response = "<b>Hello from my http server!!</b>";
response += "<p>Total awesome: " + results[0] + "</p>";
response += "<p>Total cool: " + results[1] + "</p>";
res.writeHead(200, {"Content-Type": "text/html"});
res.end(response);
}
});
}).listen(3000);
Note that we used Redis’s mget command to send in an array of keys and get back an array of values. This isn’t something that I described in my previous blog posts, but it is described in the Redis documentation. If you continue to use Redis, it would be helpful to become familiar with some of the commands and data structures that Redis offers.
Now suppose we wanted to actually style this code and include some javascript. Notice how quickly this can add complexity to our callback function:
var server = http.createServer(function (req, res) {
client.mget(["awesome","cool"], function(err, results) {
if (err !== null) {
//handle error here
console.log("error: " + err);
} else {
var response += "<script>";
response += "//script stuff here";
response += "</script>";
response += "<style>";
response += "//style stuff here";
response += "</style>";
response += "<b>Hello from my http server!!</b> <br/>";
response += "<p>Total awesome: " + results[0] + "</p>";
response += "<p>Total cool: " + results[1] + "</p>";
res.writeHead(200, {"Content-Type": "text/html"});
res.end(response);
}
});
}).listen(3000);
This is definitely an unsustainable approach. In fact, if you’re coming from an apache or rails background, this likely seems a little ridiculous. Does this mean we need to hard-code static html files as strings in our code?
That’s where the Express web framework comes in. Express is a web application framework for Node.js that is inspired by the Sinatra framework for Ruby. It creates a layer on top of the http library which provides these features among others. In this post, I’ll describe how to install express and create an Express application. Then I’ll talk a little bit about how to edit and create routes in your application.
Start by firing up your Vagrant virtual machine and ssh’ing into it. Next, you’ll want to add express to your package.json file. After adding it, your file should look something like this:
{
"name": "tutorial",
"description": "a tutorial on using node, twitter, redis, and express",
"version": "0.0.1",
"dependencies": {
"express": "3.1.x",
"ntwitter": "0.5.x",
"redis": "0.8.x"
}
}
And now we can use npm install to install it in our node_modules directory.
$ npm install --no-bin-links
Next, we’ll create a very basic express application in a new file. Create a file called app.js that looks like this:
// We need to 'require' the
// following modules
var express = require("express"),
http = require("http"),
path = require("path"),
app = express();
// This is our basic configuration
app.configure(function () {
// Define our static file directory, it will be 'public'
app.use(express.static(path.join(__dirname, 'public')));
});
// Create the http server and get it to
// listen on the specified port 3000
http.createServer(app).listen(3000, function(){
console.log("Express server listening on port 3000");
});
app.get("/", function (req, res) {
//send "Hello World" to the client as html
res.send("Hello World!");
});
Save this file and run it using Node.
$ node app.js
You should see that Express is listening on port 3000, and now you should be able to point your browser to localhost:3000 and see “Hello World!”
We can specify other routes by providing other calls to the get method in the app object. For example, add the following to your code after the “Hello World” route:
app.get("/goodbye", function (req, res) {
//send "Goodbye World" to the client as html
res.send("Goodbye World!");
});
app.get("/login", function (req, res) {
res.send("You need to login!");
});
Run your server and then visit “localhost:3000/goodbye” and “localhost:3000/login” in your web browser. You should see the message associated with the route.
We can add any number of routes to our server, but we’re really interested in serving up static html, css and JavaScript files so that we don’t have to create long strings. To do this, we’ll create a public directory in our app folder:
mkdir public
In our express configuration, we’ve specified that this directory will be used to serve our static content. Inside that directory, let’s make an “index.html” file with the following content.
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<p>This is a static file!</p>
<script src="http://code.jquery.com/jquery-1.9.1.min.js"></script>
</body>
</html>
We should now be able to go to “localhost:3000/index.html” in our browser and see the contents of this file displayed as HTML. Now we’ll create a route that returns JSON from our server, and we’ll have our client side JavaScript request it via Ajax. To start with, we’ll create a route that connects to redis and gets our awesomeCount out. Make sure you’ve let the twitter.js program run for a few seconds so that Redis is populated with a count.
Since we don’t have the Redis module included in our app.js file, let’s modify the require section of our code to look something like this:
var express = require("express"),
http = require("http"),
path = require("path"),
redisClient = require("redis").createClient(),
app = express();
Next, add the following route to your express app:
app.get("/counts.json", function (req, res) {
redisClient.get("awesome", function (error, awesomeCount) {
if (error !== null) {
// handle error here
console.log("ERROR: " + error);
} else {
var jsonObject = {
"awesome":awesomeCount
};
// use res.json to return JSON objects instead of strings
res.json(jsonObject);
}
});
});
Now if we go to “localhost:3000/counts.json” in our web browser, we should see the file as a server-generated JSON object. We’ll continue by creating a client-side JavaScript file to consume the JSON object. Inside your “public” directory, create a directory called javascripts and a file called client-app.js with the following content:
var main = function () {
$.getJSON("/counts.json", function (response) {
$("body").append("<p>awesome:"+response.awesome+"</p>");
});
};
$(document).ready(main);
And now include that file in your index.html with a script tag, right below the jQuery script tag:
<script src="/javascripts/client-app.js"></script>
After that, if you navigate your browser back to localhost:3000/index.html, you should see the number of times the word “awesome” has appeared.
It should now be straightforward to create any number of routes to pull data from Redis as JSON objects. If you’ve successfully used mget to pull data from Redis on the server as in the previous example, it should be relatively easy to modify the counts.json route callback to return an array of all of the counts. Do that now and have your client display all of them with jQuery.
