How to get data from MongoDB into Node.js

So you’ve got a Node.js website, and created a MongoDB database - the big question – how do you connect the two? When we installed MongoDB we also installed the native MongoDB drivers for Node.js, so that’s a good start!

If you have just arrived at this page, and want the context and framework for the rest of the files, check out my posts Creating an MVC framework for our Node.js page – getting ready for scalability and How to create a MongoDB database

In the MVC approach we will need to look at the ‘model’ code and the ‘controller’ code.

Make your code ‘non-blocking’

We will start off by looking at the controller. In a traditional stack you say:

  1. I’m going to go and get the data from the database
  2. When I’ve got it I will do this with it

This is no good for Node.js as it is a ‘blocking’ operation – nothing else can happen until you have the data back. This is essentially what we did in the MVC framework with this bit of code in home.js.

var teamlist = test_data.teamlist;  
var strTeam = "",  
  i = 0;
for (i = 0; i < teamlist.count;) {  
  strTeam = strTeam + "<li>" + teamlist.teams[i].country + "</li>";
  i = i + 1;
}

` In that particular instance it was correct to do it this way, as we knew that the data-retrieval operation was non-blocking – we instantly pushed back a JSON object into the teamlist variable.

As pulling data from a database is a ‘blocking’ operation we need to handle it differently. We need to say:

  1. Go get this data from the database
  2. When you’ve got it, do this with it
  3. Don’t disturb me until you’re done, I’ve got other things to do

Sound like a callback function to you? It does to me!

Creating the controller

So let’s take a look at the new controller code and see what’s going on. I have created a new file for this, called home-mongo.js.

var template = require('../views/template-main');  
var mongo_data = require('../model/mongo-data');  
exports.get = function(req, res) {  
  mongo_data.teamlist("D", function(err, teamlist) {
    if (!err) {
      var strTeam = "",
        i = 0;
      for (i = 0; i < teamlist.count;) {
        strTeam = strTeam + "<li>" + teamlist.teams[i].country + "</li>";
        i = i + 1;
      }
      strTeam = "<ul>" + strTeam + "</ul>";
      res.writeHead(200, {
        'Content-Type': 'text/html'
      });
      res.write(template.build("Test web page on node.js", "Hello there", "<p>The teams in Group " + teamlist.GroupName + " for Euro 2012 are:</p>" + strTeam));
      res.end();
    } else {
      res.writeHead(200, {
        'Content-Type': 'text/html'
      });
      res.write(template.build("Oh dear", "Database error", "<p>Error details: " + err + "</p>"));
      res.end();
    }
  });
};

This is very similar to our *home.js *file, just a few important differences.

Firstly, we ‘require’ a different model file, mongo-data.js that we will look at shortly.

Secondly, rather than pulling the data and doing something with it, we are using a callback function to say “do this with the data when you’ve got it”. What we are doing with the data is exactly the same as before. Note that we are sending the string “D” through as a variable; this is the string we will be searching for in the database.

Thirdly, through the callback function we are also sending instructions for what to do if an error occurs getting the data.

Using Node.js to query the MongoDB data

Before we look at the code, let’s think about what we’re looking to achieve:

  1. Connect to the MongoDB database
  2. Have a function that is expecting a string variable and a callback
  3. Search the database for the variable (a specific key of ‘GroupName’)
  4. Create a JSON object if successful
  5. Catch the error if not successful
  6. Pass the JSON object or the error to the provided callback function
  7. Close the database connection

Easy? Not as hard as you might think. So here’s the code for the *mongo-data.js *file:

var mongo = require('mongodb'),  
  Server = mongo.Server,
  Db = mongo.Db;
var server = new Server('localhost', 27017, {  
  auto_reconnect: true
});
var db = new Db('euro2012', server);  
var onErr = function(err, callback) {  
  db.close();
  callback(err);
};
exports.teamlist = function(gname, callback) {  
  db.open(function(err, db) {
    if (!err) {
      db.collection('teams', function(err, collection) {
        if (!err) {
          collection.find({
            'GroupName': gname
          }).toArray(function(err, docs) {
            if (!err) {
              db.close();
              var intCount = docs.length;
              if (intCount > 0) {
                var strJson = "";
                for (var i = 0; i < intCount;) {
                  strJson += '{"country":"' + docs[i].country + '"}'
                  i = i + 1;
                  if (i < intCount) {
                    strJson += ',';
                  }
                }
                strJson = '{"GroupName":"' + gname + '","count":' + intCount + ',"teams":[' + strJson + "]}"
                callback("", JSON.parse(strJson));
              }
            } else {
              onErr(err, callback);
            }
          }); //end collection.find 
        } else {
          onErr(err, callback);
        }
      }); //end db.collection
    } else {
      onErr(err, callback);
    }
  }); // end db.open
};

The first five lines are all about setting up the MondoDB server and connecting to our database. You can learn more about that on the MongoDB Native Driver GitHub page.

Then we have a function onErr that we will call if we catch an error at any point.

Now comes the fun part! The MongoDB Native driver comes with a number of non-blocking functions built in. We are using three, to:

  1. Open our database (db.open)
  2. Find the ‘collection’ we specify (db.collection)
  3. Find the documents in that collection that match the query we specify (collection.find().toArray)

These all run as nested callbacks, which is where the code can get tricky to follow. To help with this I recommend adding a comment at the end of each callback function, like you can see in the code.

Counter-intuitively, the collection.find method doesn’t actually execute the query, that’s done by the toArray method. So that’s where we pass our next callback function. This part is where we take the docs returned by the database and use them to create a JSON object in the format we want.

The finished JSON object should look like this:

{
  "GroupName": "D",
  "count": 4,
  "teams": [{
    "country": "England"
  }, {
    "country": "France"
  }, {
    "country": "Sweden"
  }, {
    "country": "Ukraine"
  }]
}

When the JSON object is complete we finally use the callback function, sending the object through.

The last thing we need to do – if you’ve been playing along at home – is update our router file to call the controller.

So in router.js file just change the home controller to be home-mongo, like so:

if (path === '/' || path === '/home') {  
  require('./controllers/home-mongo').get(req, res);
}

If you run the site, you shouldn’t really see any change, as all of the data manipulation is happening behind the scenes.

Next up we’ll see if we can simplify this process by using Mongoose, Node.js and MongoDB together.

Get the source code

**The source code for this and my other tutorials on Node.js is available on GitHub here: https://github.com/simonholmes/knowing-node/