Node JS nginx & pm2 config

This is for personal reference and I make no guarantees any of this will work for you. I no longer need it, but don’t want to lose this either. Future reference, and all that.

upstream pm2_nodejs_upstream {
    server node:4000;
    keepalive 64;
}

server {
    listen 80;
    server_name api.example.com;
    root /data;

    # location / {
    #     return 200 "Hello from Example.com";
    # }
    
    location / {
    	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    	proxy_set_header Host $http_host;
    	proxy_set_header X-NginX-Proxy true;
    	proxy_http_version 1.1;
    	proxy_set_header Upgrade $http_upgrade;
    	proxy_set_header Connection "upgrade";
    	proxy_max_temp_file_size 0;
    	proxy_pass http://pm2_nodejs_upstream/;
    	proxy_redirect off;
    	proxy_read_timeout 240s;
    }

    error_log /var/log/nginx/api.example.com_error.log;
    access_log /var/log/nginx/api.example.com_access.log;
}

 

node-gyp rebuild Error on Ubuntu 15.04 with Node 4.x.x

I’m going to start with the tl;dr version:

if you are experiencing this issue, and you have recently installed Node 4.x.x, then this is likely your issue.

Solutions?

Find another package, or use an older version of node for now if you absolutely must use that package.

The Longer Version

I had been trying to install “webkid-react-starterkit@0.3.1” on my Ubuntu dev box for ages last night, and spent a wasted 25 minutes again on it tonight.

The error was staring at me right in the face honestly:

gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:270:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Linux 3.19.0-30-generic
gyp ERR! command "/usr/bin/nodejs" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/chris/Development/php/dunglas-api-test/react-example/node_modules/jest-cli/node_modules/jsdom/node_modules/contextify
gyp ERR! node -v v4.2.1
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok 
npm ERR! Linux 3.19.0-30-generic
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! node v4.2.1
npm ERR! npm  v2.14.7
npm ERR! code ELIFECYCLE

npm ERR! contextify@0.1.14 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the contextify@0.1.14 install script 'node-gyp rebuild'.
npm ERR! This is most likely a problem with the contextify package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls contextify
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/chris/Development/php/dunglas-api-test/react-example/npm-debug.log

It’s likely not as easy to see here, but on my terminal output, it’s colour coded and it’s rather glaringly obvious:

npm ERR! Failed at the contextify@0.1.14 install script 'node-gyp rebuild'.

I went round many houses.

From trying to install contextify globally, to completely reinstalling node, to even getting down to specifying the exact version of the v8 profile I wanted (npm install v8-profiler@3.6.2-1 ).

Nothing worked.

Anyway, it turns out it’s incompatible with this version of node all along.

NodeJS 4.0.0 only tagged as stable on 8th September 2015, and here I am running 4.2.1.

Buckle in, it’s going to be a wild ride.

Models not working in Loopback Passport Module

I’ve been having a heck of a time trying to get Passport, or more specifically, the Loopback Passport Component to work with my Strongloop API project.

I should say, this is an existing project into which I have installed the loopback passport component (npm i loopback-component-passport --save) rather than using the Loopback Passport Example – which I also have had plenty of problems with.

Today’s error was when loading the Strongloop Composer (v1.8.1), I could see my datasources, but the models were empty:

strongloop-loopback-empty-models-in-slc-arc

When opening up the Chrome developer tools > console, I could see this:

TypeError: Cannot read property 'dataSource' of undefined
    at Array. (model.services.js:179)
    at Object.forEach (angular.js:323)
    at model.services.js:161
    at angular-resource.js:627
    at processQueue (angular.js:13248)
    at angular.js:13264
    at Scope.parent.$get.Scope.$eval (angular.js:14466)
    at Scope.parent.$get.Scope.$digest (angular.js:14282)
    at Scope.scopePrototype.$digest (hint.js:1955)
    at Scope.parent.$get.Scope.$apply (angular.js:14571)

But Googling this error wasn’t coming up with much in the way of suggestions.

Another strange thing was that I could add in a new Model, save it, but then if I refreshed the page, it would disappear. The model itself had been created in the common/models dir. I could see the .js and .json files in my WebStorm project, and if I tried to create a new Model with the same name it wouldn’t let me. But the slc arc Composer just wouldn’t display them.

Though I cannot remember exactly how I was getting to this point, I was also infrequently seeing errors in the console, with this error message:

{"error":{"name":"Error","status":404,"message":"Unknown "DataSourceDefinition" id "undefined".","statusCode":404,"code":"MODEL_NOT_FOUND","stack":"Error: Unknown "DataSourceDefinition" id "undefined".\n    at Function.convertNullToNotFoundError (/usr/local/lib/node_modules/strongloop/node_modules/loopback/lib/persisted-model.js:81:17)\n    at invokeRestAfter (/usr/local/lib/node_modules/strongloop/node_modules/loopback/node_modules/strong-remoting/lib/rest-adapter.js:445:25)\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:607:21\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:246:17\n    at iterate (/usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:146:13)\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:157:25\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:248:21\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:612:34\n    at /usr/local/lib/node_modules/strongloop/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:626:11\n    at execStack (/usr/local/lib/node_modules/strongloop/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:460:7)"}}

I’m leaving that in, in the hope that any future Googlers may save themselves some time.

My Fix

To get this working I ended up creating a brand new project (slc loopback) and then piece by piece, copying my existing project across and watching for where it broke.

It turns out it was largely the Passport module config that was breaking the project, so here are the relevant, working files:

Check your versions. Loopback changes a lot.

// package.json
{
  "name": "tester",
  "version": "1.0.0",
  "main": "server/server.js",
  "scripts": {
    "pretest": "jshint ."
  },
  "dependencies": {
    "compression": "^1.0.3",
    "cors": "^2.5.2",
    "errorhandler": "^1.1.1",
    "express-session": "^1.11.3",
    "loopback": "^2.14.0",
    "loopback-boot": "^2.6.5",
    "loopback-component-passport": "^1.5.0",
    "loopback-connector-mysql": "^2.2.0",
    "loopback-datasource-juggler": "^2.19.0",
    "passport-local": "^1.0.0",
    "passport-tumblr": "^0.1.2",
    "serve-favicon": "^2.0.1"
  },
  "optionalDependencies": {
    "loopback-explorer": "^1.1.0"
  },
  "devDependencies": {
    "jshint": "^2.5.6"
  },
  "repository": {
    "type": "",
    "url": ""
  },
  "description": "tester"
}

In the model-config.json, note the inclusion of "./node_modules/loopback-component-passport/lib/models" in the meta > sources, the UserCredential, and UserIdentity models:

// server/model-config.jsom
{
  "_meta": {
    "sources": [
      "loopback/common/models",
      "loopback/server/models",
      "../common/models",
      "./models",
      "./node_modules/loopback-component-passport/lib/models"
    ],
    "mixins": [
      "loopback/common/mixins",
      "loopback/server/mixins",
      "../common/mixins",
      "./mixins",
      "../node_modules/loopback-ds-timestamp-mixin"
    ]
  },
  "User": {
    "dataSource": "db",
    "public": true
  },
  "AccessToken": {
    "dataSource": "db",
    "public": false
  },
  "ACL": {
    "dataSource": "db",
    "public": false
  },
  "RoleMapping": {
    "dataSource": "db",
    "public": false
  },
  "Role": {
    "dataSource": "db",
    "public": false
  },
  "UserCredential": {
    "dataSource": "db",
    "public": false
  },
  "UserIdentity": {
    "dataSource": "db",
    "public": false
  },
  "user": {
    "dataSource": "db",
    "public": true
  },
  "userCredential": {
    "dataSource": "db",
    "public": true
  },
  "userIdentity": {
    "dataSource": "db",
    "public": true
  },
  "accessToken": {
    "dataSource": "db",
    "public": true
  },
  "acl": {
    "dataSource": "db",
    "public": true
  },
  "roleMapping": {
    "dataSource": "db",
    "public": true
  },
  "role": {
    "dataSource": "db",
    "public": true
  },
  "scope": {
    "dataSource": "db",
    "public": true
  }
}

Why the duplicates, e.g. User and user?

Well, you have to define the base model, and then your own model which extends the base. But both have to be defined.

This bit left me confused, so here are each of the relevant models for reference:

// common/models/access-token.json 
{
  "name": "accessToken",
  "plural": "accessTokens",
  "base": "AccessToken",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/acl.json 
{
  "name": "acl",
  "plural": "acls",
  "base": "ACL",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/role.json
{
  "name": "role",
  "plural": "roles",
  "base": "Role",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/role-mapping.json
{
  "name": "roleMapping",
  "plural": "roleMappings",
  "base": "RoleMapping",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/scope.json
{
  "name": "scope",
  "plural": "scopes",
  "base": "Scope",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/user.json
{
  "name": "user",
  "plural": "users",
  "base": "User",
  "properties": {
    "accessTokens": {
      "type": "hasMany",
      "model": "accessToken",
      "foreignKey": "userId"
    },
    "identities": {
      "type": "hasMany",
      "model": "userIdentity",
      "foreignKey": "userId"
    },
    "credentials": {
      "type": "hasMany",
      "model": "userCredential",
      "foreignKey": "userId"
    }
  },
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/user-credential.json
{
  "name": "userCredential",
  "plural": "userCredentials",
  "base": "UserCredential",
  "properties": {},
  "validations": [],
  "relations": {
    "user": {
      "type": "belongsTo",
      "model": "user",
      "foreignKey": "userId"
    }
  },
  "acls": [],
  "methods": []
}
// common/models/user-identity.json
{
  "name": "userIdentity",
  "plural": "userIdentities",
  "base": "UserIdentity",
  "properties": {},
  "validations": [],
  "relations": {
    "user": {
      "type": "belongsTo",
      "model": "user",
      "foreignKey": "userId"
    }
  },
  "acls": [],
  "methods": []
}

Once the models are displaying again, feel free to change the datasources:

// server/datasources.json 
{
  "db": {
    "name": "db",
    "connector": "memory"
  },
  "mysql": {
    "host": "your.server.ip.here",
    "database": "dbname",
    "password": "dbpassword",
    "name": "mysql",
    "connector": "mysql",
    "user": "dbuser"
  }
}

Note, those are the settings created by Strongloop slc arc. If you enter them manually, using the data / fields in the manual, you may end up with duplicates – e.g. user and username.

My Loopback Passport Module Config

And lastly, and probably the most important is the server.js file.

// server/server.js
var loopback = require('loopback');
var boot = require('loopback-boot');

var app = module.exports = loopback();

var PassportConfigurator = require('loopback-component-passport').PassportConfigurator;
var passportConfigurator = new PassportConfigurator(app);

app.start = function() {
  // start the web server
  return app.listen(function() {
    app.emit('started');
    console.log('Web server listening at: %s', app.get('url'));
  });
};

// Bootstrap the application, configure models, datasources and middleware.
// Sub-apps like REST API are mounted via boot scripts.
boot(app, __dirname, function(err) {
  if (err) throw err;

  // start the server if `$ node server.js`
  if (require.main === module)
    app.start();
});

// Enable http session
app.use(loopback.session({ 
  secret: 'keyboard cat',  
  resave: true,
  saveUninitialized: true 
}));

// Load the provider configurations
var config = {};
try {
  config = require('./providers.json');
} catch(err) {
  console.error('Please configure your passport strategy in `providers.json`.');
  console.error('Copy `providers.json.template` to `providers.json` and replace the clientID/clientSecret values with your own.');
  process.exit(1);
}
// Initialize passport
passportConfigurator.init();

// Set up related models
passportConfigurator.setupModels({
  userModel: app.models.User,
  userIdentityModel: app.models.UserIdentity,
  userCredentialModel: app.models.UserCredential
});
// Configure passport strategies for third party auth providers
for(var s in config) {
  var c = config[s];
  c.session = c.session !== false;
  passportConfigurator.configureProvider(s, c);
}

The ordering here seems to be the most important part.

If your passport config tries to call the models before they are ready, that seems to cause a lot of problems. It sounds obvious in hindsight, but when you have a lot of things going on in your server.js file, sometimes it can be hard to see the wood for the trees.

How to get the Latest NPM Dependencies

I have been doing a fair amount of work lately with the Node.JS Loopback framework.

The more I use the framework, the more I find myself enjoying it.

marcus-wareing-great-british-menu
I liked your dish, but…

But, there’s one thing about Loopback – and pretty much every Node development project I have worked on – that I find needlessly time consuming:

Finding the latest versions of your NPM dependencies.

Up until very recently, the way I was doing this was comically laborious.

I would open the package.json file, take a look at the installed packages:

node-js-package-json-dependencies

And then… *cringe*, head over to NPMJS, manually search for each project, then take a note of the latest version:

npmjs-latest-version

And pop that info into my package.json file. Rinse and repeat for all the NPM dependencies in my project.

This is 2015. There has to be a better way.

And there is.

Using Node to Update Your NPM Dependencies

There’s a really useful package for keeping your NPM dependencies up to date with the very latest versions – it’s called NPM Check Updates (NCU) by Tomas Junnonen.

Straight from the ReadMe:

npm-check-updates is a command-line tool that allows you to find and save the latest versions of dependencies, regardless of any version constraints in your package.json file (unlike npm itself).

npm-check-updates maintains your existing semantic versioning policies, i.e., it will upgrade your"express": "^4.11.2" dependency to "express": "^5.0.0" when express 5.0.0 is released.

npm-dependency-update-with-ncu

A word of caution, however.

Whilst this seems like a great time saver – and it is – be sure to have some way of checking (ahem, tests) that the latest and greatest packages haven’t inadvertently completely broken your project.

A great example of how this can go wrong would be in the screenshot above. Here, I pulled in the Loopback Passport Example project, ran the ncu utility, and promptly ended up with a project that won’t even start.

As a side note, if you are wanting to add tests to a Node project, I highly recommend Tape. Initially, it’s a little different to normal testing libraries, but having taken the plunge, I really like it.