Models not working in Loopback Passport Module

I’ve been having a heck of a time trying to get Passport, or more specifically, the Loopback Passport Component to work with my Strongloop API project.

I should say, this is an existing project into which I have installed the loopback passport component (npm i loopback-component-passport --save) rather than using the Loopback Passport Example – which I also have had plenty of problems with.

Today’s error was when loading the Strongloop Composer (v1.8.1), I could see my datasources, but the models were empty:

strongloop-loopback-empty-models-in-slc-arc

When opening up the Chrome developer tools > console, I could see this:

TypeError: Cannot read property 'dataSource' of undefined
    at Array. (model.services.js:179)
    at Object.forEach (angular.js:323)
    at model.services.js:161
    at angular-resource.js:627
    at processQueue (angular.js:13248)
    at angular.js:13264
    at Scope.parent.$get.Scope.$eval (angular.js:14466)
    at Scope.parent.$get.Scope.$digest (angular.js:14282)
    at Scope.scopePrototype.$digest (hint.js:1955)
    at Scope.parent.$get.Scope.$apply (angular.js:14571)

But Googling this error wasn’t coming up with much in the way of suggestions.

Another strange thing was that I could add in a new Model, save it, but then if I refreshed the page, it would disappear. The model itself had been created in the common/models dir. I could see the .js and .json files in my WebStorm project, and if I tried to create a new Model with the same name it wouldn’t let me. But the slc arc Composer just wouldn’t display them.

Though I cannot remember exactly how I was getting to this point, I was also infrequently seeing errors in the console, with this error message:

{"error":{"name":"Error","status":404,"message":"Unknown "DataSourceDefinition" id "undefined".","statusCode":404,"code":"MODEL_NOT_FOUND","stack":"Error: Unknown "DataSourceDefinition" id "undefined".\n    at Function.convertNullToNotFoundError (/usr/local/lib/node_modules/strongloop/node_modules/loopback/lib/persisted-model.js:81:17)\n    at invokeRestAfter (/usr/local/lib/node_modules/strongloop/node_modules/loopback/node_modules/strong-remoting/lib/rest-adapter.js:445:25)\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:607:21\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:246:17\n    at iterate (/usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:146:13)\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:157:25\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:248:21\n    at /usr/local/lib/node_modules/strongloop/node_modules/async/lib/async.js:612:34\n    at /usr/local/lib/node_modules/strongloop/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:626:11\n    at execStack (/usr/local/lib/node_modules/strongloop/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:460:7)"}}

I’m leaving that in, in the hope that any future Googlers may save themselves some time.

My Fix

To get this working I ended up creating a brand new project (slc loopback) and then piece by piece, copying my existing project across and watching for where it broke.

It turns out it was largely the Passport module config that was breaking the project, so here are the relevant, working files:

Check your versions. Loopback changes a lot.

// package.json
{
  "name": "tester",
  "version": "1.0.0",
  "main": "server/server.js",
  "scripts": {
    "pretest": "jshint ."
  },
  "dependencies": {
    "compression": "^1.0.3",
    "cors": "^2.5.2",
    "errorhandler": "^1.1.1",
    "express-session": "^1.11.3",
    "loopback": "^2.14.0",
    "loopback-boot": "^2.6.5",
    "loopback-component-passport": "^1.5.0",
    "loopback-connector-mysql": "^2.2.0",
    "loopback-datasource-juggler": "^2.19.0",
    "passport-local": "^1.0.0",
    "passport-tumblr": "^0.1.2",
    "serve-favicon": "^2.0.1"
  },
  "optionalDependencies": {
    "loopback-explorer": "^1.1.0"
  },
  "devDependencies": {
    "jshint": "^2.5.6"
  },
  "repository": {
    "type": "",
    "url": ""
  },
  "description": "tester"
}

In the model-config.json, note the inclusion of "./node_modules/loopback-component-passport/lib/models" in the meta > sources, the UserCredential, and UserIdentity models:

// server/model-config.jsom
{
  "_meta": {
    "sources": [
      "loopback/common/models",
      "loopback/server/models",
      "../common/models",
      "./models",
      "./node_modules/loopback-component-passport/lib/models"
    ],
    "mixins": [
      "loopback/common/mixins",
      "loopback/server/mixins",
      "../common/mixins",
      "./mixins",
      "../node_modules/loopback-ds-timestamp-mixin"
    ]
  },
  "User": {
    "dataSource": "db",
    "public": true
  },
  "AccessToken": {
    "dataSource": "db",
    "public": false
  },
  "ACL": {
    "dataSource": "db",
    "public": false
  },
  "RoleMapping": {
    "dataSource": "db",
    "public": false
  },
  "Role": {
    "dataSource": "db",
    "public": false
  },
  "UserCredential": {
    "dataSource": "db",
    "public": false
  },
  "UserIdentity": {
    "dataSource": "db",
    "public": false
  },
  "user": {
    "dataSource": "db",
    "public": true
  },
  "userCredential": {
    "dataSource": "db",
    "public": true
  },
  "userIdentity": {
    "dataSource": "db",
    "public": true
  },
  "accessToken": {
    "dataSource": "db",
    "public": true
  },
  "acl": {
    "dataSource": "db",
    "public": true
  },
  "roleMapping": {
    "dataSource": "db",
    "public": true
  },
  "role": {
    "dataSource": "db",
    "public": true
  },
  "scope": {
    "dataSource": "db",
    "public": true
  }
}

Why the duplicates, e.g. User and user?

Well, you have to define the base model, and then your own model which extends the base. But both have to be defined.

This bit left me confused, so here are each of the relevant models for reference:

// common/models/access-token.json 
{
  "name": "accessToken",
  "plural": "accessTokens",
  "base": "AccessToken",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/acl.json 
{
  "name": "acl",
  "plural": "acls",
  "base": "ACL",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/role.json
{
  "name": "role",
  "plural": "roles",
  "base": "Role",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/role-mapping.json
{
  "name": "roleMapping",
  "plural": "roleMappings",
  "base": "RoleMapping",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/scope.json
{
  "name": "scope",
  "plural": "scopes",
  "base": "Scope",
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/user.json
{
  "name": "user",
  "plural": "users",
  "base": "User",
  "properties": {
    "accessTokens": {
      "type": "hasMany",
      "model": "accessToken",
      "foreignKey": "userId"
    },
    "identities": {
      "type": "hasMany",
      "model": "userIdentity",
      "foreignKey": "userId"
    },
    "credentials": {
      "type": "hasMany",
      "model": "userCredential",
      "foreignKey": "userId"
    }
  },
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}
// common/models/user-credential.json
{
  "name": "userCredential",
  "plural": "userCredentials",
  "base": "UserCredential",
  "properties": {},
  "validations": [],
  "relations": {
    "user": {
      "type": "belongsTo",
      "model": "user",
      "foreignKey": "userId"
    }
  },
  "acls": [],
  "methods": []
}
// common/models/user-identity.json
{
  "name": "userIdentity",
  "plural": "userIdentities",
  "base": "UserIdentity",
  "properties": {},
  "validations": [],
  "relations": {
    "user": {
      "type": "belongsTo",
      "model": "user",
      "foreignKey": "userId"
    }
  },
  "acls": [],
  "methods": []
}

Once the models are displaying again, feel free to change the datasources:

// server/datasources.json 
{
  "db": {
    "name": "db",
    "connector": "memory"
  },
  "mysql": {
    "host": "your.server.ip.here",
    "database": "dbname",
    "password": "dbpassword",
    "name": "mysql",
    "connector": "mysql",
    "user": "dbuser"
  }
}

Note, those are the settings created by Strongloop slc arc. If you enter them manually, using the data / fields in the manual, you may end up with duplicates – e.g. user and username.

My Loopback Passport Module Config

And lastly, and probably the most important is the server.js file.

// server/server.js
var loopback = require('loopback');
var boot = require('loopback-boot');

var app = module.exports = loopback();

var PassportConfigurator = require('loopback-component-passport').PassportConfigurator;
var passportConfigurator = new PassportConfigurator(app);

app.start = function() {
  // start the web server
  return app.listen(function() {
    app.emit('started');
    console.log('Web server listening at: %s', app.get('url'));
  });
};

// Bootstrap the application, configure models, datasources and middleware.
// Sub-apps like REST API are mounted via boot scripts.
boot(app, __dirname, function(err) {
  if (err) throw err;

  // start the server if `$ node server.js`
  if (require.main === module)
    app.start();
});

// Enable http session
app.use(loopback.session({ 
  secret: 'keyboard cat',  
  resave: true,
  saveUninitialized: true 
}));

// Load the provider configurations
var config = {};
try {
  config = require('./providers.json');
} catch(err) {
  console.error('Please configure your passport strategy in `providers.json`.');
  console.error('Copy `providers.json.template` to `providers.json` and replace the clientID/clientSecret values with your own.');
  process.exit(1);
}
// Initialize passport
passportConfigurator.init();

// Set up related models
passportConfigurator.setupModels({
  userModel: app.models.User,
  userIdentityModel: app.models.UserIdentity,
  userCredentialModel: app.models.UserCredential
});
// Configure passport strategies for third party auth providers
for(var s in config) {
  var c = config[s];
  c.session = c.session !== false;
  passportConfigurator.configureProvider(s, c);
}

The ordering here seems to be the most important part.

If your passport config tries to call the models before they are ready, that seems to cause a lot of problems. It sounds obvious in hindsight, but when you have a lot of things going on in your server.js file, sometimes it can be hard to see the wood for the trees.

AWS in Plain English

Amazon Web Services (AWS) LogoYou’ve likely heard of Amazon Web Services (AWS).

And, if you are anything like me, you have likely been left – more than once – scratching your head as to what these crazily named services – ElasticBeanstalk anybody – actually do.

As if the naming wasn’t confusing enough, with over 50 different cloud-based offerings, it’s really quite difficult to even know where to begin.

So, the friendly chaps at Expedited SSL have come up with a Layman friendly buzzword breakdown of all the different AWS offerings:

amazon-web-services-explained
A Layman friendly buzzword decoding

Who’s That Pokémon

Now that you’re bona fide Amazon Web Services expert, why not play the delightfully geeky:

Is it Pokémon or Big Data technology?

How to get the Latest NPM Dependencies

I have been doing a fair amount of work lately with the Node.JS Loopback framework.

The more I use the framework, the more I find myself enjoying it.

marcus-wareing-great-british-menu
I liked your dish, but…

But, there’s one thing about Loopback – and pretty much every Node development project I have worked on – that I find needlessly time consuming:

Finding the latest versions of your NPM dependencies.

Up until very recently, the way I was doing this was comically laborious.

I would open the package.json file, take a look at the installed packages:

node-js-package-json-dependencies

And then… *cringe*, head over to NPMJS, manually search for each project, then take a note of the latest version:

npmjs-latest-version

And pop that info into my package.json file. Rinse and repeat for all the NPM dependencies in my project.

This is 2015. There has to be a better way.

And there is.

Using Node to Update Your NPM Dependencies

There’s a really useful package for keeping your NPM dependencies up to date with the very latest versions – it’s called NPM Check Updates (NCU) by Tomas Junnonen.

Straight from the ReadMe:

npm-check-updates is a command-line tool that allows you to find and save the latest versions of dependencies, regardless of any version constraints in your package.json file (unlike npm itself).

npm-check-updates maintains your existing semantic versioning policies, i.e., it will upgrade your"express": "^4.11.2" dependency to "express": "^5.0.0" when express 5.0.0 is released.

npm-dependency-update-with-ncu

A word of caution, however.

Whilst this seems like a great time saver – and it is – be sure to have some way of checking (ahem, tests) that the latest and greatest packages haven’t inadvertently completely broken your project.

A great example of how this can go wrong would be in the screenshot above. Here, I pulled in the Loopback Passport Example project, ran the ncu utility, and promptly ended up with a project that won’t even start.

As a side note, if you are wanting to add tests to a Node project, I highly recommend Tape. Initially, it’s a little different to normal testing libraries, but having taken the plunge, I really like it.

 

 

Composer Lock & Colleague Hair Loss

beethoven the grumpy composer
Right, that’s it sonny, your name is going on my list

One of the more confusing aspects of Composer is the purpose of the composer.lock file.

Firstly, it doesn’t appear until you have initially run a composer update.

Except when it does – which is when you join a project that’s already in progress.

If you clone an existing modern PHP project, you *should* get the composer.lock file, alongside the expected composer.json.

If you open your composer.lock file, it’s a big old file, packed full of JSON, but because it has the extension of .lock, it has no syntax highlighting.

Ok… close file. That was scary. What was all that?

Locking in the Fun

When a modern PHP project is born, usually a small handful of entries are added to your composer.json file.

Things like Symfony, or PHPUnit, or any of the 69,632 other packages that live on packagist. (Hey, that figure was accurate when I wrote this).

Then you run composer install, and aside from pulling down all those remote files and stashing them helpfully in your vendors directory – and creating the vendors/autoload.php file – composer creates a record of the exact installed versions of each of those dependencies.

symfony-composer-json-example
An example of the ‘require’ section in a composer.json file, taken from a Symfony project

So What?

quake-1-lan-party
A Quake 1 LAN party, according to Google.

You may wonder why this is important. Without giving it much initial thought, I felt the same way.

The reason is: stuff changes.

I’m not talking about code you change – I’m talking about the code in your vendors folder.

Development is often tricky enough with your own code. Letting your dependencies update at will is like playing Quake 1 on Nightmare.

It’s a fair assumption to think those Symfony dependencies are going to be used together frequently.

Symfony’s test suite likely encompasses the various interactions relatively well, and 99% of the rest of them are going to be caught by the many, many users of Symfony.

But what about the third party dependencies you have specified for your particular project?

third-party-dependencies-in-composer-json

Chances are, many fewer people will have used this particular combination of dependencies.

And just look – two packages are set to dev-master – whilst sometimes unavoidable, this is exactly the sort of thing that’s going to lead to sad panda situations.

Colleagues and the Composer.lock File

If you can live with your dependencies potentially breaking your project in subtle, and unexpected ways – then that’s your call.

However, if you are part of a team, the composer.lock file suddenly becomes very, very important indeed.

conmposer lock out of sync

This is the sort of situation that leads to bad times.

The problem here is that whilst someone updated the composer.json, and they may very well have ran a composer update afterwards, and the dependencies they got still led to some nice green tests…

But they forgot to commit the resulting composer.lock file.

And that means that we too now need to run the composer update command.

The thing is, if we have dependencies on projects that are using dev-master, for example, we are very likely to get a different commit to what our team mate got when they originally changed the composer.lock file.

This leads to sometimes subtle, sometimes glaringly obvious bugs. Bugs of someone else’s making. The worst kind of bugs.

Not to mention all the other weird issues that you might run in too.

Let’s imagine that the composer.json file is telling you that Symfony is now at 2.7, but you might hit strangeness like this when trying to add in a new dependency:

a classic case of composer hatred

That might set you back ten minutes, half an hour, a couple of hours on a particularly bad afternoon.

But what about a junior team member?

I Never Use dev-master, Am I Safe?

No!

Take a look at this guide on understanding Composer’s versioning syntax.

I think the tilde operator is the most confusing of all.

And even if you use definitive versions, you still can’t be sure someone didn’t delete and recreate the tag you are relying on.

Only the composer lock file holds the definitive proof because it records the exact commit hash / signature, and then until the composer lock file is next updated, everyone who installs using composer will get the expected version of each dependency.

Clever, but confusing 🙂

TD;DR;

Can’t be bothered reading all that eh?

Well, if you open up any composer.lock file, right there at the very top you’ll find :

    "_readme": [
        "This file locks the dependencies of your project to a known state",
        "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#composer-lock-the-lock-file",
        "This file is @generated automatically"
    ],