Here's a way to get environment variable values into your [Next.js](https://github.com/zeit/next.js/) app.



## Why in the Environment?

Why might you want to put variables into the environment?  These might be values that are sensitive that you don't want to commit to source control.  They might be things that change per environment (staging vs production).  They might be things you want to be able to adjust without changing your code.

An example of a thing that we might want to change in an environment variable is the url for an API host that our Next.js site connects to for data.

## Node Environment Variables

In Node, and thus in Next.js, we access environment variables via `process.env`.  This is a map of different values available from the environment.  If I set the variable in the environment and then ran my next app:

```bash
export API_HOST=https://exampleapi.com
next 
```

Then I could access the variable in my Node code with:

```js
process.env.API_HOST
```

Instead of typing `export` every time I want to modify the environment for my app, I create a list of variables for the app process to have automatically every time I run it.  I can do this with a little tooling and setup.  

## Dotenv Module

First, create a `.env` file in your project root:

```bash
API_HOST=https://exampleapi.com
```

Put your environment variables in that file, one per line.

Then install a [dotenv](https://www.npmjs.com/package/dotenv) module for your Next.app to use to load this file:

```bash
npm install dotenv
```

In your node code, you need to invoke `dotenv` to load the `.env` file contents into the environment.  This is done with:

```js
require('dotenv').config()
```

But where should this call go in your Next.js app?

## Next.js Config Modifications

Out of the box, the Next.js config doesn't have anywhere specifically to put environment variables.  But Next.js does provide extensibility via a couple interesting hooks in the `next.config.js` file in your project root.  One hook is the `webpack` hook, where we can access and change anything in webpack.

We need to provide the environment variables to webpack so that when it builds the browser bundle of JavaScript it can make the needed string replacements for the browser environment.

Specifically, `process.env` isn't available in the browsers (only Node.js), but your Next.js code runs in both places.  So the browser bundle that is generated by webpack needs to replace, for instance, `process.env.API_HOST` with `"https://exampleapi.com"`.

To do this, edit `next.config.js` to include:

```js
const { parsed: localEnv } = require('dotenv').config()
const webpack = require('webpack')

module.exports = {
  webpack(config) {
    config.plugins.push(new webpack.EnvironmentPlugin(localEnv))

    return config
  }
}
```

Here, we call `dotenv.config()`, and we destructure out the parsed object -- usually set on `process.env`, but now given to the `webpack.EnvironmentPlugin` -- which will do the string replacement.

Now inside your Next.js code, you can use the environment variable `process.env.API_HOST`, and the final bundle will be output with `"https://exampleapi.com"`.  You're probably using this inside of `getInitialProps` or wherever you're calling your API.

What alternate methods do you use to set environment variables in your Next.js projects?

Get environment variable values into your Next.js app

Environment Variables in Next.js

Add typing to your `process.env` access in Node.js.



## process.env Access

`process.env` is a globally-available object on which environment variables are accessible to your Node.js program. 

When you ask for an environment variable, such as `process.env.MY_VAR`, it may or may not be set.  If it's set, it'll be returned as a string.  If it's not set, it'll be returned as `undefined`.

## process.env Original Type

Because of the above behavior, the flowtype definition for `process.env` is a map of nullable strings:

```js
{ [string]: ?string }
```

## process.env Nullable Access

If you're trying to access and use the values of an environment variable, you have to do null-checking before you can be sure when using it.

This code is unsafe:

```js
process.env.MY_CSV.split(',')
```

And provides the following flowtype error:

```txt
call of method `split`. Method cannot be called on possibly null value
```

To make it safe, we have to null-check first:

```js
if (process.env.MY_CSV)
  process.env.MY_CSV.split(',')
```

## Null Checking En-masse

Usually you'll have multiple env vars in your code that you want to use.  Many of these are required for your program to run. And usually you don't want to be checking for nulls on required values in the middle of other logic.  You want a non-nullable type that you can rely on.

There are many ways to accomplish this.  Here is one:

Create a `config.js` abstraction around your `process.env` access.  In the initialization of this abstraction, read from `process.env`, do all the null checking at once, and from then on only use the config abstraction in the rest of your code.

This `config.js` code might resemble:

```js
type Cache = {
  databaseUrl: string,
  // ...
}

let cache: Cache = {
  databaseUrl: '',
  // ...
}

const getVar = name => {
  if (!process.env[name]) throw new Error(`process.env.${name} required`)

  return process.env[name]
}

const load = (seed: ?Cache): Cache => {
  cache = Object.freeze(Object.assign(cache, seed, {
    databaseUrl: getVar('DATABASE_URL'),
    // ...
  }))

  return cache
}

module.exports = load()

```

When you call `load()` in the above code, if there are missing env vars, your program will error out.

The `cache` is initially assigned to an object of empty strings.  This is required to assure flowtype that the value really is always non-null.

There is a lot of type defining that's going on here.  This is all in the name of safety.  It's definitely more of a pain when you write this, but if you could guarantee that you access of env vars is safe, might it not be worth it?

What do you think?  How else have you accomplished this?

Add typing to your process.env access in Node.js.

Flowtype process.env

Here's a way to setup for types on a Node.js backend app.



## Flowtype Requires Transpilation

[Flowtype](https://flow.org/) will provide types for JavaScript, which originally has so little type information that it has essentially none.

You'll need to transpile from flowtyped-javascript into vanilla javascript that Node can run.

My favorite compiler for this is [babel](http://babeljs.io/).

## Lightweight Babel

The great thing about Node these days is that most of the time, you don't need to compile at all.  With flowtype, you will need to compile.  

The most lightweight babel integration we can get is via [`babel-cli`](https://babeljs.io/docs/usage/cli/).

First, install it.  While we're add it, grab the babel preset for flowtype:

```bash
npm install babel-cli babel-preset-flow --save-dev
```

To compile, run the cli in your terminal:

```bash
babel src/ --out-dir dist/ --no-babelrc --presets=flow --ignore spec.js
```

This will take all the Node code in the `src/` directory, compile it, and save it to the `dist/` directory.  Note that the babel preset we're using will remove the flowtype annotations, making our code again runnable in the Node.js runtime.

## Development Build

In your local dev environment, you'll want changes that you make in source code immediately transpiled and run by Node. Let's setup an npm script to help.

We'll first install another couple [useful helper](/post/some-helpful-packages-npm-scripts/)s:

```bash
npm install concurrently nodemon --save-dev
```

`concurrently` helps us run multiple processes at once.  `nodemon` will take care of restarting our Node server when code changes.

And then setup our `package.json` snippet:

```json
"scripts": {
  "build": "babel src/ --out-dir dist/ --no-babelrc --presets=flow --ignore spec.js ",
  "dev": "concurrently \"npm run build -- --watch\" \"nodemon dist/server.js\"",
}
```

Now if we run `npm run dev`, we'll get our compiled code running in Node.  Note that we're passing the extra `--watch` flag to the `build` script to enable this.  Also note that this assumes your app's entry point is `server.js`.

## Production Build

The production build setup is going to be even easier. 

Let's assume that we're deploying to the heroku.  We want to add a script that will run after we push new code to our server.  Heroku's version of this is called `heroku-postbuild`.  Again, in `package.json`:

```json
"scripts": {
  "heroku-postbuild": "npm run build",
  "start": "node server/dist"
},
```

We run the build after the server deploy, and we start our app by pointing to our compiled code.

Now we're set to add types via flow to our backend code.  Does this setup work well for you?  What have you done to make it even more slick?

Here's a way to setup for types on a Node.js backend app.

Setup Flowtype for a Node.js Backend

How to upsert in Knex.js.



## Postgres Upsert

Upsert is a fantastic feature in SQL that works kind of like "insert a new row, and if it already exists, update it".

In [Postgres](https://www.postgresql.org/docs/9.5/static/sql-insert.html), the syntax is handled as:

```sql
INSERT INTO table_name 
(column_name) VALUES (expression) 
ON CONFLICT (conflict_target)
DO UPDATE SET column_name = expression;
```

There are quite a few keywords there, but I think it reads rather close to the English sentence above.

In a scenario where we're inserting new users where the primary key is `email`, we could write this sql:

```sql
INSERT INTO users
(email, name) VALUES (:email, :name) 
ON CONFLICT (email)
DO UPDATE SET users.name = :name;
```

## Knex.js Upsert

[Knex.js](http://knexjs.org/) is a handy SQL query builder, database connection manager, and ORM for Node.  You already know you're in trouble, right?

Without Knex, you'd write your own upsert command, and in this case, that might be easier because Knex doesn't support this `ON CONFLICT` clause out of the box.  Surprise! But you can write it with Knex APIs fairly easily.  

The example SQL above, written in code with Knex is something like:

```js
const util = require('util')
const knex = // app's db instance

async function createUser({ email, name }) {
  const insert = knex('users')
    .insert({ email, name })
    .toString()

  const update = knex('users')
    .update({ name })
    .whereRaw('users.email = ?', [email])
  const query = util.format(
    '%s ON CONFLICT (email) DO UPDATE SET %s',
    insert.toString(),
    update.toString().replace(/^update\s.*\sset\s/i, '')
  )

  await knex.raw(query)
}
```

It's cool how Knex can output the queries that are build using `toString()`.  Those queries are later used as raw sql in the call to `db.raw()`.

Note the use of `whereRaw` instead of just `where` in the update.  This is required to get the table alias on the column.  Otherwise just the `email` column name in the where clause will be ambiguous.

:boom: 'Ya done been upserted!

Is there another way that you've gotten Knex to do this?

For a working example of this code, see the [jaketrent/demo-knex-upsert](https://github.com/jaketrent/demo-knex-upsert) repo.

Upsert in Knex.js

A long dependency list in a Node project is part of the philosophy.



## Different Stacks

There are different tech stacks for different folks.  Much of code is art, and there are so many ways that people prefer to experience art.  I've heard people who are unused to Node look at a Node codebase and marvel at the length of the dependency list.  Or I've heard people marvel at the thousands of packages that exist in NPM and exclaim that no tech stack could possibly use all that.  Well, maybe not.  But they're there, and dependency lists often get long quickly in a Node project.  

## Generally Small

There are some really small libraries on npm.  Famously, [left-pad](https://www.npmjs.com/package/left-pad) is one such example.  A small library is not unusual or undesired in the Node ecosystem.  Value can be shared, even in small doses.  It's easier and faster to get to the point of publishing something when it's small.

Small libraries are easy to reason about.  You can grasp and use the API surface area quickly.  

Granular dependencies means you get more of exactly what you need and less cruft that you don't.

You can [change faster](/post/speed-small-solutions/) with small solutions.

Small libraries are easier to keep cohesive and avoid becoming sprawling swiss army knives.

## Working Together

Because many libraries in Nodeland are small, they need to work together.  Small bits working together for a greater, interesting purpose has roots in Unix philosophy, where tools act in much the same way.

Because there are bits that can be combined in more than one way, the potential uses of these compositions multiply.  

The tools remain generally useful in more situations for longer periods of time.

## Some Tradeoffs

There are some tradeoffs to working with lots of small dependencies.  You simply have more dependencies to track.  You have to keep many versions updated.  You often have to provide your own glue code.  Sometimes composing very different packages is awkward or feels forced.  Sometimes it's harder to find what you want in a sea of options.  And probably more.

What do you think?  Do you enjoy an ecosystem of small dependencies?  What are the up or downsides that you see?

A long dependency list in a Node project is part of the philosophy.

Lots of Node Dependencies

Here are a couple helpful packages that will help you write better npm-scripts.



## rimraf

[rimraf](https://www.npmjs.com/package/rimraf) is a package for removing directories.  It is the equivalent of running `rm -rf`.  You could, of course, use this unix command directly in your npm-scripts, but then your windows users would be hosed.

First, install:

```bash
npm instlal rimraf --save-dev
```

Then add the script to your `package.json` that will blow away your `dist` (or other) directory:

```json
{
  "scripts": {
    "clean": "rimraf dist"
  }
}
```

And run with:

```bash
npm run clean
```

Simple, helpful, cross-platform directory removal.  So clean!

## cross-env

And speaking of cross-platform, why does every system have to have its own syntax for setting environment variables?  Never mind that any more!  From within npm-scripts, now you can set them in a consistent way, using [cross-env](https://www.npmjs.com/package/cross-env).

Install:

```bash
npm install cross-env --save-dev
```

Then imagine that we have a need to set the `NODE_ENV` environment variable to `'production'` when we run our `server.js` script.  That script in `package.json` might look like:

```json
{
  "scripts": {
    "start": "cross-env NODE_ENV=production node server"
  }
}
```

You can have multiple of these space-separated `key=value` pairs before your command.

## concurrently

Finally!, here's package that is good at forking children.  With this package you can use one npm-scripts target to run multiple things at once.  The package is called [concurrently](https://www.npmjs.com/package/concurrently). Let's say you want to simultaneously run a webpack build process, generating some static output, *and* spin up a simple http service to serve that static output directory.

Let's install our helper package:

```bash
npm install concurrently --save-dev
```

Then update our `package.json`:

```json

{
  "scripts": {
    "start": "concurrently \"webpack --watch\" \"http-server dist\"",
  }
}
```

Note the escaped use of the `\"` quotes inside the script.  Also note that these other utilities must be installed dependencies as well.  We can run as many processes as we need, separated by spaces, surround by quotes.

These are a few of my favorite utility helpers that are generally useful in writing npm-scripts.  What are some of your favorite packages to use when writing npm-scripts?

Here are a couple helpful packages that will help you write better npm-scripts.

Some Helpful Packages for NPM Scripts

Learn the syntax to export in these two styles in Node.js.



## Single Export

`module.exports` is the export style for a single export in a module.  If you want to be able to require, alias, and use a single thing from a module, you would use this style.  You would have `a-module.js` like:

```js
module.exports = () => { /* ... */ }
```

Which would be usable like:

```js
const aModule = require('./a-module')
aModule()
```

This style is nice because you can immediately invoke the function by the variable name that you've assigned it to on import.

## Multiple Named Exports

For multiple, named exports, the `exports` style is usable.  In `a-module.js`, you would export like:

```js
exports.aName = () => { /* ... */ }
exports.anotherName = () => { /* ... */ }
```

And use like:

```js
const aModule = require('./a-module')
aModule.aName()
aModule.anotherName()
```

You could also destructure your import with:

```js
const { aName, anotherName } = require('./a-module')
aName()
anotherName()
```

This style of export is nice because you can provide multiple things in a single module, each identified by some name in that collection.

## Export Multiple with an Object

To get multiple exports another way, you could group them together in a single object and export that.  To export a single thing, we're using `module.exports` again:

```js
module.exports = { 
  aName: () => { /* ... */ }
  anotherName: () => { /* ... */ }
}
```

In commonjs, this is used just as the multiple named exports above:

```js
const { aName, anotherName } = require('./a-module')
aName()
anotherName()
```

## Mix of Export Styles

Finally, if you would like to have something exported in a single export style and then have multiple, named exports dangling off it, you might define that module like this:


```js
module.exports = () => {}
module.exports.aName = () => {}
```

Which would then be usable like this:

```js
const aModule = require('./a-module')
aModule()
aModule.aName()
```

This style allows you to mix both export styles together in a single module.

Those mixtures all form quite the brew, eh?  What other strange mixture of exports do you find useful?

Learn the syntax to export in these two styles in Node.js.

Mixing module.exports with exports

The most lightweight way to get an async/await function.



## No Async Function?

It sure would be nice if you could `await` something without needing to wrap it in an `async` function.  But, language-wise, these two go together.  If you use `await` outside an `async` function, it's an invalid token.  Inside the `async` function, however, `await` is perfectly legal (on supported platforms).

## Simplest Async Function

So the next simplest thing to do is write it in a way that doesn't seem like it's a function at all: a self-executing function.  Here's an example:

```js
;(async _ => {
  await somethingAsync()
})()
```

Note that the whole function is wrapped in parenthesis, and then there are open/closed parens at the end, invoking the inner function.  The inner function is marked with `async` so that `await` can be used inside.  Bam!  No name on the function and no named invocation.  As light as it gets.  

Or can you get it lighter?

The most lightweight way to get an async/await function.

Simplest async/await Function

Here's an easy example of how to send email using the aws-sdk on Node.js.



## Install SDK

You have the full power of Amazon's AWS SDK as soon as you install it for Node/JavaScript:

```bash
npm install aws-sdk
```

## Get Authorized

Make sure you setup a user in Amazon's [IAM](https://console.aws.amazon.com/iam) service that has access to the SDK.  You'll need a key id and a secret access key.  You can set these AWS credentials for your whole system in `~/.aws/credentials` or you can apply them only in the project that will use them.  We'll choose the latter.

To store credentials, we're going to use environment variables.  To load them we'll use the venerable `dotenv` library:

```bash
npm install dotenv
```

Create a gitignored file `.env` in your project root:

```txt
AWS_REGION=us-west-2
AWS_ACCESS_KEY_ID=yourKey
AWS_SECRET_ACCESS_KEY=yourSecret
```

First thing in our new `server.js` app file, let's load these variables making them available to the `aws-sdk`.  Let's run:

```js
require('dotenv').config()
```

## Verify Email Addresses in SES Sandbox

When you first sign up for Amazon [Simple Email Service (SES)](https://us-west-2.console.aws.amazon.com/ses), you'll want to verify your sending domain and email address (both to and from).  Amazon requires this, as well as keeping you in a non-prod sandbox with plenty of limits, in order to protect its users, the Internet, and your own email's trustworthiness to other mail servers.  Take care of this verification before trying to send any email.  If you don't, it'll just error out.  Later once this is all setup and working, you can request prod access.

##  SES in Node

Since we have the `aws-sdk` already installed, it's ready for us to use.  Let's require it and instantiate an SES instance:

```js
const AWS = require('aws-sdk')

const ses = new AWS.SES()
```

This instance has all the top-level [APIs for SES](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SES.html) that we need, notably `ses.sendEmail`.  All `ses.sendEmail` needs is some config for the to and from addresses and email content that you want to send.  Set it up, and press send:

```js
const params = {
  Destination: {
    ToAddresses: ['to-email@example.com']
  },
  Message: {
    Body: {
      Html: {
        Charset: 'UTF-8',
        Data:
          'This message body contains HTML formatting, like <a class="ulink" href="http://docs.aws.amazon.com/ses/latest/DeveloperGuide" target="_blank">Amazon SES Developer Guide</a>.'
      },
      Text: {
        Charset: 'UTF-8',
        Data: 'This is the message body in text format.'
      }
    },
    Subject: {
      Charset: 'UTF-8',
      Data: 'Test email from code'
    }
  },
  ReturnPath: 'from-email@example.com',
  Source: 'from-email@example.com'
}

ses.sendEmail(params, (err, data) => {
  if (err) console.log(err, err.stack)
  else console.log(data)
})
```

The params can take a formatted email body (`Message.Body.Html`) and an alternate text body (`Message.Body.Text`).  The `ReturnPath` is the place where bounce-backs will be sent back to.  Also note that the asynchronous method takes a callback.  The `data` that is returned in the callback to the callback will have the shape of:

```js
{ ResponseMetadata: { RequestId: 'f96abf23-5108-11e7-a0d8-5fe3d188b3b6' },
  MessageId: '0101015ca6e24030-7913ca02-de24-42d4-9a5d-83f877114974-000000' }
```

That's the shape you'll see if it sends once you run `node server.js`.  If it didn't work, you'll get one of a number of possible [error codes](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/api-error-codes.html) console logged instead.  

For a full example, see [this hello-ses repo](https://github.com/jaketrent/hello-ses).

Pretty easy, huh?  Is this how you send email over SES in Node?  What cool tricks have you learned while doing this?

An example of how to send email using the aws-sdk on Node.js.

Send Email with aws-sdk on Node.js Example

Let's setup Next.js to be able to serve markdown content, such as blog posts.



## Next.js Custom Server

Next.js has a nice feature set out of the box.  If you have a site that's more on the site end of the spectrum (rather than the app end), you'll likely be able to use the `next` server as-is. 

If, however, you'd like to do app-ish things on the backend, you'll likely need your own custom server.  This is not hard.

Serving markdown seems to be one of the things that needs a custom server.  I tried rendering the Markdown straight from React components, and it didn't seem to work.  This after adding custom markdown loaders to the webpack config and all that.  It still didn't work.  The big hang up, for some unknown reason, was that the markdown modules could not be found in order to be imported correctly.  

My solution was to build a custom server.  

## Basic Custom Server Setup

In Next.js terms, a custom server is simply some JavaScript that starts the next app in addition to doing any of the extra app stuff you need.

First, change your `package.json` `start` script:

```
{
  "scripts": {
    "start": "node server.js"
  }
}
```

Then do the basics in your `server.js` to initialize `next`:

```js
const { createServer } = require('http')
const next = require('next')

const dev = process.env.NODE_ENV !== 'production'
const app = next({ dev })
const handle = app.getRequestHandler()

;(async _ => {
  await app.prepare()
  createServer(async (req, res) => {
    // ...more custom stuff here later

    return handle(req, res, parsedUrl)
  })
  .listen(process.env.PORT || 3000, (err) => {
    if (err) throw err
    console.log('> Ready on http://localhost:' + (process.env.PORT || 3000))
  })
})()
```

This can be done in Express, Koa, or any other http server library.  This example uses the Node built-in `http.createServer`.  Note that all this is doing is creating a `next` app, then using that app's request handler to handle all incoming requests.  Custom additions are yet to come.

## Reading Markdown

Since we're on the server, we have full access to the Node API.  This includes the `fs` filesystem API.  To get our markdown content, let's read markdown files out of a `posts` directory.  Once the file strings are read, let's use a library called [`front-matter`](http://npm.im/front-matter) to parse metadata out of the front matter (not the markdown content yet).  First install:

```bash
npm install front-matter
```

And let's put all this in a `posts-repo.js`:

```js
const path = require('path')
const { promisify } = require('util')
const frontMatter = require('front-matter')
const fs = require('fs')

const readdir = promisify(fs.readdir)
const readFile = promisify(fs.readFile)

const deserialize = parsed => ({
  title: parsed.attributes.title,
  date: parsed.attributes.date,
  body: parsed.body
})

async function fetch() {
  const postsDir = path.join(__dirname, 'posts')
  const filenames = await readdir(postsDir)
  
  return Promise.resolve(filenames.map(async filename => {
    const markdown = await readFile(path.join(postsDir, filename), 'utf-8')
    const parsed = frontMatter(markdown)
    return deserialize(parsed)
  }))
}

exports.fetch = fetch
```

## Serving Markdown

Now we have a module that can read and load into memory all the parsed markdown files.  Now we have to serve this markdown with our custom server.  

First, we need to detect that our client is making a request for the markdown.  We'll do an overly-simple check of the request path.  If the request url matches '/api/', we'll assume we're asking for this single endpoint to fetch all the markdown posts.  We can make this more complicated later.  

In `server.js`, we'll make the adjustments.  Inside the `createServer` callback, we don't just pass everything on to the Next.js `handle` function any  more.  We now parse the URL and call our `postsRepo.fetch` instead:

```js
const { parse } = require('url')

const postsRepo = require('./posts-repo')

const serialize = data => JSON.stringify({ data })

// ...

  createServer(async (req, res) => {
    const parsedUrl = parse(req.url, true)
    const { pathname } = parsedUrl

    if (pathname.includes('/api/')) {
      const posts = await postsRepo.fetch()

      res.writeHead(200, { 'Content-Type': 'application/json'})
      return res.end(serialize(posts))
    }

    return handle(req, res, parsedUrl)
  })
```

## Requesting Markdown from Next.js Component

Now we have an endpoint that can serve the markdown-based posts.  We need to get the React components to request the data they need.  In Next.js, this is done on the server and in the client.  We'll choose an http library that can function in both places.  We'll also install a markdown parser:

```bash
npm install isomorphic-fetch react-markdown
```

We're ready to rock and roll in our React component.  The top-level component in Next.js, called a "page", has a special hook called `getInitialProps`.  Here in `pages/indexx.js` we can do our data fetching:

```
import Post from '../components/post'

export static class Index extends React.Component {
  static async getInitialProps({ query }) {
    const posts = await Post.fetch()
    return { posts }
  }
  render() {
     {this.props.posts.map(p => <Post post={p} />)}
  }
}
```

And then `components/post.js`:

```js
import 'isomorphic-fetch'
import Markdown from 'react-markdown'
import React from 'react'

const Post = props =>
  <div>
    <h1 className="title">{props.title}</h1>
    <time className="date" dateTime={props.date}>{props.date}</time>
    <Markdown source={props.body} />
  </div>

class PostContainer extends React.Component {
  static async fetch() {
    const res = await fetch('<yourHost?/api/v1/posts')
    const json = await res.json()
    return json ? json.data : {}
  }
  render() {
    return <Post {...this.props.post} />
  }
}

export default PostContainer
```

Here we have the http code to fetch the post from the server, and we're finally rendering markdown.

There are a few steps here, aren't there.  We can still get the coolness of Next.js' server-rendering of React while doing our own custom markdown fetching on top of it.

Is this how you render markdown content in Next.js?  How have you improved upon the method?

Setup Next.js to be able to serve markdown content

Serve Markdown with a Next.js Server

There are a couple ways to split up and import parts of Node packages.  Using a slash (`/`) in the import statement is one of them.



## Main Entry Point

In the `package.json` for a Node package, the entry point of the package is defined in the `main` attribute:

```json
{
  "name": "some-package",
  "main": "index.js"
}
```

This defines the file that will be linked to when importing the package by name -- here called `some-package`. A common entry point is an `index.js` file in the root of the package:

```
├── index.js
└── package.json
```

From there we can do other interesting things in an import.

## Slash in the Require Statement 

Node packages are conventionally small and focused, and this is good.  Sometimes, however, it makes sense to group similar things in a package and publish them together.  If a package is a collection of things published together, there is going to be a user someday who doesn't want that whole collection.  Or perhaps they just want to conveniently import the subset of features that they want at that moment.

One way to get this subset of the package is to use a slash in the require statement.  Imagine such a statement:

```js
const subThing = require('some-package/sub-thing')
```

Because Node `require` statements use the filesystem as an API for addressing package locations, we can deduce that `sub-thing` is one of two places:

- `./sub-thing.js`
- `./sub-thing/index.js`

If you want to allow the users of your package to import subsets of your package this way, make them easily addressable on the filesystem.  

## Make Deep Directories Easy

Sometimes, if you have a more complicated directory/file setup, you'll want to create some convenience for users that want to import things this way.  Let's say that `buried.js` is deep in the directory structure, but you want it importable via `require('some-package/convenience')`.  You might set this up as:

```
├── convenience.js
├── deep
│   └── deeper
│       └── buried.js
├── index.js
└── package.json
```

Where `convenience.js` contains:

```js
module.exports = require('./deep/deeper/buried')
```

Do you import packages with slashes in this way?  Do you think it's a good thing to do?  When you publish your next package, is there some convenience for this method you could provide?

How to use a slash in the Node package import statement to get only a part of the package.

Import Parts of Node Packages with Slash

Most native Node APIs use callbacks.  If you want to use Promises to work with these APIs instead, it's easy.



## Native Node APIs

In Node, most of the async APIs in the core APIs use callbacks to notify of task completion and resume a code path.  This works just fine.  The pattern is solid.  If you like callbacks, keep using them.  That original callback method of an async file read might look like this:

```js
const fs = require('fs')

fs.readFile('myFile.txt', 'utf8', (err, data) => {
  if (err) return console.error(err)
  
  // ... process data
})
```

This works great.  It does require care to not recreate a biblical tower of doom with nested callbacks. 

There is a new native kid in Node.js town that might help.  Promises have been there for a long while, and constructing towers of terror is possible with these as well.  But now [async functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function) are available as of Node [7.x](http://kangax.github.io/compat-table/es2016plus/#node7_6).  `async` and `await` are keywords that you can use without a code compilation step.  This syntax will help flatten out your code, and it's quite nice. That might be one reason to want to get back on the Promise train and use the built-in Node APIs as if they were Promise-based as well.

## Converting Node Functions to Use Promises

But how do you make the conversion?  There are a couple Promise libraries that provide a function that's generally called `promisify`.  To use one, we'll install:

```bash
npm install promisify-node
```

This is a wrapper function.  Whatever API that was callback-based can be wrapped using `promisify` to make it Promise-based.  To wrap a whole Node API like the `fs` module, use it in place of the `require` call.  The above code changes to look like:

```js
const promisify = require('promisify-node')
const fs = promisify('fs')

async function processMyFile() {
  try {
    const data = await fs.readFile('myFile.txt', 'utf8')
    // ... process data
  } catch (err) {
    console.error(err)
  }
}
```

Note that the `await` keyword has to be used in the enclosing `async` function.  If you'd like to promisify a single function, that's possible as well:

```js
const promisify = require('promisify-node')
const fs = require('fs')

const promiseBasedReadFileFunction = promisify(fs.readFile)
// then use async/await as above...
```

Pretty cool, huh?  Although I was a little disconcerted when I saw some of the [guts](https://github.com/nodegit/promisify-node/blob/master/index.js#L61) of how this is accomplished by this library.  :)

## Built-in Support in v8.0.0

Now things are even better!  If you're on a new enough version of Node, this is built in. As of [node 8.0.0](https://nodejs.org/en/blog/release/v8.0.0/#improved-support-for-promises), the ability to `promisify` is a core API itself:

```js
const { promisify } = require('util')
const fs = require('fs')

const promiseBasedReadFileFunction = promisify(fs.readFile)
// then use async/await as above...
```

Do you `promisify` your usage of core Node APIs?  Why or why not?

Most native Node APIs use callbacks.  Use Promises instead.

Use Node APIs with Promises

[Next.js](https://github.com/zeit/next.js/) is a framework that most notably allows you to write server-rendered React apps easily -- amongst other cool things.



## It's a Framework

Next.js is opinionated and provides structure for how you build your app.  Like Rails, the file-system is often the API.  You name your file something special and put it in a special directory, and it Just Works.  In this way, Next.js feels like a framework.

## The Main Event: Server-Rendering

Anyone can create an `index.html` page and call `ReactDOM.render` on it to get a client-side React app running.  Something that simple hasn't existed for the server.  Next.js attempts to be that thing.  "Put a file in this directory," it muses, "and I will create a Node.js server that renders that React component as HTML at this route".  And it really does seem that easy.

Next.js allows for things like data-fetching and app state via Redux and all that.  But you must bring that stuff to the party yourself.  And it'll work just fine.  If your game is rendering HTML via React from server and client, Next.js has that all in the box.

## CSS 

The default CSS approach in Next.js is [styled-jsx](https://github.com/zeit/styled-jsx).  You define string blocks of CSS as a child `<style />` tag in the component you want the styles to apply to, a la:

```js
const MyComponent = _ =>
  <div>
    <p>Style only applies to this paragraph</p>
    <style jsx>{`
      p { color: red; }
    `}</style>
  </div>
```

In definition, this feels a lot like the defunct [scoped css](http://caniuse.com/#feat=style-scoped).  In practice, it outputs selectors that match data- attributes that Next.js adds to the component's HTML output:

```html
<p data-jsx="123456789">Style only applies to this paragraph</p>
<style>p[data-jsx="123456789"]</style>
```

There are (so many!) other options for CSS.  Your able to customize this to your own flavor.

## Pages

The most magical directory in a Next.js project is `/pages`.  This is where your top-level React component modules will live.  These files will be used to generate routes in your app.

If you have a `./pages/about.js` component file, your app will have an `/about` route that will execute the component in that file when requested.

These top-level components have a Next.js-specific lifecycle hook, called `getInitialProps`.  It has access to route-related data like the requets (`req`), response (`res`), etc.

Pages are cool because, with a couple exceptions, they feel just like any other React component.  A valid (and boring) About page component:

```js
export default _ => <div>The About page!</div>
```

## Routing and Splitting

Next.js does automatic code splitting.  The splitting is done at page boundaries.  This means that you can be certain that any page you request from your Next.js app only includes the JavaScript that is needed to run that page.  How nice to not have to configure that.

There are other cool things that Next.js provides in order to make this even better.  It has its own router that supports lazy loading.  It allows for dynamic route creation.  There is a `<Link />` component that allows adding an attribute, `<Link prefetch />` that will fetch the route contents and scripts for quick execution down those paths.

## Escape Hatches

The power of Next.js and any framework is the mileage it gets you if its use cases match up to yours.  You can use the framework happily, and it gives you what you want.  Use cases rarely match up 100% of the time.  Next.js seems to realize this and gives you some useful escape hatches to customize what the framework provides by default:

- You can use your customized CSS-in-JS solution, eg postcss
- You can add your data fetching library, eg [`isomorphic-fetch`](https://www.npmjs.com/package/isomorphic-fetch)
- You can add your app state management, eg redux
- You can add a type system, eg Typescript
- You can override the babel config
- You can override the webpack config
- You can provide your own error handling
- You can override some of the Next configuration, eg provide your own magic directories

I haven't used Next.js on anything worthy of interesting overrides, but I am glad for the forethought here.

## Easy Deployment

With all your app wrapped up nicely inside the Next.js framework, it's not surprising that it's easy to deploy.  Ahhhh, yes.  Love me something easy to deploy.  To get a Next.js app to Heroku, add three scripts to your `package.json`:

```
{
  "scripts": {
    "build": "next build",
    "start": "next start -p $PORT",
    "heroku-postbuild": "npm run build"
  }
}
```

Procure your app on Heroku and push your code, and you're one with the Internet.

```bash
heroku create my-next-app
git push heroku master
```

Next.js seems pretty impressive.  That React server render -- so easy!

What have you used Next.js for?  What do you think its sweet spot is?

Next.js is a framework for writing server-rendered React apps.

What is Next.js?

There is a great path API in Node.  Still, you can get mixed up in how to specify the correct path.  Let's break down a few options.



## OS-agnostic Paths

All OSes have a file structure, where directories are nested within directories, with files existing in each.  But between OSes there is a difference in the symbol used to separate directory names.  When showing a file path in MacOS or Linux, forward slash `/` is used, as in `dev/magic.txt`.  In Windows, back slash `\` is used, as in `dev\magic.txt`.

You don't need to detect which OS you're on yourself.  Node will do that for you and then provide you with the correct file separator if you use this API:

```js
const path = require('path')

path.join('dev', 'magic.txt')
```

On Mac, this would output:

```txt
"dev/magic.txt"
```

## Process-relative Paths

Let's say that we're opening a file from within our Node app, so we need to specify a file path.  The code we're executing is in a module called `util/file-opener.js`, and the file we want is `files/secret-numbers.txt`.  The app was started in the project root directory, calling `Node app.js`.  What is the path that `file-opener.js` should use to access `secret-numbers.txt`?:

```
.
├── app.js
├── files
│   └── secret-numbers.txt
└── util
    └── file-opener.js
```

The answer is `path.join('files', 'secret-numbers.txt')`.  The reason is that by default relative paths are relative to where the process is being run from.  The entry point for the application is in the root project directory.  Paths will simply be subdirectories from there in this example.

## File-relative Paths

Sometimes it's confusing to consider where the app process is running from. Sometimes that has the potential to change over time, so you don't want to couple file paths to that fact.  Sometimes it can be easier to think about the path relative to the module that you're *accessing* the file from.  In this case, this means the relative path from `file-opener.js` to `secret-numbers.txt`.

There is another global variable in Node that allows us to know what directory or path the current module is in.  Thus, we can navigate from inside the `util` directory out and into the `files` directory using a relative file from `__dirname`, the current directory:

```js
path.join(__dirname, '..', 'files')
```

## Absolute Paths

Many times, relative paths will be just fine.  But there are some APIs that require an absolute path.  That should be easy to get.  You may know an absolute path that you are willing to hard code into your program.  But often you want to avoid that, since your program might run on a different host machines, and then that absolute path is likely wrong.  

To get an absolute path, start with a relative path, and have Node convert it for you with `path.resolve`.  If run from the `util` directory:

```js
const path = require('path')

path.resolve(path.join(__dirname, '..', 'files'))
```

...will output something like:

```txt
/Users/jaketrent/dev/secret-tunnels-project/files
```

With those basic tools and understanding, you're ready to blaze some paths all over your app.  Are there any other ways of approaching pathing that helps you keep it straight?  Are there other Node APIs that you find useful in this regard?

There is a great path api in Node.  Still, you can get mixed up in how to get the right path.  Let's break it down.

Paths in Node

Node has a built-in debugger.  It requires no extra tooling or purchased IDE.  Try it out.  It's easy.



## Commandline Debugging

Node's debug module is available in the terminal.  Normally you would start your app up with `node app.js`.  If you adjust it slightly, you'll be ready to debug:

```bash
node inspect app.js
```

Once you start the app this way, you'll see a debug prompt instead of your normal terminal prompt:

```bash
< Debugger listening on port 9229.
< ...
break in server.js:1
> 1 (function (exports, require, module, __filename, __dirname) { const http = require('http')
  2
  3 http.createServer((req, res) => {
debug>
```

You're debuggin'!

## Set Breakpoints

Previous to starting the debugger, you can set breakpoints inside your Node app by include `debugger` lines in your source code, like this:

```js
const http = require('http')

http.createServer((req, res) => {
  debugger

// ...
```

Now, if I execute this program using the `node inspect app.js` command, the first time the `debugger` line is hit, execution of the program will pause.  You can also set breakpoints and do other things from inside the debugger.

## Debug Control

To see what options you have inside debug mode, type: `help` in the debugger.

There are a few controls that any interactive debugger will have to help in navigation.  You can use the following:

- `sb` - set breakpoint - interactive equivalent of putting `debugger` in your source code
- `c` - continue - let the program execute until it encounters another breakpoint
- `n` - next - let the program advance to the next instruction
- `s` - step into - step into the next instruction (eg, inside a function call)
- `o` - step out - continue execution until the stack is popped

## Inspect Local Variables

One of the coolest things about interactive debugging is the ability to see what the values of variables are as the program runs.  To do this in the Node debugger, set a breakpoint at the place where you'd like to stop and look around, then let the program run.  Once you're paused at the right point, type `repl` into the debugger.  You've now entered a new repl mode.  Here, you can type the variable names of anything within scope and see the values printed to the screen.

For the source code above, we could pause and investigate:

```
break in server.js:4
  2
  3 http.createServer((req, res) => {
> 4   debugger
  5
debug> repl
Press Ctrl + C to leave debug repl
> req
{ _readableState: ReadableState,
  readable: true,
  domain: null,
  _events: EventHandlers,
  _eventsCount: 0,
  ... }
> req.url
'/'
```

To exit, press `Ctrl-c` once.

It's not a particularly beautiful UI, but it's keyboard accessible!  I like it because it's always available to use -- it's built-in tooling.  

What other ways do you do your debugging in Node?

Debug your Node.js app without having to buy or install other tools.

Debug Node with Built-in Tools

Local auth for an API in [koa](http://koajs.com/) will take a few pieces working together.  Here's one way to do it.



## The Flow

We're going to require authentication on our rest APIs for a single page app web client.  We're going to maintain a session in the API.  If you're logged in, you have a session.  When a request is made to an API endpoint, a session cookie is passed.  In the API, the cookie value is used to lookup the session.  If there's a valid session matching the cookie, you're allowed to continue on an fetch the resource you addressed.  If there is not a valid session, it means you're not authenticated, and the an error response is sent back to the client.

## Session Libs

There are many libraries that will help you establish a sesssion.  We're going to use [`koa-generic-session`](https://github.com/koajs/generic-session).  This requires another library choice for the session store.  There are many options.  We're going to use postgres, so [`koa-pg-session`](https://github.com/TMiguelT/koa-pg-session) is a great choice.  

`koa-generic-session`, as of this writing, doesn't natively support koa v2 middleware, so you need to use [`koa-convert`](https://github.com/koajs/convert) to make it compatible.

We're also going to use [`koa-passport`](https://github.com/rkusa/koa-passport) to give us access to `login`, `logout`, and `isAuthenticated` functions.

Install them all:

```
npm install koa-generic-sesion koa-pg-session koa-convert koa-passport --save
```

## Session Store

I made a `auth/session.js` module:

```js
const PgStore = require('koa-pg-session')

const store = new PgStore(process.env.DATABASE_URL)

function init() {
  return sessionStore.setup()
}

exports.init = init
exports.store = store

```

A few things to note:

- It's important to connect your `PgStore` with a full database connection string, like: `postgres://username:password@host:port/database`.  Don't store this in source code.  We're loding this from the environment variable `DATABASE_URL`. 
- We expose `store` for use in multiple places later, and it's important that it's the same instance, so this is a singleton in our app. 
- We'll use `init` later... stay tuned. 

## Session Middleware Stack

Now it's a matter of getting your koa (v2, btw) middleware setup the correct way.  I'll lay out what I have in my `server/index.js` file first, and we'll talk through it:

```js
const convert = require('koa-convert')
const koa = require('koa')
const passport = require('koa-passport')
const session = require('koa-generic-session')

const { store } = require('./auth/session')

// ..
const app = new koa()
app.keys = [process.env.SECRET_KEY]
app.use(convert(session({
  cookie: {
    path: '/',
    httpOnly: false,
    maxAge: 24 * 60 * 60 * 1000,
    rewrite: true,
    signed: true
  },
  key: 'myAppId',
  store
})))
app.use(passport.initialize())
app.use(passport.session())
// ..
```

- The setting of `app.keys` is for the cookie signing.  Make sure you have this set or you'll get a funky error saying that there's a cookie setting error and the key needs to be a Buffer.  (agh!)  For me, I'm setting this in an environment variable, `SECRET_KEY`.
- The first middleware is the `koa-generic-session` middleware.  Make sure you `convert` it to make it compatible with koa v2.
- The `cookie` attribute contains all the options for how cookies are set in the browser.  Notice we're setting `signed: true`, hence the need for `SECRET_KEY`.
- The `key` attribute is the cookie name, customizable by you.
- The `store` is the singleton `PgStore` that we instantiated in the `auth/session.js` module.  Remember to always use this one instance.

## Session Store Initialization

Before you start up your app and listen for incoming requests, you want to make sure that the `store` you plugged into your session middleware above is fully armed and operational.  To do this, make sure you're calling `store.setup()` and waiting until that async action is done before going on.  We wrapped this in the `auth/session.js` module's `init` function.  We might call that in this order:

```js
const session = require('./auth/session')

async function startApp() {
  await session.init()
  app.listen(process.env.PORT)
}
```

Now your app is fully initialized and ready to receive requests.  Now, to protect the endpoints.

## API Endpoint Authentication

To protect an endpoint, we are going to add a middleware that checks for the existence of a valid, active session.  First let's write that middleware in `auth/login.js`:

```js
module.exports = async function requireLogin(ctx, next) {
  if (ctx.isAuthenticated()) {
    await next()
  } else {
    ctx.status = 401
    ctx.body = {
      errors: [{ title: 'Login required', status: 401 }]
    }
  }
}
```

- `ctx.isAuthenticated` comes from `koa-passport`.  This is checking for a valid session.  
- If we are authenticated, let the request continue.
- If we are not authenticated, the request cannot continue, and we return a response of `401`, "Unauthorized".

Now let's use that middleware, putting it in front of a resource that we want to protect.  Let's say that our `/ringsofpower` are precious to us, so we want to require authentication there.  We might write this in the controller:

```
const koa = require('koa')
const route = require('koa-route')

const requireLogin = require('./auth/login')

const app = new koa()

// ..
app.use(requireLogin)
app.use(route.delete('/:id', mountDoom))
```

- We're using another library here, `koa-route`, which simply lets you define urls for your endpoints, mapping them to request handlers.  You definitely don't need this library for this to work, but it's really nice.
- `requireLogin` becomes the local alias for the `auth/login.js` function we wrote above.
- We stack `app.use(requireLogin)` before our precious resources later in the `app.use` stack in order to require its execution before anything else.

At this point, all of our requests will start to fail with `401`s because we haven't created any way to actually login and create a new session.

## Valid Users

To have a set of valid users, we'll need to create user records and store them in our database.  We will need to take extra care in storing passwords -- in fact, we'll just store password hashes, no more -- using [bcrypt](https://github.com/kelektiv/node.bcrypt.js).  For the down low on how this might look, check out the post [on storing local user passwords using bcrypt](/post/store-local-user-password-bcrypt/).  Once we have valid users in the system, we're ready to login.

## Logging In

We're assuming that your web app client has a UI for a user to enter their username and password.  We're just going to look at the server-side flow for logging in for now.  We're going to make a module, `auth/index.js` that will be the controller to create our new session.  First, we register our route for session creation:

```js
const koa = require('koa')
const route = require('koa-route')

const app = new koa()

async function create(ctx) {
  // ..
}

app.use(route.post('/', create))
```

This is all as expected.  Now what goes in `create`?  It might look like this:

```js
const bcrypt = require('bcrypt')

const repo = require('./repo')

function serialize(user) {
  return {
    data: {
      username: user.username
    }
  }
}

async function create(ctx) {
  const { username, password } = ctx.request.body
  try {
    const user = await repo.find(username)
    if (!user) {
      ctx.status = 401
      return ctx.body = { errors: [{ title: 'User not found', status: 401 }]}
    }

    const matches = await bcrypt.compare(password, user.passwordHash)
    if (matches) {
      ctx.status = 201
      ctx.body = serialize(user)
      return ctx.login(user)
    } else {
      console.log('u, p', username, password)
      ctx.status = 401
      return ctx.body = { errors: [{ title: 'Password does not match', status: 401 }]}
    }
  } catch (err) {
    ctx.status = 500
    return ctx.body = { errors: [{ title: err.message, status: 500, stack: err.stack }]}
  }
}
```

There's a lot here:

- The `username` and `password` come from the client via the `ctx.request.body`.
- We lookup the user using a `repo.js` module.  This is some code that takes a username and matches it to a user in your database.  The implementation is up to you and not super important here.
- `bcrypt.compare` is the line that determines if the passwords match.
- The `serialize` function makes sure we only expose fields we desire to the client.
- `ctx.login` is provided by `koa-passport`.  It will actually trigger the session creation, storage in our session store, and setting of the cookie.

Now for the response options based on the logic branches in this function:

- 401 - if username is not in our user database
- 401 - if the password does not match
- 500 - if any part of this fails
- 201 - if password matches a valid user

There is one more piece to setup.  We need to teach passport how we want the user record that we're `ctx.login`ing in with to be serialized.  So in this same file, we can add:

```js
const passport = require('koa-passport')

passport.serializeUser((user, done) => {
  done(null, { username: user.username })
})

passport.deserializeUser((user, done) => {
  done(null, user)
})
```

In this configuration, we show we only want to store the user's `username`.

## Logging Out

Now the user has had exposure to the precious resources of our REST API long enough.  Before the grow wizen and corrupt, let us revoke their access.  They might not come willingly, so we must be firm.  Thankfully, revoking access is much simpler than revoking just about anything else in real life.  Let's register another endpoint in `auth/index.js`, and we'll be ready:

```js
async function destroy(ctx) {
  ctx.logout()
  ctx.status = 204
}

app.use(route.delete('/', destroy))
```

That's all there is to it.  The session record in the store will be removed as will the session cookie.

So it turns out there are lot of things to get in place for a full auth flow through an API.  Do you do this in a similar way?  What could be simplified or made better?

Local auth for an API in Koa will take a few pieces working together.  Here's one way to do it.

Local Authentication in a Koa API

Be careful when you store a user password locally.  You can use bcrypt to make the password hash hard to break.  Here's a partial implementation in Node.



Sometimes you might want to store a password with a local user record in your app.  You might do this if you didn't want to let Twitter, Facebook, or something other 3rd-party site take care of auth via OAuth or something like that (those are good options too).

## The User Record

First, we need a set of valid users.  With an SQL-ready database like Postgres, we can create a table in the database that will hold our valid userbase.  The migration script might look something like this:

```
begin;

create table auth_user (
  username varchar(255) primary key,
  password_hash varchar(255) not null,
  created date default now()
);

end;
```

Of course, you can add other fields as desired.  Side note: Don't try to create the table named `user` because Postgres already has one of those built in. :)

## Input Username and Password

Now to create a code abstraction to transfer data in and out of the database.  We'll make a module that can do a user `create` to insert a user into the database.

We'll use a light ORM here that I've come to like: [`massive`](https://github.com/dmfay/massive-js).  I'll leave the finer points of the ORM setup to the reader to implement based on the documentation.  For our purposes, it's enough to know that we have an ORM, and the intent of the code will still make sense.  For instance, `create` will perform an insert:

```js
function create(db, user) {
  return new Promise(async (resolve, reject) => {
    const record = await serialize(user)
    db.auth_user.insert(record, err => {
      if (err) return reject(err)

      resolve(user)
    })
  })
}
```

A few things to note:

- `create` returns a `Promise` to account for the async nature of database interactions
- We're using the `async/await` syntax for `Promise`s instead of `then` chaining.  The result is the same.
- `db` is an instance of the database (the ORM), which has a reference to the table we're `insert`ing into: `auth_user`.

And what about that `serialize` function?  Well...

## Serialize the Database Record

The user that we'll insert will have a `username` and `password` field.  But we don't want to store the `password` as plaintext in the database.  Therefore we need to let [`bcrypt`](https://github.com/kelektiv/node.bcrypt.js) create a hash of the password first.  

A hash of the original password text is a one-way operation, meaning that you cannot decrypt or rediscover the original value of the password by starting with the stored hash.  You must, yourself, know the original password *first*, and then another hash can be generated and then compared.  Thus storing this hash is much safer than storing plaintext passwords, which we should never do.  So, `serialize` might look like this:

```js
const bcrypt = require('bcrypt')

async function serialize(user) {
  const saltRounds = 10
  const hash = await bcrypt.hash(user.password, saltRounds)
  return {
    username: user.username,
    password_hash: hash
  }
}
```

- We're finally using `bcrypt`.  Make sure to `npm install --save bcrypt` first.
- `bcrypt.hash` is potentially long-running, so it's async.  
- `saltRounds` is a number that controls essentially how long and processor-intensive it is to recreate the hash.  I just took the `bcrypt` docs default value.  You can [be a lot more precise](https://security.stackexchange.com/questions/3959/recommended-of-iterations-when-using-pkbdf2-sha256/3993#3993) about it.
- This function returns an object that is usable by the ORM to insert as a record into the database `auth_user` table.

Now you have a user record in the database, password safely stored.

## Comparing Hashes

Of course, sometime you'll want to retrieve the user from the database and check to see if a user who is trying to log in is actually a valid user with a matching password in your system.  So you'll probably end up with a `find` function that returns a `user` object.  This user will have a `passwordHash`, and you'll be able to use another `bcrypt` function to recreate the hash:

```js
const bcrypt = require('bcrypt')
async function login(username, password) {
  // ..
  const user = await find(db, username)
  const matches = await bcrypt.compare(password, user.passwordHash)
  if (matches) {
    // .. login
  } else {
    // .. error
  }
}
```

Again, `bcrypt.compare` is async.  It takes the input `password` to check against the `user.passwordHash`, which we generated and stored previously.  If they match, you're golden.  If not, you've got an imposted.

We've made the round trip.  Bcrypt helps us keep the password safe.  How! could Success we make our implementation better?

Be careful when you store a user password record locally.  Here's how to protect the password in node.

Store Local User Password with bcrypt in Node

Create-react-app is Facebook's no-config solution to starting a React project.  This setup does not support a server out of the box.  You can create that easily enough.  Node.js can be a good choice for your app server.  Once it's made and functioning, you might want to deploy your static app and app server.  Here's a few tips on getting them up onto Heroku.



## Two Builds

Create-react-app comes with two npm-scripts targets.  One is the live dev server that builds assets in a development configuration and serves them out of webpack-dev-server.  This is run with `npm start`.  When you're ready for prod, however, there is a `npm run build` script that will build all assets for a production setting.  In this configuration, it's up to you to provide the server to serve such assets.

## Serving Static Assets on Koa

Koa is a nice web framework for Node.js that is fun to write little app servers in.  If you want to tack on web server abilities like serving static assets, that's easy to do as well.  

Assuming that you have the basics of your koa server already going, you now want to:

```
npm install koa-static --static
```

Now you need to use the static middleware.  The main thing you need to know in order to use it is the location of your static assets.  When you previously ran `npm run build`, create-react-app puts the built files into a `build/` directory in the root of your project.  You want to point the middleware there:

```js
const koa = require('koa')
const path = require('path')
const static = require('koa-static')

const app = koa()
app.use(static(path.resolve('build')))
// ...
```

Now requests for static files that your app might make will route to that `build/` directory.  For instance, your main js bundle, probably called something like `/static/js/main.f682b6a1.js` will be addressable because of this middleware.  

It's important to note that the assets in the `build/` directory will only exists (or be updated) when `npm run build` is run.  The directory does *not* contain the live-updated assets that you might be used to seeing in your development cycle of changing a file, refreshing your browser, and getting the change.

## Serving Your React App

Since you've decided to serve all static assets via your app server, this includes the `index.html` which will bootstrap your React app.  Create-react-app is setup to build this file as well, depositing it at `build/index.html`.  You'll want a route in your app server that can serve this as the html that will request the other js and css static assets needed to get your React program fired up.

Koa has a nice helper for establishing route patterns.  Let's grab that:

```
npm install koa-route --save
```

And then setup our index route:

```js
const fs = require('fs')
const route = require('koa-route')

function* index() {
  this.body = fs.readFileSync(path.resolve(path.join('build', 'index.html')), 'utf8')
}

app.use(route.get('*', index))
```

The reason to map the index route to `*` instead of `/` is because your React app likely has *client*-side routes.  So if the client-side router has an `/about` route to the about page, you'll want a user to be able to deep link straight to that experience: the user refreshes the page, makes a request to your koa server at your domain, it doesn't have a server route for `/about`, but drops to the `*` route handler, serves the `index.html`, which invokes your React app, starting the client router, which has a route for `/about` and serves that UI via React.

## Build When Deploying

Now that you have an app that is served successfully from a koa server locally, you want to get that on Heroku.  That's one of the best things about Heroku: the ease with which you can put your app on the Internet.  Now you just need to get the deployment to automatically do what you did locally.  

After you pushed your app to Heroku, you could manually run:

```
heroku run npm run build
```

Then your static assets would be in the expected spot.  But you don't want to have to remember to do this every deploy.  Thankfully, Heroku gives you a nice hook to automatically do things after you run a deploy.  

In your `package.json` file, add a new script:

```json
{
  "scripts": {
    "heroku-postbuild": "npm run build"
  }
}
```

Commit and push this to Heroku, and you'll get your assets building automatically on every deploy.  Create-react-app with a Node.js server on Heroku will be yours!

Are there better or simpler ways that you've found to get this done?

Create-react-app sets up your static asset build.  Now learn how to deploy that easily to an app server on Heroku.

Deploy create-react-app to Heroku on Node.js

When testing a unit, often you'll want to stub out that unit's dependencies.  Some libraries will help you do this.  But you can do it with plain JavaScript, and it works quite well.



## Stubbing Dependencies

We're unit testing a module in Node.  That unit has dependencies on some other sub-unit.  For our example, these units are modules. The subject under test is `prep-for-fight.js`.  It has a dependency on `eat-corn.js`.  We want to eventually stub out `eat-corn.js` within our unit test for `prep-for-fight.js`.

```js
const eatCorn = require('./eat-corn')
module.exports = function prepForFight() {
  const nutrients = eatCorn()
  if (nutrients > NUTRIENT_LEVELS.EAGLE_EMPOWERMENT)
    return { hasCape: donHomemadeCape() }
  // ...
}
```

A [stub](/post/sinon-spies-vs-stubs/) is a testing fake that you create in place of the real thing.  While testing our module `prepForFight`, we'll stub out the sub-unit module `eatCorn`.  That module is tested elsewhere in another unit test.  We don't want to conflate the two tests, coupling them strongly together.  This is because if the sub-unit's implementation changes, we don't want to have to change our current unit's test.  

The tradeoff is that we are going to couple our `prepForFight` unit test to the implementation of that module.  This is generally known as white box testing, where we care about the internal implementation of our source code within our tests.  If we did the opposite, and treated `prepForFight` as a black box and just tested its final output, there really is no need to stub anything or ever care about implementation as long as `prepForFight` continues to do its job.  For today's example, we'll choose to stub so that we can have a more isolated unit and focused test, drawing our unit boundaries strictly around code that exists in `prepForFight` proper.

## Stubbing Libraries

There are libraries that help us stub.  Since we have a `require`d module for `eat-corn.js`, there is one in particular that would do well for us called [proxyquire](https://github.com/thlorenz/proxyquire).  It allows targeting the `./eat-corn.js` import path and replacing it with your own module at test runtime.  With a couple caveats, it usually works quite well.  We're going to *not* use it and see how we fare.

## Stub by Passing the Dependency

An easy way to get a dependency into `prepForFight` is to pass it as a function argument.  The rewrite might look like this:

```js
module.exports = function prepForFight(eatCorn) {
  const nutrients = eatCorn()
  if (nutrients > NUTRIENT_LEVELS.EAGLE_EMPOWERMENT)
    return { hasCape: donHomemadeCape() }
  // ...
}
```

Now we have no `require` statement, and `prepForFight` gets the dependency it needs.  Given this implementation, we can exercise our two code paths in our test:

```js
const test = require('ava') // or whatevs

const subject = require('./prep-for-fight')

test('many nutrients dons cape', t => {
  function eatCornStub() {
    return NUTRIENT_LEVELS.EAGLE_EMPOWERMENT + 1
  }
  t.truthy(subject(eatCornStub).hasCape)
})

test('fewer nutrients remains cape-less', t => {
  function eatCornStub() {
    return NUTRIENT_LEVELS.EAGLE_EMPOWERMENT - 1
  }
  t.falsy(subject(eatCornStub).hasCape)
})
```

By passing in our stub directly, we control the branching inside the function.

## Leaking Dependencies

By exposing `eatCorn` as a function parameter, we're telling all consumers that we rely on `eatCorn`.  We've leaked our dependency, lessening our encapsulation.  For the function to work as written, it always needs the consumer to send it the `eatCorn` argument when `prepForFight` is called.  Let's give it a default, and make the consumer code care about our dependencies a bit less.  The default will be our original `require`d module.

```js
const defaultEatCorn = require('./eat-corn')
module.exports = function prepForFight(eatCorn = defaultEatCorn) {
  const nutrients = eatCorn()
  if (nutrients > NUTRIENT_LEVELS.EAGLE_EMPOWERMENT)
    return { hasCape: donHomemadeCape() }
  // ...
}
```

Now if `eatCorn` is *passed* as an argument, it will be used.  Otherwise, `defaultEatCorn`, which is the normal imported dependency, will be used.  This is great because now consumers don't necessarily have to care about the dependency, except to override, which for now is just a thing our test wants to be able to do.

## Stub Without Changing Your Signature

If putting `eatCorn` in your function parameter list bothers you, here's another potential solution.  

```js
const eatCorn = require('./eat-corn')
exports.prepForFight = function prepForFight() {
  const nutrients = eatCorn()
  if (nutrients > NUTRIENT_LEVELS.EAGLE_EMPOWERMENT)
    return { hasCape: donHomemadeCape() }
  // ...
}

exports.withEatCornForTest = function withEatCornForTest(eatCornOverride) {
  eatCorn = eatCornOverride
}
```

Now you can call `withEatCornForTest` before you exercise your subject under test:

```js
const test = require('ava')

const subject = require('./prep-for-fight')

test('many nutrients dons cape', t => {
  function eatCornStub() {
    return NUTRIENT_LEVELS.EAGLE_EMPOWERMENT + 1
  }
  subject.withEatCornForTest(eatCornStub)
  t.truthy(subject.prepForFight().hasCape)
})
```

This overrides the imported `eatCorn` module much like proxyquire does.  This is nice because your `prepForFight` function remains untouched, but I think there are a few drawbacks.

- We had to change our single export module to a multiple named export module in order to add the extra API for setting the dependency.
- We have code in our src that is there specifically for testing.  The `*ForTest` suffix is a particularly clear flag of that.  But if we remove the `*ForTest` suffix, we simply cloud that fact and make something still test-specific look like it's for general use.
- We have made our `prepForFight` function impure, because now its output can change depending on when or if we call the `withEatCornForTest`, creating a module-global side effect.
- `withEatCornForTest` is further away from the `eatCorn` usage inside of `prepForFight`.  Thus, we could read `prepForFight` and never know that it's possible for the `eatCorn` implementation to be switched out from under us without examination of more code outside that function.

## Decouple Function Parameter Order

Previous to reading Sandi Metz' [POODR](http://www.poodr.com/) book, I hadn't considered this, but she posits that a parameter list has coupling because of the order of the parameters.  To lessen the coupling, she proposes changing the function signature to take an argument hash instead.  This has the benefits of not requiring a specific order, letting consumers name the arguments, creating clarity on the consuming side, and having the consistency of a single argument for most/if not all functions that take input.

We can take advantage of these attributes and realize one of our own in our `prepForFight` function.  If we have multiple dependencies, we can put these dependencies and their defaults inside the argument hash, and no consumer has to know anything about it.  There's no ordering problem.  There's no null arugment passing.  We just specify the keys that we care to specify and have defaults for the rest.  A minor refactor might yield some destructuring of a single object sent to the function:

```js
const defaultEatCorn = require('./eat-corn')
module.exports = function prepForFight({ eatCorn = defaultEatCorn /*, more... */ }) {
  const nutrients = eatCorn()
  if (nutrients > NUTRIENT_LEVELS.EAGLE_EMPOWERMENT)
    return { hasCape: donHomemadeCape() }
  // ...
}
```

I think that's probably our final refactor for now.  What could we do to make this better?  What are other stubbing methods that you've gotten a lot of mileage out of?

How to stub your own dependencies in node without using a library like proxyquire

Stub Dependencies in Node Without Proxyquire

NewRelic is a monitoring vendor that has good support for Node.js apps.  When your Node app goes down, you'll want to tell NewRelic about it.  It's surprisingly easy.



## `newrelic` npm Package

The `newrelic` npm package is super easy to integrate with.  First install:

```
npm install newrelic
```

Call it from early in your program (before errors can start to occur).  NewRelic docs say on the first line of your program.  I load config first.  Everything is fine.

```js
require('newrelic')
```

Then in order to let NewRelic have the data that it needs to run, you'll need to setup the config file.  The lib gives you a seed file to start from.

```
cd <proj_root>
cp node_modules/newrelic/newrelic.js .
```

Make sure to edit it with your changes.  With that file, you'll want to set your app name, your license key, and any other [options you desire](https://github.com/newrelic/node-newrelic/blob/master/lib/config.default.js).

## Calling NewRelic on `uncaughtException`

When your Node app goes down, you have [one final moment](post/handle-errors-node-app/) to send out your SOS signal -- this is in the `uncaughtException` handler.

Usually NewRelic will detect errors automatically, such as those that you return as HTTP responses to your clients of status 500 with some error json.  But in this case, we'll need to send the error manually.  The `newrelic` lib gives us a way to do this with the `notifyError` function.  It takes the error object and optionally some custom parameters.

So, at first glace, we'd do this:

```js
const newrelic = require('newrelic')

process.on('uncaughtException', err => {
  log.fatal({ err }, 'unhandled error')

  newrelic.noticeError(err)
  process.exit(1)
})
```

But this doesn't deterministically give NewRelic time enough to send the error off to their server.  The lib apparently sends errors on some sort of "harvest" cadence.  `process.nextTick` doesn't seem to provide enough time either.  But NewRelic does give another API that should help.  The `shutdown` function cleans up the agent.  It also allows to flush all the pending notifications via `collectPendingData` option previous to shutdown, which is what we want.

So to make this reliable, we change to:

```js
process.on('uncaughtException', err => {
  log.fatal({ err }, 'unhandled error')

  newrelic.noticeError(err)
  newrelic.shutdown({ collectPendingData: true }, err => {
    if (err) log.error({ err }, 'error shutting down newrelic agent')
    process.exit(1)
  })
})
```

And it works like a charm.  Any additional data that you send to monitoring when your app crashes?  Any ways that you know to make this more reliable?  Have fun logging those crashes!

When your app goes down, call newrelic.notifyError with enough time to send

Notify NewRelic of Error on UncaughtException

As with any software, here you can expect the unexpected.  Node apps experience errors as well.  Let's say that an error crops in our Node API -- what should we do about it?



Errors can be handled in a lot of different ways.  "Handling" in this article will essentially mean error response messaging.

#### Specific Messages are Best

When handling errors, we should do so as specifically and quickly as possible.  As an example, if we can respond to a request that causes an error with a more specific error message, this would be better than responding with a generic, catch-all message.  This first message:

```
{ errors: [{ title: 'Title field missing', source: '/data/attributes/title' }] }
```

is better than this:

```
{ errors: [{ title: 'Bad request' }] }
```

But _some_ error messaging is better than nothing.

So, starting with the most specific handling cases and going to the most generic.

## Handle Specific Business Errors

Let's say that we have an endpoint that creates a new book in our books database, requested at `POST /books`.  It validates the request body data shape to ensure sufficient and correct data is entered for new books.  Let's say that a user submits a new book without the required title field.  It would be great to have specific response that tells the client what exactly is wrong.  The controller code, in part, might look like this in [Express](https://expressjs.com/):

```js
router.post('/books', (req, res, next) => {
  const errors = validate(req.body.data)
  if (errors) {
    saveBook(req.body.data, (_, newBook) => {
      res.status(201).json({ data: newBook })
    })
  } else {
    res.status(400).json({ errors })
  }
})
```

That last line is the error response with specific error messages.  These messages are presumeably generated by the `validate` function.  We can provide a specific `bad request` response status code with a `400`.  We can provide the specific error response body, which would look like:

```
{ errors: [{ title: 'Title field missing', source: '/data/attributes/title' }] }
```

## Catch All Other Errors*

Now there are other errors that might occur that we either haven't, don't want, or will never be able to anticipate enough to provide specific handling for.  In these cases, we still want to handle the error, but we'll only be able to provide minimal value and insight into the nature of the error to clients in the responses.

Let's adjust our controller code to handle potentially errors that might happen in the book saving process.  What could happen?  Anything... like database issues with connections, constraints, locks or just something bad in our code.  In our callback, let's do something with that potential error:

```js
router.post('/books', (req, res, next) => {
  const errors = validate(req.body)
  if (errors) {
    saveBook(req.body, (err, newBook) => {
      if err return next(err) // new line

      res.status(201).json({ data: newBook })
    })
  } else {
    res.status(400).json({ errors })
  }
})
```

#### Required Error Contract

The `err` that comes back should be an `Error` or a subtype that at least follows the error contract and has the 3 required fields of `name` to identify the type of error, `message` for the human readable main issue of the error, and `stack` a string of the accurate location and stack trace of the error.

#### Express' `next` for Errors

The `next` function in [Express](https://expressjs.com/) will advance to the next middleware.  In the case of passing a non-null value to `next`, remaining normal middleware will be skipped and the first error middleware will be executed.  Error middleware functions have an arity of 4 parameters instead of the usual 3, where `err` is the first parameter.  Here's a basic catch-all error handler:

```js
app.use((err, req, res, next) => {
  res.status(500).json(serializeError(err))
})
```

`serializeError` is just going to take that and transform it into something to push across the network in the response.

```
function serializeError(err) {
  return { errors: [{ status: 500, title: err.message, stack: err.stack }] }
}
```

#### Don't Leak Stacktraces

Let's add a little something else here.  We probably don't want to leak the stack trace to our users in production, so let's protect it by detecting `NODE_ENV` and update it to be something like:

```
function serializeError(err) {
  const body = { status: 500, title: err.message }
  if (process.env.NODE_ENV !== 'production')
    body.stack = err.stack
  return { errors: [payload] }
}
```

Even better.  Generic errors handled.

## Crash On the Rest

Now note the asterisk on the previous section.  We aren't actually able to catch *all* errors in that generic "catch-all" error handler.  To illustrate, let's create two new routes.  The first is a route that does nothing but throw a new `Error`:

```js
router.get('/debug/error', (req, res, next) => {
  throw new Error('Test explosion')
})
```

The above route, if requested at `GET /debug/error`, would throw a new Error and it would be caught by the generic error handler of the previous section.  This is because `throw` delivers that new `Error` synchronously.  It stays in the context of the current call stack of the request through Express middleware.  And Express can catch the `Error` and call your first error-handling middleware.

#### Out-of-Context Errors

But we can easily break out of this context and bypass the catch-all handler entirely.  All we have to do is use a `setTimeout`.  Calling `setTimeout` will queue a new message for the event loop to, at some later tick of the clock, run the enclosed function.  Within that function, we'll throw another `Error`:

```js
router.get('/debug/crash', (req, res, next) => {
  setTimeout(_ => {
    throw new Error('Error outside request context')
  }, 1000)
})
```

#### Let the Process Die

And now there's really nothing that Express can do to help us.  The `Error` will instead bubble up to the Node process running our app.  That process gives us one final opportunity to know about the occurrence of such a fatal error.  Once we know about it, we can log it.  Perhaps we can get off a call to our monitoring service.  After that, we should assume our process is compromised, potentially unstable, and just crash.  Then use a tool like [forever](https://github.com/foreverjs/forever) to detect the downed process and restart it.  Such a crash handler might look like:

```js
  process.on('uncaughtException', err => {
    log.fatal({ err }, 'uncaught exception')

    process.nextTick(_ => process.exit(1))
  })
```

The call to `process.nextTick` is meant to give some leeway for just enough processing time to finish those last ditch logging/monitoring efforts.

Now we have caught all the errors, in hopefully the best and most helpful way possible.

What other things have you done in your app to make Node error handling better?

##### Great Resources

- [Joyent Production Practices](https://www.joyent.com/node-js/production/design/errors) for all-around design practices
- [Express Error Handling](https://expressjs.com/en/guide/error-handling.html) for framework-specific understanding
- [JSON-API Error Objects](http://jsonapi.org/format/#error-objects) for sensible info to include in an error response
- [MDN Event Loop](https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop) for basics on the message queue in the event loop

How to handle 3 types of errors in an Express API in Node.

Handle Errors in a Node App

With deep object structures, `console.log` can leave you wondering what's down in the depths of the objects you're printing out.  Here's a quick help from Node stdlib.



You can usually see the objects that you're `console.log`ing, and life is good:

Given:

```js
const stickers = {
  are: 'fun'
}
console.log('stickers', stickers)
// Outputs: stickers { are: 'fun' }
```

We can see the whole object, and we're happy.  We can even do objects up to a depth of 3 sub objects, and it works great:

```js
const stickers = {
  are: {
    even: {
      more: 'fun'
    }
  }
}
console.log('stickers', stickers)
// Outputs: stickers { are: { even: { more: 'fun' } } }
```

If, however, we have a potentially deeply-nested object, we'll get some of our data truncated:

```js
const stickers = {
  are: {
    even: {
      more: {
				than: {
					outrageously: 'fun'
				}
			}
    }
  }
}
console.log('stickers', stickers)
// Outputs: stickers { are: { even: { more: [Object] } } }
```

`[Object]`s are cool and all, but not very informative.  

To get at this information, use [`util.inspect`](https://nodejs.org/api/util.html#util_util_inspect_object_options) from the [stdlib](https://nodejs.org/api/util.html#util_util_inspect_object_options).  Here, you can set a depth option to be higher, letting you peer into those deep objects in the console:

```js
const util = require('util')
const stickers = {
  are: {
    even: {
      more: {
				than: {
					outrageously: 'fun'
				}
			}
    }
  }
}
console.log('stickers', util.inspect(stickers, { depth: 4 }))
// Outputs: stickers { are: { even: { more: { than: { outrageously: 'fun' } } } } }
```

Your data is yours again!

Deep objects need some extra help to log all their contents

Console.log Deep Objects

We are hiring a bunch of Node.js developers at the moment.  You may be as well.  I'm sure you have a good list of skills that you're looking for in developers that join your posse.  Here are a few _technical_ things I think about when getting the opportunity  to sit down with a Node developer.  


What might you add to the list?

## Async Programming
- What are options for async flow control?  Favorites?  Least?  Pros?  Cons?
- Node event loop -- How does it work?  What applications does it fit?

## Error Handling
- Limitations/advantages in handling async errors?
- Handle errors in Promises?
- Handle errors in callbacks?
- What errors should be thrown?

## Functional
- What is immutability?  Why want it?
- Basic Array.prototype methods?  
- What does shared state give you?
- Advantages to pure functions?
- Applications of higher-order functions?

## API
- Specifically, how does CORS work?
- Familiar with which request and response formats?  Favorite?  Worst?  Why?
- How does middleware work?  When do you like to use it?

## Ops
- How can you use all CPUs on a server?
- How to associate logs together with a single request path?
- How to recover from fatal errors?
- How do you load per-environment configuration?
- How do you handle versioning of your apps and libs?

## Modules
- Why use modules?
- How do you organize your API apps' modules?
- Flavors of exports?  When do use each?

## ES Next
- Which new features have you used?  Like, not like?
- async/await - Used instead of what?
- Object literal extensions - How used?
- Spread operator - Helpful usecases?
- Arrow functions - What do they help with?

A collection of Node.js skills to explore in an interview

Some Node Skills to Explore

Tools for storing and then later easily running scripts alias names are awesome.  Npm is one of those tools.  Rake is another.  Rake has a cool feature of allowing you to list the scripts available to you.  Npm has not -- until now.



## Npm Scripts

You can specify scripts in your `package.json` for later use.  They go in the `scripts` attribute:

```json
{
  "scripts": {
    "start": "node server.js",
    "dist": "webpack -p"
  }
}
```

To run these scripts, you use the [`npm run`](https://docs.npmjs.com/cli/run-script) command.  For example, type `npm run build` to actually execute the `webpack -p` command in your terminal.  `start` and `test` are specially commands to npm and do not require `run`.  Instead, you can use `npm start`.

## Npm List Scripts

In Rake, you run `rake -T` in your terminal and all the available scripts for your project are displayed.  This allows you to see the what's potentially helpful.  You can't be expected to remember all of them, especially months later, right?

I certainly was having a hard time remembering all of my scripts and getting tired of typing `cat package.json`.  It didn't feel as cool as the `rake -T` experience.  I wrote a library called [`npm-ls-scripts`](https://www.npmjs.com/package/npm-ls-scripts).  To install, run:

```bash
npm install npm-ls-scripts -g
```

I like installing this package globally because then the CLI binary is available globally -- it does seem applicable to all projects.  You could instead include it under project `devDependencies` if you'd like.

If installed globally, you can then run it with:

```bash
ls-scripts
```

If I was to run this in the above project, it would print:

```bash
NPM - ls scripts
---
start - node server.js
dist - webpack -p
---
```

This prints the exact code that can be executed for each script.  If I would like to add some prose around any or all of the available scripts, I can do so by adding more config data to my `package.json` that will print when I run `ls-scripts`:

```json
{
  "config": {
    "scripts: {
       "dist": "Builds project for distribution"
    }
  }
}
```

Now if I were to run the `ls-scripts` command, I would see:

```bash
NPM - ls scripts
---
start - node server.js
dist - Builds project for distribution
---
```

Cool, right?  Complete with Shakespearean prose.

## New: Npm Run

After I had developed the above `npm-ls-scripts` tool, Npm built this feature natively into the core tool -- finally!  Now, to list available scripts, the command is intuitively to leave off the target script and instead simply type:

```bash
npm run
```

On the above example, this would yield:

```bash
Available scripts in the package:
  start
    node server.js
  dist
    webpack -p
```

So, they work about the same, and now it's built in!  What other cool features could you imagine this having?

A CLI tool that will help you list available npm scripts to run.

List Npm Scripts

[Express-validator](https://github.com/ctavan/express-validator) is a [middleware for Express](http://expressjs.com/api.html#middleware) on Node.js that can help you validate user input.  It's a lovely library.  Here's a pattern for testing code that uses it.



## Install

To get `express-validator` in your project, install with npm:

```bash
npm install express-validator --save
```

## Plug in the Middleware

To use the middleware in your code, crack open the part of your app that sets up Express middlewares, and add in a reference to this lil beaut:

```coffeescript
expressValidator = require 'express-validator'

# ...

app.use express.json()
app.use expressValidator()
```

Note that you should insert the new `express-validator` middleware directly after the `json` middleware.  Also note that the `json` middleware, used in conjunction with `urlencoded` now [replace `bodyParser`](http://andrewkelley.me/post/do-not-use-bodyparser-with-express-js.html) to avoid deprecation warnings on startup.

## Validate Something

Let's say you're writing a validator for new posts to a blog.  That code might look like this.

```coffeescript
module.exports = (req) ->

  req.checkBody('title', 'Title is required').notEmpty()
  req.checkBody('body', 'Body is required').notEmpty()

  !req.validationErrors() or req.validationErrors().length is 0
```

It checks to see if there is a title and a body given.  If there is not, the validator will return false, and there will be a validation errors array on the `req` object.  There are many [other assertions (provided internally by validator.js)](https://github.com/chriso/validator.js) besides `notEmpty` that you can use.

## Validating the Validator

First, setup a test helper that stubs the request, `req`, for validation:

```coffeescript
expressValidator = require('express-validator')()

exports.stubForValidation = (done) ->
  req =
    query: {}
    body: {}
    params: {}
    param: (name) ->
      @params[name]

  expressValidator req, {}, ->
    done(req)
```

There's nothing too incredibly special here.  It's just abstracted out of your test into a reusable helper.  The `req` object has empty objects to slap fixture data onto in your tests.  It includes a stubbed `param` function that can return params by name.  Finally, the `express-validator` middleware is called with the stubbed `req` object.

Now, in your test, you can simply:

1. Stub the request before each test
2. Setup your fixture data to make the test pass (or not)
3. Assert validation errors' existence and messages

```coffeescript
stubReq = require('req').stubForValidation
validateNew = require 'blog-new-validator'

describe 'blog-new-validator', ->

  req = null

  beforeEach (done) ->
    stubReq (r) ->
      req = r
      done()

  it 'is invalid without title', ->
    validateNew(req).should.be.false
    req.validationErrors(true).title.msg.should.eql 'Title is required'

  it 'is invalid without body', ->
    validateNew(req).should.be.false
    req.validationErrors(true).body.msg.should.eql 'Body is required'

  it 'is valid with title and body', ->
    req.body.title = 'New Blog Title'
    req.body.body = 'The body of the blog.'
    validateNew(req).should.be.true
```

What do you think?  Is there an easier way?  A way to get better assertions?

Express-validator is a great library for validating input.  Here's how you might test code that uses it.

Testing Express Validator

[Mongoose](http://mongoosejs.com) is an object modeler for [MongoDb](http://mongodb.org).  It has a particularly great feature, called [population](http://mongoosejs.com/docs/populate.html).  Population allows you to include other objects in a specified object.  It makes the process a lot easier and more automatic.



## Document Design

Document databases provide a lot of flexibility in how data is modeled.  Just as the name suggests, data will often mimic real-world documents.  MongoDb isn't a document database per se, but it has enough flexibility to be treated as one.  Mongoose population can help.

A document is often optimized for reads.  All the data is included, such as it would be on a real-world piece of paper.

In Mongo, data is divided into collections.  Mongoose helps define discrete schemas for data in each.  This separates and makes data fields explicit.  This helps with data organization, storage, and writing.  Population helps bring data from different collections back together again easily.  From the [docs](http://mongoosejs.com/docs/populate.html):

> Population is the process of automatically replacing the specified paths in the document with document(s) from other collection(s).

So, let's try it out with a couple examples.

## Data Definition

Mongo will add a primary surrogate key to each object, called [ObjectId](http://docs.mongodb.org/manual/reference/object-id/), referenced as `_id` in the data.  This will be the key used to refer to objects in other collections.  If I wanted to model players in leagues -- foosball leagues of course --  I might create schemas like this:

```coffeescript
mongoose = require 'mongoose'
Schema = mongoose.Schema

playerSchema = new mongoose.Schema
  display_name: String
  leagues: [
    type: Schema.Types.ObjectId
    ref: 'League'
  ]

Player = mongoose.model 'Player', playerSchema
```

```coffeescript
mongoose = require 'mongoose'
Schema = mongoose.Schema

leagueSchema = new mongoose.Schema
  display_name: String
  created_by:
    type: Schema.Types.ObjectId
    ref: 'Player'
  players: [
    type: Schema.Types.ObjectId
    ref: 'Player'
  ]

League = mongoose.model 'League', leagueSchema
```

A few points of interest:

- Since we are using `_id` to refer to other objects, we use the `ObjectId` type in the Mongoose definition.  You can use other fields.  Just make sure the type matches.
- The `ref` attribute must match *exactly* the model name in your model definition.  Otherwise you'll get something like this little beauty: `MissingSchemaError: Schema hasn't been registered for model "Player".`.
- Note that `League.players` is an array.  Just surround the field definition in square brackets to get this functionality.

## Data Read

Reading data is where Mongoose population really shines.  This is the magic that makes reads of documents very straightforward, easy, and fast.  The magic is in the `populate()` function.

### Populate One Field

If I want to populate a single field in the query for an object, I specify the name of that field in a string to the `populate` function:

```coffeescript
Player
  .findOne({ _id: 'abc123' })
  .populate('leagues')
  .exec (err, player) -> #...
```

`leagues` will be populated with an array of full `League` objects when the resulting json returned, just like magic.

### Populate Multiple Fields

Populating multiple referenced objects is similarly easy:

```coffeescript
League
  .find()
  .populate('created_by players')
  .exec (err, league) -> #...
```

Just separate the field names in your `populate` parameter with spaces.  This query will return an array of `League` with the `created_by` and `player` fields populated with the associated `Player` objects.

### Populate Partial Objects

Populating objects like this can quickly bloat your payload size.  To limit included objects to only a subset of fields, you can specify exactly what parts you want populated.  For instance, if my client UI only needed to show a list of leagues that a player belongs to, I could ask for just the `display_name` of the included `League` object by using a 2nd parameter:

```coffeescript
Player
  .findOne({ _id: 'abc123' })
  .populate('leagues', 'display_name')
  .exec (err, player) -> #...
```

For listing multiple parts, separate the attribute names with spaces.

### Forget to Populate

You might get so used to having objects populated for some of your queries that you might wonder why they're not populated in your latest query.  You probably just forgot to call `populate()` in your query.  You must do this explicitly to get the inclusions you desire.  Otherwise, you the data you query will just include the `_id` values.

### Explicitly Exclude Field

It might not be that you forgot to populate, but that on some queries you don't want to populate.  In these cases, you might not want to be sending around unneeded `_id` values, especially if they make up a large portion of your data size.  You can explicitly exclude such fields.  For instance, if you wanted all `Player` models but weren't going to populate `leagues`, you might query:

```coffeescript
Player
  .find()
  .select('-leagues')
  .exec (err, players) -> #...
```

Note the `-` sign in the `select` clause.  This removes the field from the results.

## Data Write

When you go to read the data, it's quick, easy, and automatic.  But that's because some work was done previously to reference the correct objects and make sure these references are saved.  Therefore, the naturally more work-intensive part of the population story is the data writing.

When we write to our example models, we need to save the proper references.  For instance, when a new `League` is created, let's say it automatically needs a `created_by` `Player` reference saved and the creating `Player` will automatically join the league:

```coffeescript
league =
  display_name: myLeagueName
  created_by: currentPlayer._id

League.create league, (err, league) ->
  if err?
    # ... do smart things that are never shown in a tutorial
  else

    Player.update { _id: currentPlayer._id },
      $push:
        leagues: league._id
    , (err, numberAffected, raw) ->
      if err?
        # you know...
      else
        res.json league
```

A few points of interest:

- Lowercase `league` is just the json to populate a `League` model.  `league` is also the shadowed variable name in the `League.create` callback.
- `currentPlayer` is just an imagined `Player` reference that has an `_id` that you will use to associate `Player` to this `League`.
- [`$push`](http://docs.mongodb.org/manual/reference/operator/push/) is a special `update` attribute that appends new elements to a model's array.

Depending on how complex your model relationships become, you may opt for a difference code strategy besides nesting callbacks.  Don't like your pyramid of doom?  Try the awesome [async.js](https://github.com/caolan/async).

I don't know if Mongoose population is going to change your life, but I was very happy when I found this feature.  I had been doing junk like this manually.  What are some other great use cases that you've found?

Mongoose is an object modeler for MongoDb in Node.js.  It has a great feature, population, for including references to other Mongoose models.

Mongoose Population

LiveReload is a wonderful tool/strategy for being able to develop websites more quickly.  You can make a change and see it immediately shown in the browser without requiring you to manually refresh.  To do this in NodeJs with Express, you'll need just a few packages.



## LiveReload Express

If you're doing a website on Nodejs these days, you're probably using Express.  Express uses the Connect library for its middleware scheme.  Just grab one more middleware to handle the LiveReload.  It will add a little snippet of JavaScript to each of the html responses that is requested through this middleware.  First, install:

```
npm install connect-livereload --save-dev
```

Next, import:

```coffeescript
app.configure 'development', ->
  app.use require('connect-livereload') 35729
```

Note that we only want to import and use the middleware when in development.  This matches the fact that we are only declaring this package as in `devDependencies`.

## LiveReload Grunt

Now you need to something to start a livereload server, watch files for changes, and broadcast their availability.  The rest of this will be done via Grunt.

First, install:

```
npm install grunt-cli -g
npm install grunt grunt-regarde grunt-contrib-livereload --save-dev
```

- `grunt-cli` will allow you to run the `grunt` command in the terminal.  Global puts the bin in your path.
- `grunt-regarde` is a file watcher.  It senses changes to the filesystem.
- `grunt-contrib-reload` will handle the pushing of new code to the browser.

Next setup the `Gruntfile.coffee`:

```coffeescript
module.exports = (grunt) ->
  grunt.initConfig

    regarde:
      style:
        files: ['app/static/css/*.css']
        tasks: ['livereload']

  grunt.loadNpmTasks 'grunt-contrib-livereload'

  grunt.registerTask 'be-lively', [ 'livereload-start', 'regarde' ]
```

Now when you run:

```
grunt be-lively
```

A livereload server is going to be setup.  Then we watch for filesystem changes.  The only things we're watching are css files in our app.  You could expand the `files` array to watch more if you wish.

Yay!

## LiveReload all the Things

There are many ways to make this happen.  There are numerous Node libraries, Grunt plugins, and browser plugins.  Do what works for you.  Just make it happen fast and tend it well from the beginning to the end of your project.  Almost every solution I've found is somewhat high maintenance.  This one has proven to be the same.  What have you found to be your favorite LiveReload solution, and in what environment?

Livereload allows you to make code changes and see your changes in the browser without refreshing.

LiveReload for NodeJs

Xip.io is a DNS wildcard domain that was initially created for use on Ruby projects.  But its application is desirable across platforms.  Recently, I wanted to use Xip.io on a NodeJs project.  It was easy to make happen.



## Update: "Proxying"

After having a handful of problems with these home-grown proxies, my search eventually led me to know that Pow 0.4.0+ includes proxying as a built-in feature.  So, you don't need your own `config.ru` any more.  Once you install pow, choose the port that your app will run on and at what pow address you want to access your app, and run one command in your terminal:

```
echo 3000 > ~/.pow/myapp
```

Then you'll access your app, running in the background on port 3000 via the web browser at `http://myapp.dev`.  Way better.  Skip the rest of this article.

## Install Pow and Powder

Pow is a development server that uses Xip.io.  Powder is a cli that makes working with Pow even easier.  To start, make sure that [Ruby](http://www.ruby-lang.org/en/downloads/) is setup with [Rubygems](http://rubygems.org) in your environment.  Then install Pow.  They have a short little install script you can run, as [referenced on the Pow site](http://pow.cx/).

Then install Powder:

```
gem install powder
```

## Proxy your NodeJs app

Pow only knows how to handle Rack apps, so we have to create one to be loaded into Pow.  Since we're writing the guts of our app on another platform -- in this case, Node -- we'll just create a proxy Rack app.

I found a bit of [proxy app code](https://github.com/linjunpop/node-with-pow), which you should save in your project root as `config.ru`, and which I paste here:

```ruby
require "net/http"

class ProxyApp
  def call(env)
    begin
      request = Rack::Request.new(env)
      headers = {}
      env.each do |key, value|
        if key =~ /^http_(.*)/i
          headers[$1] = value
        end
      end
      http = Net::HTTP.new("localhost", 3000)
      http.start do |http|
        response = http.send_request(request.request_method, request.fullpath, request.body.read, headers)
        [response.code, response.to_hash, [response.body]]
      end
    rescue Errno::ECONNREFUSED
      [500, {}, ["Server is down, try $npm start"]]
    end
  end
end
run ProxyApp.new
```

The only change you might have to make is the port number for your app (line 13).  In this case, I'm using port 3000 for my NodeJs app.

## Link Your App

Now all we have to do is link our app to Pow.  Powder helps us do this easily.  Go to your project root directory and type:

```
powder link <project_name>
```

Now run your app with whatever command starts NodeJs.  It's probably something like this:

```
node app.js
```

Now you should be able to access your app via your web browser at `http://<project_name>.dev`.  One of the great things about Xip.io is that it makes running your local app [extremely easy from a local Virtual machine](http://jaketrent.com/post/run-pow-server-on-vm/) as well.  To get the address of your app to use in your VM, type this from your native terminal:

```
powder open --xip
```

And copy the resulting URL into your VM web browser.

Ah, the fusion of Node and Ruby is pretty sweet.

Xip.io is a DNS wildcard initially created for use on Ruby projects.  It makes development much nicer in some respects, so it's worth a few steps to get working on a NodeJs project.

Access NodeJs App on Xip.io

Yeoman is great developer tool that will help you generate a project format that is easy to get up and running quickly.  Out of the box, Yeoman only generates static sites.  But, with a few small tweaks, you can get it running on an app server, Nodejs, and up on Heroku.



## New Project With Yeoman

To install Yeoman, run:

```
npm install -g yo
```

Now navigate to your new project directory and generate the scaffolding by running:

```
yo webapp
```

There are other generators, specifically for angularjs or a handful of other frameworks.  To see more options, run:

```
yo --help
```

## Add Node for Heroku

Since Yeoman creates a static site, it should be read to serve out of any old web server.  But Heroku specifically wants an app server.  Node is one of the options, and it's a great option for a lightweight app server.

To get the node dependencies you need, create a `package.json` via:

```
npm init
```

Then pull down the dependencies you'll need:

```
npm install gzippo express coffee-script --save
```

Now create your app server, `web.coffee`.  It's about as light as they come:

```coffeescript
gzippo = require 'gzippo'
express = require 'express'

app = express()
app.use express.logger 'dev'
app.use gzippo.staticGzip "#{__dirname}/dist"
app.listen process.env.PORT || 5000
```

It's going to be serving the static resources that you build w/ Yeoman/Grunt that end up in the `dist` directory.  In order for code to get to Heroku, it needs to be commited to your git repo.  Thus, you need to remove the "dist" line from your `.gitignore` file, and don't forget to add and commit the files to git as you build them.  You may also have to add more tasks to the `Gruntfile.js` to copy over any other assets to `dist` that your app needs to run.

Finally create a `Procfile` file for Heroku in the root of your project that points to the `web.coffee` server:

```
web: coffee web.coffee
```

Now it's time to push to Heroku.  You should have the Heroku Toolbelt installed and create a heroku url by running:

```
heroku create <my_project>
```

To test your yeoman app as it will be run in production, first build it:

```
grunt build
```

Then run the grunt server:

```
grunt server:dist
```

Or if you have foreman and want to run the app using your `Procfile`, run:

```
foreman start
```

It's time.  Commit your code, especially that `dist` directory, and... deploy!

```
git push heroku master
```

There are quite a few steps there.  What did we miss?  Or what needs more explanation?  Perhaps we just need a `yo heroku` target.

Yeoman will generate a static site.  Make a few changes, and you can have it running on Heroku.

Deploy Yeoman App on Heroku

When your app requires users to be using the `https` protocol to make requests and get responses, it's helpful to have an automatic redirect so the user is always in the right place.  Here's how to do that for a Node Express app on Heroku.

![NodeJs Https Redirect](https://i.imgur.com/5rZ8H.png)



## Secure Express Requests

The [Express](http://expressjs.com) request object has a bunch of great information.  One of its attributes is `secure`.  It's a boolean.  Usually, `secure == true` will mean that you're on `https`.  When that is not true, you're ready for a redirect.  

Unfortunately for apps hosted on Heroku, `request.secure` will always be false.  The way that Heroku routing works, it will just never be set.  

## Heroku Https Header

Instead, Heroku forwards an http header that allows us to do the same "is secure" test.  On Heroku, `request.header('x-forwarded-proto')` will contain the actual protocol string (eg, 'http' or 'https').

## Express Middleware SSL Redirect 

If you're using the Express framework on Node, then you have it easy.  There's already a great middleware mechanism for you to send any or all requests through.  If you set your Express app (v4) up like this:

```js
const app = express()
```

Then you can use the `app.use` functionality to specify a middleware.  Since I only have certificates and want the redirect to happen in the production environment, I will likewise wrap this middleware inside a `process.env.NODE_ENV` check for prod.  This redirect will be pretty rudimentary, but it's just that simple, so here it goes:

```js
if(process.env.NODE_ENV === 'production') {
  app.use((req, res, next) => {
    if (req.header('x-forwarded-proto') !== 'https')
      res.redirect(`https://${req.header('host')}${req.url}`)
    else
      next()
  })
}
```

If it's not `https` already, redirect the same url on `https`.  If it is, that's what I want, and you can pass on through my middleware function.  Note that this middleware will protect *all* urls on the site with an `https` redirect.  Your middleware could be more selective.  You could even create this as a stacked middleware per route if you wanted.  We could even enhance our middleware to use inspect both the http header *and* the `secure` flag.

Heroku threw us a for a minor, unexpected loop, but it was nothing that we couldn't easily code for.

Make a NodeJs app on Heroku redirect to an https address when it is not using one.

Https Redirect for Node on Heroku

Pinterest currently does not have an official webservice API.  It seems kind of crazy in this day and age.  They really should have one.  I can't think what the business reasons might be for not having one.  

They've not had one for long enough that it's high time we write our own.  It'll be surprisingly easy with a few choice tools



## Webservice API on NodeJs 

[NodeJs](http://nodejs.org/) is just a fun platform to write IO-heavy applications for the web.  We're going to write a quick [RESTful](http://en.wikipedia.org/wiki/Representational_state_transfer) endpoint using the [Express](http://expressjs.com/) library that allows us to consume real Pinterest content that's not available via a pre-existing service.

#### Screen Scrape Pinterest

Given no API, we're left to our own devices.  The data for Pinterest is only exposed via the UI on the website.  We're going to have our service visit that UI and grab the data that we need as a user of a web browser would see it.  This is screen scraping.  There a lot of downsides here, but we wouldn't be trying it if there was an API already.

One down side is that our service will be brittle.  If Pinterest ever changes the layout of the page, our service won't be able to bring back the right data.  Our solution will be simple, so it'll be easy to update, but this should be a red flag not to do anything mission critical via screen scraping unless you're giving it your full attention.

Another down side might be speed.  Screen scraping a UI is not the fastest way to get data.  We'll try and help mitigate this with the fastest tools that we have.  NodeJs is a blasted fast web server.  A library called [cheerio](https://github.com/MatthewMueller/cheerio) is supposedly best-in-class for screen scraping (advertised as 8x faster than [jsdom](https://github.com/tmpvar/jsdom)).

#### Caching

To make this retrieval even faster for repeat use, caching could be very helpful.  We could cache in our service what we get back from pinterest via some datastore or we could cache in our client.  Best practices here will be very dependent on your use case.  These kind of enhancements have been made over and over again and would only clutter the simple Pinterest interaction, so I will exclude them for now.

## Getting Pinterest Data

Here's the final solution in all its glory.  This snippet only includes only the code inside the Express route.

```js
// request is a library for making http requests
var request = require('request');

// cheerio is a lib for screen scraping
var cheerio = require('cheerio');

// req and res are express vars for the request and response
exports.list = function (req, res) {

  // this is the actual request to the pinterest page I care about
  request('http://pinterest.com/jaketrent/pins/', function(err, resp, body){
  
    // get ready for scraping
    var $ = cheerio.load(body);
    var pins = [];
    var $pin;
    var $pins = $('.pin');

    var i = 0;
    
    // scraping only the 1st 10 pins;  you could get them all
    for (i; i < 10; ++i) {
      $pin = $pins.eq(i);
      
      // Finding the bits on the page we care about based on class names
      pins.push({
        img: $pin.find('.PinImageImg').attr('src'),
        url: 'http://pinterest.com' + $pin.find('.ImgLink').attr('href'),
        desc: $pin.find('.description').text()
      });
    }
    
    // respond with the final json
    res.send(pins);
  });
};
```

When I wrote it out for my own use, I was surprised at the brevity.  I love it.  Given, there's no handling of any errors or attempts to make this semi-robust.  This just gets us the data on a good day.

The final json that's exposed at our chosen Express endpoint looks like it this:

```js
[ {
    "img": "http://myimg.jpg",
    "url": "http://pinterest.com/pin/mypin",
    "desc": "Sweetest pin ever"
  }, {
  // ...
]
```

It's ready for use by a json-ready client.  So stinkin easy.  We're connecting the web together, and it's awesome!  Now the world will know of the baked goods and flower arrangements that we love the most.

Pintereset doesn't currently have an API.  Here's how to use NodeJs to make a simple Pinterest webservice.

How to Make a Pinterest Webservice

Sometimes you'll want to use [npm](http://npmjs.org/) to install a package that doesn't exist in the official npm registry for some reason.  Maybe you're developing the package yourself.  Maybe it's an internal use-only package. Maybe the package writer never published it.  Here are a few ways to install a local package with npm.



## Link the Local Node Package

If you're developing a package locally, you're likely making lots of changes to the source.  You don't want to publish after each change in order to install this package as a dependency elsewhere.  Npm has provided the [link](http://npmjs.org/doc/link.html) keyword.  

This allows your put a global label on your package that tells your local npm cache about it:

```bash
cd <pkg_dir>;
npm link;
```

And then go to the local project that you're using the package in and point to the dependency by package name:

```bash
cd <project_dir>;
npm link <pkg_name>;
```

Now you have symlinked a directory called `<pkg_name>` in the project's node_modules directory.  Changes you make in your package will appear and be usable from within your project.

## Tarball the Local Node Package

Npm can [install](https://npmjs.org/doc/install.html) either a path or url to a local tarball.  To create a tarball that npm will like, you want to run npm [pack](https://npmjs.org/doc/pack.html):

```bash
cd <pkg_dir>;
npm pack
```
This will create a tarball a la `<pkg_name>-<pkg_ver>.tgz`.  Now just store your new tarball somewhere like a local filesystem or server that is accessible when you run npm install on your dependant project.  

You can ask for the tarball manually:

```bash
npm install http://localServer/my-pkg-0.0.1.tgz
```

Or put the reference in your package.json for later:

```js
{
  "dependencies": {
    "my-pkg": "http://localServer/my-pkg-0.0.1.tgz"
  }
}
```

You probably have a node package that's only available locally for a reason.  To guarantee that it's not accidentally published to the world via the npm registry, add this beaut to your package.json:

```js
{
   "private": true
}
```

## Replicate the Npm Registry

If you're going all out, like creating a corporate npm registry, you always have the option of [replicating](https://npmjs.org/doc/registry.html#Can-I-run-my-own-private-registry) the Npm registry internally.  It's CouchDb, and '[replication](http://www.imdb.com/title/tt0120029/quotes?qt=qt0192627)' is its Chinese neighbor's middle name.  You can publish your own packages there so they're still private to your and your organization.  And you also get local references to the public packages.

## The Glory of Npm

Npm is great.  The ability to install published or local packages is an example of its flexibility and usefulness.  What are the other ways that you've found to make non-public packages installable -- for instance, that package that the writer just never bothered to publish to npm?

How do you install a package with npm that's not in the npm registry?  Perhaps you're developing a package locally.  A few options.

Npm Install Local Files

Sometimes things change in different environments.  For one, I wear a tie to church but try not to otherwise.  In Node, you might want variables to change per environment.  And it turns out people have wanted this exact thing, creating marvels of code for us to use as we please.  Enter, nconf.



## Nconf

[Nconf](https://github.com/flatiron/nconf) provides "hierarchical node.js configuration with files, environment variables, command-line arguments, and atomic object merging."  It is a great little package provided by the folks at [NodeJitsu](http://nodejitsu.com/).  

To install, run a little:

```
$ npm install nconf
```

#### Environment Variables in Node

Often, NodeJs environments are specified through the aptly-named environment variables.  These are the variables that are set on a system level, available to yours and other applications.  In Unixy environments, you'd set one with:

```
$ export NODE_ENV=production
```

And it turns out that `NODE_ENV` is the single most-used environment variable of them all.  Really, it's the big kahuna that triggers other environment variables or code for activation. 

For instance, in an [Express](http://expressjs.com/) app, you'll see this a lot:

```js
if ('production' == app.get('env')) {
  app.set('dbUri', 'n.n.n.n/prod');
}
```

Actually, you see [`configure()`](http://expressjs.com/api.html#app.configure), but it's all just sugar around a check to `process.env.NODE_ENV`.

#### Set NODE_ENV and Start Node

If you want to start node and set the environment variable all in a one liner, try:

```
$ export NODE_ENV=production && node app.js
```

#### Multiple, Variable Sources

You could just grab all your variables out the process environment, but the cool thing that nconf gives you is a hierarchy.  The [docs](https://github.com/flatiron/nconf) give the probable example of the following hierarchy:

1. Command-line arguments
2. Environment variables
3. A file

One could access these sources in code with this beaut:

```js
var nconf = require('nconf');
nconf.argv()
     .env()
     .file({ file: 
       'config.json' 
     });
```

You could even have a config file specific to each environment that ships with your app and load it with a slightly spiffier:

```js
var nconf = require('nconf');
nconf.argv()
     .env()
     .file({ file: 
       process.env.NODE_ENV + '.json'
     });
```

#### Using the Data

Finally, after loading, just access your environment variables and incorporate them in your code as you will:

```js
nconf.get('NODE_ENV');
```

Declare some variables per environment.  Likely candidates are database connection strings or web service endpoints or feature flags.  These will help you avoid as many nasty if-else branches in your code.

Do you need to change variables per environment in NodeJs?  There's a great nconf package that helps out.

Environment Specific Variables in NodeJs

Have you every wished you could change your node script and not restart the server?  Restarting your node server after changing your script can get annoying.  This is especially true if you're in guess and check mode, changing and checking a lot.  Luckily, there are a few tools to save you the pain.



## Node-supervisor

Isaac Schlueter has created a [node-supervisor](https://github.com/isaacs/node-supervisor).  Install it with a quick npm:

    sudo npm install supervisor -g

Use `supervisor`, instead of `node` to start your app:

    supervisor app.js

By default, this will run with these options:

    DEBUG: Running node-supervisor with
    DEBUG:   program 'app.js'
    DEBUG:   --watch '.'
    DEBUG:   --ignore 'undefined'
    DEBUG:   --extensions 'node|js'
    DEBUG:   --exec 'node'

There are other options for things like watching only specific extensions or not restarting on app errors.

## Nodemon

Remy Sharp has created [nodemon](https://github.com/remy/nodemon).  It also is a quick npm install:

    sudo npm install nodemon -g

Again, run with `nodemon` instead of `node`:

    nodemon app.js

By default, runs:

    [nodemon] v0.6.23
    [nodemon] watching: .
    [nodemon] starting `node app.js`

There are similar options here for what to watch.

## Comparison of tools

Both tools will get the job done.  Choose either.  Both can specific directories or file extensions to watch.  Both can run other executables (like [forever](https://github.com/nodejitsu/forever), which is awesome, btw).

Both tools are circa 2010.  Node-supervisor is the slightly older brother (by 8 months).  Both projects seem to have slow, steady development development activity.  What is there to add to tools that do their job already?

In general, `nodemon` has a few more options.  For example, it has explicit support for coffee-script.  I like `supervisor` more for silly reasons:  I found it first, it works fine, feels better to type, and has fewer terminal command expansion conflicts.

*Update*

Recently I have found one clear reason to use `supervisor` over `nodemon`.  `nodemon` doesn't allow starting a process with node and monitoring coffee script files.  The conditionals in the source code rule that possibility right out.  `supervisor` does this just fine with the command:

```
supervisor --extensions 'node|js|coffee' app.coffee
```

Another couple observations:

- `supervisor` doesn't have in-terminal colors
- `supervisor` seems to be more explicit in its logging

Change Your Node Script Without Restarting

A few routes in a NodeJs app -- no problem.  Just put them in your `app.js`.  More than a handful?  You'll probably want to look at organizing your routes a different way.  This will help keep your core `app.js` clean and make your actual routes easier to find.



## Express Routes

The favorite http framework on [NodeJs](http://nodejs.org/) seems to be [Express](http://expressjs.com/).  I personally like it quite a bit.  It's simple, declarative, and allows you to set up all the normal HTTP commands with ease.

Did you know that Express comes with a binary/cli command called `express`.  This command will setup a project structure and a lot of the boilerplate code for you.  The docs contain a guide on [how to run this](http://expressjs.com/guide.html#executable).  It has support for other cool options like session support or various css preprocessors.

A sample run and the file structure that it will create looks something like this:

    $ express --sessions --css stylus --ejs myapp

    create : myapp
    create : myapp/package.json
    create : myapp/app.js
    create : myapp/public
    create : myapp/public/javascripts
    create : myapp/public/images
    create : myapp/public/stylesheets
    create : myapp/public/stylesheets/style.styl
    create : myapp/routes
    create : myapp/routes/index.js
    create : myapp/views
    create : myapp/views/index.ejs

In that generated code, your `app.js` references a route like this:

```js
var routes = require('./routes');
app.get('/', routes.index);
```

And the route, `index.js`, looks like:

```js
exports.index = function(req, res){
  res.render('index', { title: 'Express' });
};
```

I like this, but it's still not the best.  I don't like having to list the `app.get()`, etc. in `app.js`.  This list will get long.  So, how would I adjust?

## All http commands in separate files

Instead, I want to tell `app.js` about my routes, but I want each route file to hook itself up to express directly.

Showing the adjustments, `app.js`:

```js
var app = express();
var indexRt = require('./routes/index')(app);
```

This time, more goodies in `index.js`:

```js
module.exports = function (app) {
  app.get('/', function (req, res) {
    res.render('index');
  });
}
```

The difference is somewhat slight.  Now, instead of each http command listed in your app, only each resource (in REST terms) is listed in app.  Each command for a resource is listed in that resource's route.js.

There are surely many ways to organize your NodeJs routes.  Express gives a great starting pattern.  Our adjustment adds a slight optimization.  Organizing your routes will help your main `app.js` file stay way cleaner, and you'll be able to find your routes more easily.

But this surely can't be the best way for everyone.  What are the ways that you've found successful in organizing your routes?

Note: this post uses express 3 syntax.

How to Organize NodeJs Routes

When you run a shell script or some other process from NodeJs and want to get the output in a stream to a client, you can redirect or pipe the output from the process that originally outputs to stdout into your socketio stream.  



First, setup socketio on your server as you normally would:

```js
var io = require('socket.io').listen(80);
```

    io.sockets.on('connection', function (socket) {
      runMyProcess(socket);
      socket.on('my other event', function (data) {
        console.log(data);
      });
    });

Second, setup your process to run [using spawn](http://rockycode.com/blog/run-shell-script-nodejs/).  For instance:

```js
var spawn = require('child_process').spawn;
var myProcess = spawn('sh', [ 'myShellScript.sh' ]);
```

And setup your `myProcess` variable with the needed callbacks.  The 'data' callback is called whenever there is output from the process.  So, if you want to catch data from `stdout` and `stderr`, catch those and emit the data into your socketio stream, a la:

```js
myProcess.stdout.setEncoding('utf-8');
myProcess.stdout.on('data', function (data) {
  socket.emit('process_data', data);
});
myProcess.stderr.setEncoding('utf-8');
myProcess.stderr.on('data', function (data) {
  socket.emit('process_data', data);
});
```

An important note from [the docs](http://nodejs.org/api/stream.html#stream_event_data) is that:

> The 'data' event emits either a Buffer (by default) or a string if setEncoding() was used.

If I didn't set the encoding so strings were returned, they did not transfer well over socketio.  Well, there you have it.  You can now get output from child processes to stream through socketio.  In my case, I was wanting to show realtime output in a web client.  It's a fun combination of abilities that should produce interesting applications.  Have fun!

When you run a shell script or some other process from NodeJs and want to get the output in a stream to a client, you can redirect or pipe the output from t

Pipe stdout to SocketIO

NodeJs can do many things.  It can even run things that it can't do itself.  For instance, I have a shell script that I don't want to rewrite in Javascript, so I want my Node app to run the shell script.  No problemo.  



(Updated 22 Aug 2016)

The [docs](http://nodejs.org/api/child_process.html) indicate a number of ways you can kick off child processes.  In my case, I wanted to use [`spawn`](http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options).

Import the `child_process` library:

```js
const { spawn } = require('child_process')
```

Now you have `spawn` available to call.  Just send it the right stuff.  I have a `deploy.sh` script that I want to run.  The command I'll actually run is `sh`, and I'll pass the name of the script to it in the 2nd parameter.  I can also specify other options in the 3rd parameter.  For instance, I want this to execute in a certain directory and with certain commands available to it via the `PATH` variable.  And so:

```js
const deploySh = spawn('sh', [ 'deploy.sh' ], {
  cwd: process.env.HOME + '/myProject',
  env: Object.assign({}, process.env, { PATH: process.env.PATH + ':/usr/local/bin' })
})
```

This will execute the shell script.  There are other callback events that you may/not want to set to grab the data that comes back from the process, such as 'data', 'end', or 'exit'.

Rock that!