Here's how you can change the default rendering of code blocks using marksy.



## An MDX Alternative

[MDX](https://github.com/mdx-js/mdx) is a popular superset of Markdown that allows rendering custom components from within the usual markdown formatting. [Marksy](https://www.npmjs.com/package/marksy) is a competing library. It does the same thing, and at least in my experience was even easier to set up.

## Compilation Configuration

Markdown must be compiled from its original format in order to be rendered as  HTML. 

My compilation function looks like this:

```javascript
import marksy from "marksy"
import { createElement } from "react"
import { renderToStaticMarkup } from "react-dom/server"

function compileMarkdown(markdown) {
  const compile = marksy({
    createElement, 
    highlight,     // implemented elsewhere
    elements,      // implemented elsewhere
  })
  const result = compile(markdown, {})
  result.string = renderToStaticMarkup(result.tree)
  return result
}
```

Marksy allows setting your own `createElement` function. In my case, this is being provided by React. Final markup is also done by React.

## Customizing Highlighting

Code highlight can come from anywhere. I chose to use the [PrismJS](https://prismjs.com/) library. It's used on the server, so I'm using the Node API. The `highlight` function is as follows:

```javascript
import Prism from "prismjs"

const highlight = (language, code) => {
  return Prism.highlight(code, Prism.languages.javascript, language)
}
```

## Customizing Elements

Marksy allows for overrides to how the markdown is represented in HTML. This is provided in the `elements` property given to marksy.

I wanted to change the code block of my markdown. (You know, those triple backtick blocks?) I wanted to set a `tabIndex` on the `code` element. I did that like this:

```javascript
import { createElement } from "react"

const elements = {
  code({ language, code }) {
    return createElement(
      "pre",
      {},
      createElement("code", {
        dangerouslySetInnerHTML: { __html: highlight(language, code) },
        tabIndex: 0
      })
    )
  }
}
```

Note that I'm using React but not using JSX here. But if your environment constraints are different, you likely would use it. Because of the way marksy is composed, I also have to call my `highlight` method from within the `code` function.

Hook this set of functions into your markdown compilation pipeline, and you're set.

Do you have any suggestions for improvements on this method of overriding the code blocks in marksy?

Change the default rendering of code blocks using marksy

Override the Code Block Element in Marksy

Let's setup Next.js to be able to serve markdown content, such as blog posts.



## Next.js Custom Server

Next.js has a nice feature set out of the box.  If you have a site that's more on the site end of the spectrum (rather than the app end), you'll likely be able to use the `next` server as-is. 

If, however, you'd like to do app-ish things on the backend, you'll likely need your own custom server.  This is not hard.

Serving markdown seems to be one of the things that needs a custom server.  I tried rendering the Markdown straight from React components, and it didn't seem to work.  This after adding custom markdown loaders to the webpack config and all that.  It still didn't work.  The big hang up, for some unknown reason, was that the markdown modules could not be found in order to be imported correctly.  

My solution was to build a custom server.  

## Basic Custom Server Setup

In Next.js terms, a custom server is simply some JavaScript that starts the next app in addition to doing any of the extra app stuff you need.

First, change your `package.json` `start` script:

```
{
  "scripts": {
    "start": "node server.js"
  }
}
```

Then do the basics in your `server.js` to initialize `next`:

```js
const { createServer } = require('http')
const next = require('next')

const dev = process.env.NODE_ENV !== 'production'
const app = next({ dev })
const handle = app.getRequestHandler()

;(async _ => {
  await app.prepare()
  createServer(async (req, res) => {
    // ...more custom stuff here later

    return handle(req, res, parsedUrl)
  })
  .listen(process.env.PORT || 3000, (err) => {
    if (err) throw err
    console.log('> Ready on http://localhost:' + (process.env.PORT || 3000))
  })
})()
```

This can be done in Express, Koa, or any other http server library.  This example uses the Node built-in `http.createServer`.  Note that all this is doing is creating a `next` app, then using that app's request handler to handle all incoming requests.  Custom additions are yet to come.

## Reading Markdown

Since we're on the server, we have full access to the Node API.  This includes the `fs` filesystem API.  To get our markdown content, let's read markdown files out of a `posts` directory.  Once the file strings are read, let's use a library called [`front-matter`](http://npm.im/front-matter) to parse metadata out of the front matter (not the markdown content yet).  First install:

```bash
npm install front-matter
```

And let's put all this in a `posts-repo.js`:

```js
const path = require('path')
const { promisify } = require('util')
const frontMatter = require('front-matter')
const fs = require('fs')

const readdir = promisify(fs.readdir)
const readFile = promisify(fs.readFile)

const deserialize = parsed => ({
  title: parsed.attributes.title,
  date: parsed.attributes.date,
  body: parsed.body
})

async function fetch() {
  const postsDir = path.join(__dirname, 'posts')
  const filenames = await readdir(postsDir)
  
  return Promise.resolve(filenames.map(async filename => {
    const markdown = await readFile(path.join(postsDir, filename), 'utf-8')
    const parsed = frontMatter(markdown)
    return deserialize(parsed)
  }))
}

exports.fetch = fetch
```

## Serving Markdown

Now we have a module that can read and load into memory all the parsed markdown files.  Now we have to serve this markdown with our custom server.  

First, we need to detect that our client is making a request for the markdown.  We'll do an overly-simple check of the request path.  If the request url matches '/api/', we'll assume we're asking for this single endpoint to fetch all the markdown posts.  We can make this more complicated later.  

In `server.js`, we'll make the adjustments.  Inside the `createServer` callback, we don't just pass everything on to the Next.js `handle` function any  more.  We now parse the URL and call our `postsRepo.fetch` instead:

```js
const { parse } = require('url')

const postsRepo = require('./posts-repo')

const serialize = data => JSON.stringify({ data })

// ...

  createServer(async (req, res) => {
    const parsedUrl = parse(req.url, true)
    const { pathname } = parsedUrl

    if (pathname.includes('/api/')) {
      const posts = await postsRepo.fetch()

      res.writeHead(200, { 'Content-Type': 'application/json'})
      return res.end(serialize(posts))
    }

    return handle(req, res, parsedUrl)
  })
```

## Requesting Markdown from Next.js Component

Now we have an endpoint that can serve the markdown-based posts.  We need to get the React components to request the data they need.  In Next.js, this is done on the server and in the client.  We'll choose an http library that can function in both places.  We'll also install a markdown parser:

```bash
npm install isomorphic-fetch react-markdown
```

We're ready to rock and roll in our React component.  The top-level component in Next.js, called a "page", has a special hook called `getInitialProps`.  Here in `pages/indexx.js` we can do our data fetching:

```
import Post from '../components/post'

export static class Index extends React.Component {
  static async getInitialProps({ query }) {
    const posts = await Post.fetch()
    return { posts }
  }
  render() {
     {this.props.posts.map(p => <Post post={p} />)}
  }
}
```

And then `components/post.js`:

```js
import 'isomorphic-fetch'
import Markdown from 'react-markdown'
import React from 'react'

const Post = props =>
  <div>
    <h1 className="title">{props.title}</h1>
    <time className="date" dateTime={props.date}>{props.date}</time>
    <Markdown source={props.body} />
  </div>

class PostContainer extends React.Component {
  static async fetch() {
    const res = await fetch('<yourHost?/api/v1/posts')
    const json = await res.json()
    return json ? json.data : {}
  }
  render() {
    return <Post {...this.props.post} />
  }
}

export default PostContainer
```

Here we have the http code to fetch the post from the server, and we're finally rendering markdown.

There are a few steps here, aren't there.  We can still get the coolness of Next.js' server-rendering of React while doing our own custom markdown fetching on top of it.

Is this how you render markdown content in Next.js?  How have you improved upon the method?

Setup Next.js to be able to serve markdown content

Serve Markdown with a Next.js Server

Once you get a bunch of data that you want to put on your site, it's common to reach for a database.  Then you'll write an app that consumes that database and renders a dynamic site.  What if, instead, you could create a static site from that data.  What would that buy you?



## Dynamic Database

If you have a dynamic site backed by a database (eg, postgres), you'll have some good and some bad.  The good:

- Your data is well-defined, segregated into clean columns of data
- You can plug your data in anywhere in your dynamic site
- You can change data without having to redeploy your site

The bad:

- Your code is more complicated.  There are more moving parts to support the dynamic render of the site.
- Your site requires more computing power to run.  Simple crud is no big deal per se, but your site could be stressed under load.
- Your dynamic site is more expensive to host.  You have tons of options, but many will cost money.

## Markdown as a Database

But what if you could use markdown files as a database.  Instead of columns accessed by a database process, your data is stored on the filesystem as files.  The most database-like part of a markdown file is the front matter.  This is a meta data section at the top of the document that maintains some info about the document itself.  What if it contained all or some of the significant data of your doc?  

Front matter can be presented in a couple of formats like yaml, toml, or json.  I prefer yaml.  The front matter is demarcated with `---` on both sides, like:

```
aName: "for my front matter data"
```

Hmm, there are possibilities here.  It almost looks like a key-value store.  Once you have a key-value pair in the front matter, you can pull it out in the template and have access to it.  For instance, in [Hugo](http://gohugo.io/), a static site platform, you can access custom variables in your templates on the page's `.Params` object, like: `{{ .Params.aName}}`.

## Database for a Static Site

When you use markdown files as the database for your site, you get some great things:

- Good-enough data segregation with key-value pairs in the front matter.
- You get a simple, light-weight site that will take a load beating. 
- Likely cheaper and simpler infrastructure

And a couple downers:

- You redeploy to update data.  If your site doesn't change much, this might not matter to you.  Use automated build and deploy tools to make this less time-consuming.
- Mass updates are harder and less precise.  You are not writing sql update statements.  You're writing `sed` statements for string find and replacement.
- You cannot join over data sets in any meaningful way.
- You lose the flexibility to code everything else that you might want a dynamic site for

## Export Your Postgres Data

If you want to make the conversion, get your data out of your database.  There are tools for this, like [pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html).  I'll use my data in postgres on heroku as an example:

```
heroku pg:backups:capture            # export binary dump of database
heroku pg:backups:download           # get the dump onto your computer
pg_restore -f latest.sql latest.dump # convert to plaintext .sql file
```

Now cleanup your data in the .sql file.  Get rid of all that table, key, and sequence creation in the .sql file that won't matter in this new world.  The dumps use the postgres `copy` command.  Pare down to one data collection per file, just the rows of data for the `copy` command.  For me, a couple of those rows might look like:

```
25	The Innovator’s Dilemma	The simultaneous need and danger to reinvent oneself and one's business is the dilemma.  Do we disrupt ourselves or do we double-down on core competencies?  A dilemma indeed!	https://i.imgur.com/exW4j8F.jpg	2014-08-20		2014-11-28 19:05:28.153598	2014-11-28 19:05:28.153598	Craig Christensen	http://www.amazon.com/gp/product/B00E257S86/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B00E257S86&linkCode=as2&tag=jaktre-20&linkId=5CDH2BHJK7TZT52R
26	Steve Jobs	Jobs is revealed as a man of many passions.  Reading the formative moments of the personal computing technology and industry is very interesting!  However, the coarse language in the book causes me to not recommend it.	https://i.imgur.com/hpwhvgx.jpg	2014-09-10		2014-11-28 19:08:29.530743	2014-11-28 19:08:29.530743	Walter Isaacson	http://www.amazon.com/gp/product/B004W2UBYW/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B004W2UBYW&linkCode=as2&tag=jaktre-20&linkId=KPZUBQVLGURNA3RW
```

## Create the Markdown Files

If you have a lot of data or just a little or just like writing code, you might want to automate this.  I chose to write my script in javascript.  Here it is (or this [gist](https://gist.github.com/jaketrent/87d0695ab865d4a739be59434261bbd0)) -- helper functions are at the top; actual steps are shown at the bottom.  If you read the steps at the bottom, you should see rather quickly what this script is meant to do:

```js
const fs = require('fs')

const separateColumns = line => line.split('\t')
const nameColumns = line => ({
  id: line[0],
  title: line[1],
  description: line[2],
  coverUrl: line[3],
  completionDate: line[4],
  reviewUrl: line[5],
  createdAt: line[6],
  updatedAt: line[7],
  author: line[8],
  affiliateUrl: line[9]
})
const nonEmptyColumns = line => line.id !== ''
const deriveColumns = line => Object.assign({}, line, {
  descriptionSummary: line.description.substr(0, 150),
  fileName: __dirname + '/books/' + line.title.split(' ').join('-').toLowerCase() + '.md',
  reviewUrl: line.reviewUrl.replace('http', 'https')
})
const deriveFrontMatter = line => Object.assign({}, line, {
    fileContents: `---
affiliateUrl: "${line.affiliateUrl}"
author: "${line.author}"
date: "${line.completionDate}"
categories:
  - "Book"
comments: true
description: "${line.descriptionSummary}"
draft: false
metaKeywords: ""
layout: "book"
image: "${line.coverUrl}"
reviewUrl: "${line.reviewUrl}"
title: "${line.title}"
---

${line.description}
`})
const writeFiles = line => fs.writeFileSync(line.fileName, line.fileContents)

// the steps!
const file = fs.readFileSync(__dirname + '/latest.sql', 'utf8')
const lines = file
  .split('\n')
  .map(separateColumns)
  .map(nameColumns)
  .filter(nonEmptyColumns)
  .map(deriveColumns)
  .map(deriveFrontMatter)
  .forEach(writeFiles)
```

What do you think about this idea?  Would it be a good idea for any of your projects to push your database into markdown files?  Would you have anything to gain by it?

You have a site with a cms and a database.  Would it work as a static site based on markdown?  Maybe.

Convert a Database to Markdown

Several weeks ago, I decided it was time for me to move on from Octopress.  It's given me a good run.  But there were a couple of causes that pushed me over the edge.  My experience has been good with a few pains in the conversion.



## Outgrowing Octopress

The most fundamental and important reason for me converting from Octopress is that it was getting slow to build my site.  I'd write an article, run `rake generate`, and it'd take (anecdotally) 45 seconds.  Wow, impatient, right?  Well it was apparently enough to make it less fun to write on as a platform.

Updating my site's layout on Octopress was also troublesome.  I had written my own theme, but I had worked my custom theme into the existing `sass` and template hierarchy that exists in Octopress' default theme.  These files are a tangle and hard to work with.

Octopress has made rumblings about being updated and improved upon.  But these efforts have been silent as of late.  The last commits on Jekyll's Ferrari (v3), as of this writing, were a full year ago.

Finally, I usually welcome an opportunity to learn something new.  I was on the hunt for a platform that would enccourage my writing.

## Tried Hexo

I looked at Hexo.  It looked rather promising.  It's based on Node.  The docs were extensive.  Initial speed tests were pretty good.  It seems like a pretty cool project.  But then I pushed close to 300 articles into it, and it slowed down significantly -- about 30 seconds to build the site.  Then I read some github issues about how people were having similar problems.  Then I tried to extend it with different types of content beyond blog posts (a core use case for the next generation of my blog), and got lost in creating new Hexo generators.  So I kept looking.

## Fun with Hugo

Hugo was another static site generator that appeared high on a number of top 10 lists.  It is written in golang.  I picked it up.  I identified with a marketing slogan high on their list: "Make the web fun again."  Yes!  This is essentially how I wanted to feel when I thought about adding another post to my blog.

### Hugo is Fast

Initial tests with speed were mind blowing -- near instantaneous.  Then I added almost 300 articles.  The live server and blog generation were still quite fast (606ms):

```
278 regular pages created
210 other pages created
2 non-page files copied
287 paginator pages created
198 tags created
8 categories created
total in 606 ms
```

This is a fresh build.  The caching, however, makes it even faster.  Two builds in a row will yield a second build that's faster (barely, 596ms). 

I realized that part of the fast speed was because it wasn't doing code highlighting, which I wanted on.  I read through the docs at how to turn those on.  They're actually run in python by pygments, and the docs say they're slow.  Yes, the build time went way up (15s). But again, caching still helps significantly here (in a tests, anywhere from a total 800ms to 3s) -- and is a lot more important.  The dev cycle response time is enjoyably fast.

### Templates are Easy & Intuitive

As I have rebuilt my blog on Hugo, I've been pleasantly surprised at how easy things have been to figure out and implement.  Templates are written in go templates, which are pretty nice.  There are good built-in parameters for the site and each page available in the templates.

Creating arbitrary types of content is super easy with its cli tool (via `hugo new`) that doesn't really assume anything and doesn't try to be too clever.  I can easily start new "sections" of content.

Most things are driven by a template.  They go in intuitive locations and are easy to override -- for different sections, the sitemap, and rss feeds.  There's good template fallbacks with the `_default` section.  I can find simple instructions for desired features in the docs.

### Things I Wished Hugo Had

The theme install instructions are odd.  I suppose if you want a plug n play theme, they're fine.  But I have just put my custom theme straight into my blog git repo, not using a git submodule as instructed.  

There are a few features that I miss from Octopress, including the ability to give code snippets file names.  That's apparently not generally-accepted markdown, but it's super nice.  I could write a shortcode, equivalent to Jekyll's plugins, but I'm trying to keep my Markdown vanilla parseable.  It also seems that Jekyll was more forgiving of my sloppy markdown formats.

## Data Conversion

In order to make the migration to Hugo from Octopress, my source content markdown files needed some changing.  Lots of this was in the front matter.  There were a few issues in the body of the content as well where I was using things that Octopress supported but that Hugo did not.

### Date Format

The publish date in the front matter for Octopress was in the format of:

```
date: 2015-02-28 16:20
```

Hugo wanted it in the format of:

```
date: "2015-02-28"
```

I changed this with a sed script:

```
find ./ -type f -exec sed -i '' -E "s/(date: )([^ ]+).*$/\1\"\2\"/" {} \;
```

### Split Categories

As far as I know, Octopress didn't support categories.  So I mixed my categories in as tags for my posts.  Hugo can treat them as separate things, so I split out what looked like this:

```
from: categories: [Code, heroku, dns]
```

Into this:

```
categories: 
  - "Code"
tags: [heroku, dns]
```

This is partial step, not the final format.  We'll finish it soon. 

To get categories onto a new line, it turned out that I needed a new version of `sed`.  Instead of the BSD version that's on the Mac by default, I had to install the GNU version:

```
brew install gnu-sed
```

This is apparently because the BSD `sed` can't read newline charaters `\n` in a regex.  It needs the literal new line (as in, press enter).  That seems like a pain for a nice, copy-pasteable regex.  So, in this command, I used `gsed`:

```
find . -iname "*.md" -exec gsed -i -E "s/categories: \[(\w+)(, ?)(.*)$/categories:\n  - \"\1\"\ntags: \[\3/" {} \;
```

### Format Tags

The remaining comma-separated array of tags that I was using in Octopress land wasn't apparently real yaml at all.  Anyway, Hugo didn't like it.  So, I was going to make a bulleted list.  To go from this:

```
categories: [heroku, dns]
```

To this:

```
tags:
- heroku
- dns
```

I needed to run a couple `sed` commands in a row (because I couldn't figure out how to do this with a single command).  The first command is for the posts that have one tag, then 2 tags, and so on.  I'll show you the first two.  You can extrapolate from there:

```
find . -iname "*.md" -exec gsed -i -E "s/tags: \[([a-z-]+)]$/tags:\n  - \"\1\"/" {} \;
find . -iname "*.md" -exec gsed -i -E "s/tags: \[([a-z-]+)(, ?)([a-z-]+)]$/tags:\n  - \"\1\"\n  - \"\3\"/" {} \;
```

### Moving Cover Image Into Front Matter

For a couple years, I've put a cover image in the body of the post text, right at the end of the summary section.  I wanted to move this into the front matter.  It would give me a little more visual control.  I tried many, many `grep`, `sed`, and even `perl` commands to try to make this happen.  In the end, I went through articles manually.  That turned out to be a blessing in disguise (at least I tried to squeeze lemonade out of it), and I could make other adjustments to data, where my style has changed in various ways over the years.

I found files with images using this `grep` command:

```
grep -o -c "^\!\[.*\]$.*$" * | awk -F: '{if ($2 > 0){print $1}}'
```

### Remove Code Snippet Filenames

One of the features I really liked from Octopress was the out-of-the-box support for showing file names on code snippets blocks (the ones that use 3 backticks surrounding the block). Apparently, however, this is not standard Markdown trickery, or at least Hugo doesn't like it and stops all formatting if it detects it.  It would be much nicer if it just ignored it.

Again, I was stifled in my automated solution to this.  I spent a bit of time on this, but since it affected a relatively small amount of files, I learned my lesson from the pain of time spent on the cover image transform and just went for the manual editing. I think the problem had to do with the backticks not being well-supported (or at least understood by me) in the `sed` replacement command.  So I just found the files like this:

```
grep "^$\`\{3\}[a-z]\+$ [a-zA-Z_\-\.\/\-]\+" *.md
```

### Strict Hash Separation

In markdown, you make header text using the leading hashes.  Apparently, I had hashes that butted right up next to the header text in some cases.  Octopress dealt with that ok, but this new parser used by Hugo didn't like it and would render hash literals.  To make the strict separation, I ran:

```
find . -iname "*.md" -exec gsed -i -E "s/^(#+)(\w+)/\1 \2/" {} \; 
```

In all, the Hugo conversion has led to a fast development cycle of a fast static site.  It has brought some of the blogging fun back.  I did a visual redesign at the same time, and that's been fun too.  If it still looks like a painterly celestial scene when you read this, let me know what you think of it.  

Are you using Hugo?  What do you think of it so far?

A story of my migration from Octopress to Hugo static site generator