Creating a "DSL" for GitHub's API (rewriting philschatz/octokit.js)

I originally wrote philschatz/octokit.js to interact with GitHub’s API using Promises. I started by forking an existing API michael/github and rewrote it using CoffeeScript and jQuery Promises. Then, thanks to contributions, they removed the dependence on Underscore and I removed the dependence on jQuery (and added support for several promise implementations).

But the code was getting too large as people incrementally added features so I thought it was time to rewrite with a focus on implementing as much of GitHub’s API concisely and consistently (with the hope of getting it adopted by GitHub officially).

Given my interest in Programming Languages, I decided to make a “Domain Specific Language” for GitHub (it’s in quotes because technically it’s still just Javascript but read on.

The results are at philschatz/octokat.js.

Necessary Features

This new library needed to:

1. support ~100% of the GitHub API from the start
2. have as little code as possible to maintain
3. work in NodeJS and the browser
4. have multiple source files
5. have tons of unit tests
6. support NodeJS callbacks and Promises

Let me quickly go through each of these and the challenges each presented.

Support ~100% of the GitHub API

This library abstracts all the authentication, status codes, caching, headers, HyperMedia (URL templates), and pagination returned from GitHub.

Abstracting Requests and Responses

Understanding how to form a request and how to parse the response is the bulk of the API; the logic is in src/request.coffee and src/replacer.coffee.

The other part is a chaining function that constructs a valid request.

Constructing Valid Requests

In the original philschatz/octokit.js there was a ton of copy/pasta dedicated to just constructing valid URLs.

Instead, this library has a regular expression that validates all URLs before calling GitHub and constructs objects dynamically through chaining (see src/grammar.coffee).

By reading the documentation at https://developer.github.com/v3/ a developer implicitly constructs a URL and then issues a request by calling one of the verb methods.

For example, to list all comments on an issue convert the documentation URL to the following:

GET /repos/:owner/:repo/issues/:number/comments

    .repos(owner, repo).issues(number).comments.fetch()

Here are 3 ways to list all comments on an issue:

octo = new Octokat()

REPO = octo.repos('octokit/octokit.rb') # for brevity

# Option 1: using callbacks
REPO.issues(1).comments.fetch (err, comments) ->
  console.error(err) if err
  console.log(comments) unless err

# Option 2: using Promises
REPO.issues(1).comments.fetch()
.then (comments) ->
  console.log(comments)

# Option 3: using methods on the fetched Repository object
REPO.fetch()
.then (repo) ->
  # `repo` contains returned JSON and additional methods
  repo.issues(1).fetch()
  .then (issue) ->
    # `repo` contains returned JSON and additional methods
    issue.comments.fetch()
    .then (comments) ->
      console.log(comments)

There are several verb methods to choose from:

• .fetch() sends a GET and yields an Object
• .read() sends a GET and yields raw data
• .create(...) sends a POST and yields an Object
• .update(...) sends a PATCH and yields an Object
• .remove() sends a DELETE and yields a boolean
• .add(...) sends a PUT and yields a boolean
• .contains(...) sends a GET and yields a boolean

Pagination and HTML templates

GitHub returns headers for lists of results. These are automatically converted to nextPage, previousPage, firstPage, and lastPage methods on the resulting JSON.

Paging through all the issues on a repository looks like this:

# Create an issues object (but does not fetch anything yet)
ISSUES = octo.repos('octokit/octokit.rb').issues

# Option 1: with callbacks
ISSUES.fetch (err, issues) ->
  console.log(issues)
  issues.nextPage (err, moreIssues) ->
    console.log(moreIssues)

# Option 2: with Promises
ISSUES.fetch()
.then (issues) ->
  console.log(issues)
  issues.nextPage()
  .then (moreIssues) ->
    console.log(moreIssues)

Maintain a Minimal Amount of Code

With this API you are prevented from constructing an invalid URL because every request is validated against the URL_VALIDATOR regular expression before sending the request to GitHub.

Instead of writing largely copy/pasta code that constructs classes I opted for a regular expression that represents the entire GitHub API.

This can be found in src/grammar.coffee.

Works in NodeJS and the Browser

Getting this to work was a bit challenging.

NodeJS and AMD have slightly different syntaxes; enough to require adding in some boilerplate code to convert between the two.

Each of the source files contains 2 lines of boilerplate on the top of the file and the bottom.

Attempt 1

Originally, I had the entire library in a single file.

I was able to have very little boilerplate. Just something like the following:

@define ?= (cb) -> cb((dep) -> require(dep.replace('cs!', '')))

define (require) ->

  foo = require 'foo'

  ...

  module?.exports = Octokat # For NodeJS
  window?.Octokat = Octokat # For browsers
  return Octokat            # For browsers using AMD

This worked well but resulted in a single large file.

Attempt 2

I then split up the library into multiple files and got it working but ran into a problem when trying to build everything into one file.

I spent about a week trying to get jrburke/r.js and jrburke/almond to build a single file but could not get NodeJS, AMD, and the single file to all work at the same time.

Attempt 3

Finally, I opted for compiling the coffee files and concatenating them together with a custom define method similar to what other libraries do (see the require() method in less/less.js.

With this option, the tests run:

• on the source coffee files in NodeJS
• on the source coffee files in the browser
• on the built dist/octokat.js file in the browser

Finally, multiple source files and tests running in all 3 environments!

Fixtures and Recording HTTP Requests

The library currently runs about 400 tests. There are about 120 unique tests, 80 are alternates using callbacks instead of promises, and 200 are the same tests but running in the browser.

In order to not pummel GitHub with duplicate requests I use linkedin/sepia to generate “cassettes” to replay the HTTP requests.

Unfortunately, sepia only runs on NodeJS so I wrote philschatz/sepia.js which plays back (and can record) linkedin/sepia tests.

Support NodeJS Callbacks and Promises

In order to support both callbacks and promises, the asynchronous methods (called verb methods) all support a function as the final argument and return a Promise.

This way, you can always end each line with a callback or with a .then() and the code just works.

The client only returns a Promise if one of the supported Promise libraries are detected (jQuery, angular, Q, or native Promises).

If you use one of those libraries but still prefer to use callbacks just make sure the callback is the last argument and the code will just work^TM.