Creating a "DSL" for GitHub's API (rewriting philschatz/octokit.js)25 May 2014
I originally wrote philschatz/octokit.js to interact with GitHub’s API using Promises. I started by forking an existing API michael/github and rewrote it using CoffeeScript and jQuery Promises. Then, thanks to contributions, they removed the dependence on Underscore and I removed the dependence on jQuery (and added support for several promise implementations).
But the code was getting too large as people incrementally added features so I thought it was time to rewrite with a focus on implementing as much of GitHub’s API concisely and consistently (with the hope of getting it adopted by GitHub officially).
The results are at philschatz/octokat.js.
This new library needed to:
- support ~100% of the GitHub API from the start
- have as little code as possible to maintain
- work in NodeJS and the browser
- have multiple source files
- have tons of unit tests
- support NodeJS callbacks and Promises
Let me quickly go through each of these and the challenges each presented.
Support ~100% of the GitHub API
This library abstracts all the authentication, status codes, caching, headers, HyperMedia (URL templates), and pagination returned from GitHub.
Abstracting Requests and Responses
Understanding how to form a request and how to parse the response is the bulk of the API; the logic is in
The other part is a chaining function that constructs a valid request.
Constructing Valid Requests
In the original philschatz/octokit.js there was a ton of copy/pasta dedicated to just constructing valid URLs.
Instead, this library has a regular expression that validates all URLs before calling GitHub and constructs objects dynamically through chaining (see
By reading the documentation at https://developer.github.com/v3/ a developer implicitly constructs a URL and then issues a request by calling one of the verb methods.
For example, to list all comments on an issue convert the documentation URL to the following:
GET /repos/:owner/:repo/issues/:number/comments .repos(owner, repo).issues(number).comments.fetch()
Here are 3 ways to list all comments on an issue:
octo = new Octokat() REPO = octo.repos('octokit/octokit.rb') # for brevity # Option 1: using callbacks REPO.issues(1).comments.fetch (err, comments) -> console.error(err) if err console.log(comments) unless err # Option 2: using Promises REPO.issues(1).comments.fetch() .then (comments) -> console.log(comments) # Option 3: using methods on the fetched Repository object REPO.fetch() .then (repo) -> # `repo` contains returned JSON and additional methods repo.issues(1).fetch() .then (issue) -> # `repo` contains returned JSON and additional methods issue.comments.fetch() .then (comments) -> console.log(comments)
There are several verb methods to choose from:
GETand yields an Object
GETand yields raw data
POSTand yields an Object
PATCHand yields an Object
DELETEand yields a boolean
PUTand yields a boolean
GETand yields a boolean
Pagination and HTML templates
GitHub returns headers for lists of results. These are automatically converted to
lastPage methods on the resulting JSON.
Paging through all the issues on a repository looks like this:
# Create an issues object (but does not fetch anything yet) ISSUES = octo.repos('octokit/octokit.rb').issues # Option 1: with callbacks ISSUES.fetch (err, issues) -> console.log(issues) issues.nextPage (err, moreIssues) -> console.log(moreIssues) # Option 2: with Promises ISSUES.fetch() .then (issues) -> console.log(issues) issues.nextPage() .then (moreIssues) -> console.log(moreIssues)
Maintain a Minimal Amount of Code
With this API you are prevented from constructing an invalid URL because every request is validated against the
URL_VALIDATOR regular expression before sending the request to GitHub.
Instead of writing largely copy/pasta code that constructs classes I opted for a regular expression that represents the entire GitHub API.
This can be found in
Works in NodeJS and the Browser
Getting this to work was a bit challenging.
NodeJS and AMD have slightly different syntaxes; enough to require adding in some boilerplate code to convert between the two.
Each of the source files contains 2 lines of boilerplate on the top of the file and the bottom.
Originally, I had the entire library in a single file.
I was able to have very little boilerplate. Just something like the following:
@define ?= (cb) -> cb((dep) -> require(dep.replace('cs!', ''))) define (require) -> foo = require 'foo' ... module?.exports = Octokat # For NodeJS window?.Octokat = Octokat # For browsers return Octokat # For browsers using AMD
This worked well but resulted in a single large file.
I then split up the library into multiple files and got it working but ran into a problem when trying to build everything into one file.
Finally, I opted for compiling the coffee files and concatenating them together with a custom
define method similar to what other libraries do (see the
require() method in less/less.js.
With this option, the tests run:
- on the source coffee files in NodeJS
- on the source coffee files in the browser
- on the built
dist/octokat.jsfile in the browser
Finally, multiple source files and tests running in all 3 environments!
Fixtures and Recording HTTP Requests
The library currently runs about 400 tests. There are about 120 unique tests, 80 are alternates using callbacks instead of promises, and 200 are the same tests but running in the browser.
In order to not pummel GitHub with duplicate requests I use linkedin/sepia to generate “cassettes” to replay the HTTP requests.
Unfortunately, sepia only runs on NodeJS so I wrote philschatz/sepia.js which plays back (and can record)
Support NodeJS Callbacks and Promises
In order to support both callbacks and promises, the asynchronous methods (called verb methods) all support a function as the final argument and return a Promise.
This way, you can always end each line with a callback or with a
.then() and the code just works.
The client only returns a Promise if one of the supported Promise libraries are detected (
Q, or native Promises).
If you use one of those libraries but still prefer to use callbacks just make sure the callback is the last argument and the code will just workTM.