It was a project last year, where yet another inline documentation syntax was “created” and suggested to be used. That triggered our brains and they started spinning caused by the unhappiness of reinventing the wheel, especially for something that is still too much of a step-child for frontend engineers and doesn’t get the necessary dedication anyway, the JavaScript inline docs. So let’s give them some love :-).
Fast forwarding … what came out of it is the API docs viewer which, we here at uxebu, built over the christmas holidays. It is completely client-based, for creating the documentation and retreiving all the information, no server-side code is needed anymore, it’s all done in dojo.
If you want to see a detailed introductional video, sit back, relax and enjoy the following screencast:
Inline doc styles
There are a couple of JavaScript inline documentation tools and styles out there, but no real standard has evolved yet. Maybe because of JavaScript’s dynamic nature it’s hard to fix it to just one style. Though this just makes the task more interesting, actually.
Let’s take a step back and take a look at just three of the various different inline doc styles.
The jQuery way, pretty much textual.
// (You can seed the arguments with an array of args, but this is
// only used internally.)
each: function( callback, args ) {
The YUI way, as you will see it’s very JavaDoc like, the method is documented above the actual definition and uses the @ sign for identifiers.
* Returns a string without any leading or trailing whitespace. If
* the input is not a string, the input will be returned untouched.
* @method trim
* @since 2.3.0
* @param s {string} the string to trim
* @return {string} the trimmed string
*/
trim: function(s){
The dojo way, with proprietary identifiers, inside the function code!
// summary:
// Trims whitespace from both sides of the string
// str: String
// String to be trimmed
// returns: String
// Returns the trimmed string
// description:
// This version of trim() was selected for inclusion into the base due ...
We see that there are different levels of depth and explicitness in the inline comments. But they all have in common, that important and interesting information are contained inside the source code, that should be exposed to the users of those APIs.
The __doc__ approach
In python, a certain type of comments is available as the docstring property doc of the function it is in. This was actually the initial idea, to provide the docs in this kind of way:
dojo.trim.<strong>doc</strong> = {
summary:"Trims whitespace from both sides of the string",
returns:{type:"String", summary:"Returns the trimmed string"}
}
It’s is pretty obvious, that this is is pretty ugly source code in the first place and secondly a comment is not a comment anymore but becomes source code. Prone to errors, and not really leading to higher productivity, we realized that this approach was actually just a learning stage.
Still it is pretty handy to use the command line in FireBug and have the docs at hand via autocompletion e.g. just typing dojo.trim.doc__ and the docs are there (this feature is currently also implemented, but the __doc property must be generated by a function call afterwards and it is not in the main focus for the current state of the project).
The less disruptive approach
Since the doc approach has the ugly side effect that library authors would have to change all their source code it was dismissed pretty quickly. New ways had to be explored. Driven by the flexibility of JavaScript and it’s dynamic nature we found a nice mix of reflection and minimal file parsing that allows us to extract the information from the existing raw (unpacked and not minified!) source code, without the need to modify it upfront.
First comes the reflection. Using the simple for-in loop over the prototype and the object itself allows us to extract all the methods and properties. Applying a little bit of logic using toString() and alikes and we know if the implementation of this object overrides, inherites or creates this method, we find out the default value of properties and a lot more. In the special case of dojo we can also easily find out the parent class(es). The reflection is actually worth a separate article, since it is quite powerful and allows deep insights into a class’s inner values.
Second step is simple file parsing. Since we know that object “foo” has the method “bar”, we know that in the source code we can find the definition by looking for the following patterns:
// OR
foo = {
bar: function(){}
}
// OR
foo = new function(){
this.bar = function(){}
}
There are a couple of special but rare other cases, but the patterns are very simple to detect as you can see. So our FileParser doesn’t really need to understand the JavaScript syntax, it is just looking for exact patterns and is therefore quite fast (we can surely still improve on the performance!).
Using regular expressions we also find out the parameters and the actual docstring itself, which then is parsed by the DocStringParser.
Let’s summarize. We have an object’s/class’s (if you want to call it a class) methods, properties, their according docstring and the docstrings of the class itself. We know the inheritance structure, can determine the type (public, private, etc.) from the name (for now we assume it is private when it starts with an “_”) or soon we will use the docstring identifier “tags” (which was just shortly introduced into dojo as you can see it used here). And we have example code, which is contained in a lot of docstrings. So this is already quite an amount of useful data. Let’s make them useable.
Plugging it all together
The pieces are there, they just need to be plugged together and hopefully a useful API docs viewer will arise. For this special dojo use case, we also had to build a mapper which enables the mapping of a namespace to the correct file(s). E.g. dojo.query() is actually a function inside the dojo object, which would naturally mean it is inside dojo.js, but in this case this method is in a separate file query.js which is “the knowledge” of the mapper.
Building the namespace tree was another challenge, loading all the files in the browser and inspecting the entire namespace via JavaScript would just be an aweful waste of resources. So currently there is a small python script which generates the JSON files which contain the contents of the tree to the left. But the high flexibility of using dynamic extending of objects (like e.g. NodeList-fx does) mixins and other things make it hard to rely purely on the generated contents of the tree, so they are partly hand-crafted too.
The UI is purely dojo, showing off some of the neat features, like the slick ExpandoPane. In my eyes it looks very nice, with a lot of attention to the detail. The UI was actually the part that needed the least time to build, thanks to Nikolai’s deep knowledge it was a piece of cake. There are of course a couple of glitches like the wrapping of the tabs, but hopefully this is just another kick in the right direction to speed up those kind of fixes. Because what is there is already much more than just the base of a GUI toolkit with some widgets, it is a full-fledged-ready-to-go-batteries-included-kick-ass set of UI components. (Ok, enough wallowing, yes I am biased :-).) So look for yourself and let us know what you think.
The future
As you know it too, working on those kind of projects always spins off a lot of new ideas. So we want to share some with you and maybe even get some help on some of them. First: we will put the source code out on google code (or alike) and it will be open for contributions, of course. We just need to do some renaming to ensure future-proof enhancements won’t just clobber this thing and make it unmaintainable.
Another idea is to configure the API docs viewer for custom namespaces and as long as the inline comments are dojo style it should be just a snap of a finger to get your own project’s API docs out. Well, actually the snap of a finger is the creation of the above mentioned JSON file, which contains the tree structure. But one day we will also have an auto-detector which builds the tree initially out of traversing through a given existing namespace (given that it is entirely loaded).
Idea No. 734 
was to provide the API viewer for other libraries too. The architecture is made this way, that the reflection and file parsing are separated and could be extended to work with another inline doc style syntax (we made first experiments with it). In order to do this right we think about having a common interface where to put and get the apidoc data from. A json-schema for API doc data is required. I have started on one, but this task really is huge when done right.
Of course the doc property will be provided. Currently when working with libraries or function that I don’t know I mostly use toString() on it and learn from the source code. I could also very well imagine myself using doc if it was there. So keep looking for that, I think it can change the way one uses the FireBug console and learns about libraries.
Since we know all the objects, methods on the one hand and the according docs on the other hand we can tell how good the doc coverage is. This heads into the direction of continuous integration and imho that is really where JavaScript still has a long way to go.
And there are a couple more ideas, like running the inline doctests in the UI, integrating the API viewer with Bespin and allowing to throw the api viewer onto any page using a bookmarklet, which would nicely let you browse the JavaScript API of any site, no matter if you want to learn, fix something or whatever. To mention just some of the ideas …
Now if you want to get your hands dirty and play with the code, see how it can integrate with your own code or just have offline dojo api documentation, drop us an email ( labs (a t) uxebu.com ) and we will make sure a copy of the code lands in your mailbox. Also, if you find bugs, have ideas, want to contribute, contact us and we will exchange ideas to make this tool the best available javascript api viewer out there.
Pingback: Get real with ES6 (and webpack) - uxebu | React...()