Hasen el Judy

Interesting hack to create html templates in javascript

Fri, 17 May 2013 02:41:00 -0600

Edit: various people pointed out that this trick is non-standard and not guaranteed to work across browsers, for instance, Firefox. So take this only as an interesting hack .. don’t use it in production

In a quest to have “reusable” knockout.js components, I asked on the mailing list if anyone has had any experience dealing with the problem of packaging the templates.

To distribute your HTML templates for reusability, they are usually written in html files and are meant to be included in your main html file using some server side technology, so for example if you’re using flask/jinja2 you’d use the include tag: {% include 'ko_widget.html' %}

This obviously is not ideal.

At first, I came up with a solution (that now I see is rather crappy) that revolves around a compiler script that crates a little javascript file that will recreate the template and insert it to the DOM using the document.body.insertAdjacentHTML method.

This is the core of it. Very simple. Takes html content, returns javascript:

def process_content(content):
    js_content = json.dumps(content)
    return "document.body.insertAdjacentHTML('beforeend', {0})".format(js_content)

But then someone showed me a different approach: a magic way to include html snippets in javascript. Actually it’s a hack to create multiline strings.

https://github.com/xdenser/kc-tree-habr/blob/master/tree.js#L106

It’s amazing what this can do:

function hereDoc(f) {
  return f.toString().
      replace(/^[^\/]+\/\*!?/, '').
      replace(/\*\/[^\/]+$/, '');
}

This function will extract the text content of block comments inside a function definition and return it as a string!

So like ….. you can create a multiline string like this:

var mlstring = hereDoc(function() {
    /*
        This is a multiline string ..
        Really .. it is!

        it also contains html:

            <a href="http://google.com">Google</a>
    */
    })

Printing the content of mlstring in the Chrome console:

> mlstring
"
            This is a multiline string ..
            Really .. it is!

            it also contains html:

                <a href="http://google.com">Google</a>
        "

Building on this

I created a function that takes a multiline string in this way and then wraps it in a script tag (with type set to text/tempalte), then sets a unique (random) id to this tag, then inserts it to the dom (or if the dom is not ready, waits for the dom to get ready then does the insertion) and return the generated id.

/**
    Takes a namespace and a function who's inside is a comment containing the content of an html template.

    Generates a randomized id (based on the name space, i.e 'template..random.digits'

    Returns the id

    Guarantees the templates gets inserted into the dom: either now or when it's ready (if the page is still loading).
 */
insertTemplate = function(namespace, f) {
    if(!namespace) { namespace = 'generic'; }
    var text = hereDoc(f);
    var random = (Math.random() * 9000).toFixed(0);
    var timestamp = (new Date()).getTime();
    var id = "template." + namespace + "." + random + "." + timestamp;

    var node = document.createElement("script");
    node.setAttribute("id", id);
    node.setAttribute("type", "text/template");
    node.innerHTML = text;

    var insertNode = function() {
        console.log("Inserting template");
        document.body.appendChild(node);
    }

    // insert node now or when body is loaded
    if(document.readyState == "loading") {
        document.onreadystatechange = function () {
            if (document.readyState == "interactive") {
                insertNode();
            }
        }
    } else {
        insertNode();
    }

    return id;
}

So now with this, it’s possible to include the html templates inside the javascript files. This should make it easier to package and distributes knockout.js based components.

Experimental "ajax" library

Sat, 13 Apr 2013 08:53:00 -0600

Lately I’ve been feeling like jQuery is becoming rather obsolete for me.

It used be a great library for dom manipulation, but for doing a real interactive UI, knockout.js is much better.

It also used to be great for abstracting away the different way to send asynchronous requests (aka ajax calls) in various browsers, but now that I don’t care about IE anymore, this abstraction is not needed all that much.

I’m not saying that jQuery is completely obsolete. I’m saying that I’d prefer if I don’t have to include it just for its ajax function.

If I need to do some neat UI effects, sure, I’ll use jQuery. But I’d rather not include it or depend on it if I don’t have to. If I only want a simple ajax library, I’d rather not include jQuery.

Actually even an ajax library is not really terribly necessary. If you target only modern browsers (Chrome and Firefox) you can use the XMLHttpRequest api directly, it’s simple and straight forward.

If one was to write an ajax library (now), it should be simple, very simple, designed to handle 90% of the use cases. There’s no need to add any complexity to make it more flexible; if you ever need more control over your ajax requests, just use XMLHttpRequest directly.

So, what would make for a nice-to-use ajax api? Well first of all it should not include the term “ajax” or “xml”, because it’s not about xml; it’s about asynchronous requests, and the format is json most of the time.

A few methods with names like “get” or “post” to create the request object:

requests.get("/data")
requests.post("/messages")

This would not send the request yet; just create the request object. Then more things about the request can be specified via methods that can be chained together.

.params(json) sets the arguments in the request url, for instance:

requests.get("/contacts").params( { "page": 10 } )

would set the request url to /contacts?page=10

.data(json) sets the json data to be sent (for PUT/POST/PATCH requests).

requests.post("/messages").data( { contact_id: 3423, text: "See you soon" });

.header(name, value) sets a request header

.success(fn) adds a success callback

.error(fn) adds an error callback

.complete(fn) adds a complete callback

.send() actually sends the request.

The success/error/complete callbacks will receive one parameter: the native “xhr” object, which has a “response” field that can be used to read the response (and a “responseType” field that can be used to check if the response is json or not).

However like I said I don’t like to call it “xhr”; I prefer to call it ‘areq’, as in “async request”.

Here’s what I have so far in a gist: https://gist.github.com/hasenj/5378660

setTimeout for python

Wed, 10 Apr 2013 01:46:00 -0600

In Javascript, there’s a function you can call to execute some function in the future:

setTimeout(fn, delayMillis)

This will call fn() after delayMillis milliseconds.

To get a similar functionality in python, we need first to install a package apscheduler, then we can implement a similar functionality like this:

from apscheduler.scheduler import Scheduler
import datetime as dt

sched = Scheduler()
sched.start()

def timeout(job_fn, *fn_args, **delta_args):
    """Like setTimeout in javascript; returns a job object

    First argument is the function to be called.

    Positional arguments will be passed to the function when it's called.

    Keyword arguemnts will be passed to datetime.timedelta

    Usage:
        # calls `fn()` after 3 seconds
        timeout(fn, seconds=3)

        # calls `fn(foo, bar)` after 10 seconds
        timeout(fn, foor, bar, seconds=10)
    """
    time = dt.datetime.now() + dt.timedelta(**delta_args)
    return sched.add_date_job(job_fn, time, fn_args)

Here, the function timeout works sort of like setTimeout, but the delay must be specified using keyword arguments; it accepts the same things that are accepted by datetime.timedelta

For instance:

def hello(): print "Hello"

timeout(hello, seconds=4)

The thing about this though, is you have to keep the program from quitting. Unlike node.js, the python script will quit when it reaches the end of the script, even if you have scheduled delayed functions.

If you are running a web server, you don’t need to do anything as the web server will continue running for ever (listening for incoming connections).

Otherwise, you need to somehow block or sleep.

The simplest way to do that is time.sleep

Simplest example:

Here’s a simple example.

def hello_spam(): 
    print "Hello"
    timeout(hello_spam, seconds=3)

hello_spam()

import time
time.sleep(10)

This will print “Hello” about four times and then die.

Additional arguments

You can also pass additional arguments to the function you want to call.

Example:

def hello_spam(name): 
    print "Hello {0}".format(name)
    timeout(hello_spam, name, seconds=1)

hello_spam("Dude")

Positional arguments to timeout are going to be passed to the function you’re delaying.

Notice that this means if you do this: timeout(fn, 2000) expecting that you’ll call fn after 2 seconds, you would be making a mistake. You’re telling timeout to call fn(2000) but you’re not telling when you want that call to happen. It will freak out and raise an exception.

Here’s the demo file on a gist: https://gist.github.com/hasenj/5352608

Back to SQL

Wed, 10 Apr 2013 00:55:00 -0600

For the past year or two I was trying to jump around from one nosql system to another. I was unsatisfied with traditional relational database systems, and looking for something more flexible. My main annoyance with sql databases was this:

The data format is rigid, and migrations are annoying.

Really. That’s it. That was my only complaint. At the time I didn’t realize it of course; I thought that “migrations are annoying” is a genuine reason to hate SQL.

Well, it kinda is, if there was an alternative system that has all the benefits/usefulness of SQL but without the annoyance of migrations. And that’s what I thought I would get from the nosql systems. But it turned out that each system has its own set of annoyances, and sometimes they are much worse than the migrations annoyance.

At the end of the day, it turns out that it’s much easier to develop for SQL databases, simply because:

They’ve been around longer, so they’re more mature, and there exists mature and full-featured ORMs/libraries for them.
They provide a flexible and powerful declarative querying language (the whole point of SQL: structured querying language).
Imposing a structure on your data is actually a good thing. It offloads some of the cognitive effort which you’d otherwise have to carry around in your head all the time.

While migrations are annoying, you just have to learn to deal with it and come up with a personal system for yourself to make it as routine and mindless as possible.

Adding web sockets to Flask apps

Mon, 17 Dec 2012 18:03:00 -0700

This is a walkthrough for integrating push sockets with a Flask app using gevent-socketio

I will try my best to make this tutorial short and sweet and to the point. No fluff. Only the bare minimum. I won’t bother explaining what everything means (partly because even I haven’t completely wrapped my head around all of it)

You will find a git repository with each commit documenting a step in this walkthrough here: https://bitbucket.org/hasenj/flask-sockets-tutorial

That repo is a very simple flask app: a page that receives push messages from the server and displays these messages as they arrive (in real time).

The server will listen for shouts on /shout?msg=, and broadcast each shout/msg to all connected clients.

Note: I’m not an expert on the subject. I wrote this because I wanted to do a simple thing (send a message over a socket from inside a flask request handler) but the information on the net was scattered and I wanted to just get to the core of it and get the smallest possible example working.

Note2: The example here only works with one server instance. If you run multiple instances of the server, it won’t really work as expected. For multiple-server setups, people usually use some kind of a message queue or a redis server to act as a central server that can notify all instances of the app server of what message to send over what socket.

With that cleared out of the way, let’s proceed.

1. Setup a basic Flask project

Commit: https://bitbucket.org/hasenj/flask-sockets-tutorial/commits/eb524632b66a608edbeb01032322b7f8

This is very simple and there isn’t much to it. If you’ve worked with virtualenv, Flask and jQuery before, everything here should make perfect sense.

Source the venv_init.sh file to initialize your virtual environment and install flask:

 source venv_init.sh

Then run the server with:

python main.py

The server is now running on http://localhost:6020

In main.js, we have a Javascript function addMessage that receives a string and adds it to the screen. This is the function that we will call when we receive messages through the push socket.

Try it! Open the Javascript console on your browser and run:

addMessage("Hello");

You should see the word “Hello” in big orange-ish color. Try it again, and the new message will appear on top of the “Hello”. The idea being, as the page receives messages, it will stack them on top of each other just like that.

2. Add socket.io to the client and server

Commit: https://bitbucket.org/hasenj/flask-sockets-tutorial/commits/188ea626622dff233452baeae0a09873

Socket.io is a library that make working with sockets easier.

I grabbed the Javascript client from here:

curl https://raw.github.com/LearnBoost/socket.io-client/0.9.11/dist/socket.io.min.js &gt; static/socket.io.js

For the server, we’ll use gevent-socketio. Install it via pip (while the virtual environment is active!) and don’t forget to add it requirements.txt

pip install gevent-socketio

We have to switch the server from the built-in flask server to a gevent server. gevent-socketio comes with the SocketIOServer function that helps glue things together.

app.debug = True
port = 6020
SocketIOServer(('', port), app, resource="socket.io").serve_forever()

However, this will lose the auto reloading. Luckily, werkzeug has a run_with_reloader decorator that can help us here:

@werkzeug.serving.run_with_reloader
def run_dev_server():
    app.debug = True
    port = 6020
    SocketIOServer(('', port), app, resource="socket.io").serve_forever()

So now our “main” entry just calls run_dev_server:

if __name__ == "__main__":
    run_dev_server()

Also, and this is very important, we have to call gevent.monkey.patch_all()

from gevent import monkey
# ..
monkey.patch_all()

I placed it at the top.

3. Setting up the socket endpoint on the server

Commit: https://bitbucket.org/hasenj/flask-sockets-tutorial/commits/192eb79c5ba0e3ead19a59075c38e18b2145e57d

When a client connects to a socket.io enabled server, it will always go to /socket.io/. Even though you may never explicitly visit this url on the client side, this is where it goes when you connect to the server using socket.io.

@app.route('/socket.io/<path:rest>')
def push_stream(rest):
    try:
        socketio_manage(request.environ, {'/shouts': ShoutsNamespace}, request)
    except:
        app.logger.error("Exception while handling socketio connection",
                     exc_info=True)

Please note that we ignore the rest parameter. It’s something that socket.io takes care of.

The function socketio_manage takes care of setting up the socket for that connection. The second argument maps “urls” to “namespace classes”.

You can think of the url as a channel, and the namespace class manages sockets in that channel.

Read the docs: https://gevent-socketio.readthedocs.org/en/latest/namespace.html

The important bit in the following code is that we’re adding each new socket to a list that can be accessed globally (as a class variable) and we add a class method that can broadcast a message to all connected sockets:

class ShoutsNamespace(BaseNamespace):
    sockets = {}
    def recv_connect(self):
        print "Got a socket connection" # debug
        self.sockets[id(self)] = self
    def disconnect(self, *args, **kwargs):
        print "Got a socket disconnection" # debug
        if id(self) in self.sockets:
            del self.sockets[id(self)]
        super(ShoutsNamespace, self).disconnect(*args, **kwargs)
    # broadcast to all sockets on this channel!
    @classmethod
    def broadcast(self, event, message):
        for ws in self.sockets.values():
            ws.emit(event, message)

The methods recv_connect and disconnect are called by the socketio framework. We’re overriding them to hook into events so that when a new client is connected, we add its socket to the list, and when it disconnects, we remove its socket from the list.

Note: the method disconnect does some necessary work behind the curtains, so it’s important to call the same method on the superclass!

Now we have things setup so that when clients connect to a socket on “/shouts”, we can broadcast something to all of them by calling ShoutsNamespace.broadcast

4. Connect the client to the server

Commit: https://bitbucket.org/hasenj/flask-sockets-tutorial/commits/d158b2214120bcfcef41e038e39eeeaeca053771

The client side code is probably the simplest part in all of this. Simply call io.connect('/shouts'); to get a socket object, then write some event handlers on the socket.

$(document).ready(function() {
    var socket = io.connect('/shouts');
    socket.on('connect', function() {
        console.log("socket connected");
    });
    socket.on('disconnect', function() {
        console.log("socket disconnected");
    });

    socket.on('message', function(data) {
        console.log("Got message:", data);
        addMessage(data);
    });
});

The important thing here is the ‘message’ event handler: it calls our function addMessage.

In this case, we’re expecting ‘data’ (the thing sent over the socket) to be a string. This is not a limitation! It can be a complex nested JSON object. I’m just using strings here to make things simple.

5. Broadcast messages from the server to the client

Commit: https://bitbucket.org/hasenj/flask-sockets-tutorial/commits/9e2e854a46e1a8bc233dc5e879795db40ef00550

We’ll create the shout endpoint:

@app.route("/shout", methods=["GET"])
def say():
    message = request.args.get('msg', None)
    if message:
        ShoutsNamespace.broadcast('message', message)
        return Response("Message shouted!")
    else:
        return Response("Please specify your message in the 'msg' parameter")

This allows anyone to shout a message via the /shouts end point.

Try it yourself! Open http://localhost:6020/shout?msg=PushIt in a new window and watch the main page get updated instantaneously! Open several pages and then shout a message again, and behold as they all updated in real time!

Summary

In this tutorial we did the following:

Change the flask app setup to use the SocktIOServer
Create a simple socketio namespace class that can broadcast a message.
Connect to the socket from the client side and receive push events.

Notes

We barely touched the surface! There’s a lot more! For instance gevent-socketio has mixins that can help with broadcasting.
We used socket.io (a framework) instead of dealing with sockets directly. Socket.io handles events and namespaces, and it also handles automatically reconnecting to the socket server if it gets disconnected.
It’s all self contained, and we’re using only python! No redis, no message queues server, no node.js! Just python! (talking about the server side here).
Originally, I tried to make this work with pypy, but unfortunately gevent is not compatible with pypy.
This setup seems to lose the werkzeug debugger. I tried to re-enable it using wekzeug’s DebuggedApplication class, but it seemed to breaks sockets. I haven’t figured out why exactly.

Knockout.js vs Backbone.js

Mon, 12 Nov 2012 14:48:00 -0700

What the hell am I talking about? Backbone? Knockout?

These are 2 popular javascript frameworks, supposedly designed to help/facilitate the building of rich client application on the web.

To me, this means web apps with non-trivial interactive UI, which would be difficult to manage with just jQuery.

What’s wrong with jQuery, you say? Well, jQuery is awesome, but it’s too low-level as far as app-development is concerned. How are you going to manage it when the complexity increases? It gets hairy very quickly.

If you’ve been developing mostly in jQuery, and you’re getting a bit tired of it, you should be looking for alternatives; higher level frameworks/libraries to help you manage the complexity. However, you may find the information on the internet to be very confusing. What should you use? Which framework/library is better? You might have heard of Backbone.js or Knockout.js, and you don’t know which one you should go with. And to make things worse, Backbone.js and Knockout.js are not the only games in town; there are plenty others as well.

So, I’m going to spare you some of the trouble by telling you straight away which one is better: Knockout.js or Backbone.js

Some people will disagree, but they are ugly and stupid.

Knockout.js is awesome

Backbone.js is pointless

Backbone.js doesn’t solve any problem I’ve ever had. When I look at what Backbone provides, I can’t help but wonder: what’s the point? None of it made me think “Ooooh I wish I had that a long time ago!!”. No, quite the opposite, most of the time I was thinking “Why would I need this?” and “How would this help me?”.

As far as I’m concerned, the only problem that Backbone.js solves is the buzzword compliance problem. “Building a complex client app? You should use MVC”. Good for you; you may now say on your resume “I have experience with MVC”.

I even asked this question on StackOverflow, and the answer was “it helps do MVC”.

Knockout.js actually solves a real problem; a real pain point for anyone who’s ever done front-end work. When you learn how to use Knockout.js … when you go through the tutorials, you’ll constantly feel like “Wow man, this is awesome! Why didn’t I have this before?”

Knockout.js (hereafter referred to as KO) automates what amounts to 70% of Javascript development work, which is listening to events and binding callbacks so that you can do things like:

Update variables/data when user changes form values
Hide or show elements of the page when something happens (or: add/remove elements, instead of simply hiding/showing them)
Change the entire “page” to a different component (a special case of the previous point)
Displaying “items” and “objects” as a complex HTML structure (aka widgets).
Displaying (and updating the display of) a list of items/objects.

KO automates all that manual work. It’s a high level framework so you almost never have to deal with “events” directly; it’s abstracted away so you don’t have to think about the low level aspects of it.

KO also does a great job of really binding the data to the UI. You almost never have to assign a special “class” or “id” for the purpose of manipulating elements and their content from Javascript, as often happens in jQuery-based development.

Tiny disclaimer

To be slightly fair, Knockout.js and Backbone.js have very different purposes.

Backbone.js is not really pointless; it serves as a pretty good ORM for REST APIs.

However, in this article I’m evaluating its usefulness for building interactive UIs on the web.

You may say that this is not really what it’s designed for, but in practice, that’s what most people think Backbone.js is about: helping you build single-page interactive web applications.

That’s what most people end up using it for.

A Closer Look

So far I’ve just been trolling basically. Let me backup my opinion by explaining the reasoning behind it.

Backbone.js

Backbone.js (hereafter referred to as BB) provides you with “Models”, “Collections”, and “Views”.

Models are object wrappers that fire events when any field in the object gets changed. For example, you have a user model. In one corner of your application, you can say:

user.set("name", new_name);

And, in another corner, you can say:

user.on("change:name", function() {
    // do something here
    // yes, this is an event handler function/callback

    // example of what typically goes in here:
    $el.find(".name").text(user.get("name"))
});

In essence, you are still dealing with events, and writing event handlers, explicitly. And what’s worse; inside the event handler, you’ll be manipulating the DOM using jQuery (or a similar low-level API).

BB has a “views” system, but it’s not really much of a system; it’s more of a convention. Each view has a model and an element (a DOM element). Convention has it that the view should have a render() function, which uses the model data to render HTML into the element. Convention also states that render() should return the view.

ViewName = Backbone.View.extend({
    // ...
    render: function() {
        var $el = $(this.el);

        // initialize the element's content by some template
        $el.html(loadSomeTemplate("by_name"));

        // fill various template blocks, or something like that ...
        $el.find(".name").html(this.model.get("name"));

        return this; // if you forget this, dragons might hunt you
    },
    // ...
});

It’s typically inside the view definition that you bind to your model’s “change” events, and that’s basically how you build the interactivity in your application.

And, at the end of the day, it all comes down to the same old low level jQuery mess. You end up adding special classes and ids to your DOM elements, so you can access them easily (from your code) for the purpose of manipulating them. You write a lot of code to listen for events and do stuff.

One of the pain points I remember having with BB (I’m sure it’s been experienced by a lot of other people as well) is the management of views within views, aka nested view, or sub-views. Particularly annoying was the management of views that are meant to display a “Collection” (list) of items. The view itself contains a list of subviews, matching the items in the collection. You end up having to manually write the code to update the view each time the collection changes.

Handling updates to lists/collections can be very complicated, and hard to do efficiently, so most people (including me) end up writing a very generic render function that simply wipes out all the previous subviews and builds new ones. This means, if you have a collection of a 100 items, and you add one new item, you will destroy a 100 DOM elements (each which contain some number of child elements), destroy a 100 view objects, clear out all the bindings between these view objects and the collection items (models), create 101 new view elements, creating a DOM element for each of those, and attaching it to the page (body) DOM.

Naturally, it would be much more efficient to simply add 1 new view and append it to the list, and append its DOM element to the list view. But, like I mentioned above, managing all the various ways in which a list can be updated is quite tricky, so we end up with this solution that just “works”. I’m sure plenty of people had to go through the same trouble, over and over. And most likely, no one bothers to tweak the list-view rendering algorithm to make it smarter, because it’s just not worth it.

BB also suffers from many other complications.

In short, when it comes to building rich front-ends on the web, I found BB to not be effective at all.

However, to be fair, (as I’ve already mentioned), BB is very good at being an ORM for REST APIs.

Yes, perhaps I’m evaluating BB against standards which it was not really intended for: managing the complexities involved in building rich interactive UI. But, as I said, that’s what most people actually use it for, and I find that it’s not really useful in that department at all.

Backbone.js does come with a pretty good url router component, but I think of this as more of a “batteries included” kind of thing. There are other libraries that provide similar functionality, for instance, page.js.

Knockout.js

KO provides a sort of declarative language that allows you to bind DOM elements to your data.

On any DOM element, you can say: this elements takes its value from this variable, or, this element is only displayed when some value is “truthy”, and so on.

KO calls these things “bindings”. A “value” binding on a text field keeps the content of the field in sync with some JS variable. A “visible” binding controls the visibility of an element based on the truthiness of some expression. There are other sorts of bindings, and you can define your own, too.

To have the DOM update automatically when data changes, there must be a way to “listen” to these changes, and the solution KO provides you with is “observables”. An observable holds a value and wraps it up in an object so that you can update that value and have the rest of application see that you made a change.

name = ko.observable(initial_value);

// ....
// sometime later
name(new_value) // change the value

// ...
// to read the value:
val = name()

When you update the value of ‘name’, the rest of the application will automatically adjust. If an element is bound to display the content of ‘name’, it will update itself. If an element is bound to be visible only when name is not empty, it will recalculate its visibility .. and so on.

KO handles all the event bindings for you; you never have to think about it.

KO also provides another very powerful concept: “computed” values. A “computed” object wraps a function (in the functional-programming sense; i.e. it should have no side-effects) and automatically tracks its dependencies.

What the hell does that mean? Let’s illustrate with a simple example. Let’s say that “Full Name” is simply firstName + " " + lastName. If we would write that as a function:

 var fullName = function() {
      return firstName + " " + lastName;
 }

If you were writing low level Javascript, you’d have events that listen to when the user changes the firstName and lastName fields, and then you call this fullName() function, take the result, and display it inside some div, which you probably put an ‘id’ on so you can access: $("#fullName").html(fullName())

With KO’s “computed” concept, you define fullName like this:

 var fullName = ko.computed(function() {
      return firstName() + " " + lastName();
 });

As far as you care, KO magically figures out the dependency graph for this “computed”, and so every time firstName changes, fullName also gets updated.

You can then bind the text value of some div to fullName, and bind the value of some text field to firstName, and as the user edits the name in the text field, the div holding the full name will automatically be updated, without you having to do anything extra.

And, as I’ve mentioned before, ‘text’ and ‘value’ are only 2 of the basic bindings that KO provides, and you can also define your own bindings.

Naturally, inside the binding’s definition, you will be doing low-level DOM manipulation using jQuery, but once you have that binding written; you can use it everywhere.

KO comes with a built-in ‘foreach’ binding that iterates over a list and displays the items inside it. As you can expect, the first version of this binding probably wasn’t very smart; it probably just re-rendered the entire element every time an item got changed. But, because it’s a built-in binding that’s used by a lot of people, it can be improved, and as far as I can tell, it’s now quite intelligent; it can find precisely which items have changed, and update only the items that need updating. The official docs say:

> (…) The binding will efficiently update the UI to match - inserting or removing more copies of the markup, or re-ordering existing DOM elements, without affecting any other DOM elements. This is far faster than regenerating the entire foreach output after each array change.

The only downside with KO is that the binding syntax is somewhat obtrusive.

This is more or less the first example from the tutorials:

<p>First name: <strong></strong></p>
<p>Last name: <strong></strong></p>
<p>Full Name: <strong></strong></p>


// Javascript
function AppViewModel() {
    var self = this;
    self.firstName = ko.observable("Bert");
    self.lastName = ko.observable("Bertington");
    self.fullName = ko.computed(function() {
      return self.firstName() + " " + self.lastName();
 });
}

rootVM = new AppViewModel();
// Activates knockout.js
ko.applyBindings(rootVM);

Basically, you define bindings on elements using the attribute “data-bind”, in it, you specify a binding and an expression that goes as an input to that binding. We have here “text” bindings, binding the elements to the “firstName”, “lastName”, and “fullName”.

Then we have the “ViewModel”. Its essentially an object with fields. “firstName” is a field in the “root” ViewModel. Since it’s an observable, you can update it, by running this in the console for example rootVM.firstName("Jack"), and the page will immediately reflect that change. Not only will the “First Name” element update itself, but the “Full Name” will also reflect that change and update itself.

KO very nicely un-entangles your messy jQuery code.

That’s why it’s awesome.

Here’s the example on jsFiddle: http://jsfiddle.net/FEVbz/

I’ve extended it to also include text fields so you can change values in real time.

I strongly encourage you to use the interactive KO tutorial; you will be amazed: http://learn.knockoutjs.com/

Buzzwords

What is that you say? Knockout.js is not MVC? Well, if you care about buzzwords, here’s one for you: MVVM. As far as I’m concerned, MVVM is gibberish, and so is MVC.

Buzzwords don’t matter. Terms like “MVC” are merely a condensed description of a solution to a problem that developers face in real life.

If you give me a framework that doesn’t solve my problem, I don’t care what buzzword you attach to it.

If you give me a framework that solves my problem beautifully and elegantly, then I’m happy with it, even if it doesn’t fill any buzzword compliance requirement.

Obligatory “web-scale” video.

Coroutines vs explicitly async APIs

Wed, 12 Sep 2012 06:41:00 -0600

I’ve expressed before (rather distastefully) an opinion that Node.js is not a good choice for developing web applications, because its async APIs will usually turn your code to a callback hell.

Many people responded negatively, perhaps rightly so. But on the other hand, I think some of them didn’t actually understand why the explicitly async APIs of Node.js are not a good way to handle the “blocking I/O” problem; or rather, these people didn’t understand that gevent’s coroutines and Go’s goroutines provide a much cleaner solution. They seem to think that if it’s not callback hell, it won’t scale. I want to show that they’re wrong.

I’m going to try to explain the difference between an API that’s explicitly asynchronous and one that appears “normal” but is ‘asynchronous’ under the hood.

Why asynchronous IO?

The short answer is that OS threads don’t scale. Meaning, if you spawn a new thread per request, your server will not scale to tens of thousands of simultaneous requests.

I don’t know much about the details of this, but it appears that OS level threads are expensive (relatively .. and only after you’ve spawned thousands of them).

Request handlers typically do some sort of a blocking I/O operation, such as reading a file, or making a network connection (to the database, for instance). To handle more connections, you have to do it in parallel, which is usually done by spawning new threads, but we just mentione d that they don’t “scale”.

So for scalability, there’s a need for a different mechanism of handling large amounts of concurrent connections. A mechanism that does not involve spawning a kernel-level thread per request.

One such method is using asynchronous I/O, because it’s non-blocking. What you basically do is tell the system: read data from here, and when it arrives, call this function; meanwhile, I will keep working on other stuff. Because I/O operations don’t block, you don’t have to spawn a thread for every connection.

This is basically what Node.js does. This is how it can scale to heavy loads while while doing everything in one thread.

A typical request handling function merely reads data from files/databases and does simple computations that don’t take more than a few nanoseconds on modern hardware.

Let’s suppose we have a request handler that takes the userid and fetches some “items” that belong to this user so they can be displayed to him.

In a typical synchronous IO style, it might look something like this:

def serve_items(request):
    user_id = grab_user_id(request) # simple function
    items = fetch_user_items_from_db(user_id) # blocking IO - blocks current process or thread
    items = sort_items_by_score(items) # super fast operation
    return json_response(items) # respond with the items in json format

That would be the synchronous version of this request handler.

In Node.js, it would look more like this:

var serve_items = function(request, response) {
    var user_id = grab_user_id(request); // simple function
    var on_items_ready = function(items) {
        items = sort_items_by_score(items); // simple computation
        response.send_json(items); // respond with the items in json format
    };
    fetch_user_items_from_db(user_id, on_items_ready);
};

The way you write the code is different.

Now, if you pay attention, you will find that this function returns almost immediately! The function doesn’t wait for the data fetch to complete before returning, it returns as soon as it fires the fetch request. The actual sorting of the items and sending the response happen later - after the data has arrived.

The disadvantage here is the way you write the code:

More cognitive effort is required to express the control flow in a soup of callback handlers.
The resulting code is harder to read.

I’d argue that from 1 and 2, we can conclude that the resulting code is harder to maintain, because maintenance requires both reading and writing.

I’m not going to delve into more detailed examples. There is a lot of material on the net. Just Google it.

http://bjouhier.wordpress.com/2011/01/09/asynchronous-javascript-the-tale-of-harry/

There are some solutions to this within the Node.js ecosystem. Here are some examples:

http://alexeypetrushin.github.io/synchronize/docs/index.html

https://github.com/Sage/streamlinejs

But now I’d like to take a look at another approach to the issue.

Goroutines

What makes blocking code “bad” or “undesirable” is that process and threads are expensive.

Why are they expensive? Can’t we make them cheap? If we can’t make them cheap, can’t we come up with other concurrency constructs that are cheap?

Well, it turns out we actually can. This is exactly what goroutines are: cheap concurrency constructs.

When a goroutine blocks on IO, it’s not a problem, because it’s very cheap to create new goroutines. You can create and teardown hundreds of thousands of goroutines in under a second; that’s how cheap they are. So “blocking” on I/O does not cause any scalability issues.

So in Go, you can write code in the usual synchronous style, and it will scale. It just works!

Actually goroutines still do asynchrunous I/O; but it all happens under the cover. The APIs of the language are not explicitly asynchronous. The APIs are usual, synchrunous.

Coroutines and Continuations

Goroutines are based on the concept of coroutines. The term “coroutine” comes from “cooperative routines”. We call them “cooperative” because they don’t get scheduled by the OS or any other scheduler, they just run for a while then “yield”. When a coroutine yields, other coroutines get to run. When a coroutine gets control back, it continues running from where it left off last time.

This tutorial gives a nice overview of coroutines in lua.

The concept of suspending and resuming a coroutine is closely related to the concept of continuations.

When a coroutine suspends, we can think of this as a snapshot being taken of the coroutine’s execution context. This snapshot can then be invoked at a later time and it will resume the execution of the coroutine.

Let’s look at a simple example of a yielding operation:

print("hello")
sleep(10) // milliseconds
print("world")

The call to sleep basically says: capture the state of the current coroutine (i.e.: save the current continuation), and don’t resume it before 10 milliseconds pass. After the sleep time passes (10 milliseconds, in this case), the coroutine may resume whenever it’s given the chance.

The same can happen with I/O operations that usually block. Depending on the implementation, coroutines may “yield” execution when they make, for example, an http request, and don’t resume execution until the response comes back. So, while the request is waiting to return, other coroutines can continue to run (inside the same process).

If you’re familiar with javascript, the sleep example might remind you of something like this:

console.log("hello")
setTimeout(10, function() {
    console.log("world")
});

Basically, in javascript, you have to manually create something similar to a continuation, but instead of a continuation, it’s a closure, and you pass it to the “sleep” function to say: here, forget about me, I’ll be gone now, but when you return, do this thing that I’m telling you about right here.

Instead of capturing the state of the current execution context, you terminate the current execution context and set the stage for a new execution context to be created in the future.

You may think continuations are conceptually the same thing as closures, but they are not. A continuation includes the call stack, so when a continuation is resuming, it may hit a “return” statement, and when it does that, the function returns the result to its caller.

Consider this:

# python syntax, but don’t assume that this is python per se ..

def logError(errorCode):
    message = getExplanation(errorCode)
    time = getTime()
    print(time + ": error " + code + ": " + message)

def getExplanation(code):
    resp_text = http_get_request("http://api.service.example/errors/" + code) # this line dispatches the request and yields execution
    resp_json = json.parse(resp_text)
    return resp_json.human_readable_explanation

Now, in this example, http_get_request is assumed to be a function that makes an http request, and just like sleep in the above example, it will capture the current continuation and yield execution to other coroutines, and when the request comes back, the captured continuation maybe invoked, to resume the execution of the coroutine.

Now, when this continuation is resumed, the entire execution context is resumed, including the fact that getExplanation was called from inside logError and that logError is waiting for getExplanation to return a string.

So, when the http response comes back, and the continuation is resumed, and we hit the line return resp_json.human_readable_explanation, we actually take this value and return it, and then continue execution at the line message = getExplanation(errorCode) inside the caller .. and so on.

You can’t do this with closures.

If the getExplanation function is to be written in the javascript callback style, it can’t really “return” anything.

Here, let’s try to make it return something. It will be absurd - and it won’t work as expected - but let’s do it just for fun:

var logError = function(errorCode) {
    var message = getExplanation(errorCode);
    var time = getTime();
    console.log(time + ": error " + code + ": " + message);
}

var getExplanation = function(code) {
    http_get_request("http://api.service.example/errors/" + code, function(resp_text) {
            var resp_json = json.parse(resp_text);
            return resp_json.human_readable_explanation; // absurd!!!
    });
    // getExplanation will infact return here, and it will return null
}

To help see the absurdity of this return attempt more clearly, let’s rewrite the async http request:

var getExplanation = function(code) {
    var url = "http://api.service.example/errors/" + code;
    var callback = function(resp_text) {
        var resp_json = json.parse(resp_text);
        return resp_json.human_readable_explanation; //absurdity!!!
    };

    http_get_request(url, callback);
    // getExplanation will infact return here, and it will return null
}

There’s no way to tell the callback “oh btw, return to logError, please!”. No! You can’t do that. It’s a brand new execution context; all information about the call stack is completely lost.

So, what you end up having to do instead, is pass a callback to getExplanation itself!

// notice the "caller_callback"
var getExplanation = function(code, caller_callback) {
    var url = "http://api.service.example/errors/" + code;

    var callback = function(resp_text) {
        var resp_json = json.parse(resp_text)
        var result = resp_json.human_readable_explanation;
        caller_callback(result);
    };

    http_get_request(url, callback);
    // getExplanation will infact return here, and it will return null
}

// and rewrite the `logError` function too
var logError = function(errorCode) {
    var callback = function(message) {
        var time = getTime();
        console.log(time + ": error " + code + ": " + message)
    };

    getExplanation(errorCode, callback)
}

So, I hope that I have illustrated why closures are not equivalent to continuations, and are in fact inferior to them.

Note that when I say a “continuation”, I’m talking about a concept; not any particular implementation detail. When a OS process is suspended and then resumed, we may also think of this as a continuation being resumed. When a thread is suspended and resumed, we may also think of this as a continuation being resumed.

The key aspect of coroutines is that they are much cheaper than threads or processes, and are not managed by the kernel.

The gevent library for python does just what I described about coroutines and continuations. It wraps an asynchronous I/O with a coroutine yielding mechanism, thus providing a synchronous API on top of an asynchronous API.

The obvious conclusion:

With an explicitly asynchronous API, such as the one that Node.js gives you, you must manually construct “callback” functions that emulate continuations. You get the benefit of asynchronous IO but lose all the benefits of synchronous IO APIs.

With things like gevent and goroutines, you get all the benefits of asynchronous IO while maintaining all the benefits of the tranditional synchronous IO APIs.

Another way to look at it is related to closures and continuations.

With Node.js, you have to create closures, and call the async IO APIs with these closures as the thing to execute when the response arrives.

With coroutines, the language/compilers does it for you such that when the response comes back, the current continuation is resumed.

Notice that what the compiler generates for you (continuations) is much more powerful than what you have to write by hand (closures).

Footnote:

Now, if only it was possible to specify the callbacks as continuations … it might then be possible to easily wrap async IO functions and make them appear synchronous

I’m imagning something like this:

var wrap_async_api = function(async_fn) {
    var sync_version = function() {
        return call_with_current_continuation as cc {
            var callback = function(data) {
                resume_continuation cc {
                    return data
                }
            }
            var args = Array.prototype.slice.call(arguments);
            args.push(callback)
            async_fn.call(this, args);
        }
    }
    return sync_version;
}

Not sure if this is a reasonable way to imagine using continuations to provide a synchronous API on top of an async one. I haven’t actually used call/cc in any real program.

For a good understanding of scheme’s continuations and call/cc, I recommend this article:

http://community.schemewiki.org/?call-with-current-continuation

It’s really good; read it slowly and carefully.

Footnote #2:

It appears that there are continuation-based solutions for node.js:

https://github.com/laverdet/node-fibers

Gevent? Coroutines for Flask?

Sun, 17 Jun 2012 17:36:35 -0600

Yesterday I was reading random stuff on the internet, one thing led to another, and I discovered there’s something called “greenlets” in python, which seem to be essentially coroutines of some sort.

The Flask docs mention that you can deploy flask apps using gevent:

http://flask.pocoo.org/docs/deploying/wsgi-standalone/#gevent

Gevent

Gevent is a coroutine-based Python networking library that uses greenlet to provide a high-level synchronous API on top of libevent event loop

It seems to monkey-patch existing synchronous apis to make them work with greenlets/coroutines, or something to that extent.

I haven’t actually figured out what’s going on, but this seems to be interesting.

A short introduction to call-with-current-continuation

Thu, 20 Oct 2011 23:58:09 -0600

A short introduction to call-with-current-continuation:

Learning about lisp/scheme continuations .. wow .. what a mind twisting concept!

And there I thought lisp lacked structured exit points like “break” and “continue” and “return” ..

This is just brilliant .. I’m so humbled.

When I finally understood how the coroutine implementation works on top of call/cc (it wasn’t obvious how that worked), I felt a sense of awe at the beauty and magnificence. I almost want to cry :”]

Sketching helped me avoid getting stuck

Mon, 17 Oct 2011 19:39:00 -0600

For the past couple months, I’ve been feeling stuck. Every time I sit down and try to get something done, I can’t. I find myself “dreading” the thought of going through all the troubles involved (working out some minor low level details, etc). So instead of doing work, I’d procrastinate on youtube or facebook or what have you.

I tried to pin down why I was having this mental block that prevented me from doing work. I thought: maybe I’m worrying about the code quality. I know that my code won’t be perfect and elegant, and that disturbed me. Maybe that was one of the reasons. Maybe I was worried about my “reputation”; it’s not that I have any reputation online (I’m pretty much a nobody), but when potential employers are reviewing my work (say, on github), they may think I’m a messy programmer when they see my messy code.

So, I made a mental note (I also wrote down these thoughts on a file) of these worries and how they could be stopping me, but that didn’t quite get rid of my problems. I still felt “stuck”. I still couldn’t get much done.

And then one day, I decided to do some “doodling” (after I read an article on HN about how doodling helps your brain when you’re bored). I doodled random non-sense for about a minute, and then I found myself sketching a UI for an iPad app that’s been in my head for a while now. I didn’t just draw some boxes; I also wrote down some notes about how various elements should behave. Then after that, I found myself sketching out another version of the same app but for the web. I didn’t only sketch it, but I also sketched out the basic structure of the code behind it. I basically wrote a high level view of the code that would make it run. Then I worked out some of the mathematical details. All that on paper. All these ideas have been in my head for a while, but I wasn’t able to express them as code; I was stuck; I had a mental block. But now, I expressed them on paper. This got me excited, so I went to my laptop right away and started translating this stuff into code.

This time, I didn’t have anything to worry about. I didn’t have any psychological worry about how things will turn out. I was basically code-monkeying my sketch. I wasn’t thinking while I was coding, I had already figured things out on paper, so my coding consisted of me looking at the sketch and writing it down in the programming language I was doing my app in. I was plagiarizing from my sketch.

There were a few details which I didn’t get right on paper, but they were easy to fix. And in very little time, I got a working prototype of my idea.

And then I realized that this is the way to go. And I also realized what exactly was happening, and why sketching is the solution.

I’m an INTP, and I think this applies to xNTP types. I don’t know if it applies the same to xNTJs.

What happens when I have lots of ideas that I want to express in code is the following: I have a rough/vague picture in my head representing the structure of my program. As soon as I start writing code, I have to concentrate on the details of the code I’m writing now, and this means that there’s no more room in my mind for that general “picture” I just mentioned. This is a problem because: I don’t want to lose this picture in my head; if I lose it, I won’t get it back easily. It’s basically the same thing when you are in flow and then suddenly you’re interrupted. You need a lot of time to build the house of cards, and you can’t afford to let it fall apart; it’s mentally devastating.

This is where sketching comes in. When you sketch the idea on paper, you have the entire picture in your head, and you’re translating it on paper. Because it’s just paper, you don’t have to worry about details; you’re not writing code; you’re just sketching; there is no need for your mind to do any context switching. After you layout the general structure, you can then dive into some corners and work out some of the details in a very rough/general manner. Again, because it’s on paper, you get to control how deep you want to dive in, and because the structure is already written on paper, you don’t have to rebuild it in your head; you just look at it again and delve into the next detail (if needed).

Then you can translate this into code, and you will not have the problem of context switching. Why? Because you’re not worried about preserving the “mental picture” of the general structure of your idea. This way, you don’t have to think a lot while you’re coding, you’ve already done the thinking and produced this sketch. Now you can just follow your own sketch, like a code-monkey.

Why is this related to psychological type? Well, It’s related to how you organize and express your thoughts. In the classical MBTI/Jung theory, INTPs organize logical structure internally (introverted thinking) and express ideas and intuitions externally (extroverted intuition).

When thinking, you get alot of ideas, and you want to express all of them. The problem is, if you try to express them in code directly, you can’t, because it will take quite some time to express each idea, and working on one idea at a time kills you, because you have all these other ideas that also want to be expressed, at the same time! And you know that if you start detailing some idea before the other ideas got a chance to be expressed, you’ll lose them. So, by righting down a sketch of these ideas on paper, you can express all of them in a rough basic form. This way, all of your ideas get to be expressed at the “same time”, and this clears up “cognitive space” in your brain to start working on the details of each idea.

Like I said, maybe this doesn’t apply to INTJs, who constitute about half the programmers/hackers out there, but I’d be interested to know if any INTJ tries this and finds it helpful.

This sketching business is not entirely new to me. I used to do it a lot in the past. In fact, now that I think about it, most of my non-trivial programs in the past were sketched out on paper first before I started coding them, otherwise they probably wouldn’t have been possible. I just didn’t realize how important/pivotal it was. I don’t remember how/when did I stop doing it. Maybe somehow I got the impression that real hackers just sit down and churn out code.

SuperAgent: ajax library without the suck (by visionmedia)

Wed, 05 Oct 2011 07:05:35 -0600

SuperAgent: ajax library without the suck (by visionmedia)

LiveReload: automatically reload browser page when editing local files

Fri, 02 Sep 2011 05:03:54 -0600

LiveReload: automatically reload browser page when editing local files:

The official site doesn’t make it clear, but it works on linux too

Fix for when mongodb won't start on Ubuntu

Wed, 17 Aug 2011 14:04:14 -0600

Fix for when mongodb won't start on Ubuntu:

Kinda lame ..

A different kind of writing platform

Thu, 07 Jul 2011 01:28:00 -0600

I’ve been thinking for a long time about building a “writing” platform. But when I try to put it into simple words, it sounds like “oh, yet another blog”. In fact I’ve even written a post before titled “yet another blog engine”. It’s been probably over a year, and the thing is still only in my head.

Most current blog engines are like magazines. You write something, and it gets filed under the date it was published in! If you write an essay, why should it matter whether you wrote it around 15/4/2006 or around 8/10/2007? Most of the time, the exact date doesn’t matter. Just a general time line is usually enough, “2006”. If the date does matter for a certain essay, you can add it as a note somewhere.

When I think about an essay here, I’m thinking about things that might go in books, more so than newspapers. And in books, you don’t usually get a date around every essay or chapter. Books are a bit more “timeless” than magazines in this sense. There’s usually only one date on a book: the year it was published.

This is actually inspired by Paul Graham’s essays page.

If most current writing platforms (manifested as blog engines) are designed more around a magazine concept, then there ought to be a place in the market for writing platforms that are designed more around the “book” concept.

It’s certainly possible to write essays on top of current blog engines; I’ve seen several people do it, but it’s somewhat annoying, and you get a lot of other little problems. I’m sort of doing something similar with Tumblr here, though not quite so.

The nature of essays is that they can evolve. You write a piece, revise it, revise it, revise it, then publish it. You get lots of criticism, people comment about how stupid you are, etc, and maybe you will want to edit it, to sort of elaborate some problem points. OK, maybe you have a thick skin and you can ignore trolls, good for you. But still, say you change your opinion about something, or you learn of something you didn’t know before, so you want to go back and revise your essay.

So in this sense, each essay is like its own project, you work on it, publish it, but you can never really say it’s “done”; there’s always room for changes and improvements. So it’s ought to be possible for you to completely “own” your project, independently of the web platform where you’re publishing it. You should be able to work on it locally on your machine. Blogs don’t usually give you that kind of freedom. There might be some APIs for importing and exporting blog posts, but that doesn’t quite cut it. I’m thinking of ownership here in terms of how you own your project that you’ve published on github. You have the git repository on your local machine, all the time. If github disappears, you can publish your local git repository somewhere else. If Google disappears, all your blog posts and Google Docs also disappear. (It’s much more likely for your computer to break down than for Google to disappear, but it’s always possible that Google may suspend your account and delete your content).

Another important thing for this kind of writing platform is how the content is presented. Most blogs allow you to make your blog ugly and stupid in all kinds of different ways. Silly color schemes, useless widgets everywhere, picture backgrounds, etc. I much prefer there be a single, good, minimal theme, or just a few of them to choose from.

Pen.io seems like a such a writing platform, though it doesn’t fulfill the content-ownership I talked about.

Google Docs also allows you to do pretty much the same, and it’s much better at being a writing tool than pen.io and other blogs, but it’s not “webby” at all. You can “publish” your writings, but they have a long cryptic URL, and there are no interaction features what so ever.

Any writing and publishing platform on the web needs to have some way of interacting with readers and the rest of the web. Some kind of a commenting system seems necessary. I’m not sure if comments should be right on the essay page (like most blogs) or in a separate place. For some reason I prefer to have the comments somewhat “detached” from the essay itself, but at the same time readily accessible from it. A simple way to do that is by having a link at the top and/or bottom of the essay to a “discussion page”. The discussion page would also have a link to the original essay at the top and/or the bottom.

Webpage titles

Sun, 22 May 2011 03:13:09 -0600

I just had this idea. The title of a wepage should try to be as short as possible. Perhaps a 5-letter word.

Titles (along with icons) are used to recognize tabs in a multi-tabbed browser. Keep that in mind. For the title to be useful, it has to be short, otherwise the browser will cut it.
Titles are also used when creating bookmarks; long titles force the user to do the useless task of editing a bookmark name from “YouTube! Broadcase yourself!” to “uTube”. Why not just make it “uTube” to begin with?

PySide + QtQuick Tutorial

Fri, 13 May 2011 02:51:36 -0600

PySide + QtQuick Tutorial:

I’m just looking around for tutorials, and just found this (haven’t even read it yet, but seeing as its in the official wiki, it’s gotta be good).

bash: mkdir + cd with one command

Sat, 23 Apr 2011 07:19:50 -0600

codedump:

function mkcd() {
  [ -n "$1" ] && mkdir -p "$@" && cd "$1";
}

Example:

user@host:~$ cd work
user@host:~/work$ mkcd website.com/{html,src,stuff}
user@host:~/work/website.com/html$

Synthesizing music

Sun, 17 Apr 2011 08:31:00 -0600

Please be warned again that this is not my area of expertise. Don’t take anything I say too seriously. At best anything I say it just a reasonable approximation that happens to work for me.

I’m assuming you’re trying to generate musical tones, so I’ll talk about frequencies as if you already know what frequency you want to produce. I will also be using coffee-script and the Firefox Audio Data API.

The max value of a sample is 1.0 and the min value is -1.0 so all samples will be between these two values (these are the “peaks” of the volume).

Frequency / Period

If you consider a sound sample to be a sequence of a repeating shape, then count how many times per second does the shape repeat? This is called the “period” (e.g. the period of the sample/signal). The frequency then will be 1/p (where p is the period).

If your sample rate is SRATE, then that’s how many samples are there in a second. So, to get how many samples are required to generate one period p, it’s srate/freq.

Technically this is the “fundamental frequency” but I’m not too deep on the math of signal processing.

Sine waves

Sine waves produce a pure “peeeeeeep” sound. Generally speaking a sine wave is a good start when you’re hacking around, but I don’t know if you can get far enough with it. The formula for generating a sound wave doesn’t require calculating the period or anything like that; just the frequency.

The formula for it is simple mathematically: given a point in a long sound sample (think of point as index):

sine = (freq, point) ->
    k = 2 * Math.PI * freq / SRATE
    Math.sin(k * point)

This is not very efficient as you keep calculating k over and over, so:

sine = (freq) ->
    k = 2 * Math.PI * freq / SRATE
    (point) -> Math.sin(k * point)

Now sine(440) returns a function that takes a point and returns a sample value.

Sine waves are not very interesting like I already said. I thought we’d build more complex sounds on top of them, but it turns out they’re not needed at all when synthesizing string sounds using the karplus-strong algorithm. Perhaps sine waves are useful when generating sounds in another way.

There are a few extra things you should take note of, I already mentioned them here:

Quick notes on generating sine waves

Basically you’ll need to dampen the signal using e^(-5x) (you can change 5 to 4 or something else — feel free to experiment with it). To get x you have to know when do you want the signal to end.

In code would look like this:

dampen = (point, lastpoint) ->
    x = point / lastpoint
    Math.pow(Math.E, x * -5)

When you write the output, you multiply the sample from sine by the damp factor from dampen.

And the pink noise process is dead simple: you multiply the signal by 100/freq

s = sinefn(point)
d = dampen(point, lastpoint)
p = 100/freq
output[point] = s * d * p

Wavetables

Another way to generate sound is to use a pre-populated list. For some reason this list is called a wavetable. Please don’t confuse this table with hash-tables or dictionaries: it’s just an array of values.

For a sound of frequency freq, the size of the wavetable is SRATE/freq. Now this actually doesn’t produce exactly the desired frequency, but a close enough frequency that the ear can’t differentiate really.

See, if you want to generate a wavetable for the frequency 440, here’s what would happen:

coffee> SRATE = 96000
96000
coffee> freq = 440
440
coffee> SRATE/freq
218.1818181818182
coffee> Math.floor SRATE/freq
218
coffee> SRATE / 218
440.3669724770642

The period length would be 218, but that’s the period associated with frequency 440.4. This difference is like I said small enough that the human ear doesn’t notice a thing.

So if you pre-populate an array of this size, you can generate the samples on the fly:

length = Math.round SRATE/freq
wavetable = new Float32Array(length)
# fill table somehow
(point) ->
    wavetable[point % length]

Since the process of pre-populating itself can be time-consuming, we can do it lazily:

(point) ->
    if point < length
        wavetable[point] = wavefn(point)
    else
        wavetable[point % length]

This leads us to the Karplus-Strong method of synthesizing guitar strings, which I already wrote about here:

Very simple implementation of Karplus-Strong algorithm in coffeescript

Note that it doesn’t make use of sine waves at all. Instead, it starts with a noise sample and causes it to decay in a very simple-minded way. It’s very surprising (to me at least) that it manages to generate a sound like that.

OK, this sums up my experience so far. Unfortunately I haven’t come across the ultimate function that can generate any kind of musical instrument :) but that ok, since it wasn’t my goal to begin with.

Very simple implementation of Karplus-Strong algorithm in coffeescript

Mon, 11 Apr 2011 00:52:00 -0600

I’m a bit over excited as I just managed to successfully implement Karplus-Strong’s guitar string algorithm.

http://en.wikipedia.org/wiki/Karplus-Strong_string_synthesis

The Wikipedia description was too complicated for me. I had to get the original paper and try to study it. It’s rather old and the terminology was a bit weird, so it took me a while to really understand it.

This snippet assumes you have SRATE defined somewhere (the sample rate), and that you already managed to setup the audio output and the mixer and everything else. I’m using audiodata.js to output audio using Firefox4’s Audio Data API.

The function guitar here returns a function which takes a sample index/point and returns the sample. It assumes that it will be called with all the samples sequentially, otherwise it won’t work. Something like:

signal_fn = guitar(440) # 440 being the frequency we want to sound

point = 0
while not done
    sample = signal_fn(point++)
    # output sample

Not exactly that, but something like that.

Here’s the code:

period_len = (freq) -> Math.floor (SRATE/freq)

avg = (a, b) -> (a + b) / 2

ks_noise_sample = (val) ->
    # get either val or -val with 50% chance
    if Math.random() > 0.5
        val
    else
        -val

# karplus strong algorithm
guitar = (freq) ->
    samples = period_len freq
    table = new Float32Array(samples)
    getsample = (index) ->
        point = index % samples
        if index == point
            table[point] = ks_noise_sample(1)
        else
            prev = (index - 1) % samples
            table[point] = avg(table[point], table[prev])

Note that this is coffee-script, where the last expression in a function is returned, and A = B expressions return B. So, guitar returns the getsample function, and getsample return table[point] after we calculate it.

Here’s what this does:

Fill a table of length p with random values (actually: fill it with either A or -A, with 50% chance each). Where p is the period of the signal, which is the sample_rate/frequency.

To produce samples, just keep cycling on this table. That is, to get any point t, take table[t % p].

BUT, as you loop on the table, average each two adjacent samples. That is, everytime you get table[t] (except the first time around), let table[t] be (table[t] + table[t-1]) / 2.

This causes the signal to decay overtime, or something like that.

That’s it.

Quick notes on generating sine waves

Tue, 05 Apr 2011 20:39:52 -0600

A few notes based on my experiments the past couple of days.

First, when you manage to generate a sine wave, you’ll notice there’s a “clicking” sound at the end. Where does it come from and how to deal with it? I think it comes from having the volume suddenly drop to 0. So to make it go away, you gotta make the wave form fade away slowly and gradually.

The graph of e^(-2x) looks like a good smooth path for that.

Second, you’ll notes that high frequencies sound way louder than low frequencies. To work around that, multiply the samples by 1/freq (see: pink noise). This makes each tone sound the same loudness. (In reality you use a multiple of 1/freq, for for instance, I used 100/freq.

I realize this is way out of context and perhaps won’t make sense to most people. I’ll talk more about generating sound waves later. But I wanted to jot down these quick notes before I forget.