The problems involved with developing Synaptic Web applications

Just as the foundation of modern RISC computers was a set of basic design and performance requirements put forth in the mid 1970′s, so too there are a few basic requirements for a synaptic web application.

Paraphrased and somewhat shortened from synapticweb.org they are:

• They connect two or more categories of things together
• They create or derive new/novel meaning or utility from implicit connections
• The connections they enable adjust in real or near-real time to changes
• They bias towards implicit connections driven by user behaviour
• They use the web as the platform
• They apply a variety of inputs to extend existing applications
• They become stronger through network effects
• They are defined by usership and information flows and are untethered from any destination site
• One of their primary inputs and/or outputs is the stream

Some of these requirements are arguably almost trivial to implement. Most web developers already know how to use the web as the platform with various API‘s and many of the most popular applications already base almost their entire ecosystem on extensions and pluggable widgets.

However there are some very hard nuts to crack in there. Namely the two biggest problems are creating a rating algorithm, because all connections need ratings/weights lest they are useless, that would comply and creating connections between several different kinds of things.

The Connections Problem

ON the face of it, creating connections between completely different and practically unrelated concepts is nearly impossible. How would you, for example, form connections between a platypus and a frog?

A human can [almost] easily make the connection, after all, a human came up with it. The association is quite simply that they both live mostly in water, but aren’t fish.

But how could a computer come up with such a leap of logic? It can’t do this with machine learning, simply isn’t possible by any means we know today and there is no set of organised data large enough to figure out intricate connections like that. Perhaps if you had the whole of wikipedia processed and a few years of supercomputer time … then perhaps, perhaps you could crack it.

Did you notice it? The answer was right there, in those two paragraphs.

No? Well let me help you out, the answer is to use humans to make the connections. This might not work for every application out there, but humans in general are very good at making illogical and non-obvious connections. The real kicker? These are the connections that really matter to us. We don’t, personally, care that water on Phobos implies life. We care that our neighbour’s cat reminds us of Mr. Fluffy we had as a child.

The Rating Problem

Because all connections and pretty much anything about information is rather useless, or only marginally useful, without some sort of weights acting on the whole mishmash of technologies, we soon clash with the toughest nut to crack with all of the Synaptic Web.

Eric Schmidt has said that the biggest problem of this age is discovering a link rating algorithm that works with a real-time information stream. The answer, in my opinion, is a synaptic application.

Right now most ratings are considered globally, are created very explicitly, and barely tell us anything about what they’re trying to rate. Google Reader suggests things as very very awesome even when they were “liked” by people I’ve never heard of. TweetMeme promotes stuff every-fucking-body in the world has retweeted, but there is nothing even remotely considering whether I‘m going to like it.

The synaptic web model predicts a future where applications understand my actions (some of this is implicitly creating connections between things) and dynamically adjust their ratings of content accordingly. One of my biggest pet peeves, for example, is that last.fm says my favouritest band in the world is something I haven’t listened to in almost a year. Wtf?

Creating such algorithms is very problematic though. The only real way of even scratching the surface of this problem is by applying a lot of meta-heuristics, like simulated annealing, and doing a lot of reinforcement machine learning – something that isn’t as perfectly researched and understood as supervised, unsupervised and semi-supervised machine learning algorithms – to the problem of weighing connections and things in databases.

And even after you’ve created a good reinforcement learning environment with a lot of simulated annealing going on, you’re still left wondering how to even rate user action. Sometimes clicking a link is good because the person liked what they were suggested, at other times they opened and shared it just so they could warn their friends not to bother with this crap.

Final Summation

In short, creating a real synaptic application is hard and those that do make one are bloody amazing guys.

This entry was written by swizec, posted on December 29, 2009 at 10:21 am, filed under Algorithms, synaptic web and tagged Add new tag, Application programming interface, Artificial intelligence, Google, Google Reader, Machine learning, People, Research Groups, Web application. Bookmark the permalink.
Follow any comments here with the RSS feed for this post.