When we first started the Preona project one of our most important assumptions was that actively liking things is very cumbersome and few people will want to do it in order to personalise the reading suggestion system.

What we wanted to build was a system that could automagically learn from what the user is doing, be intriguingly implicit and invisible.

This was wrong.

After observing what people are doing online and how they behave, we, or rather I, have come to the conclusion that liking is a very important action that transcends cultural and technological bounds. And that people do it for everything but improving the ratings of what they’re reading.

The reason people like liking things is simple: Feedback.

And I don’t mean giving feedback to developers, or recommendation systems. No, they’re giving feedback on a completely personal level. The act of liking things online has gained a significant cultural and sociological dimension – it’s a lot like giving flowers to people, the meaning depends wholly on the context.

But this makes the whole act very difficult to interpret and base your machine learning systems on. When the context defines the action’s meaning, rather than the action itself that’s a very big problem. Sure, in general you can assume that liking means people would like to read more of that particular content, but what about when they’re just expressing agreement, or mutual dislike of something? Or expressing condolences? Nobody wants to read a lot of sad things …

This sounds like a very difficult problem to solve, but perhaps the solution is incredibly simple.

Implement liking as a purely conversational feature and ignore it completely from a machine learning perspective? I don’t know, we’re going to have to try and see what works best.

An interesting experiment would be to try predicting whether a person will “like” something with Google‘s new prediction API. This probably calls for a weekend project, hopefully I’ll have a chance to play with that in the following days and try it out.

Related articles by Zemanta

• LinkLog: Google Prediction API (dorai.wordpress.com)
• BigQuery and Prediction API: Get more from your data with Google (googlecode.blogspot.com)
• Machines that learn better (physorg.com)
• Google’s search for the perfect learning machine (news.cnet.com)

Lately facebook has been taking a lot of flak over user privacy regarding the changes they’ve implemented to their API and how easy it is for 3rd parties to access user’s private data … or more to the point, how much pain users have to go through to hide their personal data.

My personal opinion is that, yes, facebook is being a bit of an arse about the whole issue. At first they were promising everyone a walled garden free of outside scrutiny where you can behave just like you were locked up in your little dorm room having fun with a few mates.

And people got used to that.

For example you’ll never see anyone complain about the fact their tweets are public, or that whatever they post on their blogs is public, or flickr or forums or irc and a miriad other services. The whole online world is … public. And nobody bats an eyelid.

Then facebook becomes slightly more public.

Public outrage!

Villages, internet and old women peeking out of windows

The fact of the matter is that facebook was never private even though it promised to be. In fact I’ll wager it’s more private now than it was ever before!

Wait what?

It all boils down to rumors and basic human behaviour.

if you’re worried about looking like an idiot, don’t be an idiot when someone is looking

Now I’m not a psychologist or sociologist or anything like that, but because I’m a computer scientist I like to pretend I can understand anything algorithmically complex – like people.

Also I’ve talked to one or two people who have lived in a small enough village to explain this effect to me.

When you have a small population of people living in a confined area everybody knows a little bit about everybody else. It doesn’t matter who you are or how many friends you’ve got. Someone will always know what you did last summer or how drunk you got on that party on Friday and not to mention what big of an arse you turned out to be when you dumped that poor girly.

Everybody! Will! Know, When. You, Fuck Up! They just will.

But when you live in a big city, it’s a little different. Suddenly there are so many people on so big an area you hardly know anyone. Most of us don’t even know any of the people living in the same building we are. If you’re lucky you’ll have a vague idea of what profession they’re in.

This gives us a super huge expectation of privacy and we’ve grown so used to being anonymous and unknown that we want this notion of privacy to extend into every little facet of our world.

So where lies the problem?

The problem is when these city slickers reach the online world. It looks anonymous. It smells anonymous. And it certainly likes to talk anonymous. No names. No identity. Nothing

Hoorah! I can make a total idiot of myself and no-one will know! yay

Well no, not really. Google and a bunch of other things are tracking your every move, every keystroke almost. They do this to make more money off of you through targeted advertising and some other unimportant stuff.

We’ve gone full circle, suddenly the online world behaves a lot like a village. Everybody knows everybody and so on.

So why such an outcry over privacy issues?

Well, because us city slickers don’t really know how to handle this kind of information. We’re not used to knowing so much about people around us, we’re not used to always having a pair of eyes on our back through the curtains of the window across the street.

That’s why we do strange google searches of people we want to hire, we research people we want to date, we … I don’t know, we basically dig up every little detail of anybody we encounter and think is important enough.

We want all that info.

But we can’t handle it.

A long time ago people in villages learned that being drunk at a party 20 years ago does not reflect poorly on a man today. The rest of us still need to figure that out so there won’t be cases of people not getting hired due to an obscure image found online or a drunk photo posted on their facebook etc.

Until then, for fuck’s sake people, if you’re worried about looking like an idiot, don’t be an idiot when someone is looking!

Related articles by Zemanta

• Google Offers Lip Service Response to Privacy Commissioners (readwriteweb.com)
• Make Up Your Mind Facebook, Because Your Privacy Controls Are A Mess (techcrunch.com)
• Why You Should Still Quit Facebook [Rant] (gizmodo.com)
• EMBARRASSING: Facebook Board Member’s Profile Hacked (businessinsider.com)
• Yet another privacy risk with Facebook: leaked IP addresses (boingboing.net)
• Has Facebook Jumped the Shark? (markevanstech.com)
• The Facebook Trap: How Do You Stay Connected Without Compromising Your Own Privacy? [Facebooked] (jezebel.com)
• INFOGRAPHIC: The History Of Facebook’s Default Privacy Settings (allfacebook.com)

Yesterday I read an interesting post about the speed at which information travels between two nodes in a network. It was about a so called Data Singularity and the basic premise was that nowadays information flows are so horribly fast that only computers talking to computers can handle the stream at all and that humans are even less than mediaries, but can only really comprehend meta-meta-data.

To recap the changing history of what the concept of “real-time” information means (btw this is a talk I was supposed to have at WebCampLj last month but then lost the artistic inspiration):

• there was once a time when any news that came to the listener in a week was considered real-time information, depending on how far away the event was (snail mail)
• then real-time became the speed of light with human routers (telegraph, early telephone)
• real-time then became instant, but not constant (phone, TV etc.)

But right now we are in a world where there is an instant and constant global conversation going on. Everybody is multicasting, if not downright broadcasting, very many random thoughts that pop into their heads. People are having conversations all over the place and it’s all right there; always.

Hell, even Google has decided to start thinking about something as horrible as real-time RSS feeds by integrating pubsubhubbub into their Reader.

It’s almost as if the whole world is conspiring against the lonely infonaut who just wants to be able to do something while still getting all the information they crave so deeply. I’m not sure where I’ve read it or how far ago, but it was something along the lines of the average person these days being completely and totally addicted to information and that this addiction is made worse than heroin by the simple fact society expects it.

That is to say, if you’re not addicted to information, you’re being quite odd and strange.

But where does this attitude lead us?

There are 25 million tweets made every day, that is to say 289 tweets EVERY FUCKING SECOND! Even if you follow a very small subset of those people, that still makes at least 10 tweets every minute!

Count RSS feeds into that … Reader already pushes updates to the interface so very real-time that when I click “mark all as read” there are usually 5 new items waiting for me in the time it takes for the interface to clear. Yes, it’s That Bad(tm). Considering that following only a handful of RSS feeds (about 100) means I personally get on average 500 to 600 new articles every day … yeah, that makes one article every two bloody minutes!

And they want to make that process even faster.

So essentially, as an infonaut, the internet expects me to read twenty tweets and one long-ish article every two minutes of my day. And that’s not even counting E-Mail, Facebook, Forums and a bunch of other things.

Just to give you a sense of how little two minutes is: It takes three minutes to steep a cup of tea.

Now obviously I am quite incapable of processing information constantly, all day, at a rate of at least 240-ish words per minute. For a little sense of what this means, the world record for typing is 150 words per minute sustained over 50 minutes.

Yeah that’s right, those very very crazy typists that type so fast it looks like magic … type much much slower than you are expected to read these days.

Quite apparent to anyone paying attention is that this situation is very unsustainable and people saying that “Oh you can just take an hour every day and skim through the titles of everyting” are downright bollocking crazy! What can be done is a matter of long debate, what will be done remains to be seen.

But something has got to be done ’cause this is insane.

Related articles by Zemanta

• myScoop now fetches your articles in REAL-TIME (nickduncan.co.za)
• What is PubSubHubbub and How to Implement It In WordPress (maketecheasier.com)
• Google Indexing In Near-Realtime (search.slashdot.org)
• 3 Ways To Use Google Reader As An Online Archive (makeuseof.com)

Word on the street has it that semantic has become a curseword and people should avoid using it at all cost, especially in relation to any sort of internet business or technology.

Hearing about this led me to wondering about why and how it was that the semantic web failed us so badly that even just using the word is bad for business. The first step on my adventure was figuring out what the semantic web promised us, because you can’t see where things went wrong if you don’t know what a movement tried to produce.

After a lot of googling and searching my memory for what people have said/promised over the years this is the best description I could find:

I have a dream for the Web [in which computers] become capable of analyzing all the data on the Web – the content, links, and transactions between people and computers. A ‘Semantic Web’, which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines. The ‘intelligent agents’ people have touted for ages will finally materialize.

– Tim Berners-Lee, 1999

Far as I can understand this means that the web will be organised in such a fashion machines could understand it perfectly and use it as some sort of huge collective brain – a bit like the Borg perhaps. This would indeed be a wonderful future, machines talking to machines, data flowing to and fro with perfect understanding of why, streamlined transactions optimised to perfection through better understanding of their environment …

The big flaw here is: machines talking to machines.

It would be difficult to say how exactly somebody came up with a concept of the internets that so deeply misses the whole point of why the internet is as popular as it is. Perhaps the reason is that the quote comes from 1999 when the web was a wholly different beast, but why then, pray tell, did entrepreneurs latch onto this idea only around 2005 when not much was different than today? If you’re looking for those answers, I’m not the man to ask, but I can tell you what the internet is about and how the synaptic web helps there.

The internet is people talking to people

When it comes to the internet it’s all about being social, about following your friends and having conversations. Nobody cares anymore about finding data, that’s all on wikipedia, at worst a google search away, nobody cares anymore about computers talking to computers. In fact I’d wager a nice lump of money that most user’s eyes would glaze over when you started talking about all of that.

This is where I believe the semantic web has failed us, the users, and investors and pretty much anyone who wanted to do anything serious with it. Semantics are a good background protocol, a lovely standard for obsessive webdevs to follow, an amazingly good technology … but in and of itself without a purpose.

The only people that needed the semantic web were [are] developers, there is no product to be found therein, Zemanta‘s tool is probably the closest one can come to a semantic product and even that probably has far more magic in everything but semantics.

We need computers that understand people and this is something you, of course, need semantics for. It’s also something that produces immediate value to the only person that really matters – The User.

tl;dr version: The Synaptic Web will not fail us because it focuses on users.

Related articles by Zemanta

• Top 5 Web Trends of 2009: Structured Data (readwriteweb.com)
• Will The Semantic Web Have a Gender? (readwriteweb.com)
• A Yarn About Twine — ISWC 2009 Keynote — Nova Spivack (slideshare.net)
• Sir Tim BernersLee demands more ‘raw data’ from Government (telegraph.co.uk)

Just as the foundation of modern RISC computers was a set of basic design and performance requirements put forth in the mid 1970′s, so too there are a few basic requirements for a synaptic web application.

Paraphrased and somewhat shortened from synapticweb.org they are:

• They connect two or more categories of things together
• They create or derive new/novel meaning or utility from implicit connections
• The connections they enable adjust in real or near-real time to changes
• They bias towards implicit connections driven by user behaviour
• They use the web as the platform
• They apply a variety of inputs to extend existing applications
• They become stronger through network effects
• They are defined by usership and information flows and are untethered from any destination site
• One of their primary inputs and/or outputs is the stream

Some of these requirements are arguably almost trivial to implement. Most web developers already know how to use the web as the platform with various API‘s and many of the most popular applications already base almost their entire ecosystem on extensions and pluggable widgets.

However there are some very hard nuts to crack in there. Namely the two biggest problems are creating a rating algorithm, because all connections need ratings/weights lest they are useless, that would comply and creating connections between several different kinds of things.

The Connections Problem

ON the face of it, creating connections between completely different and practically unrelated concepts is nearly impossible. How would you, for example, form connections between a platypus and a frog?

A human can [almost] easily make the connection, after all, a human came up with it. The association is quite simply that they both live mostly in water, but aren’t fish.

But how could a computer come up with such a leap of logic? It can’t do this with machine learning, simply isn’t possible by any means we know today and there is no set of organised data large enough to figure out intricate connections like that. Perhaps if you had the whole of wikipedia processed and a few years of supercomputer time … then perhaps, perhaps you could crack it.

Did you notice it? The answer was right there, in those two paragraphs.

No? Well let me help you out, the answer is to use humans to make the connections. This might not work for every application out there, but humans in general are very good at making illogical and non-obvious connections. The real kicker? These are the connections that really matter to us. We don’t, personally, care that water on Phobos implies life. We care that our neighbour’s cat reminds us of Mr. Fluffy we had as a child.

The Rating Problem

Because all connections and pretty much anything about information is rather useless, or only marginally useful, without some sort of weights acting on the whole mishmash of technologies, we soon clash with the toughest nut to crack with all of the Synaptic Web.

Eric Schmidt has said that the biggest problem of this age is discovering a link rating algorithm that works with a real-time information stream. The answer, in my opinion, is a synaptic application.

Right now most ratings are considered globally, are created very explicitly, and barely tell us anything about what they’re trying to rate. Google Reader suggests things as very very awesome even when they were “liked” by people I’ve never heard of. TweetMeme promotes stuff every-fucking-body in the world has retweeted, but there is nothing even remotely considering whether I‘m going to like it.

The synaptic web model predicts a future where applications understand my actions (some of this is implicitly creating connections between things) and dynamically adjust their ratings of content accordingly. One of my biggest pet peeves, for example, is that last.fm says my favouritest band in the world is something I haven’t listened to in almost a year. Wtf?

Creating such algorithms is very problematic though. The only real way of even scratching the surface of this problem is by applying a lot of meta-heuristics, like simulated annealing, and doing a lot of reinforcement machine learning – something that isn’t as perfectly researched and understood as supervised, unsupervised and semi-supervised machine learning algorithms – to the problem of weighing connections and things in databases.

And even after you’ve created a good reinforcement learning environment with a lot of simulated annealing going on, you’re still left wondering how to even rate user action. Sometimes clicking a link is good because the person liked what they were suggested, at other times they opened and shared it just so they could warn their friends not to bother with this crap.

Final Summation

In short, creating a real synaptic application is hard and those that do make one are bloody amazing guys.