I’ve contributed to the Metrics Insider column on Mediapost. Have a read, and post a comment 🙂
Prior to 2006, I had sort of heard of the Semantic Web. To be honest, I didn‚Äôt know much ‚Äì it was just another buzzword. I‚Äôve been hearing about Microformats for years, and cool but useless initiatives like XFN. However to me it was simply just another web thing being thrown around.
Then in August 2006, I came across Adrian Holovaty‚Äôs article where he argues journalism needs to move from a story-centric world to a data-centric world. And that‚Äôs when it dawned on me: the Semantic web is some serious business.
I have since done a lot of reading, listening, and thinking. I don‚Äôt profess to be a Semantic Web expert ‚Äì but I know more than the average person as I have (painfully) put myself through videos and audios of academic types who confuse the crap out of me. I‚Äôve also read through a myriad of academic papers from the W3C, which are like the times when you read a novel and keep re-reading the same page and still can‚Äôt remember what you just read.
Hell ‚Äì I still don‚Äôt get things. But I get the vision, so that‚Äôs what I am going to share with you now. Hopefully, my understanding will benefit the clueless and the skeptical alike, because it‚Äôs a powerful vision which is entirely possible
1) The current web is great for humans; useless for machines
When you search for ambiguous terms, at best, search engines can algorithmically predict some sort of answer that partially answers your query. Sometimes not. But the complexity of language, is not something engineers can engineer to deal with. After all, without ambiguity of natural languages, the existence of poetry is impossible.
What did you think when you read that? As in: ‚ÄúI‚Äôve had it ‚Äì fine!‚Äù which is like another way of saying ok or agreeing with something. Perhaps you thought about that parking ticket I just got ‚Äì illegal parking gets you fined. Maybe you thought I am applauding myself by saying that was one fine piece of wordcraftship I just wrote, or said in another context, like a fine wine.
Language is ambiguous, and depending on the context with other words, we can determine what the meaning of the word is. Search start-up company Powerset, which is hoping to kill Google and rule the world, is employing exactly this technique to improve search: intelligent processing of words depending on context. So by me putting in ‚Äúit‚Äôs a fine‚Äù, it understands the context that it‚Äôs a parking ticket, because you wouldn‚Äôt say ‚Äúit‚Äôs a‚Äù in front of ‚Äòfine‚Äô when you use it to agree with something (the ‚Äòok‚Äô meaning above).
But let‚Äôs use another example: ‚ÄúHilton Paris‚Äù in Google ‚Äì the worlds most ‚Äòadvanced‚Äô search engine. Obviously, as a human reading that sentence, you understand because of the context of those words I would like to find information about the Hilton in Paris. Well maybe.
Let‚Äôs see what Google comes up with: Of the ten search results (as of when I wrote this blog posting), one was a news item on the celebrity; six were on the celebrity describing her in some shape or form, and three results were on the actual Hotel. Google, at 30/70 ‚Äì is a little unsure.
Why is Paris Hilton, that blonde haired thingy of a celebrity, coming up in the search results?
Technologies like Powerset apparently produce a better result because it understands the order of the words and context of the search query. But the problem with these searches, isn‚Äôt the interpretation of what the searcher wants ‚Äì but also the ability to understand the actual search results. Powerset can only interpret so much of the gazilions of words out there. There is the whole problem of the source data, no just the query. Don‚Äôt get what I mean? Keep reading. But for now, learn this lesson
Computers have no idea about the data they are reading. In fact, Google pumping out those search results is based on people linking. Google is a machine, and reads 1s and 0s ‚Äì machine language. It doesn‚Äôt get human language
2) The Semantic web is about making what human‚Äôs read, machine readable
Tim Berner‚Äôs Lee, the guy that invented the World Wide Web and the visionary behind the Semantic Web, prefers to call it the ‚Äòdata web‚Äô. The current web is a web of documents ‚Äì by adding this extra data to content ‚Äì machines will be able to understand it. Metadata, is data about data.
A practical outcome of having a semantic web, is that Google would know that when it pulls up a web page regardless of the context of the words ‚Äì it will understand what the content is. Think of every word on the web, being linked to a master dictionary.
The benefit of the semantic web is not for humans ‚Äì at least immediately. The Semantic Web is actually pretty boring with what it does ‚Äì what is exciting, is what it will enable. Keep reading.
3) The Semantic web is for machines to interpret, not people
A lot of the skeptics of the semantic web, usually don‚Äôt see the value of it. Who cares about adding all this extra meta data? I mean heck ‚Äì Google still was able to get the website I needed ‚Äì the Hilton in Paris. Sure, the other 60% of the results on that page were irrelevant, but I‚Äôm happy.
I once came across a Google employee and he asked ‚Äúwhat‚Äôs the point of a semantic web; don‚Äôt we already enough metadata?‚Äù To some extent, he‚Äôs right ‚Äì there are some websites out there that have metadata. But the point of the semantic web is so that machines once they read the information, can start thinking like how a human would and connecting it to other information. There needs to be across the board metadata.
For example, my friend Michael was recently looking to buy a car. A painful process, because there are so many variables. So many different models, different makes, different dealers, different packages. We have websites, with cars for sale neatly categorised into profile pages saying what model it is, what colour it is, and how much. (Which may I add, are hosted on multiple car sites with different types of profiles). A human painfully reads through these profiles, and computes as fast as a human can. But a machine can‚Äôt read these profiles.
Instead of wasting his (and my) weekends driving around Sydney to find his car, a machine could find it for him. So, Mike would enter his profile in ‚Äì what he requires in a car, what his credit limit is, what his prior history with cars are ‚Äì everything that would affect his judgement of a car. And then, the computer can query every online website with cars to match the criteria. Because the computer can interpret these websites across the board, it can evaluate and it can go back to Michael and say ‚Äúthis is the car for you, at this dealer ‚Äì click yes to buy‚Äù.
The semantic web is about giving computers the information to be able to interpret data, so that it can do what they do really well ‚Äì compute.
4) A worldwide database
What essentially Berner‚Äôs Lee envisions, is turning the entire world wide web into a database that can be queried. Currently, the web looks like Microsoft Word ‚Äì one swab of text. However, if that swab of text was neatly categorised in an Excel spreadsheet, you could manipulate that data and do what you please ‚Äì create reports, reorder them, filter, and do whatever until your heart is content.
At university, I was forced to do an Information Systems subject which was essentially about the theory of databases. Damn painful. I learned only two things from that course. The first thing was that my lecturer, tutor, and classmates spoke less intelligible English than a caterpillar. But the second thing was that I learned what information is and how it differs from data. I am now going to share with you that lesson, and save you three months of your life.
You see, data is meaningless. For example, 23 degrees is data. On its own, it‚Äôs useless. Another piece of data in Sydney. Again, – useless. I mean, you can think all sorts of things when you think of Sydney, but it doesn‚Äôt have any meaning.
Now put together 23 degrees and Sydney, and you have just created information. Information is about creating relationships between data. By creating a relationship, an association, between these two different pieces of data ‚Äì you can determine it‚Äôs going to be a warm day in Sydney. And that is what information is: Relationship building; connecting the dots; linking the islands of data together to generate something meaningful.
The semantic web is about allowing computers to be able to query the sum of human knowledge like one big database to generate information
You are probably now starting to freak out and think ‚ÄúTerminator‚Äù images with computers suddenly erupting form under your computer desk, and smashing you against the wall as a battle between humans and computers begins. But I don‚Äôt see it like that.
I think about the thousands of hours humans spend trying to compute things. I think of the cancer research, whereby all this experimentation occurring in labs, is trying to connect new pieces of data with old data to create new information. I think about computers being about to query the entire taxation legislation to make sure I don‚Äôt pay any tax, because it knows how it all fits together (having studied tax, I can assure you ‚Äì it takes a lifetime to only understand a portion of tax law). In short, I understand the vision of the Semantic web as a way of linking things together, to enable computers to compute ‚Äì so that I can sit on my hammock drinking my beer, as I can delegate the duties of my life to the machines.
All the semantic web is trying to do, is making sure everything is structured in a consistent manner, with a consistent dictionary behind the content, so that a machine can draw connections. As Berner‚Äôs Lee said on one of the videos I saw: ‚Äúit‚Äôs all about creating links‚Äù.
The process to a Semantic Web is boring. But once we have those links, we can then start talking about those hammocks. And that’s when the power of the internet – the global network – will really take off.
I recently was pointed to a presentation of John Hagel who is a renowned strategy consultant and author on the impact the Internet has on business. He recently joined Deloitte and Touche, where he will head a new Silicon Valley research institute. At the conference (Supernova 2007), John outlined critical research questions regarding the future of digital business that remain unresolved, which revolved around the following:
What happens after everything is connected? What are the most important questions?
I had to watch the video a few times because its not possible to capture everything he says in one hit. So I started writing notes each time, which I have reproduced below to help guide your thoughts and give a summary as you are watching the presentation (which I highly recommend).
I also have discovered (after writing these notes – damn it!) that he has written his speech (slightly different however) and posted it on his blog. I’ll try and reference my future postings on these themes here, by pinging or adding links to this posting.
Robert Scoble has put together a video presentation on how Techmeme, Facebook and Mahalo will kill Google in four years time. His basic premise is that SEO’s who game Google’s algorithm are as bad as spam (and there are some pissed SEO experts waking up today!). People like the ideas he introduces about social filtering, but on the whole – people are a bit more skeptical on his world domination theory.
Relevance is personal
When I search, I am looking for answers. Scoble uses the example of searching for HDTV and makes note of the top manufacturers as something he would expect at the top of the results. For him – that’s probably what he wants to see – but for me, I want to be reading about the technology behind it. What I am trying to illustrate here is that relevance is personal.
The argument for social filtering, is that it makes it more relevant. For example, by having a bunch of my friends associated with me on my Facebook account, an inference engine can determine that if my friend called A is also friends with person B, who is friends with person C – than something I like must also be something that person C likes. When it comes to search results, that sort of social/collaborative filtering doesn’t work because relevance is complicated. The only value a social network can provide is if the content is spam or not – a yes or no type of answer – which is assuming if someone in my network has come across this content. Just because my social network can (potentially) help filter out spam, doesn’t make the search results higher quality. It just means less spam results. There is plenty of content that may be on-topic but may as well be classed as spam.
Google’s algorithm essentially works on the popularity of links, which is how it determines relevance. People can game this algorithm, because someone can make a website popular to manipulate rankings through linking from fake sites and other optimisations. But Google’s pagerank algorithm is assuming that relevant results are, at their core, purely about popularity. The innovation the Google guys brought to the world of search is something to be applauded for, but the extreme lack of innovation in this area since just shows how hard it is to come up with new ways of making something relevant. Popularity is a smart way of determining relevance (because most people would like it) – but since that can be gamed, it no longer is.
The semantic web
I still don’t quite understand why people don’t realise the potential for the semantic web, something I go on about over and over again (maybe not on this blog – maybe it’s time I did). But if it is something that is going to change search, it will be that – because the semantic web will structure data – moving away from the document approach that webpages represent and more towards the data approach that resembles a database table. It may not be able to make results more relevant to your personal interests, but it will better understand the sources of data that make up the search results, and can match it up to whatever constructs you present it.
Like Google’s page rank, the semantic web will require human’s to structure data, which a machine will then make inferences – similar to how Pagerank makes inferences based on what links people make. However Scoble’s claim that humans can overtake a machine is silly – yes humans have a much higher intellect and are better at filtering, but they in no way can match the speed and power of a machine. Once the semantic web gets into full gear a few years from now, humans will have trained the machine to think – and it can then do the filtering for us.
Human intelligence will be crucial for the future of search – but not in the way Mahalo does it which is like manually categorising pieces of paper into a file cabinet – which is not sustainable. A bit like how when the painters of the Sydney harbour bridge finish painting it, they have to start all over again because the other side is already starting to rust again. Once we can train a machine that for example, a dog is an animal, that has four legs and makes a sound like “woof” – the machine can then act on our behalf, like a trained animal, and go fetch what we want; how those paper documents are stored will now be irrelevant and the machine can do the sorting for us.
The Google killer of the future will be the people that can convert the knowledge on the world wide web into information readeable by computers, to create this (weak) form of artificial intelligence. Now that’s where it gets interesting.
When I was in Prague two years ago, I met a bloke from Bristol (UK) that very convincingly explained how patents as a concept, are stupid. Because alcohol was involved, I can’t recall his actual argument, but it has since made me question: do you really need a patent to protect your business idea?
Narendra Rocherolle, an experienced entrepreneur, has written a good little article explaining when you should, and shouldn’t, spend money to protect your IP. Racherolle offers a good analysis, but I am going to extend it by stating that a patent can be dangerous for your business, and not just because of the monetary cost. Radar Networks is my case-study – a stealth-mode “Semantic web” company, that has received a lot of press lately because apparently they are doing something big but they are not going to tell us until later this year.
One of the highlights at Bar camp Sydney, held on March 3rd 2007, was a presentation by Martin Wells and Mike Cannon-Brookes on “How to Start a company”. Both men have a lot of wisdom to share, which was worth every penny (no pun on the fact the event was free). However despite an awesome presentation that covered a lot of ground, there was one slide that in particular annoyed me. It bothered me so much that I wanted to say so, but I thought it might be better to let it be because the guys were doing an otherwise great job.
My problem was slide number two. It listed four companies as ‘ideal’ start-up businesses for all those in the room. Those companies were Flickr, Del.icio.us, YouTube, and MyBlogLog. Why I had a problem with that, was because if that is what web entrepreneurs are being told to look up to, then we have a bit of a problem.