Frequent thinker, occasional writer, constant smart-arse

Month: August 2007

On the future of search

Robert Scoble has put together a video presentation on how Techmeme, Facebook and Mahalo will kill Google in four years time. His basic premise is that SEO’s who game Google’s algorithm are as bad as spam (and there are some pissed SEO experts waking up today!). People like the ideas he introduces about social filtering, but on the whole – people are a bit more skeptical on his world domination theory.

There are a few good posts like Muhammad‘s on why the combo won’t prevail, but on the whole, I think everyone is missing the real issue: the whole concept of relevant results.

Relevance is personal

When I search, I am looking for answers. Scoble uses the example of searching for HDTV and makes note of the top manufacturers as something he would expect at the top of the results. For him – that’s probably what he wants to see – but for me, I want to be reading about the technology behind it. What I am trying to illustrate here is that relevance is personal.

The argument for social filtering, is that it makes it more relevant. For example, by having a bunch of my friends associated with me on my Facebook account, an inference engine can determine that if my friend called A is also friends with person B, who is friends with person C – than something I like must also be something that person C likes. When it comes to search results, that sort of social/collaborative filtering doesn’t work because relevance is complicated. The only value a social network can provide is if the content is spam or not – a yes or no type of answer – which is assuming if someone in my network has come across this content. Just because my social network can (potentially) help filter out spam, doesn’t make the search results higher quality. It just means less spam results. There is plenty of content that may be on-topic but may as well be classed as spam.

Google’s algorithm essentially works on the popularity of links, which is how it determines relevance. People can game this algorithm, because someone can make a website popular to manipulate rankings through linking from fake sites and other optimisations. But Google’s pagerank algorithm is assuming that relevant results are, at their core, purely about popularity. The innovation the Google guys brought to the world of search is something to be applauded for, but the extreme lack of innovation in this area since just shows how hard it is to come up with new ways of making something relevant. Popularity is a smart way of determining relevance (because most people would like it) – but since that can be gamed, it no longer is.

The semantic web

I still don’t quite understand why people don’t realise the potential for the semantic web, something I go on about over and over again (maybe not on this blog – maybe it’s time I did). But if it is something that is going to change search, it will be that – because the semantic web will structure data – moving away from the document approach that webpages represent and more towards the data approach that resembles a database table. It may not be able to make results more relevant to your personal interests, but it will better understand the sources of data that make up the search results, and can match it up to whatever constructs you present it.

Like Google’s page rank, the semantic web will require human’s to structure data, which a machine will then make inferences – similar to how Pagerank makes inferences based on what links people make. However Scoble’s claim that humans can overtake a machine is silly – yes humans have a much higher intellect and are better at filtering, but they in no way can match the speed and power of a machine. Once the semantic web gets into full gear a few years from now, humans will have trained the machine to think – and it can then do the filtering for us.

Human intelligence will be crucial for the future of search – but not in the way Mahalo does it which is like manually categorising pieces of paper into a file cabinet – which is not sustainable. A bit like how when the painters of the Sydney harbour bridge finish painting it, they have to start all over again because the other side is already starting to rust again. Once we can train a machine that for example, a dog is an animal, that has four legs and makes a sound like “woof” – the machine can then act on our behalf, like a trained animal, and go fetch what we want; how those paper documents are stored will now be irrelevant and the machine can do the sorting for us.

The Google killer of the future will be the people that can convert the knowledge on the world wide web into information readeable by computers, to create this (weak) form of artificial intelligence. Now that’s where it gets interesting.

Google: the ultimate ontology

A big issue with the semantic web is ontologies – the use of consistent definitions to concepts. For those that don’t understand what I’m talking about – essentially, the next evolution of the web is about making content readable by not just humans but also machines. However for a machine to understand something it reads, it needs consistent definitions. Human’s for example, are intelligent – they understand that the word “friend” is also related to the word “acquaintance”, but a computer would treat them to mean two different things. Or do they?

Just casually looking at some of my web analytics, I noticed some people landed on my site by doing a google search for how many acquaintances do people have, which took them to a popular posting of mine about how many friends people have on facebook. I’ve had a lot of visitors because of this posting, and its been an interesting case study for me on how search engines work. However today was something different from other times: I found the word acquaintance weird. I know I didn’t use that word in my posting – and when I went to the Google cache I realised something interesting: because someone linked to me using that word, the search engine replaced the word ‘friend’ with ‘acquaintances’.

acquaintances

Google’s linking mechanism is one powerful ontology generator.

BarCampSydney2

Things I learned at this BarCamp

  • It was a very different crowd from the first one.
  • It’s so easy to network – it was as difficult as breathing in, breathing out! I gave a presentation, and as a consequence, I had people throughout the day approach me and introduce themselves.
  • In the morning, collaboration was a bit of a hot theme. John Rotenstein from Atlassian asked the question of how do people define collaboration: “when two or more people work together on a business purpose”, was my answer. We agreed. Everyone else, kind of didn’t.
  • How to raise money – was the afternoon’s theme. Great points were brought up by Marty Wells, Mike Canon-Brookes and Dean McEvoy who led the discussion.
  • Some things mentioned:
  1. Aussie VC’s lead you on. “Nice idea- let’s keep in touch” is their way of not burning bridges
  2. VC’s work in a cycle that are in five or so year cycles – raise money at the beginning of the cycle
  3. Rule of thumb: give 30% away on the first round, 30% on the second round
  4. Advisor’s that give out Comet grants work on a 2% commission of future venture capital that you raise.
  5. No one understands the advertising market – everyone in the room wanted something they could read to learn more (check back here soon – I promise!). For example, Google’s adwords programme is largely supported by the property market – the mortgage lending market that is affected by the current credit crisis, is going to affect start-ups relying on adsense as the money drops out of these ads.
  • I met Jan Devos, who randomly approached me and blew me away with what he has done in his life. Basically (and from the age of 17), he created an implementation of the MPEG4 compression technology (for non-tech readers – MP4 as opposed to the older MP3) and he licenses out the technology to major consumer appliance companies like Samsung, who incorporate the technology into their products.
  • I met Dave O’Flynn – self-described as a “tall Irish red-head” developer; Matt June – a former Major in the Australian military, and now pursuing a project based around social innovation; I discovered Rai of Tangler is a commitmentphobe; Mick thinks he can skip most of BarCamp because he thinks organising a wedding is so hard; Mike Canon-Brookes over beer revealed he is a Mark Zuckerberg wannabe; and Christy Dena one of the lead (un)organisers of the conference looks completely different from the person I thought she was!

I got a positive reaction to my half hour session on five lessons I have learned on successful intrapreneurship due to a large internal project I started at my employer, with people throughout the day getting into a chat with me about it. Richard Pendergast, who is starting a online parenting site, said he was going to write a blog on one the points with his own personal battle of creating credibility. Glad I helped! I said to him I was going to blog what I talked about it so we could turn it into a discussion, but I have decided, this exam I have to sit in 12 8 days might need to start getting my attention. Anyway, here were the five points I made, however given the discussion during the session by everyone, is a very rough framework as people brought up some great points when talking:

1) It is a lot easier to seek forgiveness, than permission when doing something in an organisation. Or in other words, just do it.

2) Be proactive, never reactive. By pushing the agenda, you are framing the agenda for something that works for your project. Once you start reacting to others, your idea will die.

3) The more you let go – the bigger your idea will get. Use other people to achieve your vision. Give other people a sense of ownership in it. Let them take credit.

4) It’s all about perception. It’s amazing how much credibility you can build by simply associating your idea to other things – and which in the process, builds your own personal brand to push through with more later on.

5) Hype build hype. Get people excited, and they will carry your idea forward. People get excited when you communicate the potential, and have them realise it.

Thank you to all those involved – both the organisers and the contributors – and I look forward to the next one.

You need to be persistently adaptable

Tim Bull has recently written an interesting discussion point on when is the right time to innovate. In a post titled “Steam engine time“, he asks:

If innovation is a process of the right idea, in the right place and at the right time, how do we judge what the right time is and measure what is going on around us to hit the right spot?

Some would say luck has something to do with it, although I believe that is the perception from an outsiders point of view. In my eyes, a core set of attributes are required for innovation.

Consider this quote from Calvin Coolidge, 30th president of USA:

Nothing in the world can take the place of Persistence. Talent will not; nothing is more common than unsuccessful men with talent. Genius will not; unrewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistence and determination alone are omnipotent. The slogan ‘Press On’ has solved and always will solve the problems of the human race.

I think Tim is wrong to ask when is the right time, because innovators understand their environment, adapt to it – and then push until they get there. Persistence and adaptability, in my eyes, are two crucial aspects needed in a person or even a country or company, for it to successfully move forward. However whilst persistence is key – you need determination to push forward despite the barriers you are going to encounter – adaptability is the real secret to successfully innovating.

A case-study: multiculturalism in a flat world
Although I was born and bred in Australia, I have been brought up under a very strong Greek influence. With an Australian-born father, and a fresh-off-the-boat Greek mother – I have lived a life straddled in two cultures. Going to an Anglo-Saxon school, yet at the same time doing Greek classes at 9am Saturday (but leaving early for my schools footy games) – I grew to resent Australia’s multiculturalism policy. Without going into too much detail because this will turn it into a political discussion and detract from the point I wish to make – I disliked the fact that Greeks in Australia refused to integrate into the local culture. The Australian government’s stance of officially supporting Multiculturalism, which does things like pay for that Saturday morning tuition, was to me a stupid policy.

Fast forward to 2005, when I visited the Balkans as part of my nine months traveling around Europe. Serbia’s story is one of the saddest stories in Europe. Walking around the city of Belgrade, interacting with its inhabitants, and just generally experiencing Serbia – you realise you have come across a hidden gem in Europe. Yet once you look at the statistics and talk to some of the educated, you understand otherwise: a basket case situation that has little hope.

Serbia, like a lot of other countries I discovered in my travels, have a cultural problem: they can’t let go of the past. Millions of people have died over differing interpretations of history. The Republic of Macedonia’s identity is entirely staked on the fact they are situated on the lands of Alexander the Great. Identity to the nation states of Europe, is in history. And challenges to that history, and their identity, has led to some stupid wars affecting millions of innocent lives.

So guess what? I now think multiculturalism is the best thing my country could ever do, for the simple fact we can never have a fixed identity – what it meant to be Australian 50 years ago looks very different from what it looks like now. In Europe, identity is based on ethnicity with a fixed identity tied to history, language and a religion. In Australia, our identity isn’t allowed to be based on a certain ethnicity, and forces us to find common ground on what really matters like our way of life. If it wasn’t for the policy of Multiculturalism, we would be turning into one of these static nation states within Europe who become fixed as a certain point of time. The Greeks are still mourning over the Turks capturing the Great City of Constantinople from them in 1453 (which is why Tuesday is the unlucky day of the week for them). Yet for the countries like Australia, who don’t have much of a history – they are not locked – and consequently look forward, rather than back. Multiculturalism is a crucial ingredient to our success, because with all that diversity, it means we are constantly evolving our culture to the times without any one group fixing it. And with a globalised word, Australia’s ability to adapt to circumstances will be a key competitive advantage we have over countries.

If you don’t agree with me, have a read of Thomas Friedman’s The World is Flat – a book a entrepreneur/intrapreneur suggested I read. This guy who told me about the book was a German from Argentina, working for an Indian company to set up the company’s presence in Turkey! He told me that after he read that book, he quit his job and got himself into his current role. He faced the facts, and adapted his career.

Adaptability as success
You’re probably wondering what I am trying to get at, but to tie it back to my point about adaptability, successfully innovators need to constantly adapt to their environment. What happens with people once they get an idea, is that they spend all their time trying to fit it into a world that once existed, only for the world to be a entirely new place. Successful innovators need to constantly evolve their ideas, to the changing circumstances.

In October last year, I made a proposal at my firm to implement a new technology. For the months leading up to that point, people had to some extent talked down my idea and some even flat out rejected it. October however had me find the right person to hear my idea. And yet if I look at what I originally had thought, and what it is now – it is almost a completely different thing. Because when I pitched my idea, I was asked “why” it works and “how” is it different from anything else. It was that ‘why’ question that had me spend countless hours researching and understanding – adapting – my idea to the scenario being presented to me. I successfully made my business case, because I was given the opportunity to reframe my idea and adapt it to the circumstrances I was presented. Had I not adapted my original idea and vision, I wouldn’t be doing what I am doing now.

Of course, I could have summed up the above by mentioning Charles Darwin’s theory of evolution. Survival of the fittest, right? Adapt to the Green forest like that Green lizard that looks like a leaf, and you’ll find some food (rather than being the food yourself). Adaptability in life is a key critical success factor; and with innovation, it is the hidden factor that on the outside and in retrospect by others, gets attributed as luck.

Update 20/6/07: Catching up on some reading, I just came across a great posting by Marc Andreessen, an internet pioneer, who talks about the four types of luck and which nicely complements my thoughts above.

A casual chat with a media industry insider

Today I had the chance of picking the mind of Achilles from the International Herald Tribune, who last year was appointed Vice-President, circulation and development. Achilles is a family friend and I took the opportunity to talk to him about the world of media and the challenges being faced.

The IHT is one of the three daily financial newspapers of the world, along with the Wall Street Journal and the Financial Times. It is currently owned by the New York Times, and has a global circulation of 240,000 people. I had a great chat on a lot of different themes which could have me blog about for a week straight, but here are some of the facts I picked up from our discussion, which I will summarise below as future talking points:

  • On Murdoch’s acquisition of the Wall Street Journal: “very interested to see if he will remove the paid wall”.
  • The IHT experiemented with a paid wall for it’s opinion content, but they will be removing that later this year
  • He says the Bancroft family sold it because they are emotionally detached from the product. It was just an asset to them.
  • A lot of the content is simply reedited content from the NYT and internationalising it. For example, replacing sentences like “Kazakhstan in the size of New York state” doesn’t work well for an international reader who has no idea how big New York state is.
  • On the threat of citizen journalism with traditional media: “they are a competitive threat because we are competing for the same scarce resource: the attention of readers”
  • The problem with citizen journalism and bloggers is the validity of their information – behind a newspapers brand, is trust from readers of the large amounts of research and factchecking that occur. They have no credibility.
  • A blog may develop credibility with an audience greater than the New York Times. But this poses problems for advertising as advertisers might only advertise because of its niche audience. Blogs are spreading the advertising dollars, which is hurting everyone – it’s become decentralised and that has implications which are problematic.
  • The IHT’s circulation is spread thinly across the world. For example, it has 30,000 readers in France and six in Mauritius.
  • Their target market is largely the business traveller, which has its own unique benefits and problems. For example, a business traveler will read it for two days but when they get back home, they will revert to their normal daily newspaper. It’s not a very loyal reader.
  • Readership is a more important concept than circulation as it tells advertisers how big the actual audience of a publication is. For example, the average newspaper has 2.7 readers per copy. However due to the nature of the IHT’s readers, despite having high circulation, they have low readership.
  • IHT is in a unique position of relying on circulation revenue more than advertising. For example, a normal daily relies on circulation revenue as 20% of its total revenue; the IHT counts on it for 50%.
  • It’s hard to get advertising because a readership of university professors is less desirable than fund managers that might read the WSJ. Advertisers prefer to target key decision makers.
  • It doesn’t rely on classifieds as a revenue source – a key thing hurting the newspaper industry currently.
  • Although they place more reliance on circulation revenue, they still get some good advertising opportunities as a lot of readers are politicians and government decision makers.
  • They get a lot of advertising for fashion
  • Psychographic data is more important to advertisers than circulation and it shows what type of readership a publication has.