I’ve been reading this book called “Where good ideas come from – The Natural History of Innovation” by Steven Johnson where the author attempts to unlock the secrets of generating great ideas through some of the history’s greatest innovations. The central theme of the book is that a great idea is definitely not a result of an instance we call as a ‘flash’ or an ‘epiphany’ or a ‘eureka’ moment that we all have come to romanticize with but rather a dull, time consuming hunch that gets a great deal of  help from our social networks/environments . The author then goes on to explain 7 basic rules needed for successful innovation through accounts of our history’s greatest inventions.

One of the rules that the author explains and that caught my attention is that of Serendipity. To begin with, there is something really fairy tale-like, if that’s the right way to describe it, about the way the word sounds [ser-uhn-dip-i-tee]. The fact is that the word has its roots from an actual Persian fairy tale called “The Three Princes of Serendip” whose protagonists always make discoveries by accidents and sagacity, of things they were not in quest of [Wikipedia]. The vernacular usage of the word today is in the same vein with the dictionary stating the meaning of the word as ‘the aptitude for making desirable discoveries by accident’.  The role that Serendipity played in the advancement of science and technology is unfathomable especially in the field of natural and biological sciences.

As kids most of us have learnt the basic understanding of the meanings of the words ‘discovery’ and ‘invention’. We understood that discovery is the learning of anything abstract that was already existent, so it was an act of knowledge building just that you are the first person learning about it while invention is the act of creating anything that was previously non-existent.  Organizations today in the effort to sustain competitive advantage in the market have learnt to encourage their employees to be more creative by fostering an ‘Innovative’ environment. So what is the meaning of Innovation? Well, unfortunately the dictionaries don’t give us a clear picture. They say that the meaning of the word is ‘the act of innovating’ or ‘the introduction of a new method or idea’ etc., I’m neither a linguist nor an authority on the English language but if you asked me to define it purely from the standpoint of understanding the term better, I would say that Innovation is a combination of consequences of the acts of both ‘discovery’ and ‘invention’. More in mathematical terms you could say,

Innovation = f (Discovery+Invention)

The definition I’ve assumed for ‘Discovery’ in the above statement is a bit of a white lie. Here the definition assumed is the learning of a pre-existent fact otherwise not known by ‘you’ alone and not necessarily to anybody else.  So in other words the definition for Discovery being used here is purely personal knowledge building. This is where Serendipity comes into picture. The author of this book says that accidental learning of a previously unknown fact has a potential to spark new connections between the neurons in your brain which lead to the generation of new ideas which could result in great innovations.  Remember, I’m not talking about intentional learning where you sign up for a course at college and have expectations about what you’ll learn, I’m talking about accidental learning where you come to the library to find a book on ‘programming’ but pulled a book on ‘global warming’ from the rack because you liked the title of the book etc.,

So are our modern day protocols well equipped to foster our creative spaces by allowing for this form of accidental learning? Are we positioning ourselves for Serendipity? We got to think about this.

I mean what do we do if we want information on something? We go to our favourite search engine, use a couple of keywords as queries and bingo! 9/10 times we find what we want. And for all you know the miss hit was because you did not use the right keyword in the first place. This protocol completely is counterintuitive to the idea of accidental learning. You intend to learn about something and thanks to the search engines you get what you want.  One can argue that these days there is  a great deal of knowledge sharing that happens on your social networks but even in those cases it is still not accidental learning, you are just exposed to what your friends have found and most often they share the same values or interests with you.

Think about reading a newspaper and enroute to the sports section, while flipping pages, you find an interesting article on global warming and suddenly that reminds you that you had to pay your electricity bill and today was the last day! That is serendipity. I have no argument on the fact that the web is a gold mine of information both in terms of quantity and variety but there hasn’t been any solution or software that could emulate the feeling of serendipity as much as our libraries or news papers are positioned to. As mentioned before, Serendipity has played a significant role in advancing science and technology often by creating spaces for innovation and it will continue into the future. History of course does not have any account of this as it concentrates more on the often fabricated ‘aha’ or ‘eureka’ moments of innovations.

I’ve come across this nifty plugin for gmail called Rapportive that adds a nice touch to your email experience. What this plugin does is searches the publicly available social profiles of the person and displays that information in your inbox while you are still reading the email.

The information includes a picture of the person plus links to the facebook/twitter/linkedin/Google profiles all together in the side bar where the Google ads used to appear (Curious to hear what Google thinks of this). This feature is extremely useful as it allows you to make new connections with relevant people of interest which would not have been as flexible and certainly less probable if not for this feature. What’s more? You can add a   comment about this person right there in your inbox which can aid as a memory assistant for future conversations with this person. This means you got a nice little social CRM tool running right in your email. Check the screenshot.

I’ve been using this plugin for a couple of weeks and it already has proved to be useful in making a couple of fruitful connections that I otherwise would not have discovered for sure.  The company started in early 2010 and is well funded by some top notch investors, one of them being Paul Buchheit, the creator of Gmail. Rapportive is looking forward to adding new services and expanding soon to other email clients. They also have a pretty busy crowdsourcing/feedback tool which gives us a nice overview of what features are being worked on/planned for the future etc.,

Give it a shot and install it’s definitely a keeper.

This post is a result of a random observation that I’ve made over the past couple of weeks. Targeted advertising has proven to be hugely successful both for consumers, advertisers and more importantly for companies like Google that have been the pioneers of the concept. Starting with ‘text’ ads to the current rave about ‘display’ advertising to the future which is ‘location based’ advertising, targeted advertising has revolutionized the whole e-commerce space. Here are a few thoughts on status quo and future directions.

Question # 1: Is ‘targeted’ advertising ‘smart’ advertising?

Answer: Indeed, incredibly smart. Apart from the fact that it is ‘targeted’ as it generates relevant advertisements triggered by the user’s queries here are a few more reasons.

• Aware of the user’s geographic location through the IP address and can micro-target an audience pretty much to a radius of a few kilometers.
• Based on the demographics – Gender, Age, Status etc.,
• Recognize the device from which the query is being fired and return ads accordingly,
• Using the data from services like personal search or clicks and predict the behavior of an individual and serve ads accordingly
• Using data from your social graph for example, if you ‘liked’ a product on facebook or the type of places you’ve checked in on ‘foursquare’ etc.,

Question # 2: Can ‘targeted’ advertising grow even smarter?

Answer: Again, indeed. Consider the following scenarios.

I’ve been observing a deluge of Chrome ads these days …both as video embedded text ads and single block text ads on ‘Youtube’. I like chrome and it is my favorite browser but is Google smart enough not to push these ads onto somebody already using chrome? No, at least not yet. With a growing population of people online using chrome this practice, if not corrected, will only prove increasingly fruitless. This stems a new question. Why can’t Google detect the browser information and trigger ads accordingly? I can imagine a world of opportunities if this happens.  Different browsers have different capabilities and can run applications or services that are aligned to those capabilities etc.,

I’ve also been observing another set of ads that Google has been presenting. The query site:[DOMAIN NAME.COM] triggers an ad for  ‘Google Webmaster Tools’.  But the ad gets triggered only for some domain names.  Quick research seems to reveal that these ads don’t appear for the supposedly chosen few ‘white hat’ or well know websites (though I could be quickly rebuked on that), which is fine. For the rest, the problem is that Google doesn’t seem to be checking whether the domain name used in the query already is using the advertised application or not, in this case Google Webmaster Tools.  Again, this is targeted advertising but definitely not smart enough. Why waste so many impressions when you can save on them by putting a check in place to confirm whether the user is already using Google Webmaster Tools, especially if the user is making the queries logged in ?

Now, I do understand that often these changes are not as easy as I’m proposing them and that there could either be a ton of technical constraints that are acting as bottlenecks or simply the effort v/s outcome tradeoff is not worth it, at least not yet.  But I see incredible potential here especially as the idea of OpenSocial and OAuth become more popular and with so many companies opening up their hoods through the APIs, advertising for these very applications can make use of this development and gain an understanding of what applications a user has already been using and present ads in a more smarter way.

Cloud computing is the definitely the buzz word these days. The recent global financial meltdown is said to be a significant reason for the manifestation of this new paradigm for obvious reasons of cost efficiency, flexibility, scalability blah, blah & blah. But many in the academic and research fraternity do not agree with the above point. Their argument is that like any other innovation the time has come for Cloud Computing to transcend from the innovation phase to the Utility phase.

So, according to ‘them’ the global meltdown fuelling the use of this paradigm is just no more than a coincidence. There is a ton of research being done in various fields like Technology Adoption, Information Security and Organizational Behavior. Like any other innovation there are early adopters of technology and later adopters of the technology. The early adopters are usually characterized by their risk-seeking or entrepreneurial mindsets and late adopters by their risk-averse mindsets. What is interesting is to see if the above mentioned mindset, let’s call it ‘Perceived Risk’ linearly reduces with time or not and what factors mediate this relationship. This definitely will have implications on the success or failure of an innovation and can be useful to innovators and technology adopters alike. There is an awesome video (2 parts) from a CloudCamp session in Frankfurt where Simon Wardley from Canonical, Sponsors for Ubuntu talks about the state of confusion when it comes to understanding what Cloud Computing really is. He describes how this paradigm of computing wasn’t realized overnight, how it traveled through the different phases that any innovation usually does and why its’ time really has come. Totally recommend it.

Lectures, assignments, labs and learning…..It’s been an invigorating and enriching two months of ‘back to basics’ for me. After 4.5 years of enslaving your mind to corporate duties, I really needed this break and I’m glad I got it.

Since the time I revealed my intention of pursuing the highest of all ‘academic accolades’, I got a lot of people who started asking questions/directions to help them get’ on board’. I did my bit of customizing my advice depending upon the level of awareness of the inquirer, but I thought it will be useful to churn up a write up which could be considered as sought of a primer on ‘How to get admitted into a Doctoral degree’ . Of course what will follow is a series of personal reflections and should no way be considered as an established formula.

The first thing that you need to ask yourself is why would you want to do a PhD?

• Is it because you want better life monetarily?
• If you are aiming at an International school, is it because you are attracted to higher standard of living?

Well, all I say is that getting a PhD could be necessary but not a sufficient condition to guarantee enhanced living both in terms of finance and standard of living. If you are motivated by money then you need to look at other degrees like an MBA or MS programs.

Enduring the academic rigour is going to be the main challenge. If you think that the time you spend as a student will translate into quick financial returns, again its best that you stop thinking of pursuing this avenue.

So bottom line is that only those people who are willing to sacrifice 3-4 years of their lives, irrespective of what stage they are in – fresh post-graduate, single working professional, working professional with a family etc., and people who are passionate about adding an original academic contribution to the ever growing body of knowledge should consider pursuing a PhD degree.

If you think you fit the description above then read on, for everyone else what follows is just mindless FYI.

Preparation for getting into a school:

‘A winning effort begins with preparation.’

-          Joe Gibbs

The whole process took approximately 2 years for me and this partly may have been due to the fact that I was working at the same time, nevertheless the minimum amount of time you should give yourself is a good year just for preparation. This includes giving all the required tests (GMAT, GRE, TOEFL, IELTS etc.,). Depending upon where you are planning to apply the requirements for the tests vary, please refer to individual Universities’ websites for that. Please find this useful table about the differences in the admission requirements between Universities in USA and other West European and Asia-Pacific schools

So depending upon where you want to apply your strategy changes. I decided to study in New Zealand purely because as an International Post-Graduate student my wife gets to work here for the entire duration of my course.

Roadmap and check-list with time frame for applying at the University of Auckland:

• Lodging the ‘Expression of interest’ (EOI) here: http://www.postgrad.auckland.ac.nz/doctoral/interest/ You will need a fully developed research proposal and your degree transcripts to make an EOI. This process took a good 9 weeks for me. Basically you will get an email from the school stating whether your EOI has been accepted or not. EOI is nothing but a pre-assessment of the strength of your application.  But getting through this stage is a positive sign as half the battle is won.
• Next step is to launch the formal application. The University will give you login details for you to lodge a formal application online. You will have to complement your online application with certified hard copies of your degree transcripts and certificates and sealed and signed reference letters. You can ask your referees to send the letters electronically as well. Reference letters are crucial, so please make sure that you give enough time to your referees to write the letters, as often, hastily written letters may not give a holistic description of your abilities of worse leave out important aspects of your abilities. Once you complete your formal application and the University receives your documents it should take another 9 weeks before they come back to you with a result.
• Assuming that they do return with a positive result, the next important step is the visa process. This for me by far was the most frustrating and infuriating experience as nothing realistically is under your control. The first requirement is you need a Police Clearance Certificate (PCC) and for that you will have to deal with your local police. For a detailed account of my experience with this check the following link:                                                                                               http://www.biztrivia.org/blog/2009/11/getting-a-police-clearance-certificate-is-such-a-pain/

• Then when you do get it, which is like Nirvana, you quickly have to fight the uphill battle of enduring the actual Visa process. It took a bloody long 8 months to get my visa, for reasons, well I simply don’t know. All visa applications lodged with Immigration New Zealand, though you apply in your home country, are sent to this clandestine branch called IPG (Immigration profiling group) in Wellington in New Zealand.  Neither your University nor anyone else can communicate with these folks. So you just have to wait, wait and wait. A Visa officer gets allocated to my application after about 3 months of lodging the application and then it takes another good 4 months to get the Visa. Here is the link to a forum where people with similar frustrations came together to discuss the Visa process: http://forums.immigration.com/showthread.php?305396-NZ-PhD-Student-Visa-5-Month-Processing-time

I hope that this write up would prove to be useful to all who are dilly dallying on the topic of pursuing a PhD and would make a realistic decision before they make the final leap. Again, this by no means is complete and is heavily biased towards the application and visa process in the context of a New Zealand University.

In general any visual representation of data is the most efficient form of putting your point across. A picture is worth a thousand words. It is true, not only in the art world but also in the business & academic worlds. Graphs, all sorts of them, stem, box, scatter, histogram, bar, pie & 3D are all used as windows for looking deep into the otherwise shallow looking & less informative pool of data points. Graphs have the power to harness the might of the human neural network to discover new dimensions which otherwise would always be inherent.  This is exactly why one should be really careful while choosing the type of graph in order to paint the ‘best picture’

Many online & offline magazines make use of Pictograms and 3D plots in order to present data to the public. Pictograms and 3D plots though may look very appealing can usually mislead us while interpreting the data. A couple of examples of such cases are shown below.

The above example is a pictogram that aims to show the distribution of advertising budgets across TIME, Newsweek & US News. What is clear is that TIME attracts the largest amount of advertising spending followed by Newsweek & then US News. What is interesting to note is that having a look at the figures quoted in the picture tells you that the  ratio of TIME’s to Newsweek’s share is around 1.6 and that of TIME/US News is around 2.9. This means that TIME should be only about 1.6 times larger than Newsweek and only 2.9 times larger that US News. But according to the pictogram TIME looks definitely more than the above ratios. What is this visual discrepancy?

Well what actually happened is that the picture in the pictogram was magnified and used for the three data points. While magnifying the picture both the height and width have been increased proportionally to avoid distortion. This means that the actual increase in size between the TIME’s figure and Newsweek’s figure is actually 2.56 (1.6*1.6, accounting for both height & width). Similarly the increase in size between Time’s figure and US News figure is 8.4 instead of 2.9.

So, our eyes, which capture the area of the pens rather than only the height, are misled to think that TIME’s share of budgets is way ahead of its competitors while it actually is not

The following is another example of an unnecessary use of 3D plot to present the data.

The aim of this graph is to showcase the drastic increase in the number of Starbucks outlets around the world. If you look at the graph carefully between 1999 & 2003 there is an increase from about 2000 stores to a little more than 6000. This is effectively an increase of about 200%. But the from the graph below it appears that the increase is much bigger than what the actual numbers suggest.

Using a 3D graphs brings in with it another problem, the third dimension. So when a 3D picture is magnified the height, width & breadth are all simultaneously increase to avoid distortion. This is the reason the picture looks much bigger than the 200% increase suggested by the numbers.

So essentially make use of the traditional 2D plots for the best and most accurate representation of your data.

Data mining as a field of significant academic research  is on an exponential curve. Though the theoretical underpinnings of this concept have been around for a while the practical use of this field quickly reached a level where researchers and businesses are finding great value.  Information explosion due to the success of the world wide web, general purchasing power of the common man, growing population, Globalization of commerce are only a few of the various reasons for this field to gain such popularity in such a short time.

Initially there was a lot of confusion about the difference between the fields of Inferential Statistics and Data Mining and researchers around the world immediately started working on producing tons of literature to address this issue. Though there is no single universally accepted definition for either of them, many researchers have defined these fields through their individual application perspectives. From what I’ve gathered and known about these fields, this is how I draw a line between them.

Difference between Data mining & Inferential Statistics:

• Data Mining is a field where you discover hidden patterns from already existing large data sets. These hidden patterns discovered are later used for analysis and decision making scenarios in the area of concern. The process of using the discovered hidden patterns is also called as ‘Knowledge Discovery’. Inferential Statistics is the field where you prove or refute a pre-conceived hypothesis (or a null hypothesis) by performing classical statistical methods on a sample of a given population size.
• Data mining starts at an already existing database (usually large datasets) and Inferential Statistics generates its own database using sampling methods on data set.
• Data mining methods employed like classification, clustering, etc., scan the entire dataset in search of hidden patterns while classical statistical methods are run over only a small section of the dataset (the sample).

The above point also infers that Data mining methods are more computer intensive as they have to run through large data sets and hence should be used only when really needed.

So when do you use data mining over inferential statistics?  Well the answer is simple. If you don’t know what you are looking but want to make the best sense of the data you have then use data mining. If you know what you are looking for and want to back it up with proof by checking the data you have you should use inferential statistics.

Now,  I know that your Doctor/Mom/ and the general ‘elite’ of the society are usually against the notion that there acutally can be something useful that could be derived from a fast food joint. But trust me on this, there is something that I did find something like that.  Something that I probably would not ever find even after years of searching.  No its not all the yummy looking/smelling but not good for health food but its the bunch of people that work as a team that is holding the joint together.  Fast food joints are the best places on this planet to learn about an important trait that young starters need. – Team Work.

I’m completely convinced that Fast food joints are the best places for the young folk that most throng these joints to take a few lessons on Team Work. This is not a random observation. This is a hypothesis that I proved to be true after years of collecting data of statistical significance . No, seriously jokes apart I’ve been visiting this fast food joint since a loong time and I’ve visited at random times, a good combination of early mornings, late mornings, evenings and on both on weekdays and weekends. I can say that even at peak times never was  my order  held for more than 15 minutes, averaging at around 7 minutes in general. Believe me when I say peak time I’m speaking about at least around 10 orders being placed per minute and sometimes each order in bulk, Ex: 10 plates idli, 5 plates Dosa etc., Another observation made was that its always the same 4 people who takes the oders at the counter and then another couple of people inside the kitchen actually executing the order. What is another constant is there is continual chaos at these food joints irrespective of the time of the day. With all these unfavourable parameters one would would think how these folks have been managing to sustain its customers for all these years. The answer is simple ! They work as a team. They have all the qualities that an efficient team should have. I’ve read a few online resources that talk about the traits of a great team and when I tried to align these traits along with the few observations that I made at these fast food joint folks I actually got a good overlap of these traits.

These are some of the often found traits of members of a ‘great’ team:

1. Sharing of responsibilities & rotating roles as needed.

2. Showing of support, respect and trust.

3. Great  understanding, immaculate communication and collaboration

4. Having a unified goal oriented approach in their actions/work.

Another very inspiring story is that of the mumbai ‘Dabbawallahs’ whose business model was studied by big business schools around the world. You can find more information on them here:

http://www.mydabbawala.com/

India –  the second largest nation in terms of population,

– the seventh largest nation in terms of area,

– one of the fastest growing GDPs in the world,

– more than 500o years of rich cultural history,

– one of the world’s highest producers of engineers & doctors,

– a nuclear superpower and a soon to be economic super power,

– largest democracy in the world,

and the list goes on…

All these are indeed things to be proud of, things that can instill an instant dose of patriotism in anyone. But all of this, everything stated above  just vanishes into thin air when you see that things really don’t work the way its supposed to at the grass root level. And I put the entire blame on the Indian Bureaucracy. This stinking organization has been pulling India back 2 steps for every 1 step that it puts forward. Rajiv Gandhi once openly stated that for every rupee that the Government puts aside to spend on its people, only 15 paise or 15% of it actually comes through. This is an observation that was made in the year 1988. 20 years since a lot of things have changed , most of it superficially. You try to dig a bit deep and you see the dirty side of the country.

Now a bit more context to my random ranting. I’ve recently had to deal with a government organization for my cousin’s personal work. The job at hand was to obtain a police clearance certificate for this cousin living in another country to extend his visa. Initially, I was glad to get to know that there was an online application to be submitted. Progress ! right ??? Wrong what I had to do really is after I’ve submitted that application online, I still had to go to the office to submit the printed version of the online application form. Duhh!! Anyway, I did that assuming that it was just an 7 day ordeal as the website stated that the certificate could be collected on the 7th day after submitting the application. After the police verification was done it is the 3rd week now and I have received it yet and every time I call them to check  the status, the information I get is that the report has not even reached them. When I ask them what the reason is, I get the most insensitive answer. I mean how difficult is it for a citizen of this country, who has not had even a single complaint against him ever, has all the right documentation to get a clearance letter from the local police stating his good conduct?? Well, it is clear that it is very very difficult. My friends and parents say that if I bribe them my work would mover faster, why should I give any extra money to anybody?

I understand many people love this country and would have a bucket full of great advices for me to join the ‘system’ and be upright and shine as an example. Why should I ? I mean I don’t like to be part of the system, its a personal choice. Whats wrong with that? If everyone who has my opinion does the same thing then you will have a billion people who would be looking for jobs in this organization. I wonder what will change us, it cannot be organizational , it has to be spiritual. God save us.

I’ve been reading a lot about Click Fraud lately and its been really interesting. I wanted to share a bit of my perspective on the whole issue and point you guys to some useful resources that I’ve found extremely useful. There are tons of news articles that project this concept as theDark side of the online advertising . Many also blame that efforts to mitigate this type of fraud is not in the interest of the Ad Networks as increased clicks result in incremental revenue. The key to understanding the implications of click fraud, as it is in any other concept, is in having increased awareness of the issue.

Click Forensics, a leading solutions provider for improving traffic quality for online ad campaigns, reported that though there is an overall dip in click fraud rate in Q2 2009 over Q1, click fraud traffic from sophisticated sources and scripted programs rose again in Q2 2009. This is evident from the rise of Publisher Collusion Fraud where publishers are employing sophisticated IP rotation methods or botnets  in oder to inflate their commissions.

There is a huge incentive for the Ad Networks and Search Engines to look at this as a serious concern because any hint of discomfort advertiser will lead the advertiser and/or publisher to defect from the system which will eventually lead to revenue losses.  Infact, Click Forensics President, Tom Cuthbert states that the increased diligence from the Ad Networks and Search Engines is the main reason for the click fraud rates to steadily decrease over the past few quarters. Google for example has set up a resource center to raise awareness about this issue. All the docs, presos and videos at this link are excellent reads on the topic.

So, from a technology standpoint, Search Engines and Ad Networks are doing their bit by developing tools and techniques in order to mitigate the effects of click fraud. What is lacking though are studies that look at user, publisher and advertiser behavior towards click fraud. Do they know what click fraud is? How do they validate erratic behavior on their campaigns? Is there a certain level of awareness to help them make informed decisions ?etc., There is enough proof to tell you that often advertisers complain that a seemingly erratic behavior in their campaigns is click fraud while the real reasons for such a behavior might be something completely different. Conducting research hence is important for the Search Engine  and Ad Networks to develop a strategy around increasing awareness around this issue that actually is effective.