Need Anticipation

Please forgive me; I am here reposting a bit on Need Anticipation:

An advantage often enjoyed by e-commerce ventures not available to other types of businesses is the use of algorithms to solve technical problems too complex for humans. Historically, supply chain management, enhanced pricing discrimination, and data mining have all bolstered the income and profitability of e-commerce vendors. One promising new technology which retailers are currently investigating is consumer need anticipation - accurately predicting the desire for a good or service, and offering it at an opportune point in time. Existing data about the customer (or other customers) is leveraged to make an informed decision about what products to offer.

While marketing has long been concerned with creating the need for a product, it is important to make note of the new distinction here: consumer needs have, at the point of time of suggestion, already been created or at least lay dormant - need anticipation intends to identify the consumer's existing willingness to purchase an item and consequently offer the item. It is also important to note the possibility of this technology to build customer closeness in a way concordant to branding by allowing the customer to spend additional time browsing recommendations, allowing users to recommend products to friends, and allowing users to rate items they are interested in. These features not only keep customers on a website for longer, they give the user solid reason not to switch to a competing website (and lose the use of their ratings) and they give the user a motivation to invite their friends (to gain and give recommendations).

Amazon may well be the most successful early adopter of consumer need anticipation tactics. When a new user arrives at the site, they are greeted with the message "Hello. Sign in to get personalized recommendations. New customer? Start here." [1], along with generic recommendations of bestsellers, sale items, and seasonal items. When a user logs in, they are presented with new recommendations based on recently added products. Throughout the process of viewing products (a procedure roughly corresponding to browsing), the user is shown a histogram outlining what products were bought by users who viewed the current product, instantly showing the path of least resistance for most consumers. This entrusts that the customer can easily find the most 'purchasable' similar products. Suggestions are based on items "Frequently Bought With Items in Your Cart", "Customers Who Bought This Item Also Bought", "Frequently Bought Together" [with the currently viewed item], and "These recommendations are based on items you own and more." [2] Clearly, Amazon engineers are making use of several different easily obtained statistics to make recommendations, and it seems as though most data points are being put to use in some way or another.

Technically, the statistical approach to Amazon's approach seems to be ad-hoc and relies on the common sense translation of user data into predictive sets. Amazon does not disclose any of its algorithms, and seems mostly intent on providing for its users raw data of the type "You may like the following..." While the simple correlation of two or more items may not find a granular, specific likelihood of a user's desire for a book, Amazon's methods seem effective enough to bolster sales by offering the user a product they are likely to buy based on the purchasing patterns of similar users. Though other companies have pursued the same approach, Amazon's engineers seem to have taken the lead in accurately predicting a user's habits. [3]

Despite the success of this approach, more incisive algorithms are under development which attempt to predict a quantitative rating of a product based on prior ratings of other products: enter the Netflix Challenge[4]. Netflix has built an empire on the ability to deliver their physical product (DVD rentals) quickly, efficiently, and without hassle to the consumer, but the company is less than content to rest on their laurels. By providing anonymized data to software developers, Netflix hopes to arrive at a superior ratings prediction algorithm. Basically, entrants to the Challenge are given 100 million DVD ratings entered by 480 thousand users, and are asked to generate as accurately as possible ratings for more movies. The company justifies the $100 million US prize on the basis that the basis that "Netflix is all about connecting people to the movies they love." It is clear that Netflix is forward-looking in their desire to suggest to users movies that they will like (thereby increasing use of their service), and admirable that they are advancing the state of the art in order to increase profits. Users have so far created entries based on expert systems, neural networks, and other advanced artificial intelligence algorithms. [5] The Challenge itself is an innovative form out outsourcing that may be saving the company money: According to Vivek Ranadive, "Holding on to a competitive edge means staying one step ahead, and the more reliably one can predict the next step is often the difference between success and failure."

1 Wikipedia: Amazon

2 Amazon

3 BusinessWeek

4 http://www.netflixprize.com/rules

5 Ranadive, V. (1991). The Power to Predict. McGraw-Hill (January 2006)

2009 is Fast Approaching

...as it always has been.

I haven't read any really great predictions for 2009. What will the web see next? I have some ideas, and it's one of those times when what I think is going to happen is also what I want to happen. That, I suppose, is the definition of optimism.

I predict that in 2009, the next economies to move to the web will be professional services like medicine, law, pharmacy, engineering, and business consultancy. While of course people in these markets already find each other on the internet, I think that our changing circumstances will begin to make it cost effective and reliable to conduct these services - legal counsel, healthcare, and other informational services that require very skilled, even licensed practitioners - online and with a maximum of automation and generalization.

One of the reasons I think this is likely to happen so soon is that it is already beginning to happen, peripherally, in the form of online health records, college classes, investing sites (especially those already offering financial planning advice), and so on. I think the main drivers for this trend will be a generation of Americans slowly coming into adulthood that is able to trust the internet, a changing healthcare system in the US, and exponential growth in developing markets in countries that have fewer skilled professionals but a growing middle class able to purchase professional services over a more ubiquitous internet.

This isn't a very bold statement - sorry - but we hope that economic downturns inspire new levels of efficiency, and I'm thinking it could be the catalyst we need to move the remainder of our information services online.

I also think that the first real entirely online careers will be created - while many service jobs become automated, telepresence positions (eg. controlling a fruit-picking robot from a computer in another place or doing data entry) will become more commonplace as the manufacturing and installation of those systems plus remote labor costs falls below the local labor cost for those outsourceable but non-automatable jobs. I think it's likely that 2009 and 2010 will see a taste of this, like maybe increased use of Amazon's Mechanical Turk. The constantly falling price of computers and internet access would really be a telling factor here, I think.

Here's hoping 2009 is a good year for technology and prosperity everywhere in the world.

Things the Collective Should Do: Open Academic Web Journal

I'm suggesting here a free-content, openly accessible online repository where researchers, professors, and students publish scientific journal articles for peer review and wide distribution.

This website allows scientific research papers to be published by qualified academicians. The articles can be freely read and peer-reviewed online. Articles are translated into other languages so that they can be read worldwide. A system of of moderation provides a meritocratic means of awarding prestige and press based on quality. An accompanying print journal is provided pro bono or at nominal cost to institutions. To increase the prestige of the journal/repository, it is marketed as a trustable, progressive, intelligent institution, and content is carefully reviewed.

The current scientific publication industry relies on established branding of respectable journals and the 'publish or perish' dynamic to keep it afloat. Authors often have to pay for their articles to be published in print-bound journals, which are then sold at a high price to academic institutions. At best the publishing industry contributes little of value to the system, and at worst prevents most people from accessing information that could be useful in the hands of the general public. In essence, the current system lacks utility in spreading scientific knowledge and neither apportions prestige fairly, nor distributes knowledge widely.

Minds around the world would benefit greatly, as the results of studies would be available internationally in many different languages. Universities and authors would also benefit because they are now able to publish and access papers at lower cost. Science as a whole would benefit due to the increased volume and visibility of papers published. Because of the greater number of eyes on the articles and the increased ease of peer review, communicative openness and the scientific method would benefit.

In order to be successful, the open web journal would require buy-in from academic institutions and scientific readers. A combination of aggressive marketing and branding to entice article submissions will facilitate presenting the site as a respectable, reliable source of information. We will need to develop the website's software, decide how the site is run and edited organizationally (peer review and editing will play a huge part)

The overall progress of science will be assisted, because knowledge will be exchanged more freely. People who would otherwise not have the opportunity to read current scientific literature will have the chance to be inspired as well as educated by it. Competing with current journal models may persuade existing publishers to become more free in an economic and cultural sense. Researchers will have a website which will both distribute their knowledge to the world and grant them recognition for their work - without charging admission. Counting readers or articles would be simple metrics. Measuring changes in research job satisfaction, number of articles published worldwide, or cost of subscriptions to existing print journals would tell other sides of the story.