The following is an analysis of the merits of the Online Reputation Scoring™ (ORS™), a metric based on the Net Promoter Score™ which is used for measuring and benchmarking the commercial persuasiveness of search engine results and other content items displayed to a web user.

Since marketers need metrics to demonstrate the progress being made in the consideration phase of the marketing cycle, there needs to be a robust metric that can be used to show what success looks like.

Marketing Cycle

For the purposes of this article, we see the marketing cycle as three phases:

Brand stimulus

  1. Open to stimulus
  2. Outcome: ambient awareness, preference for certain brands set
  3. Tracking: very indirect attribution. Very vague

Consideration

  1. Informational: Information seeking and looking for answers and reassurance
  2. Buying questions answered, buying decision formed
  3. Tracking: Indirect attribution. Reasonably accurate

 Purchase

  1. Transactional: Looking for best vendor
  2. Outcome: buying decision made and executed
  3. Tracking: Direct attribution. Very accurate
buying cycle simple

Buying Cycle

 

 

Online as Direct Marketing

Online marketing is seen mainly as way of direct marketing [1] because of its ability to communicate directly with the customer and because of tracking capabilities online, it’s relatively easy to build accurate attribution for online marketing spend.

Because of Adwords relevance and accurate attribution, Adwords accounts for about 49% of all online advertising spend in the United States [2] , yet Adwords only accounts for about 6% of all click through from search engines [3].

Makeup of search traffic

For all search traffic, it is reckoned that the approximate ratio of query types is approximately [4][5]:

  1. Transactional Queries: 7.06%
  2. Navigational queries: 24.53%
  3. Informational queries: 68.41%

(These figures more or less reconcile with the Adwords overall click through.)

Conclusion

The problem facing online marketers is having a solid metric that clearly demonstrates the extent to which a consumer is being persuaded to buy in the consideration phase of the marketing cycle where attribution is harder to determine. In other words scoring the persuasiveness of content and search results online allows marketers have a benchmark metric to plan and monitor with.

Understanding the Online Reputation Scoring

In order to come up with a relevant scoring system for persuasiveness, we referred to the Net Promoter Score which is used in analysing the sentiment around a brand.

It works by typically asking: “How likely are you to recommend (our brand ) to a colleague or friend?”. The percentage of negative responses are subtracted from the percentage of positive responses and a percentile score at arrived at. [6] This Infographic gives more background information to NPS™

NPS-en

Caption: How Net Promoter Score Works (Source)

Online Reputation Scoring™ is a variant of NPS™ and works by typically asking: “How likely are you to purchase (our brand) as a result of what you see?”

ops

Direct comparison between ORS and NPS

For comparison we have a table showing how the scoring is calibrated:

2013-10-21_093343

 

Example Tables

Basic Formula for ORS™ (same formula is used for NPS®)

2013-10-21_093743

This example demonstrates the idea some more: The Online Reputation Scoring™ would therefore be 30% – 20% = ORS™: 10

2013-10-21_094039

We have three examples of skincare products where we have done ORS™ scoring:Examples of ORS in search results and reviews

  1. Boots Protect and Perfect: Keyword ‘protect and perfect’ (Search query here)
  2. Creme De La Mere skin creams: Keyword ‘creme de la mer’ (Search query here)
  3. Clinique Repairwear skin cream: Keyword ‘repairwear lift’ (Search query here)

We examined each result and tabulated them.

We have also scored a set of reviews which demonstrate how ORS™ can be used in this context. (original reviews HERE)

 

Boots Protect and Perfect ORS: 100

Commentary:

This is a massively popular brand which has had the first reliable clinical trial of any anti-wrinkle cream available on the high street, which suggests that it really does help to reduce wrinkles.

When you review the search results they are generally very popular with authority sites claiming its efficacy and review sites also endorsing this product. This is why it scores with an ORS of 100

Note: that we discount the adverts for the purposes of scoring because they are most typically neutral.

  1. Boots. Review content accessible and positive. Hence a score of 9 (Persuasive)
  2. Boots. Review content accessible and positive. Hence a score of 9 (Persuasive)
  3. Telegraph. Authority site endorsing the product compellingly. Hence a score of 10 (Persuasive)
  4. Mumsnet. Forum thread endorsing product. Hence a score of 10 (Persuasive)
  5. Daily Mail forum thread endorsing product. Hence a score of 9 (Persuasive)
  6. Guardian. Authority site endorsing the product compellingly. Hence a score of 10 (Persuasive)
  7. Thisismoney. Authority site endorsing the product compellingly. Hence a score of 9 (Persuasive)
  8. Amazon. Only 1 review, but from a verified purchaser. Hence a score of 9 (Persuasive)
  9. Marie Claire.Authority site endorsing the product compellingly. Hence a score of 9 (Persuasive)
  10. MellowMummy. Site endorsing the product compellingly. Hence a score of 9 (Persuasive)

Boots keyword ‘protect and perfect’: ORS : 100

 

Screengrab

ops - protect perfect

Boots Protect & Perfect (Click to see full size image)

 

Crème da la Mer ORS  -10

Commentary

This is a very expensive product with apparently no scientific claims set against it. The ORS™ scores demonstrate the lack of information validating the effectiveness of the product and the damming article from the Daily Mail. Note: that we discount the adverts for the purposes of scoring because they are most typically neutral Positions:

  1. Brand Site in No1 has no review content, hence a score of 7 (neutral)
  2. Brand site No2 has no review content, hence a score of 7 (neutral)
  3. Daily mail article stating that ingredients probably cost £25 for a £530 pot of the Crème de la Mer cream
  4. John Lewis. No review content , hence a score of 7 (neutral)
  5. Houseof Frasier. The reviews are not prominent and are very hence a score of 7 (neutral)
  6. Selfridges. No review content , hence a score of 7 (neutral)
  7. Ebay. No review content , hence a score of 7 (neutral)
  8. Urban retreat. Inaccessible review content which is too effusive, hence a score of 7 (neutral)
  9. Harvey Nichols. No review content , hence a score of 7 (neutral)
  10. Harrods. No review content , hence a score of 7 (neutral)

Crème de la Mer: keyword ‘crème de la mere’: ORS™ : -10

Screengrab 

ops - creme de la mere

 

Clinique Repairwear ORS: 10

Commentary

This is a mid priced product with no specific scientific claims in place. Interestingly Clinique has another product similar to this and discontinued it in favour of Repairwear Lift which has had a very mixed reception.

Note: that we discount the adverts for the purposes of scoring because they are most typically neutral.

Positions:

  1. Clinique. Brand Site in No1 has positive review content, hence a score of 9 (Persuasive)
  2. Ebay. No review content , hence a score of 7 (neutral)
  3. Makeup alley. Reviews scoring 3/5 which we see as slightly negative, hence a score of 6 (Dissuasive)
  4. Totalbeauty. Reviews scoring 3.9/10 which we see as negative, hence a score of 4 (Dissuasive)
  5. Clique reviews. Reviews scoring 3.5/5 which we see as neutral, hence a score of 8 (Neutral)
  6. Youtube. Positive and credible review of product , hence a score of 9 (Persuasive)
  7. Dailymed. Ingredients information, hence a score of 7 (Neutral)
  8. Paula’s Choice. No review content, hence a score of 7 (Neutral)
  9. Slideshare slides with coupons. No review content, hence a score of 7 (Neutral)
  10. Amazon. Only 1 review, but from a verified purchaser. Hence a score of 9 (Persuasive)

Clinique: keyword ‘repairwear lift’: ORS™ : 10

ops- repair wear lift

Review scoring using ORS™

By applying a persuasiveness score to reviews we can score this for ORS™
Sample below (Original reviews HERE & Screengrab HERE)
Star ratings gave the product 3.9 out of 10 and ORS™ gives it -43.8

Screengrab

2015-03-27_092357

 

Table of scores for the reviews

Opportunities for optimizing ORS

  1. Take into account different Click through rates for search engine positions
  2. Take into account that a negative sentiments influences purchasing decisions more than positive
  3. Use a numeric score to express how persuasive or dissuasive a page is
  4. Take into account presence of additional elements on a Google results page, such as “6 pack”, items on the right half of the page, etc.
  5. Take into account other suboptimal search results, such as affiliates, who despite improving your brand image, may still cost you more than you would want.
  6. Optimize scoring by using multiple raters and combining their scores for more reliability

We have undertaken a research project evaluating which method of scoring ORS would be most effective. One variant was to score by weighting the actual scoring of the item in question. The following explains this idea in more detail.

A detailed example of a weighted ORS scoring system.

2013-10-21_092908

We calculate the weighted ORS™ score by scoring everything out of 10. If a review or a search result qualifies as “This is very interesting and I may well buy because of this”, it might be scored as a ‘9’ because it’s genuinely persuasive. It will be included in the bucket of persuasive results and scored as ‘1’ because we subtract the ‘9’ rating from the maximum score of 10.

This down weights positive results.

For a review or a search result that qualifies as “I don’t think I will buy based on what I see” it might be scored as a ‘4’ because the information presented is just not very persuading. It would be included in the bucket of dissuading results and scored as ‘6’ because we subtract the ‘4’ from the maximum score 10.

This up weights negative results, which correlates with the power of dissuading information.

Since the much of the strength of the ORS™ scores are in the consistent way in which they are scored, thus allowing for benchmarking against comparable results, it’s is best to go for the simplest interpretation of ORS if order to build a bank of comparable results. This is why in search results we do not account for click through rate or adverts because we cannot get sufficiently robust data to be consistently confident about the results they provide.

Examples of ORS usage

ORS can be applied to:

  1. Search results on a search engine
  2. Sets of reviews
  3. Or any instance where there are separate pieces of content which vary in persuasiveness that can be analysed
  1. Tweets
  2. Forum posts
  3. Comment threads

How ORS solves the problem of measuring persuasiveness

ORS™ gives us a set framework for making comparisons between a brand’s marketing activity and a competitor’s activity. It also acts as a guide showing the progression of marketing activity over time by showing the gradual increase of ORS™, or decrease depending on the effectiveness of the brands activities.

It also gives listings of reviews more dimension because it helps overcome the up weighting found with fake reviews that rate only a maximum score.

Conclusion

By having a metric that is only focused on the question of whether something persuades a consumer to buy a product or service, we can:

  1. Benchmark based on a consistent metric and method of evaluation
  2. Make comparisons between the marketing activities of brands
  3. Make it easier to optimise marketing activities

This gives comparable metrics and increased accountability to the persuasiveness of information relating to the consideration phase of the marketing funnel.

Appendix

  1. http://en.wikipedia.org/wiki/Direct_marketing
  2. http://en.wikipedia.org/wiki/Direct_marketing#Online_Tools
  3. http://econsultancy.com/uk/blog/10586-ppc-accounts-for-just-6-of-total-search-clicks-infographic
  4. http://www.seobook.com/search-taxonomy-getting-inside-mind-searcher
  5. http://searchengineland.com/study-80-percent-of-searches-are-informational-20-are-navigational-or-transactional-13745
  6. http://www.netpromoter.com/why-net-promoter/know/
Nick Garner

Nick Garner

Nick is founder of 90 Digital. Previously he was head of search at Unibet and prior to that search manager Betfair.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.