A/B Test in Digital Science?

A/B testing (bucket tests or split-run testing) is a randomized experiment with two variants, A and B. It includes application of statistical hypothesis testing or "two-sample hypothesis testing" as used in the field of statistics. 

A/B testing is a way to compare two versions of a single variable, typically by testing a subject's response to variant A against variant B, and determining which of the two variants is more effective. - From Wikipedia.

In general, A/B testing, refers to an experiment technique to determine whether a new design brings improvement, according to a chosen metric.

In web analytics, the idea is to challenge an existing version of a website (A) with a new one (B), by randomly splitting traffic and comparing metrics on each of the splits.

Netflix is a monster at A/B testing. They’ve got data scientists, design teams, and engineers working on potential service improvements all the time, and they A/B test everything before it becomes the default experience for their massive user base. They know it’s too risky not to.

The idea of A/B testing is to present different content to different user groups, gather their reactions and use the results to build strategies in the future. 

According to this blog post written by Netflix engineer Gopal Krishnan:

If you don’t capture a member’s attention within 90 seconds, that member will likely lose interest and move onto another activity. Such failed sessions could at tiWhmes be because we did not show the right content or because we did show the right content but did not provide sufficient evidence as to why our member should watch it.

Explore the boundaries. The best ideas come from many idea explorations. Some of the best ideas are sometime from the developers or the product managers after testing out our prototypes.

Observe what people do, not what they say. When talking to users, it’s important to keep this in mind: they always say one thing but do it differently.

Use data to estimate size of opportunity

• It’s always about the whys
• Data can help shape ideas
• Check if any A/B testing are in conflict

Always question your tests and never make assumptions. A/B testing is indeed a great way to alleviate human bias when deciding on relevance of new features. However, do not forget that A/B testing still relies on a model of truth: as we have seen, there are different possible models.

In brief, A/B testing is an art of data and marketing science, and also a great tool for business to understand what the end uses do and why. 

 

 

 

 

 

 

Source: Netflix and Wikipedia