Same-different and A-not A tests with sensr. Same-Different and the Degree-of-Difference tests. Outline. Christine Borgen Linander

Transcription

1 Same-different and -not tests with sensr Christine Borgen Linander DTU Compute Section for Statistics Technical University of Denmark huge thank to a former colleague of mine Rune H B Christensen. ugust 20th 2015 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 1 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 2 / 23 Outline Outline Same-Different and the Degree-of-Difference tests 1 2 The -not protocol 3 Measures of sensitivity 4 The -not with sureness protocol 2 products 2 confusable stimuli: Chocolate bar (standard) B Chocolate bar with less saturated fat Setting: One pair of samples evaluated at each trial Question: re the samples the same or different? Stimuli: Same stimuli pairs: and BB Different stimuli pairs: B and B Same-Different test: Same Different Degree-of-Difference test: Same Different Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 3 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 4 / 23

2 Characteristics of the DOD test Giving answers τ criteria and the decision rule Same-Different: B Intensity B > τ different n unspecified test (like Triangle, Duo-Trio, Tetrad) Only 2 samples compared at each trial Easily understood test (by consumers) (O Mahony and Rousseau, 2002) No prior knowledge of products required (unlike -not ) Response bias (like -not ) τ τ Degree of difference: B Intensity B Intensity τ 1 τ 2 τ 3 B < τ same Rating scale: Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 5 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 6 / 23 Thurstonian model for the DOD test Thurstonian distributions: Difference distributions different same τ 3 τ 2 τ 1 τ 1 τ 2 different τ 3, BB 0.3 B σ 2 = B, B Same-different example Examples in R difference and similarity assessments δ Probability of answer in the j th category: P( j Same-pair) = f s (τ ) P( j Different-pair) = f d (τ, δ) Maximum likelihood estimation of parameters: likelihood f s (τ ) + f d (τ, δ) 0 δ Sample Response Same Different Total Same Different Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 7 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 8 / 23

3 The -not protocol The -not protocol The -not protocol Example: the -not test Situation: 2 products: and B ( not ) ssessors are familiarized with samples (and sometimes B samples as well) ssessors are served one sample either or B Question: Is the sample an or a not sample? Known as the yes-no method in Signal Detection Theory (Macmillan and Creelman, 2005) Example data: Sample Response Not- Total Not Null hypothesis, H 0 : products are similar lternative hypothesis, H : products are different Problem: There are many tests to choose from. What is the p-value? Can we reject H 0? Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 9 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 10 / 23 The -not protocol The Thurstonian model for the -not test The -not protocol Estimation of d with sensr Density θ d' Not Estimation of d with sensr: > library(sensr) > not(26, 55, 14, 55) Call: not(x1 = 26, n1 = 55, x2 = 14, n2 = 55) Results for the -Not test: Estimate Std. Error Lower Upper P-value d-prime Psychological continuuum d : Sensory difference θ: Decision threshold Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 11 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 12 / 23

4 The -not protocol Likelihood confidence intervals for -not tests Similarity testing The -not protocol > (ana <- not(26, 55, 14, 55)) Call: not(x1 = 26, n1 = 55, x2 = 14, n2 = 55) Results for the -Not test: Estimate Std. Error Lower Upper P-value d-prime Standard normal based confidence intervals: CI 95% = d ± 1.96se(d ) Improved likelihood based confidence intervals: > confint(ana) 2.5 % 97.5 % threshold d.prime im: Prove that products are identical Prove that products are identical Establish similarity within a similarity bound at some α-level How: Interchange the roles of the hypotheses: Example: H 0 : d is larger than 1 H : d is less than 1 Huge practical challenge: How to choose the similarity bound? Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 13 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 14 / 23 The -not protocol Similarity testing with d for -not tests Measures of sensitivity Discrimination measures in equal-variance models Use d for similarity testing: Hypotheses: H 0 : d is larger than 1 H : d is less than 1 p-value = P(Z < (d d 0 )/se(d ) H 0 ): > ## The Wald statistic: > statistic <- ( )/ > ## Compute p-value: > pnorm(statistic, lower.tail=true) [1] d = (µ 2 µ 1 )/σ is the (relative) distance between normal distributions λ = 2Φ( d /2) is the distribution overlap (0 < λ 1) Sensitivity, S = P(x 1 < x 2 ) = Φ(d / 2) is the probability that a random sample from the low-intensity distribution has a lower intensity than a random sample from the high-intensity distribution overlap d.prime UC Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 15 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 16 / 23

5 Measures of sensitivity The Receiver Operating Characteristic (ROC) curve The -not with sureness protocol Basics of the -not with sureness protocol What is a ROC curve? visual description of the discriminative ability central concept in Signal Detection Theory ( plot of False positive ratio True positive ratio (alt. Hit rate False-alarm Rate) The real number of interest the area under the ROC curve, UC: Examples in R S = UC = Φ(d / 2) nswers are given on a sureness scale with J categories The model assumes J 1 thresholds are adopted by the assessors Multinomial response: several ordered response categories Many parameters: Thresholds, θ j and effect, δ Table: Soup data Reference Not Reference Product Sure Not Sure Guess Guess Not Sure Sure Reference Test Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 17 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 18 / 23 The -not with sureness protocol Thurstonian model for the -not with sureness protocol The -not with sureness protocol Unequal variances model θ 1 0 θ 2 δ σ θ 3 Psychological continuum Multiple thresholds (θ) parameters. NOT : σ 1 = 1 NOT : σ 2 = δ σ k Psychological continuum Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 19 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 20 / 23

6 The -not with sureness protocol n unequal-variance model in practice The -not with sureness protocol Cumulative link models for -not with sureness Reference Test products products N(0, 1) N(δ, σ 22 ) δ θ 1 θ 2 θ 3 θ 4 θ 5 Sensory intensity Reference Not Reference Product Sure Not Sure Guess Guess Not Sure Sure Reference Test Table: Discrimination of packet soup (Christensen, Cleaver and Brockhoff, 2011) Christensen showed that the -not protocol (with and without sureness) is a version of a cumulative link model: ( ) θj δ(prod P(S i θ j ) = Φ i ) σ(prod i ) where σ is the ratio of scales (std. dev). This provides (optimal) ML estimates of the parameters, standard errors etc. profile likelihood confidence intervals available with confint. > fm1 <- clm(sureness ~ prod, data=my_data, link="probit") > summary(fm1) ## print d-prime etc. > confint(fm1) ## likelihood confidence interval for d-prime Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 21 / 23 Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 22 / 23 The -not with sureness protocol Discrimination measures in unequal-variance models d = (µ 2 µ 1 )/σ (no change) Distribution overlap: use the overlap function. Sensitivity: S = Φ(d / 1 + σ 2 2 ) where σ 2 is the scale ratio of the high-intensity distribution relative to the low-intensity distribution. ll measureas are equivalent in the equal-variance model, but not so in the unequal-variance model. In the unequal-variance model d can be a poor measure of discrimination. Christine Borgen Linander (DTU) Same-different and -not tests Sensometrics Summer School 23 / 23