Some basic statistical tools. ABDBM Ron Shamir

Transcription

1 Some basic statistical tools ABDBM Ron Shamir 1

2 Today s plan Multiple testing and FDR Survival analysis The Gene Ontology Enrichment analysis TANGO GSEA ABDBM Ron Shamir 2

3 Refresher: Hypothesis Testing ABDBM Ron Shamir 3

4 Hypothesis testing 101 Two alternatives for the distribution from which a sample was taken: H 0 the null hypothesis H 1 the alternative Q: Which of them is correct? Rejecting H 0 : The alternative is true Not rejecting H 0 / accepting H 1 : the null cannot be rejected A test: a range C of values for the sample outcome for which the null is rejected. C is called the rejection region ABDBM Ron Shamir 4

5 Error types Decision Reject Ho Accept Ho Truth Ho True Type 1 error correct H 1 True correct Type 2 error α = Pr(Type 1 error) = Pr H0 (C) = Pr(C H 0 true) β = Pr(Type 2 error) = Pr H1 ( C) Making one type of error can have much more harsh effect than the other. Power of a test: π=pr(reject H 0 when H 1 is true) =Pr(C H1 true) = 1- Pr H1 ( C) = 1- β ABDBM Ron Shamir 5

6 Tests of maximum power Fix α (the statistical significance of the test) Want: test that keeps Type 1 error < α and minimizes β Or: test with max power as long as Type 1 error < α Sample X={x 1, x n } from distribution P. H 0 : P=P 0, H 1 : P=P 1 Likelihood ratio: λ(x)= P 1 (X)/P 0 (X) Neyman-Pearson Lemma*: Max power test has the form C={λ(X)>K} where K is set to get significance α. Sort possible outcomes in decreasing order of λ. Put in C all outcomes as long as α is not exceeded ABDBM Ron Shamir 6 *Assumes hypotheses are simple uniquely determine the distribution.

7 P-value Sample test result : y p=prob (getting result y or more extreme Ho) reject if p α ABDBM Ron Shamir 7

8 The permutation test Input: Indep samples X=x1, xm~f1, Y=y1, yn~ F2 H0: F1=F2 H1: F1 F2 Assume a test statistic on the samples, e.g. T(x1, xm,y1, yn)= Ave(X)-Ave(Y). For our input T=t obs ; assume we reject for high T. Consider all N! permutations on the N=n+m elts Under H0 all are equally likely. Compute the values Ti for each permutation i. Permutation distribution: mass 1/N! on each Tj p-val = fraction of j for which Tj>t obs Impractical for real N. Sample instead to get an approximate/empirical p-val ABDBM Ron Shamir 8

9 Multiple testing problem Measure expression levels of 10K genes in 10 individuals, 5 sick in a disease (cases) and 5 healthy (controls) Is the level of gene #1 different between the groups? Use T and perm test to check if p-val < α But we are making 10K such tests. Chance of at least one false rejection is much higher! Setting: m hypotheses tested: H 0i vs H 1i i=1, m, p- values obtained p 1, p m ABDBM Ron Shamir 9

10 Bonferroni method Reject null hypothesis H 0i if p i < α/m Thm: prob falsely rejecting any null hypothesis < α Pf: R i : event of i-th hypothesis falsely rejected P( 1 false rejection) = P (U 1..m R i ) i P(R i ) = i (α/m) = α Highly conservative aims to avoid even one false pos error. Loses power. Assumes nothing on the distribution. ABDBM Ron Shamir 10

11 False Discovery Rate Suppose we reject all null hypotheses that fall below some threshold. Result: Decision Reject Ho Accept Ho Total Truth Ho True V U m0 H 1 True S T m1 Total R m-r m FD proportion: fraction of rejections that are incorrect. FDP = V/R (0 if R=0) FDR = E(FDP) ABDBM Ron Shamir 11

12 Benjamini-Hochberg method Reorder H 1,..H m s.t. p 1 < p 2 <. < p m Find max k s.t. p k < kα/m Reject the null hypotheses H 1,..H k Thm (BH 95): If the p-values are independent, The procedure guarantees that FDR < α Example: α =0.05., 1000 hypotheses. Bonferroni threshold: FDR: At hypothesis 20, 20 x 0.05/1000= Expect one false rejection out of the 20 ABDBM Ron Shamir 12

13 Example: m = 10, α = 0.05 i raw p FDR Thresh Bonferroni Thresh FDR significant Bonferroni significant ABDBM Ron Shamir 13

14 ABDBM Ron Shamir l/145/001/docs/lectures/nov12.html 14

15 Yoav Benjamini & Yosef Hochberg Generalization of FDR allow handling certain types of dependency between p-values. ABDBM Ron Shamir 15

16 Another viewpoint: adjusted p- values Bonferroni: pi* = pi x m FDR: (after sorting pi-s) pi* = pi x m / i In both cases check if pi* < α i raw p FDR adjusted Bonferroni adjusted ABDBM Ron Shamir 17