Closing Remarks: What can we do with multiple diverse solutions?

Closing Remarks: What can we do with multiple diverse solutions? Dhruv Batra Virginia Tech

Example Result CRF Diverse Segmentations. Now what? (C) Dhruv Batra 2

Your Options Nothing User in the loop (Approximate) Min Bayes Risk Use solutions to estimate the distribution and optimize Bayes Risk Re-ranking Pick a good solution from the list Increasing Side Information (C) Dhruv Batra 3

Interactive Segmentation Setup Model: Color/Texture + Potts Grid CRF Inference: Graph-cuts Dataset: 50 train/val/test images Image + Scribbles MAP 2 nd Best MAP Diverse 2 nd Best 1-2 Nodes Flipped 100-500 Nodes Flipped (C) Dhruv Batra 4

Interactive Segmentation Better 96% +3.62% Segmentation Accuracy 95% 94% 93% 92% 91% +0.05% +1.61% 90% 89% MAP M-Best-MAP Confidence DivMBest (Oracle) (Oracle) (Oracle) (C) Dhruv Batra 5 M=6

Your Options Nothing User in the loop (Approximate) Min Bayes Risk Use solutions to estimate the distribution and optimize Bayes Risk Re-ranking Pick a good solution from the list (C) Dhruv Batra 6

Statistics 101 Loss PCP, Pascal Loss, etc True Distribution L(y gt, ŷ) P (y gt x) = P (X ) X X MAP Expected Loss: BR(ŷ) =E P L(y gt, ŷ) Min Bayes Risk min y Y y gt Y L(y gt, ŷ) P (y gt ) (C) Dhruv Batra 7

Structured Output Problems Min Bayes Risk min y Y y gt Y L(y gt, ŷ) P (y gt ) Two Problems Intractable Intractable Approximate MBR: min ŷ DivMBest y gt DivMBest L(y gt, ŷ) P (y gt ) P (X ) X (C) Dhruv Batra 8 X MAP

Semantic Segmentation Setup Models: Hierarchical CRF [Ladicky et al. ECCV 10, ICCV 09] Second-Order Pooling [Carreira ECCV 12] Inference: Alpha-expansion Greedy Dataset: Pascal Segmentation Challenge (VOC 2012) 20 categories + background; ~1500 train/val/test images (C) Dhruv Batra 9

Large-Margin Re-ranking Diverse Segmentations CRF (C) Dhruv Batra 10

Semantic Segmentation Input MAP Best of 10-Div (C) Dhruv Batra 11

Semantic Segmentation Better 59% DivMBest (Oracle) PACAL Accuracy 56% 53% 50% 15%-gain possible Same Features Same Model 47% 44% 1 2 3 4 5 6 7 8 9 10 Rand (Re-rank) #Solutions / Image MBR MAP [State-of-art circa 2012] (C) Dhruv Batra 12

Your Options Nothing User in the loop (Approximate) Min Bayes Risk Use solutions to estimate the distribution and optimize Bayes Risk Re-ranking Pick a good solution from the list (C) Dhruv Batra 13

Large-Margin Re-ranking Diverse Segmentations CRF (C) Dhruv Batra 14

Large-Margin Re-ranking Diverse Segmentations CRF ψ(, ) (C) Dhruv Batra 15

Large-Margin Re-ranking Diverse Segmentations CRF α ψ(, ) α ψ(, ) (C) Dhruv Batra 16

Large-Margin Re-ranking Diverse Segmentations CRF min α,ξ i Discriminative Re-ranking of Diverse Segmentation α 2 + C i ξ i [Yadollahpour et al., CVPR13, Wednesday Poster] α ψ(, ) α ψ(, ) 1 ξ i loss i (C) Dhruv Batra 17

Semantic Segmentation Better 59% DivMBest (Oracle) PACAL Accuracy 56% 53% 50% DivMBest (Re-ranked) [Y.B.S., CVPR 13] 47% 44% MBR MAP [State-of-art circa 2012] 1 2 3 4 5 6 7 8 9 10 Rand (Re-rank) #Solutions / Image (C) Dhruv Batra 18

Qualitative Results: Success (C) Dhruv Batra 19

Qualitative Results: Success (C) Dhruv Batra 20

Qualitative Results: Success (C) Dhruv Batra 21

Qualitative Results: Failures (C) Dhruv Batra 22

Qualitative Results: Failures (C) Dhruv Batra 23

Qualitative Results: Failures (C) Dhruv Batra 24

Summary All models are wrong Some beliefs are useful Diverse Multiple Solutions A way to get useful beliefs out. DivMBest + Reranking Big impact possible on many applications! (C) Dhruv Batra 25

Summary What does my model believe? Posterior Summary (C) Dhruv Batra 26

Thanks! (C) Dhruv Batra 27