Deconstructing Data Science

Transcription

1 econstructing ata Science avid Bamman, UC Berkeley Info 290 Lecture 11: Topic models Feb 29, 2016

2 Topic models

3 Latent variables A latent variable is one that s unobserved, either because: e are predicting it (but have observed that variable for other data points) it is unobservable

4 Latent variables observed variables latent variables text, date, sender topic novels text, author, pub date genre, topic social netork nodes, friendship structure communities fitbit data accelerometer output steps, sleep patterns legislators netflix users voting behavior, speeches atching behavior, ratings political preference genre preference

5 Probabilistic graphical models Nodes represent variables (shaded = observed, clear = latent) y Arros indicate conditional relationships The probability of x here is dependent on y x Simply a visual ay of riting the joint probability: P(x, y) =P(y) P(x y)

6 Topic Models A probabilistic model for discovering hidden topics or themes (groups of terms that tend to occur together) in documents. Unsupervised (find interesting structure in the data) Clustering algorithm: Ho to tokens cluster into topics?

7 Topic Models Input: set of documents, number of clusters to learn. Output: topics topic ratio in each document topic distribution for each ord in doc

8 topic models cluster tokens into topics The messenger, hoever, does not reach Romeo and, instead, Romeo learns of Juliet's apparent death from his servant Balthasar. Heartbroken, Romeo buys poison from an apothecary and goes to the Capulet crypt. He encounters Paris ho has come to mourn Juliet privately. Believing Romeo to be a vandal, Paris confronts him and, in the ensuing battle, Romeo kills Paris. Still believing Juliet to be dead, he drinks the poison. Juliet then aakens and, finding Romeo dead, stabs herself ith his dagger. The feuding families and the Prince meet at the tomb to find all three dead. Friar Laurence recounts the story of the to "star-cross'd lovers". The families are reconciled by their children's deaths and agree to end their violent feud. The play ends ith the Prince's elegy for the lovers: "For never as a story of more oe / Than this of Juliet and her Romeo."

9 topic models cluster tokens into topics The messenger, hoever, does not reach Romeo and, instead, Romeo learns of Juliet's apparent death from his servant Balthasar. Heartbroken, Romeo buys poison from an apothecary and goes to the Capulet crypt. He encounters Paris ho has come to mourn Juliet privately. Believing Romeo to be a vandal, Paris confronts him and, in the ensuing battle, Romeo kills Paris. Still believing Juliet to be dead, he drinks the poison. Juliet then aakens and, finding Romeo dead, stabs herself ith his dagger. The feuding families and the Prince meet at the tomb to find all three dead. Friar Laurence recounts the story of the to "star-cross'd lovers". The families are reconciled by their children's deaths and agree to end their violent feud. The play ends ith the Prince's elegy for the lovers: "For never as a story of more oe / Than this of Juliet and her Romeo." eath

10 topic models cluster tokens into topics The messenger, hoever, does not reach Romeo and, instead, Romeo learns of Juliet's apparent death from his servant Balthasar. Heartbroken, Romeo buys poison from an apothecary and goes to the Capulet crypt. He encounters Paris ho has come to mourn Juliet privately. Believing Romeo to be a vandal, Paris confronts him and, in the ensuing battle, Romeo kills Paris. Still believing Juliet to be dead, he drinks the poison. Juliet then aakens and, finding Romeo dead, stabs herself ith his dagger. The feuding families and the Prince meet at the tomb to find all three dead. Friar Laurence recounts the story of the to "star-cross'd lovers". The families are reconciled by their children's deaths and agree to end their violent feud. The play ends ith the Prince's elegy for the lovers: "For never as a story of more oe / Than this of Juliet and her Romeo." Love

11 topic models cluster tokens into topics The messenger, hoever, does not reach Romeo and, instead, Romeo learns of Juliet's apparent death from his servant Balthasar. Heartbroken, Romeo buys poison from an apothecary and goes to the Capulet crypt. He encounters Paris ho has come to mourn Juliet privately. Believing Romeo to be a vandal, Paris confronts him and, in the ensuing battle, Romeo kills Paris. Still believing Juliet to be dead, he drinks the poison. Juliet then aakens and, finding Romeo dead, stabs herself ith his dagger. The feuding families and the Prince meet at the tomb to find all three dead. Friar Laurence recounts the story of the to "star-cross'd lovers". The families are reconciled by their children's deaths and agree to end their violent feud. The play ends ith the Prince's elegy for the lovers: "For never as a story of more oe / Than this of Juliet and her Romeo." Family

12 topic models cluster tokens into topics The messenger, hoever, does not reach Romeo and, instead, Romeo learns of Juliet's apparent death from his servant Balthasar. Heartbroken, Romeo buys poison from an apothecary and goes to the Capulet crypt. He encounters Paris ho has come to mourn Juliet privately. Believing Romeo to be a vandal, Paris confronts him and, in the ensuing battle, Romeo kills Paris. Still believing Juliet to be dead, he drinks the poison. Juliet then aakens and, finding Romeo dead, stabs herself ith his dagger. The feuding families and the Prince meet at the tomb to find all three dead. Friar Laurence recounts the story of the to "star-cross'd lovers". The families are reconciled by their children's deaths and agree to end their violent feud. The play ends ith the Prince's elegy for the lovers: "For never as a story of more oe / Than this of Juliet and her Romeo." Etc.

13 tokens, not types The messenger, hoever, does not reach Romeo and, instead, Romeo learns of Juliet's apparent death from his servant Balthasar. Heartbroken, Romeo buys poison from an apothecary and goes to the Capulet crypt. He encounters Paris ho has come to mourn Juliet privately. Believing Romeo to be a vandal, Paris confronts him and, in the ensuing battle, Romeo kills Paris. Still believing Juliet to be dead, he drinks the poison. Juliet then aakens and, finding Romeo dead, stabs herself ith his dagger. The feuding families and the Prince meet at the tomb to find all three dead. Friar Laurence recounts the story of the to "star-cross'd lovers". The families are reconciled by their children's deaths and agree to end their violent feud. The play ends ith the Prince's elegy for the lovers: "For never as a story of more oe / Than this of Juliet and her Romeo." People A different Paris token might belong to a Place or French topic

14 Applications

15 x = feature vector β = coefficients Feature Value Feature β follo clinton 0 follo trump 0 republican in profile 0 democrat in profile 0 benghai" 1 topic topic topic follo clinton -3.1 follo trump 6.8 republican in profile 7.9 democrat in profile -3.0 benghai" -1.7 topic topic topic

16 Softare Mallet Gensim (python) gensim/ Visualiation termite-visualiations

17 α θ document distribution over topics ɣ topic indicators for ords φ ords topic distribution over ords

18 Topic Models A document has distribution over topics g a q f ar love chases boats aliens family

19 Topic Models a q A topic is a distribution over ords g f e.g., P( adore topic = love) =.18

20 a g q f K=20

21 a ar love chases boats aliens family g q f???? P(topic topic distribution)

22 a ar love chases boats aliens family g q f ar??? P(topic topic distribution)

23 a ar love chases boats aliens family g q f ar aliens?? P(topic topic distribution)

24 a ar love chases boats aliens family g q f ar aliens ar? P(topic topic distribution)

25 a ar love chases boats aliens family g q f ar aliens ar love P(topic topic distribution)

26 a ar love chases boats aliens family g q f ar aliens ar love????

27 a g q f K=20

28 a ar love chases boats aliens family g q f ar aliens ar love fights alien kills marries

29 a ar love chases boats aliens family g q f???? P(topic topic distribution)

30 a ar love chases boats aliens family g q f aliens??? P(topic topic distribution)

31 a ar love chases boats aliens family g q f aliens family?? P(topic topic distribution)

32 a ar love chases boats aliens family g q f aliens family aliens? P(topic topic distribution)

33 a ar love chases boats aliens family g q f aliens family aliens love P(topic topic distribution)

34 a ar love chases boats aliens family g q f aliens family aliens love????

35 a ar love chases boats aliens family g q f aliens family aliens love ET mom space friend

36 Inferred Topics

37 Inference hat are the topic distributions for each document? a q hat are the topic assignments for each ord in a document? g hat are the ord distributions for each topic? f Find the parameters that maximie the likelihood of the data!

38 Inference Markov chain Monte Carlo (Gibbs sampling, Metropolis Hastings, etc.) Variational methods Spectral methods (Anandkumar et al. 2012, Arora et al. 2013)

39 Gibbs Sampling Markov chain Monte Carlo method for approximating the joint distribution of a set of variables (Geman and Geman 1984; Metropolis et al. 1953; Hastings et al. 1970) Josiah Gibbs

40 Gibbs Sampling 1. Start ith some initial value for all the variables 2. Sample a value for a variable conditioned on all of the other variables around it (using Bayes theorem) g f a q P ( X) = P ( )P (X ) P ( )P (X )

41 α Inference θ ɣ φ

42 α Inference θ P ( d, d ) P ( d ) P ( i d) ɣ ir( ) i Cat( i ) i φ

43 α Inference θ P ( d,, ) ɣ P ( d)p (, ) Cat( d)cat(, ) d φ

44 α Sampling θ P( θ) P( ) P( θ) P( ) norm = ɣ = = φ = =

45 Aside: sampling?

46 Sampling from a Multinomial Probability mass function (PMF) P( = x) exactly P( = x) x

47 Sampling from a Multinomial Cumulative density function (CF) P( x) P( <= x) x

48 Sampling from a Multinomial Sample p uniformly in [0,1] Find the point CF -1 (p) P( <= x) p= x

49 Sampling from a Multinomial Sample p uniformly in [0,1] Find the point CF -1 (p) P( <= x) p= x

50 Sampling from a Multinomial Sample p uniformly in [0,1] Find the point CF -1 (p) P( <= x) x

51 α Assumptions Every ord has one topic ɣ φ θ Every document has one topic distribution No sequential information (topics for ords are independent of each other given the set of topics for a document) Topics don t have arbitrary correlations (irichlet prior) ords don t have arbitrary correlations (irichlet prior) The only information you learn from are the identities of ords and ho they are divided into documents.

52 hat if you ant to encode other assumptions or reason over other observations?

53 α θ φ

54 α θ t φ

55 α θ t αt φ βt Time is dran from a Beta distribution [0,1] (ang and McCallum 2006)

56 α θ t αt φ βt P (,, t,, t, t) P ( d)p (, )P (t,, ) Cat( d)cat(, )Beta(t t, t) d t t 1 (1 t) t 1 B( t, t)

57 α θ t μ σ Time is dran from a Normal distribution φ [-, ]

58 α θ t μ P (,, t,, µ, ) φ σ P ( d)p (, ),P(t,µ, ) Cat( d)cat(, )Norm(t µ, ) d 1 2 exp (t µ ) 2 2 2

59 α θ t ψ φ Time is dran from a Multinomial distribution [1,, K]

60 α θ t ψ φ P (,,, t, ) P ( d)p (, )P (t, ) Cat( d)cat(, )Cat(t, ) d t

61 Goldstone and Underood (2014), The Quiet Transformations of Literary Studies

62 Grimmer (2010), A Bayesian Hierarchical Topic Model for Political Texts: Measuring Expressed Agendas in Senate Press Releases