Latent Variable Models and Signal Separation

Transcription

1 Machine Learning or Signal rocessing Laen Variable Models and Signal Separaion Class Oc MLS: Bhiksha Raj

2 Sound separaion and enhancemen A common problem: Separae or enhance sounds Speech rom noise Suppress bleed in music recordings Separae music componens.. A popular approach: Can be done wih pos pans marbles and expecaion maximizaion robabilisic laen componen analysis ools are applicable o oher orms o daa as well MLS: Bhiksha Raj

3 Sounds an example A sequence o noes Chords rom he same noes A piece o music rom he same and a ew addiional noes 3

4 Sounds an example A sequence o sounds A proper speech uerance rom he same sounds 4

5 emplae Sounds Combine o Form a Signal he individual componen sounds combine o orm he inal complex sounds ha we perceive Noes orm music honeme-like srucures combine in uerances Sound in general is composed o such building blocks or hemes Which can be simple e.g. noes or complex e.g. phonemes Our deiniion o a building block: he enire srucure occurs repeaedly in he process o orming he signal Claim: Learning he building blocks enables us o manipulae sounds 5

6 he Mixure Mulinomial A person drawing balls rom a pair o urns Each ball has a number marked on i You only hear he number drawn No idea o which urn i came rom Esimae various aces o his process MLS: Bhiksha Raj

7 More complex: WO pickers wo dieren pickers are drawing balls rom he same pos Aer each draw hey call ou he number and replace he ball hey selec he pos wih dieren probabiliies From he numbers hey call we mus deermine robabiliies wih which each o hem selec pos he disribuion o balls wihin he pos MLS: Bhiksha Raj

8 Soluion Analyze each o he callers separaely Compue he probabiliy o selecing pos separaely or each caller Bu combine he couns o balls in he pos!! MLS: Bhiksha Raj

9 Recap wih only one picker and wo pos robabiliy o Red urn: 1 Red = 1.71/7.31 = Red = 0.56/7.31 = Red = 0.66/7.31 = Red = 1.32/7.31 = Red = 0.66/7.31 = Red = 2.40/7.31 = robabiliy o Blue urn: 1 Blue = 1.29/11.69 = Blue = 0.56/11.69 = Blue = 0.66/11.69 = Blue = 1.32/11.69 = Blue = 0.66/11.69 = Blue = 2.40/11.69 = =Red = 7.31/18 = 0.41 =Blue = 10.69/18 = MLS: Bhiksha Raj Called redx bluex

10 wo pickers robabiliy o drawing a number X or he irs picker: 1 X = 1 red*xred + 1 blue*xblue robabiliy o drawing X or he second picker 2 X = 2 red*xred + 2 blue*xblue Noe: Xred and Xblue are he same or boh pickers he pos are he same and he probabiliy o drawing a ball marked wih a paricular number is he same or boh he probabiliy o selecing a paricular po is dieren or boh pickers 1 X and 2 X are no relaed MLS: Bhiksha Raj

11 wo pickers robabiliy o drawing a number X or he irs picker: 1 X = 1 red*xred + 1 blue*xblue robabiliy o drawing X or he second picker 2 X = 2 red*xred + 2 blue*xblue roblem: Given he se o numbers called ou by boh pickers esimae 1 color and 2 color or boh colors X red and X blue or all values o X MLS: Bhiksha Raj

12 Wih WO pickers Called redx bluex ICKER wo ables ICKER 2 Called redx bluex he probabiliy o selecing pos is independenly compued or he wo pickers MLS: Bhiksha Raj

13 Wih WO pickers Called redx bluex ICKER ICKER 2 Called redx bluex RED ICKER1 = 7.31 / BLUE ICKER1 = / 18 RED ICKER2 = 4.2 / 7 BLUE ICKER2 = 2.8 / MLS: Bhiksha Raj

14 Wih WO pickers Called redx bluex Called redx bluex o compue probabiliies o numbers combine he ables oal coun o Red: oal coun o Blue: MLS: Bhiksha Raj

15 Wih WO pickers: he SECOND picker Called redx bluex Called redx bluex oal coun or Red : Red: oal coun or 1: 2.46 oal coun or 2: 0.83 oal coun or 3: 1.23 oal coun or 4: 2.46 oal coun or 5: 1.23 oal coun or 6: RED = 3.3 / = MLS: Bhiksha Raj

16 MLS: Bhiksha Raj In Squiggles Given a sequence o observaions O k1 O k2.. rom he k h picker N kx is he number o observaions o color X drawn by he k h picker Iniialize k X or pos and colors X Ierae: For each Color X or each po and each observer k: Updae probabiliy o numbers or he pos: Updae he mixure weighs: probabiliy o urn selecion or each picker ' ' ' k k k X X X k k X k k k X k X N X N X ' ' ' ' X k X k X k X k k X N X N

17 Signal Separaion wih he Urn model Wha does he probabiliy o drawing balls rom Urns have o do wih sounds? Or Images? We shall see MLS: Bhiksha Raj

18 he represenaion AML IME FREQ IME We represen signals specrographically Sequence o magniude specral vecors esimaed rom overlapping segmens o signal Compued using he shor-ime Fourier ransorm Noe: Only reaining he magniude o he SF or operaions We will need he phase laer or conversion o a signal MLS: Bhiksha Raj

19 A Mulinomial Model or Specra A generaive model or one rame o a specrogram A magniude specral vecor obained rom a DF represens specral magniude agains discree requencies his may be viewed as a hisogram o draws rom a mulinomial FRAME HISOGRAM FRAME ower specrum o rame robabiliy disribuion underlying he -h specral vecor he balls are marked wih discree requency indices rom he DF MLS: Bhiksha Raj

20 A more complex model A picker has muliple urns In each draw he irs selecs an urn and hen a ball rom he urn Overall probabiliy o drawing is a mixure mulinomial Since several mulinomials urns are combined wo aspecs he probabiliy wih which he selecs any urn and he probabiliy o requencies wih he urns HISOGRAM muliple draws MLS: Bhiksha Raj

21 he icker Generaes a Specrogram he picker has a ixed se o Urns Each urn has a dieren probabiliy disribuion over He draws he specrum or he irs rame In which he selecs urns according o some probabiliy 0 z hen draws he specrum or he second rame In which he selecs urns according o some probabiliy 1 z And so on unil he has consruced he enire specrogram MLS: Bhiksha Raj

22 he icker Generaes a Specrogram he picker has a ixed se o Urns Each urn has a dieren probabiliy disribuion over He draws he specrum or he irs rame In which he selecs urns according o some probabiliy 0 z hen draws he specrum or he second rame In which he selecs urns according o some probabiliy 1 z And so on unil he has consruced he enire specrogram MLS: Bhiksha Raj

23 he icker Generaes a Specrogram he picker has a ixed se o Urns Each urn has a dieren probabiliy disribuion over He draws he specrum or he irs rame In which he selecs urns according o some probabiliy 0 z hen draws he specrum or he second rame In which he selecs urns according o some probabiliy 1 z And so on unil he has consruced he enire specrogram MLS: Bhiksha Raj

24 he icker Generaes a Specrogram he picker has a ixed se o Urns Each urn has a dieren probabiliy disribuion over He draws he specrum or he irs rame In which he selecs urns according o some probabiliy 0 z hen draws he specrum or he second rame In which he selecs urns according o some probabiliy 1 z And so on unil he has consruced he enire specrogram MLS: Bhiksha Raj

25 he icker Generaes a Specrogram he picker has a ixed se o Urns Each urn has a dieren probabiliy disribuion over He draws he specrum or he irs rame In which he selecs urns according o some probabiliy 0 z hen draws he specrum or he second rame In which he selecs urns according o some probabiliy 1 z And so on unil he has consruced he enire specrogram MLS: Bhiksha Raj

26 he icker Generaes a Specrogram he picker has a ixed se o Urns Each urn has a dieren probabiliy disribuion over He draws he specrum or he irs rame In which he selecs urns according o some probabiliy 0 z hen draws he specrum or he second rame In which he selecs urns according o some probabiliy 1 z And so on unil he has consruced he enire specrogram he number o draws in each rame represens he RMS energy in ha rame MLS: Bhiksha Raj

27 he icker Generaes a Specrogram he URNS are he same or every rame hese are he componen mulinomials or bases or he source ha generaed he signal he only dierence beween rames is he probabiliy wih which he selecs he urns Frame-speciic specral disribuion z z z Frameime speciic mixure weigh SOURCE speciic bases MLS: Bhiksha Raj

28 Specral View o Componen Mulinomials Each componen mulinomial urn is acually a normalized hisogram over requencies z I.e. a specrum Componen mulinomials represen laen specral srucures bases or he given sound source he specrum or every analysis rame is explained as an addiive combinaion o hese laen specral srucures MLS: Bhiksha Raj

29 Specral View o Componen Mulinomials By learning he mixure mulinomial model or any sound source we discover hese laen specral srucures or he source he model can be learn rom specrograms o a small amoun o audio rom he source using he EM algorihm MLS: Bhiksha Raj

30 EM learning o bases Iniialize bases z or all z or all Mus decide on he number o urns For each rame Iniialize z MLS: Bhiksha Raj

31 EM Updae Equaions Ieraive process: Compue a poseriori probabiliy o he z h urn or he source or each z z z z ' z ' Compue mixure weigh o z h urn z z' z' z S z ' S Compue he probabiliies o he requencies or he z h urn z z S z ' S ' ' MLS: Bhiksha Raj

32 How he bases compose he signal = he overall signal is he sum o he conribuions o individual urns Each urn conribues a dieren amoun o each rame he conribuion o he z-h urn o he -h rame is given by z zs S = S S MLS: Bhiksha Raj

33 Learning Srucures Speech Signal Basis-speciic specrograms z From Bach s Fugue in Gm Frequency z ime MLS: Bhiksha Raj

34 Bag o Specrograms LCA Model F F F Compose he enire specrogram all a once Urns include wo ypes o balls One se o balls represens requency F he second has a disribuion over ime Each draw: =1 =2 =M Selec an urn Draw F rom requency po Draw rom ime po Incremen hisogram a F MLS: Bhiksha Raj z z z F

35 he bag o specrograms F F F DRAW F =1 =2 =M F F F Drawing procedure Fundamenally equivalen o bag o requencies model Wih some minor dierences in esimaion MLS: Bhiksha Raj Repea N imes z z z

36 Esimaing he bag o specrograms EM updae rules Can learn all parameers Can learn and only given Can learn only =1 =2 =M F F F? ' ' ' ' z z z z z z z z ' ' z S z S z z ' ' ' S z S z z ' ' ' S z S z z z z z MLS: Bhiksha Raj

37 How meaningul are hese srucures Are hese really he noes o sound o invesigae les go back in ime MLS: Bhiksha Raj

38 he Engineer and he Musician Once upon a ime a rich poenae discovered a previously unknown recording o a beauiul piece o music. Unorunaely i was badly damaged. He grealy waned o ind ou wha i would sound like i i were no. So he hired an engineer and a musician o solve he problem MLS: Bhiksha Raj

39 he Engineer and he Musician he engineer worked or many years. He spen much money and published many papers. Finally he had a somewha scrachy resoraion o he music.. he musician lisened o he music careully or a day ranscribed i broke ou his rusy keyboard and replicaed he music MLS: Bhiksha Raj

40 he rize Who do you hink won he princess? MLS: Bhiksha Raj

41 Carnegie Mellon he Engineer and he Musician he Engineer works on he signal Resore i he musician works on his amiliariy wih music He knows how music is composed He can ideniy noes and heir cadence Bu ook many many years o learn hese skills He uses hese skills o recompose he music MLS: Bhiksha Raj

42 Wha he musician can do Noes are disincive he musician knows noes o all insrumens He can Deec noes in he recording Even i i is scrachy Reconsruc damaged music ranscribe individual componens Reconsruc separae porions o he music MLS: Bhiksha Raj

43 Music over a elephone he King acually go music over a elephone he musician mus resore i.. Bandwidh Expansion roblem: A given speech signal only has requencies in he 300Hz-3.5Khz range elephone qualiy speech Can we esimae he res o he requencies MLS: Bhiksha Raj

44 Bandwidh Expansion he picker has drawn he hisograms or every rame in he signal MLS: Bhiksha Raj

45 Bandwidh Expansion he picker has drawn he hisograms or every rame in he signal MLS: Bhiksha Raj

46 Bandwidh Expansion he picker has drawn he hisograms or every rame in he signal MLS: Bhiksha Raj

47 Bandwidh Expansion he picker has drawn he hisograms or every rame in he signal MLS: Bhiksha Raj

48 Bandwidh Expansion he picker has drawn he hisograms or every rame in he signal However we are only able o observe he number o draws o some requencies and no he ohers We mus esimae he draws o he unseen requencies MLS: Bhiksha Raj

49 Bandwidh Expansion: Sep 1 Learning From a collecion o ull-bandwidh raining daa ha are similar o he bandwidh-reduced daa learn specral bases Using he procedure described earlier Each magniude specral vecor is a mixure o a common se o bases Use he EM o learn bases rom hem Basically learning he noes MLS: Bhiksha Raj

50 Bandwidh Expansion: Sep 2 Esimaion 1 z 2 z z Using only he observed requencies in he bandwidh-reduced daa esimae mixure weighs or he bases learned in sep 1 Find ou which noes were acive a wha ime MLS: Bhiksha Raj

51 Sep 2 Ieraive process: ranscribe Compue a poseriori probabiliy o he z h urn or he speaker or each z z z z ' z ' z' Compue mixure weigh o z h urn or each rame z z S observedrequencies z' S z' observedrequencies z was obained rom raining daa and will no be reesimaed MLS: Bhiksha Raj

52 Sep 3 and Sep 4: Recompose Compose he complee probabiliy disribuion or each rame using he mixure weighs esimaed in Sep 2 z z z Noe ha we are using mixure weighs esimaed rom he reduced se o observed requencies his also gives us esimaes o he probabiliies o he unobserved requencies Use he complee probabiliy disribuion o predic he unobserved requencies! MLS: Bhiksha Raj

53 redicing rom : Simpliied Example A single Urn wih only red and blue balls Given ha ou an unknown number o draws exacly m were red how many were blue? One Simple soluion: oal number o draws N = m / red he number o ails drawn = N*blue Acual mulinomial soluion is only slighly more complex MLS: Bhiksha Raj

54 he negaive mulinomial N o is he oal number o observed couns nx 1 + nx 2 + o is he oal probabiliy o observed evens X 1 + X 2 + Given X or all oucomes X Observed nx 1 nx 2..nX k Wha is nx k+1 nx k+2 k i X n i o k i i o k i i o k k i X X n N X n N X n X n MLS: Bhiksha Raj

55 MLS: Bhiksha Raj Esimaing unobserved requencies Expeced value o he number o draws rom a negaive mulinomial: observedrequencies observedrequencies ˆ S N Esimaed specrum in unobserved requencies ˆ N S

56 Overall Soluion Learn he urns or he signal source rom broadband raining daa For each rame o he reduced bandwidh es uerance ind mixure weighs or he urns Ignore marginalize he unseen requencies z Given he complee mixure mulinomial disribuion or each rame esimae specrum hisogram a unseen requencies MLS: Bhiksha Raj z

57 redicion o Audio An example wih random specral holes MLS: Bhiksha Raj

58 redicing requencies Reduced BW daa Bases learned rom his Bandwidh expanded version MLS: Bhiksha Raj

59 Resolving he componens he musician wans o ollow he individual racks in he recording.. Eecively separae or enhance hem agains he background MLS: Bhiksha Raj

60 Signal Separaion rom Monaural Recordings Muliple sources are producing sound simulaneously he combined signals are recorded over a single microphone he goal is o selecively separae ou he signal or a arge source in he mixure Or a leas o enhance he signals rom a seleced source MLS: Bhiksha Raj

61 Supervised separaion: Example wih wo sources Each source has is own bases Can be learned rom unmixed recordings o he source All bases combine o generae he mixed signal Goal: Esimae he conribuion o individual sources MLS: Bhiksha Raj

62 Supervised separaion: Example wih wo sources KNOWN A RIORI all z z z Find mixure weighs or all bases or each rame Segregae conribuion o bases rom each source source1 source2 z z z z or source1 z or source MLS: Bhiksha Raj z z z or source2 z z or source2 z z

63 Supervised separaion: Example wih wo sources all z z z Find mixure weighs or all bases or each rame Segregae conribuion o bases rom each source source1 source2 z z z z or source1 z or source MLS: Bhiksha Raj z z z or source2 z z or source2 z z

64 Supervised separaion: Example wih wo sources all z z z Find mixure weighs or all bases or each rame Segregae conribuion o bases rom each source source1 z or source1 z z z or source1 z z MLS: Bhiksha Raj source2 z or source2 z z or source2 z z z

65 Separaing he Sources: Cleaner Soluion For each rame: Given S he specrum a requency o he mixed signal Esimae S i he specrum o he separaed signal or he i- h source a requency A simple maximum a poseriori esimaor z z S ˆ z or sourcei i S z z all z MLS: Bhiksha Raj

66 Semi-supervised separaion: Example wih wo sources UNKNOWN KNOWN A RIORI source1 all z z z z or source1 z z z or source1 z z source2 z or source2 Esimae rom mixed signal in addiion o all z MLS: Bhiksha Raj z z or source2 z z z

67 Separaing Mixed Signals: Examples Raise my ren by David Gilmour Background music bases learn rom 5-seconds o music-only segmens wihin he song Lead guiar bases bases learn rom he res o he song Norah Jones singing Sunrise A more diicul problem: Original audio clipped! Background music bases learn rom 5 seconds o music-only segmens MLS: Bhiksha Raj

68 Where i works When he specral srucures o he wo sound sources are disinc Don look much like one anoher E.g. Vocals and music E.g. Lead guiar and music No as eecive when he sources are similar Voice on voice MLS: Bhiksha Raj

69 Separae overlapping speech Bases or boh speakers learn rom 5 second recordings o individual speakers Shows improvemen o abou 5dB in Speaker-o-Speaker raio or boh speakers Improvemens are worse or same-gender mixures MLS: Bhiksha Raj

70 Can i be improved? Yes weaking More raining daa per source More bases per source ypically abou 40 bu going up helps. Adjusing FF sizes and windows in he signal processing And / Or algorihmic improvemens Sparse overcomplee represenaions Neares-neighbor represenaions Ec MLS: Bhiksha Raj

71 More on he opic Shi-invarian represenaions MLS: Bhiksha Raj

72 aerns exend beyond a single rame Four bars rom a music example he specral paerns are acually paches No all requencies all o in ime a he same rae he basic uni is a specral pach no a specrum Exend model o consider his phenomenon MLS: Bhiksha Raj

73 Shi-Invarian Model =1 =2 =M Employs bag o specrograms model Each super-urn z has wo sub urns One suburn now sores a bi-variae disribuion Each ball has a pair marked on i he bases Balls in he oher suburn merely have a ime marked on hem he locaion MLS: Bhiksha Raj

74 he shi-invarian model DRAW =1 =2 =M + Repea N imes z z z MLS: Bhiksha Raj

75 Esimaing arameers Maximum likelihood esimae ollows ragmenaion and couning sraegy wo-sep ragmenaion Each insance is ragmened ino he super urns he ragmen in each super-urn is urher ragmened ino each ime-shi Since one can arrive a a given by selecing any rom and he appropriae shi - rom MLS: Bhiksha Raj

76 Shi invarian model: Updae Rules Given daa specrogram S Iniialize Ierae ' ' ' ' ' ' ' ' ' ' ' S S S S S S Fragmen Coun MLS: Bhiksha Raj

77 An Example wo disinc sounds occuring wih dieren repeiion raes wihin a signal INU SECROGRAM Discovered pach bases Conribuion o individual bases o he recording MLS: Bhiksha Raj

78 Anoher example: Dereverberaion + = =1 Assume generaion by a single laen variable Super urn he - basis is he clean specrogram MLS: Bhiksha Raj

79 Dereverberaion: an example Basis specrum mus be made sparse or eeciveness Dereverberaion o gamma-one specrograms is also paricularly eecive or speech recogniion MLS: Bhiksha Raj

80 Shi-Invariance in wo dimensions aerns may be subsrucures Repeaing paerns ha may occur anywhere No jus in he same requency or ime locaion More apparen in image daa MLS: Bhiksha Raj

81 he wo-d Shi-Invarian Model F F F =1 =2 =M Boh sub-pos are disribuions over F pairs One subpo represens he basic paern Basis he oher subpo represens he locaion MLS: Bhiksha Raj

82 he shi-invarian model DRAW F F F F =1 =2 =M F ++F Repea N imes z F z F F z MLS: Bhiksha Raj

83 wo-d Shi Invariance: Esimaion Fragmen and coun sraegy Fragmen ino superpos bu also ino each and F Since a given can be obained rom any F ' ' ' ' ' ' ' ' ' ' F F F F S F F F F S F F F S F S F F S S ' ' ' ' ' ' ' ' F F F F F F F F F F F F Fragmen Coun MLS: Bhiksha Raj

84 Shi-Invariance: Commens F and are symmeric Canno conrol which o hem learns paerns and which he locaions Answer: Consrains Consrain he size o I.e. he size o he basic pach Oher ricks e.g. sparsiy MLS: Bhiksha Raj

85 Shi-Invariance in Many Dimensions he generic noion o shi-invariance can be exended o mulivariae daa No jus wo-d daa like images and specrograms Shi invariance can be applied o any subse o variables MLS: Bhiksha Raj

86 Example: 2-D shi invariance MLS: Bhiksha Raj

87 ach Locaions Discovered aches Example: 3-D shi invariance he original igure has muliple handwrien renderings o hree characers In dieren colours he algorihm learns he hree characers and ideniies heir locaions in he igure Inpu daa MLS: Bhiksha Raj

88 he consan Q ransorm Band pass Filer Band pass Filer Band pass Filer Specrographic analysis wih a bank o consan Q ilers he bandwidh o ilers increases wih cener requency. he spacing beween iler cener requencies increases wih requency Logarihmic spacing Band pass Filer MLS: Bhiksha Raj

89 Consan Q represenaion o Speech Energy a he oupu o a bank o ilers wih logarihmically spaced cener requencies Like a specrogram wih non-linear requency axis Changes in pich become verical ranslaions o specrogram Dieren noes o an insrumen will have he same paerns a dieren verical locaions MLS: Bhiksha Raj

90 ich racking Changing pich becomes a verical shi in he locaion o a basis he consan-q specrogram is modeled as a single paern modulaed by a verical shi is he Kernel shown o he le z F s z F z F z F F F Carnegie Mellon MLS: Bhiksha Raj

91 Carnegie Mellon ich racking Le: A vocalized song Righ: Chord sequence Impulse disribuion capures he melody! MLS: Bhiksha Raj

92 Carnegie Mellon ich racking Having more han one basis z allows simulaneous pich racking o muliple sources Example: A voice and an insrumen overlaid he impulse disribuion shows pich o boh separaely MLS: Bhiksha Raj

93 In Conclusion Surprising use o EM or audio analysis Various exensions Sparse esimaion Exemplar based mehods.. Relaed deeply o non-negaive marix acorizaion BD MLS: Bhiksha Raj