616 Int'l Conf. Atificial Intelligence ICAI'15 Supevied y Actualization Leaning fo Impoving Genealization Pefomance Ryotao Kamimua IT Education Cente and Gaduate School of Science and Technology, Tokai Univeiity 1117 Kitakaname, Hiatuka, Kanagawa 259-1292, Japan yo@keyaki.cc.u-tokai.ac.jp Abtact The peent pape popoe an application of potentiality leaning to upevied leaning. The potentiality ha been developed a a meaue of the impotance of component in the elf-oganizing map (SOM) to extact impotant input neuon. The main chaacteitic lie in it implicity and thu it can be eaily implemented. If it i poible to ue it fo conventional upevied leaning, bette pefomance can be expected with much imple computational method. The potentiality i defined by the vaiance of input neuon and it i incopoated into upevied leaning. Uing the potentiality inide, two data et wee ued to evaluate the pefomance. The eult how that the potentiality method outpefomed one without it and othe conventional method in tem of genealization pefomance. Keywod: y, elective potentiality, detemination, actualization, genealization 1. Intoduction 1.1 y and It Actualization Neual netwok have been applied to many poblem with bette pefomance than that by the conventional tatitical method. Though the pefomance of neual netwok ha been impoved, it can be aid that the potentiality of component of neual netwok cannot be fully exploed [1]. The potentiality i conideed a the implicit capability of neual netwok. The potentiality can be actualized o ealized in tem of a numbe of diffeent fom. Fo example, the potentiality i ealized a the popetie of component which can be ued to intepet netwok behavio o to impove genealization pefomance. One of the main poblem i that little attempt have been made to detemine the main potentiality of component of neual netwok. In the peent pape, the imple potential method i popoed with two main chaacteitic, namely, vaiance and epaation. Fit, the potentiality i uppoed to be epeented in the fom of vaiance of connection weight. The potentiality i conideed to be highe when the neuon epond to input patten a diffeently a poible. Second, the potentiality detemination and ue phae ae epaated. Thee have been many attempt to intepet netwok behavio and to impove genealization [2], [3], [4], [5], [6] [7], [8], [9]. One of the main difficultie inheent to thoe appoache i that the eo between taget and actual output ae minimized and imultaneouly genealization pefomance i impoved o intepetation i impoved. Eo minimization and pefomance impovement ae ometime contadictoy to each othe. Fo example, to have moe intepetable netwok, intenal epeentation hould be implified a much a poible, which may degade the pefomance of neual netwok. To ovecome thoe poblem, a new method i popoed, whee potentiality detemination and actualization phae ae completely epaated. Fo example, the potentiality i detemined oughly and then thi potentiality i incopoated into the poce of eo minimization. Then, contadiction between eo minimization and potentiality detemination i minimized. 1.2 Relation to the Input Neuon Selection To demontate the potentiality method, the method i applied to the detection of impotant input neuon (vaiable) [10], [11], [12], [13], [14]. The vaiable election ha played impotant ole in impoving the pefomance of neual netwok. In paticula, in application, the intepetation of input vaiable i neceay. Howeve, in thi intepetation, neual netwok ae aid to be weake than the conventional method uch a the egeion analye. The egeion analyi ha been ued in many pactical poblem, becaue the coefficient obtained by the egeion analyi can be intepeted, though the actual genealization pefomance i much weake. To have moe intepetable input vaiable o input neuon, the potentiality i intoduced. The potentiality i defined a the capability of neuon eponding to input patten a diffeently a poible. Thu, the potentiality i defined a a vaiance of connection weight. Becaue the potentiality i an abtact concept, it can be actualized. In the potentiality actualization phae, the potentiality i actualized o a to epeent the impotance of input neuon. 1.3 Outline Section 2 intoduce the potentiality in the upevied leaning. The method i compoed of two phae. Fit, the potentiality detemination phae i applied to detect the
Int'l Conf. Atificial Intelligence ICAI'15 617 impotant input neuon with highe potentiality. Then, the potentiality i nomalized and the coeponding connection weight ae modified. Then, the final fine tuning phae i pefomed. In Section 3, the method wa applied to the two data et. In both et, genealization pefomance wa impoved by the potentiality. 2. Theoy and Computational Method 2.1 y Actualization Leaning Input x k v j Wij Output o i Taget x i The potential actualization leaning aim to detemine the potentiality of neuon and actualize it potentiality a much a poible. A mentioned, in the potentiality method, the detemination of the potentiality and it actual ue i epaated to facilitate leaning. The computational pocedue i compoed of two phae, namely, potentiality detemination and actualization phae in Figue 1. In the potentiality detemination phae, the potentiality of neuon i detemined by uing the vaiance of connection weight. Then, connection weight ae given into the potentiality actualization phae a initial connection one. In addition, connection weight ae weighted by the elative potentiality to take into account the impotance o potential impotance of input neuon a hown in Figue 1(b). Thu, in the potential actualization phae, connection weight ae actually updated to take into account the potentiality and ealize o actualize potentiality. Input x k w jk * w jk (a) y detemination phae W ij Output o i y i 2.2 Individual y Fo thi, it i needed to define the potentiality of individual input neuon. The potentiality of an input neuon i defined by M v k =exp R (w jk w k ) 2, (1) j=1 whee w jk denote connection weight fom the kth input neuon to the jth hidden neuon and w k = 1 M w jk. (2) M j=1 The coefficient R detemine the intenity of the vaiation of connection weight and hould be expeimentally detemined. The potentiality i baed on the vaiance of input neuon towad output neuon. It i natual to uppoe that when input neuon epond to output neuon with lage vaiation, the input neuon uely play impotant ole. Thi mean that the neuon with lage vaiation have high potentiality to epeent input patten. In addition, by the exponential function, when the vaiation of neuon become lage, the expected potentiality inceae exponentially o exceively. Thi popety i needed to intenify a few numbe of impotant neuon. (b) y Actualization phae Fig. 1: Netwok achitectue with the potentiality detemination (a) and actualization (b) phae. 2.3 Selective y The elective potentiality i defined by uing the concept of infomation in the infomation-theoetic method [15], [16], [17], [18], [19]. When the infomation inceae in competitive leaning, only one neuon finally fie, while all the othe neuon ceae to do o. Thi concept of infomation-theoetic competitive leaning i diectly tanlated into the potentiality. When the elective potentiality inceae, finally only one neuon tend to have the maximum potentiality. Fo uing the infomation theoetic concept, it i needed to nomalize the individual potentiality p(k) = v k L l=1 v. (3) l The elective potentiality i defined by the deceae fom
618 Int'l Conf. Atificial Intelligence ICAI'15 maximum uncetainty to obeved uncetainty L k=1 I = 1+ p(k)logp(k). (4) log L When thi potentiality inceae, a malle numbe of input neuon tend to have lage individual potentiality. 2.4 y Detemination and Actualization Phae The method i compoed of the potentiality detemination and actualization phae. In the detemination phae, afte finihing the leaning, the potentiality i computed with the paamete R = L 1, (5) whee L and denote the numbe of input neuon and the leaning paamete. Then, the elative potentiality i computed and with thi potentiality, the potentiality actualization i initialized new w jk = old w jk p(k). (6) With thee connection weight, the eo between taget and output ae minimized E = 1 S N (yi o i ) 2, (7) 2S =1 i=1 whee S and N denote the numbe of input patten and output neuon, and y i ae the taget fo the output o i 1. Thi mean that the potentiality i incopoated into the leaning pocee a initial weight. The expeiment eult how that the gaduate decent leaning i much affected by the initial condition and thi method i imple and effective to take into account the potentiality. 3. Reult and Dicuion 3.1 Geman Cedit Appoval Data Set The fit data et i the Geman cedit data et fom the machine leaning databae. The numbe of input patten wa 1000 with 24 input vaiable [20]. 3.1.1 Selective y Inceae Figue 2 how the elective potentiality a a function of the paamete. A hown in the figue, the elective potentiality inceaed gadually when the paamete inceaed. Figue 3 how the elative potentiality when the paamete inceaed fom 1.0(a) to 5.0(h). When the paamete i 1.0 in Figue 3(a), the elative potentiality ditibuted almot unifomly. Then, when the paamete inceaed fom 1.2 (b) to 1.6 (d), the potentiality became gadually diffeentiated. Then, the paamete inceaed futhe fom 2.5 (e) to 5.0 (h), eveal input neuon tended to have much highe elative potentialitie. 1 The hidden and output activation function wee the the hypebolic tangent igmoid and linea one and the ealy topping method wa ued. Selective potentiality 7 x 10-3 6 5 4 3 2 1 0 1 2 3 4 5 Fig. 2: Selective potentiality a a function of the paamete fo the Geman cedit data et. Table 1: Summay of expeimental eult of genealization fo the Geman data et with ten diffeent un. Method R Aveage Std dev Min Max Potential 4.4 0.2187 40 0.1600 0.2600 Ealy topping 0.2367 73 0.1867 0.2667 SVM 0.2613 44 0.2067 0.3133 Logitic 0.2313 08 0.1733 0.2733 3.1.2 Genealization Pefomance Figue 4(a) how genealization eo a a function of the paamete. When the paamete inceaed o the elective potentiality inceaed, the genealization eo tended to deceae and eem to each the table tate. Figue 4(b) how the tandad deviation of the genealization eo. One of the main chaacteitic i that the tandad deviation inceaed when the paamete inceaed. Thi mean that the genealization eo fluctuated when the paamete R inceaed. 3.1.3 Summay of Reult Table 1 how the ummay of expeimental eult elated to the genealization pefomance. In the table, the value in bold face how the minimum value. A can be een in the table, except the tandad deviation, the potential method how the bet pefomance with the minimum value in the aveage, minimum and maximum value. On the othe hand, the tandad deviation wa the laget by the potential method. A pointed out in the peviou ection, the tandad deviation tended to be lage by the potentiality method. Expeimental eult confim that genealization pefomance wa impoved by inceaing the potentiality but the eo tended to fluctuate fo the lage paamete value.
Int'l Conf. Atificial Intelligence ICAI'15 619 (a) =1.0 (b) 1.2 (c) 1.4 (d) 1.6 (e) 2.5 (f) 3.3 (g) 4.2 (h) 5.0 Fig. 3: y p(k) of input neuon fo fou input neuon fo the Geman cedit data et. Genealization eo 0.235 0.23 0.225 0.22 Standad deviation 9 8 7 6 5 4 3 0.215 2 1.8 2.9 4.0 5.0 1.8 2.9 4.0 5.0 (a) Genealization eo (b) Standad deviation Fig. 4: Genealization eo (a) and the tandad deviation of the eo (b) by the potentiality method fo the Geman cedit data et. 3.2 Biodegadation Data Set The econd data et i alo fom the machine leaning data et whee 41 attibute and 1055 patten, which mut be claified into 2 clae (eady and not eady biodegadable) [20]. 3.2.1 Selective y Figue 5 how the elective potentiality a a function of the paamete. The elective potentiality inceaed gadually when the paamete inceaed. Figue 6 how the elative potentiality when the paamete inceaed fom 1.0(a) to 5.0(h). When the paamete wa 1.0 in Figue 6(a), the potentiality wa almot unifom. Then, when the paamete inceaed gadually, eveal potentialitie became lage. Finally, when the paamete wa 5.0 in Figue 6(h), ome potentialitie wee clealy diffeentiated. 3.2.2 Genealization Pefomance Figue 7(a) how genealization eo a a function of the paamete. The genealization eo deceaed fo the malle value of the paamete and then fluctuated. Figue 7(b) how the tandad deviation of the genealization eo. A can be een in the figue, the tandad deviation deceaed gadually fo the malle paamete value. Then, the tandad deviation became lage when the value became lage. Thee eult eem to ugget that the potentiality i not elated to the impoved genealization pefomance a hown in Figue 7(a) and 5. Thi can be explained by eeing the tandad deviation of genealization eo. When the paamete R inceaed, the genealization eo fluctuated
620 Int'l Conf. Atificial Intelligence ICAI'15 Selective potentiality 6 x 10-3 5 4 3 2 Table 2: Summay of expeimental eult of genealization fo the bio-degeneation data et with ten diffeent un. The logitic function wa ued to nomalize the data. Method R Aveage Std dev Min Max Potential 3.7 0.1184 21 96 0.1456 Ealy topping 0.1215 06 0.0823 0.1456 SVM 0.1234 0.0192 0.0886 0.1519 Logitic 0.1316 03 33 0.1646 1 0 1 2 3 4 5 Fig. 5: Selective potentiality a a function of the paamete fo the bio-degeneation data et. when the paamete R wa lage in Figue 5(a). Howeve, the tandad deviation in Figue 7(b) geatly fluctuated when the paamete became lage. Thi lage tandad deviation uely affected the oveall genealization pefomance. 3.2.3 Summay of Reult Table 2 how the ummay of expeimental eult. The potentiality method howed the bet pefomance in tem of the aveage and maximum eo. On the othe hand, fo the minimum eo, the logitic egeion method howed the bet pefomance and the potentiality method howed the econd bet pefomance. The potentiality method had the econd laget tandad deviation. The expeimental eult alo how that the peent method of potentiality i good at impoving genealization pefomance. The good pefomance i explained by two point, namely, the effectivene of potentiality and epaation of two phae. Fit, the potentiality a the vaiance of input neuon i effective in impoving the genealization pefomance. When neuon epond to input patten a diffeently a poible, the neuon play vey impotant ole in leaning. Fo example, natually, neuon, eponding only unifomly to input patten, ae conideed to be unimpotant. Second, in the method, the potentiality detemination and ue phae wee epaated. Only when the potentiality i detemined, it i ued in leaning. Thi epaation contibute to the impoved pefomance. 4. Concluion The peent pape popoe a new type of leaning called "potentiality actualization leaning". The potentiality implie the potentiality of input neuon, which i uppoed to be ealized in many diffeent fom. In thi pape, the potentiality i epeented in tem of the vaiance of input neuon. The leaning i conducted to ealize thi potentiality of input neuon. The potentiality actualization leaning i compoed of two phae. In the fit phae of potentiality detemination, the potentiality i detemined. In the econd phae of the potentiality actualization, the leaning i conducted, incopoating the infomation on the potentiality. The method wa applied to two data et, namely, Geman cedit appoval data et and bio-degadation data et. In both cae, the potentiality could be inceaed by changing the paamete. In addition, genealization pefomance wa impoved. Compaing with thoe by the othe conventional method like the SVM, pefomance wa bette. Howeve, the tandad deviation of the genealization eo tended to be lage than that by the othe method. If it i poible to educe thi lage tandad deviation by ome method, the genealization by the peent method can be moe impoved. Thu, it i needed to develop a method to tabilize leaning pocee fo the potentiality actualization leaning. Refeence [1] R. Kamimua and R. Kitajima, Selective potentiality maximization fo input neuon election in elf-oganizing map, in Poceeding of IJCNN 2015 (to appea), IEEE, 2015. [2] L. I. Nod and S. P. Jacobon, A novel method fo examination of the vaiable contibution to computational neual netwok model, Chemometic and Intelligent Laboatoy Sytem, vol. 44, pp. 153 160, 1998. [3] A. Micheli, A. Speduti, and A. Staita, Analyi of the intenal epeentation developed by neual netwok fo tuctue applied to quantitative tuctue-activity elationhip tudie of benzodiazepine, Jounal of Chemical Infomation and Compute Science, vol. 41, pp. 202 218, 2001. [4] G. G. Towell and J. W. Shavlik, Extacting efined ule fom knowledge-baed neual netwok, Machine leaning, vol. 13, pp. 71 101, 1993. [5] M. Ihikawa, Stuctual leaning with fogetting, Neual Netwok, vol. 9, no. 3, pp. 509 521, 1996. [6] M. Ihikawa, Rule extaction by ucceive egulaization, Neual Netwok, vol. 13, pp. 1171 1183, 2000. [7] P. Howe and N. Cook, Uing input paamete influence to uppot the deciion of feedfowad neual netwok, Neuocomputing, vol. 24, p. 1999, 1999. [8] R. Feaud and F. Cleot, A methodology to explain neual netwok claification, Neual Netwok, vol. 15, pp. 237 246, 2002. [9] R. Setiono, W. K. Leow, and J. M. Zuada, Extacting of ule fom atificial neual netwok fo nonlinea egeion, IEEE Tanaction on Neual Netwok, vol. 13, no. 3, pp. 564 577, 2002. [10] I. Guyon and A. Elieeff, An intoduction to vaiable and featue election, Jounal of Machine Leaning Reeach, vol. 3, pp. 1157 1182, 2003.
Int'l Conf. Atificial Intelligence ICAI'15 621 5 5 5 5 5 5 (a) =1.0 (b) 1.2 (c) 1.4 (d) 1.6 5 5 5 5 5 5 5 5 (e) 2.5 (f) 3.3 (g) 4.2 (h) 5.0 5 5 Fig. 6: y p(k) of input neuon fo fou input neuon fo the bio-degeneation data et. 0.135 Genealization eo 0.13 0.125 0.12 Standad deviation 5 (a) Genealization eo (b) Standad deviation Fig. 7: Genealization eo (a) and the tandad deviation of the eo (b), by the potentiality method fo the biodegeneation data et. [11] A. Rakotomamonjy, Vaiable election uing SVM-baed citeia, Jounal of Machine Leaning Reeach, vol. 3, pp. 1357 1370, 2003. [12] S. Pekin, K. Lacke, and J. Theile, Gafting: Fat, incemental featue election by gadient decent in function pace, Jounal of Machine Leaning Reeach, vol. 3, pp. 1333 1356, 2003. [13] J. Reunanen, Ovefitting in making compaion between vaiable election method, Jounal of Machine Leaning Reeach, vol.3, pp. 1371 1382, 2003. [14] G. Catellano and A. M. Fanelli, Vaiable election uing neualnetwok model, Neuocomputing, vol. 31, pp. 1 13, 1999. [15] R. Kamimua and T. Kamimua, Stuctual infomation and linguitic ule extaction, in Poceeding of ICONIP-2000, pp. 720 726, 2000. [16] R. Kamimua, Infomation-theoetic competitive leaning with invee Euclidean ditance output unit, Neual Poceing Lette, vol. 18, pp. 163 184, 2003. [17] R. Kamimua, Teache-diected leaning: infomation-theoetic competitive leaning in upevied multi-layeed netwok, Connection Science, vol. 15, pp. 117 140, 2003. [18] R. Kamimua, Pogeive featue extaction by geedy netwokgowing algoithm, Complex Sytem, vol. 14, no. 2, pp. 127 153, 2003. [19] R. Kamimua, Infomation theoetic competitive leaning in elfadaptive multi-layeed netwok, Connection Science, vol. 13, no. 4, pp. 323 347, 2003. [20] K. Bache and M. Lichman, UCI machine leaning epoitoy, 2013.