CS 671 ICT For Development 19 th Sep 2008

Size: px

Start display at page:

Download "CS 671 ICT For Development 19 th Sep 2008"

Angelica Bradford
5 years ago
Views:

1 CS 671 ICT For Development 19 th Sep 2008 Vishal Vachhani CFILT and DIL, IIT Bombay

2 Agro Explorer A Meaning Based Multilingual Search Engine Vishal Vachhani 2

3 Web-site for Indian farmers Farmers can submit their problems related to their crops Queries are answered by Agricultural Experts at KVK, Baramati Languages supported: Marathi, Hindi, English Vishal Vachhani 3

4 Why Need Multilingual Search Vast Amount of Information available ailable on the Web Almost 70% of the Information is in English The Indian rural populace is not English- Literate A Big Language Barrier Information has to be made available to them in their local languages. Vishal Vachhani 4

5 Why Need Meaning Based Search Most of the current Search Engines are Keyword Based. They do not consider the semantics of fthe query The result set contains a large number of extraneous documents. Search based on the Meaning of the query will help narrow down on the desired information quickly. Vishal Vachhani 5

6 Query in Hindi search System English Document Marathi Document Result in Hindi English Document Vishal Vachhani 6

7 Same Keywords Different Semantics Moneylenders Exploit Farmers Farmers Exploit Moneylenders Found 1 Result Found 0 Result Vishal Vachhani 7

8 Provides both Meaning Based Search Cross-Lingual Information Access Vishal Vachhani 8

9 System Architecture Vishal Vachhani 9

10 Vishal Vachhani 10

11 Vishal Vachhani 11

12 Vishal Vachhani 12

13 Vishal Vachhani 13

14 Vishal Vachhani 14

15 Conclusion Provides two independent d features Multi-Linguality Meaning Based Search. Because of UNL both multi-lingual and meaning based properties can be incorporated together rather than using separate language translators in search engines. The scheme admits itself to Integration of multiple languages in a seamless, scalable manner. Vishal Vachhani 15

16 UNL Universal Networking Language Vishal Vachhani 16

17 Hind i Englis h UNL Frenc h Marath i Tam il Vishal Vachhani 17

18 Direct translation - translation will be done directly - N*(N-1) translator are needed for N languages translation. Intermediate Language - intermediate language will be used for language translation - Only 2*N translators are required. Vishal Vachhani 18

19 UNL is an acronym for Universal Networking Language. UNL is a computer language that enables es computers to process information and knowledge across the language barriers. UNL is a language for representing information and knowledge provided by natural languages Unlike natural languages, UNL expressions are unambiguous. Vishal Vachhani 19

20 Although the UNL is a language for computers, it has all the components of a natural language. It is composed of Universal Words (UWs), Relations, Attributes. Knowledge :semantic graph Nodes concepts Arcs relation between concepts Vishal Vachhani 20

21 A UW represents simple or compound concepts. There are two classes of UWs: unit concepts compound structures of binary relations grouped together ( indicated with Compound UW-Ids) A UW is made up of a character string (an Englishlanguage word) followed by a list of constraints. <UW>::=<Head Word>[<Constraint List>] example state(icl>express) state(icl>country) Vishal Vachhani 21

22 A relation label is represented as strings of 3 characters or less. The relations between UWs are binary. rel (UW1, UW2) They have different labels according to the different roles they play. At present, there are 46 relations in UNL For example, agt (agent), ins (instrument), pur (purpose), etc. Vishal Vachhani 22

23 Attribute labels express additional information about the Universal Words that appear in a sentence. They show what is said from the speaker s point of view; how the speaker views what is said. (time, reference, emphasis, etc. Vishal Vachhani 23

24 Example: Ram eats rice. {unl} Ram) rice(icl>eatable)) {/unl} Vishal Vachhani 24

25 eat plc agt Ram rice Vishal Vachhani 25

26 Example: The boy who works here went to school. {unl} :01) n)) agt:01(work(icl>do), plc:01(work(icl>do),here) {/unl} Vishal Vachhani 26

27 go agt plt work :01 school plc agt here boy Vishal Vachhani 27

28 Source language Enconvertor Intermediate Language Deconvertor target language Vishal Vachhani 28

29 It s alanguage Independent Generator It can deconvert UNL expressions into a variety of native languages, using a number of linguistic data such as Word Dictionary, Grammatical Rules of each language. The DeConverter transforms the sentence represented by a UNL expression into Natural language age sentence. Vishal Vachhani 29

30 Vishal Vachhani 30

Morphology Module Syntax Planning Module Hind idoc

31 Dictionary Case Marking Rules Morphology Rules Syntax Planning Rules UNL Doc UNL Parser Case Marking Module Morphology Module Syntax Planning Module Hind idoc Language dependent Module Language Independent Module Vishal Vachhani 31

32 UNL parser module will do following tasks Check input format of UNL document Separate attributes form UWs Separate attributes form dictionary i entries Replace UWs with Hindi root words

33 Category of morpho-syntactic ti properties which distinguish the various relations that a noun n phrase may bear to a governing head. न, पर,क, स, प,etc. A rule base based on : UNL attributes lexical attributes from dictionary Vishal Vachhani 33

34 Case marking is implemented using rules. We analyze all UNL as well as dictionary attributes and decide next and previous case marker. Also we use relation with parent to extract the right case mark. Vishal Vachhani 34

35 agt:null:null:null:न Structure relname : parent previous case marker: parent next case marker: child previous case marker: child next case marker: the rest four are in form of attr'rel'relationname ti and attr will be separated by # also relation name are separated by # Vishal Vachhani 35

36 What is Morphology Study of Morphemes Their formation into words, including inflection, derivation and composition Vishal Vachhani 36

37 Noun, Verb and Adjective Morphology Depends on the phonetic properties of the Hindi word Noun Morphology Depends on gender, number and vowel ending of the noun Adjective Morphology अ छ लडक, अ छ लडक, अ छ लडक adjective अ छ changes, lexical l attribute t AdjA Verb Morphology Depends upon tense, gender, number, person etc. Vishal Vachhani 37

38 Verbs are categorized by Tense (past,present,future) Gender(male,female) Person (1 st, 2 nd, 3 rd ) Number (sg,pl) Example Ladaka khana kha raha hai. It contains present continuous tense,male, sg, and 3 rd person Vishal Vachhani 38

39 Arranging word according to the language structure Rule based module It is priority based graph traversal Vishal Vachhani 39

40 Algorithm for Syntax Planning: 1) Start traversing the UNL graph from the entry node. 2) If node has no children then add this node to final string. 3) If there is more than one child hldof one node then sort children hld based on the priority of the relations. Relation having highest priority will be traversed first. 4) Mark that node as visited node. 5) Repeat steps 3 and 4 until all the children of that node get visited. i 6) If all the children of that node get visited then add that node to final string. 7) Repeat steps 2 to 4 until all the nodes get traversed. Vishal Vachhani 40

41 Also, spray 5% Neemark solution. obj spray U 3 man obj:17 man:9 mod:5 qua:5 solution also mod mod percent Neemark qua 5 41 Vishal Vachhani

42 Entry spray Vishal Vachhani 42

43 Entry spray obj man Vishal Vachhani 43

44 Entry spray obj:17 man:9 Vishal Vachhani 44

45 Entry spray obj:17 man:9 solution Vishal Vachhani 45

46 Entry spray obj:17 man:9 solution mod mod Vishal Vachhani 46

47 Entry spray obj:17 man:9 solution mod:5 mod:5 Vishal Vachhani 47

48 Entry spray obj:17 man:9 solution mod:5 mod:5 percent Vishal Vachhani 48

49 Entry spray obj:17 man:9 solution mod:5 mod:5 percent Vishal Vachhani 49

50 Entry spray obj:17 man:9 solution mod:5 mod:5 qua:5 percent Vishal Vachhani 50

51 Entry spray obj:17 man:9 solution mod:5 mod:5 Output : 5 percent qua:5 5 Vishal Vachhani 51

52 Entry spray obj:17 man:9 solution mod:5 mod:5 percent qua:5 5 Output : 5 percent Vishal Vachhani 52

53 Entry spray obj:17 man:9 solution mod:5 mod:5 percent qua:5 5 Neemark Output : 5 percent Neemark Vishal Vachhani 53

54 Entry spray obj:17 man:9 solution mod:5 mod:5 percent qua:5 5 Neemark Output : 5 percent Neemark solution Vishal Vachhani 54

55 Entry spray obj:17 man:9 solution also mod:5 mod:5 qua:5 percent 5 Neemark Output : 5 percent Neemark Solution also Vishal Vachhani 55

56 Entry spray obj:17 man:9 solution also mod:5 mod:5 percent qua:5 5 Neemark Output : 5 percent Neemark Solution o also spray Vishal Vachhani 56

57 Output: 5 percent Neemark solution also spray 5 तशत न मअक घ ल भ छड़क 5 तशत न मअक घ ल भ छड़क Vishal Vachhani 57

58 Input sentence: Its roots are affected by bacterial infection. Module Input Output Its roots are affected by bacterial infection. UNL parser ज भ वत ज व वक स मण Case marking Morphology Syntax Planning ज भ वत ज व वक स मण स इसक जड़ ज व वक भ वत ह त ह स मण स ज व वक स मण स इसक जड़ भ वत ह त ह Output: ज व वक स मण स इसक जड़ भ वत ह त ह Vishal Vachhani 58

UNL 2005 Specifications: http://www.undl.org/unlsys/unl/unl2005/ S.Singh, M.Dalal, V.Vachhani, P.Bhattacharrya and O.

in/~vishalv) in/~vishalv) Mrugank Surve, Sarvjeet Singh, Satish Kagathara, Venkatasivaramasastry K, Sunil Dubey, Gajanan Rane, Jaya Saraswati, Salil Badodekar, Akshay Iyer,

59 UNL 2005 Specifications: S.Singh, M.Dalal, V.Vachhani, P.Bhattacharrya and O.Damani Hindi generation from interlingua MTsummit 2007 ( in/~vishalv) Mrugank Surve, Sarvjeet Singh, Satish Kagathara, Venkatasivaramasastry K, Sunil Dubey, Gajanan Rane, Jaya Saraswati, Salil Badodekar, Akshay Iyer, Ashish Almeida, Roopali Nikam, Carolina Gallardo Perez, Pushpak Bhattacharyya, AgroExplorer Group: AgroExplorer: a Meaning Based Multilingual Search Engine, International Conference on Digital Libraries (ICDL), New Delhi, India, Feb Agro Explorer : aaqua : Vishal Vachhani 59

Anaphora Resolution in Hindi Language

Anaphora Resolution in Hindi Language International Journal of Information and Computation Technology. ISSN 0974-2239 Volume 3, Number 7 (2013), pp. 609-616 International Research Publications House http://www. irphouse.com /ijict.htm Anaphora