Solr Team CS5604: Cloudera Search in IDEAL

Transcription

1 Slr Team CS5604: Cludera Search in IDEAL Nikhil Kmawar, Ananya Chudhury, Rich Gruss Tuesday May 5, 2015 Department f Cmputer Science Virginia Tech, Blacksburg

2 Outline 1. Schema design 2. Indexing 3. Custm Search

3 SystemOverview 1. CPU a. Intel i5 Haswell Quad cre 3.3 Ghz Xen 2. RAM a. 660 GB in ttal b. 32 GB in each f the 19 Hadp ndes c. 4 GB in the manager nde d. 16 GB in the tweet DB ndes e. 16 GB in the HDFS backup nde 3. Strage a. 60 TB acrss Hadp, manager, and tweet DB ndes b TB fr backup 4. Number f ndes a. 19 Hadp ndes b. 1 Manager nde c. 2 Tweet DB ndes d. 1 HDFS backup nde

4 Slr Schema: Design

5 Slr Schema: Effect Slr idx size depends n: Number f fields Stred vs. nt stred Type f field example: string vs. text Index issues: Recmmended t add H/W Alternatively, design schema and tune Java/Slr cnfiguratins

6 Slr Schema: Future Fewer stred fields NRT indexer Cnsider using facet.methd=fcs, helps during first request Index size lwering D x S + U D is the dcument cunt S is the the size f the data type ints - 4bytes, 8 bytes fr dubles, U cumulative size f the unique field values

7 Dcument ID: Design Syntax: Cllectin_name--Cunter_value Use: Nise, Reductin, HBase, Slr/Lucene

8 Dcument ID: Effect Affects Lucene indexing Des nt affect Slr index size Fastest: Zer padded sequential Slwest: Randm UUID generated using sme languages (UUID v4) Our current design Perfrmance tradeffs Smewhere in the middle

9 ID generatin in pipeline ID additin RAW NR HBase Slr Lucene

10 Dcument ID: Future Preprcessing Current: Cncurrency `test_and_set` Batch prcessing Recmmendatin: Binary encded UUID Parallel CPU hit Querying Current: Faster fetching Lwer disk I/O Recmmendatin: Sequentially assigned value r unhashed timestamp Batch prcessing vs. asynchrnus prcessing

11 Indexing

12 Hbase Indexer - Mrphline mrphlines: [ { cmmands: [ { extracthbasecells { mappings: [{ inputclumn: "riginal:text_clean" utputfield: "text" type: string surce: value } { inputclumn: "analysis:ner_peple" utputfield: "ner_peple_multiple" type: string surce: value } ] } } { split { inputfield: "ner_peple_multiple" utputfield: "ner_peple" separatr: " " } } maps t HBase clumn family and clumn qualifier defined in schema.xml

13 Indexing - Jbs

14 Results Webpages Tweets

15 Search 1: Adjust bst levels in Query Parser (slrcnfig.xml) These numbers are cnjectural. A mre disciplined apprach: Relevance = f(q,d) Fr each f 100 queries, induce a lgistic regressin mdel. Find average cntributin f each field t relevance.

16 Search

17 Search 2: Rerder dcuments n the basis f Scial Imprtance Scre. slr.searchhandler edu.vt.dlib.ideal.slr.scialbstcmpnent...

18 Search 3: Supplement result list, if necessary, by retrieving dcuments frm cllectins that include tpics related t the search. Get tpic list straight frm HBase.

19 Search Results (first 1000 results) {'query': 'electin', 'num_results': , 'precisin': 0.998, 'time': } {'query': 'elect', 'num_results': 3247, 'precisin': 0.978, 'time': } {'query': 'revlutin', 'num_results': 13048, 'precisin': 0.95, 'time': } {'query': 'uprising', 'num_results': 1769, 'precisin': 0.851, 'time': } {'query': 'strm', 'num_results': , 'precisin': 0.999, 'time': } {'query': 'winter', 'num_results': , 'precisin': 0.999, 'time': } {'query': 'ebla', 'num_results': , 'precisin': 1.0, 'time': } {'query': 'disease', 'num_results': 6802, 'precisin': 0.993, 'time': } {'query': 'bmb', 'num_results': 33924, 'precisin': 0.857, 'time': } {'query': 'explsin', 'num_results': 1224, 'precisin': 0.284, 'time': } {'query': 'crash', 'num_results': , 'precisin': 0.995, 'time': } {'query': 'plane crash', 'num_results': , 'precisin': 1.0, 'time': } {'query': 'shting', 'num_results': 5366, 'precisin': 0.744, 'time': } {'query': 'paris shting', 'num_results': 446, 'precisin': 0.446, 'time': } {'query': 'terrrist attack', 'num_results': 1143, 'precisin': 0.768, 'time': }

20 Future Wrk 1. Relevance feedback t derive lgistic regressin, evaluate with F1 scre. 2. Mre Slr ndes -> mre index shards. 3. Bst by length 4. Innvative inputs (scial graph) 5. Mre sphisticated traversal f hierarchical classificatin

21 Acknwledgement We are especially thankful t The NSF grant IIS , III: Small: Integrated Digital Event Archiving and Library (IDEAL) fr the funding that supprted the infrastructure and the data used in the prject. Dr. Edward A. Fx fr being the guide n the side fr us and fr crdinating effrts f all the teams t help us make it a successful class prject. The GTA (Sunshin), the GRA (Mhamed) and ther students f the class wh supprted us with ideas as well as effrts during the semester. The authrs and the cntributrs f pen surce prjects, blgs and wiki pages frm where we brrwed sme ideas and slutins.

22 Thank yu! Q & A