9/7/2017. CS535 Big Data Fall 2017 Colorado State University Week 3 - B. FAQs. This material is built based on

Transcription

1 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week 3-- S535 IG T FQs Programming ssignment We discuss link analysis in this week Installation/configuration guidelines for Hadoop and Spark has been uploaded Port assignment has been posted PRT. TH OMPUTING MOL FOR IG T NLYTIS. W SL LINK NLYSIS omputer Science, olorado State University 9/5/7 S535 ig ata - Fall 7 Week 3-- 9/5/7 S535 ig ata - Fall 7 Week 3--3 Today s topics Link analysis HITS Network entrality Web Scale Link nalysis. PageRank. Hyperlink-Induced Topic Search (HITS) 3. Network entrality 9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week 3--5 This material is built based on Kleinberg, Jon. "uthoritative sources in a hyperlinked environment". Journal of the M. 46 (5): Hyperlink-Induced Topic Search (HITS)

2 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week 3--6 Types of Web queries Yes/No queries oes hrome support.ogv video format road topic queries Find information about eclipse Similar-page query Find pages similar to fandango.com 9/5/7 S535 ig ata - Fall 7 Week 3--7 Ranking algorithm to find the most authoritative pages To find the small set of the most authoritative pages that are relevant to the query uthority python olorado State University iphone 9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week 3--9 hallenge of content-based ranking Most useful pages do not have the keyword hardware in the PPL page Or the IM page Pages are not sufficiently descriptive computer hardware manufacturers in IM, or PPL HITS (Hipertext-Induced Topic Search) PageRank captures simplistic view of a network uthority Web page with good, authoritative content on a specific topic Web page that is linked by many hubs Hub Web page pointing to many authoritative Web pages e.g. portal pages (Yahoo) 9/5/7 S535 ig ata - Fall 7 Week 3-- HITS (Hypertext-Induced Topic Search).K.. Hubs and uthorities Jon Kleinberg 997 Topic search utomatically determine hubs/authorities 9/5/7 S535 ig ata - Fall 7 Week 3-- Understanding uthorities and Hubs [/] Intuitive Idea to find authoritative results using link analysis: Not all hyperlinks are related to the conferral of authority Patterns that authoritative pages have uthoritative Pages share considerable overlap in the sets of pages that point to them. In practice Performed only on the result set (PageRank is applied on the complete set of documents) eveloped for the IM lever project Used by Teoma (later sk.com) Hubs uthorities

3 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week 3-- Understanding uthorities and Hubs [/] good hub page points to many good authoritative pages good authoritative page is pointed to by many good hub pages uthorities and hubs have a mutual reinforcement relationship 9/5/7 S535 ig ata - Fall 7 Week 3--3 alculating uthority/hub scores [/3] Let there be n Web pages efine the n x n adjacency matrix such that, uv = if there is a link from u to v. Otherwise uv = P P Graph with pages 9/5/7 S535 ig ata - Fall 7 Week 3--4 alculating uthority/hub scores [/3] ach Web page has an authority score ai and a hub score hi. We define the authority score by summing up the hub scores that point to it, ( a " = $ h & &" &)* This can be written concisely as, a = + h Graph with pages P 9/5/7 S535 ig ata - Fall 7 Week 3--5 alculating uthority/hub scores [3/3] Similarly, we define the hub score by summing up the authority scores, ( h " = $ a & &" &)* This can be written concisely as, h = a Graph with pages P 9/5/7 S535 ig ata - Fall 7 Week 3--6 Hubs and uthorities P 9/5/7 S535 ig ata - Fall 7 Week 3--7 Implementing Topic Search using HITS Let s start arbitrarily from a=, h=, where is the all-one vector. Repeating this, the sequences a, a, a, and h, h, h, converge (to limits x * and y * ) Graph with pages Step. onstructing a focused subgraph based on a query Step. Iteratively calculate the authority value and hub value of the page in the subgraph 3

4 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week 3--9 Step. onstructing a focused subgraph (root set) Generate a root set from a text-based search engine e.g. pages containing query words Step. onstructing a focused subgraph (base set) For each page p R dd the set of all pages p points to dd the set of all pages pointing to p Root set ase set 9/5/7 S535 ig ata - Fall 7 Week 3-- 9/5/7 S535 ig ata - Fall 7 Week 3-- Step. Initial values Step. fter the first iteration (without normalization) Nodes Hubs uthority P P Nodes Hubs uthority P 3 4 P Ranks Hub: P=== uthority: P=== Ranks Hub: P>=> uthority: P=<< 9/5/7 S535 ig ata - Fall 7 Week 3-- 9/5/7 S535 ig ata - Fall 7 Week 3--3 Step. fter the first iteration (after normalization) Step. fter the second iteration (without normalization) Nodes Hubs/Normalized uthority/normalized P 3/.375=/(+++) /.5 /.5 /.5 /.5 /.5 /.5 4/.5 P Nodes Hubs/Normalized uthority/normalized P P Ranks Hub: P>=> uthority: P=<< Normalization Original paper: using squares sum (to ) You can use sum (to ) value = value/(sum of all values) Ranks Hub: P>>> uthority: P<<< 4

5 S535 ig ata Fall 7 olorado State University 9/5/7 9/7/7 Week 3 - S535 ig ata - Fall 7 Week 3--4 Step. fter the second iteration (after normalization) Nodes Hubs/Normalized uthority/normalized P.875/.33.5/..75/ /.67.65/.7.65/.78.5/.8 /.444 S535 ig ata - Fall 7 Week 3--5 Step. onvergence of scores Repeat the calculation (step ) until the scores converge You should specify your threshold P Ranks Hub: >P>> uthority: P<<< 9/5/7 9/5/7 S535 ig ata - Fall 7 Week /5/7 Web Scale Link nalysis. PageRank. Hyperlink-Induced Topic Search (HITS) 3. Network entrality 9/5/7 S535 ig ata - Fall 7 S535 ig ata - Fall 7 Week Network entrality Week /5/7 S535 ig ata - Fall 7 This material is built based on Understanding Social Media Robert. Hanneman and Mark Riddle, Introduction to social network methods Week 3--9 dvertising on Facebook From $.6 to $.+ per click 5

6 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week 3--3 Network entrality common goal in SN is to identify the central nodes of a network What does central mean active important non-redundant 9/5/7 S535 ig ata - Fall 7 Week 3--3 What is the Network entrality "There is certainly no unanimity on exactly what centrality is or on its conceptual foundations, and there is little agreement on the proper procedure for its measurement." -- Linton Freeman Koschutzki et al. (5) attempted a classification of centrality measures Reach: ability of ego to reach other vertices Flow: quantity/weight of walks passing through ego Vitality: effect of removing ego from the network Feedback: a recursive function of alter centralities 9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week Network entrality Measures Local measure Out-degree/in-degree egree entrality Who is important based on the Network Position In each of the following networks, X has higher centrality than Y according to a particular measure Relative to the rest of network loseness (based on the average distance) etweenness (based on geodesics) igenvector ranking (PageRank, and Katz centrality) X Y Y X Y X Y X Indegree Outdegree etweenness loseness 9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week egree entrality (undirected) He or she who has many friends is most important How equal are the nodes measure of egree entrality How much variation exist in the centrality scores among the nodes Freeman s general formula for centralization (can use other metrics, e.g. gini coefficient or standard deviation): When can the degree centrality be the best centrality measure People who will do favors for you People you can talk to / have coffee with = 8 9:; [ ( 4(")] [ <4* <4= ] where (n*) is the maximum value in the graph N is the number of vertices 6

7 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week xamples Real-world example of degree centralization High centralization e.g. few investors are trading with many others 5 = = =.67 Low centralization e.g. Trades are more evenly distributed 9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week Is egree enough to describe centrality Graph with multiple sub-graphs Who can pass information across the different sub-graphs etweenness entrality How many pairs of individuals would have to go through you in order to reach one another in the minimum number of hops Who has the higher betweenness, nn () or ob () 9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week 3--4 etweenness entrality > i = $ g & (i)/g & " Where g jk = the number of geodesics connecting j and k, and g jk (i)= the number of geodesics that actor i is on. Real-world example of etweenness entrality Facebook subgraph re there any individuals with a low degree but connecting large Groups Usually these values are normalized by: > i = > i /[ n n ] 7

8 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week 3--4 Measuring etweenness entrality [/6] 9/5/7 S535 ig ata - Fall 7 Week Measuring etweenness entrality [/6] G F and are not located between any two other vertices and are located between 3 possible pairs of vertices is located between 4 possible pairs of vertices,, G, and F are not located between any pair of other vertices and are located between 8 possible pairs of vertices is located between 9 possible pairs of vertices 9/5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week Measuring etweenness entrality [3/6] F Measuring etweenness entrality [4/6] F,,,, and are not located between any pair of other vertices F is located between possible pairs of vertices 9/5/7 S535 ig ata - Fall 7 Week Measuring etweenness entrality [5/6] Worksheet What is the etweenness entrality measures of,,,, and 9/5/7 S535 ig ata - Fall 7 Week Measuring etweenness entrality [6/6] Why do and have a betweenness of They are both on the shortest paths for pairs (-, -). They will share credits:.5+.5 = s betweenness = 3.5 s betweenness =

9 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week Finding all-pairs shortest paths (PSP) Requires large memory for computing PSP over a massive graph You can use GraphX, or Pregel for Single source shortest path (SSSP) Floyd-Warshall algorithm Iterative algorithm 9/5/7 S535 ig ata - Fall 7 Week Floyd-Warshall algorithm [/3] From Matrix representation To /5/7 S535 ig ata - Fall 7 Week /5/7 S535 ig ata - Fall 7 Week 3--5 Floyd-Warshall algorithm [/3] =(d ij ) =(d ij ) ij =Shortest distance from i to j through {,,k} Floyd-Warshall algorithm [3/3] =(d ij ) 3 =(d ij 3 ) =(d ij 4 ) 5 =(d ij 5 ) /5/7 S535 ig ata - Fall 7 Week 3--5 Pseudocode for Floyd-Warshall Input: = (dij ) (the initial edge-cost matrix) Output: n = (dij n ) (the final path-cost matrix) for k = to n // intermediate vertices considered for i = to n // the from vertex for j = to n // the to vertex dij k = min{dij k-, dik k- + dkj k- } 9/5/7 S535 ig ata - Fall 7 Week loseness entrality ased on the length of the average shortest path between a vertex and all vertices in the graph < H i = [$ d i, j ] 4* &)* You can normalize the loseness entrality with: i = i N 9

10 S535 ig ata Fall 7 olorado State University 9/7/7 Week 3-9/5/7 S535 ig ata - Fall 7 Week loseness entrality M = < 4* &)* d(, j) N = * =.4