P2P Content Distribution BitTorrent and Spotify

P2P Content Distribution BitTorrent and Spotify Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 1 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 2 / 62

Possible Solutions for Content Distribution Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 3 / 62

Client-Server Model Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 4 / 62

Client-Server Model Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 5 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 6 / 62

The Client-Server Model Problems Scalability? Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 7 / 62

The Client-Server Model Problems Scalability? Single Point of failure? Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 7 / 62

The Client-Server Model Problems Scalability? Single Point of failure? Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 7 / 62

Client-Server Systems Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 8 / 62

The Client-Server Model Problem Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 9 / 62

Scalable and Fault-Tolerant Client-Server Model Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 10 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 11 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 12 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 13 / 62

Peer-to-Peer Model Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 14 / 62

Peer-to-Peer (P2P) Model Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 15 / 62

P2P Challenges I Churn in the system I Free-riding problem I Bottleneck in the overlay network I Connectivity problem, e.g., NAT Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 16 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 17 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 18 / 62

How To Discover Data? Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 19 / 62

Possible Solutions - First Generation Central directory Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 20 / 62

Possible Solutions - Second Generation Flooding Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 21 / 62

Possible Solutions - Third Generation Distributed Hash Table (DHT) Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 22 / 62

P2P Content Distribution Applications - File Sharing Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 23 / 62

P2P Content Distribution Applications - Media Streaming Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 24 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 25 / 62

BitTorrent BitTorrent is a system for efficient and scalable replication of large amounts of static data. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 26 / 62

BitTorrent Players Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 27 / 62

Files Files are broken into pieces of size between 64KB and 1MB. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 28 / 62

.torrent Files Metadata Contains: URL of tracker Information about the file, e.g., filename, length,... Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 29 / 62

The Core Idea A peer obtains.torrent file. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62

The Core Idea A peer obtains.torrent file. It, then, connects to the tracker. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62

The Core Idea A peer obtains.torrent file. It, then, connects to the tracker. The tracker tells the peers from which other peers to download the pieces of the file. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62

The Core Idea A peer obtains.torrent file. It, then, connects to the tracker. The tracker tells the peers from which other peers to download the pieces of the file. Peers use this information to communicate with each other. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62

The Core Idea A peer obtains.torrent file. It, then, connects to the tracker. The tracker tells the peers from which other peers to download the pieces of the file. Peers use this information to communicate with each other. The peers send information about the file and themselves to tracker. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 31 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 32 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 33 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 34 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 35 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 36 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 37 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 38 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 39 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 40 / 62

What About Free Riders? Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 41 / 62

Tit-For-Tat Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 42 / 62

Question From which peers download the pieces? Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 43 / 62

Peer Selection Use choking algorithm to choose peer to download pieces. Decision to choke/unchoke based on tit-for-tat. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 44 / 62

Discover More Cooperating Peers Optimistic unchoking Allocate an upload slot to a randomly chosen uncooperative peer Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 45 / 62

Snubbed Peers If all its peers choke it. Increase the number of optimistic unchokes. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 46 / 62

Question Which piece? Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 47 / 62

Piece Selection Rarest first: common parts left for later Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 48 / 62

Piece Selection Rarest first: common parts left for later Random first piece: start-up need to get a complete piece Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 48 / 62

Piece Selection Rarest first: common parts left for later Random first piece: start-up need to get a complete piece Endgame mode: broadcast for all remaining blocks Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 48 / 62

BitTorrent Extension Distributed tracker Peer-exchange Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 49 / 62

Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 50 / 62

Spotify Active users: over 50 million Number of songs: over 20 million Number of songs added per day: over 20000 Number of playlists: over 1.5 billion created so far Available in 58 countries Legal Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 51 / 62

The Core Idea Request first piece from Spotify servers. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 52 / 62

The Core Idea Request first piece from Spotify servers. Meanwhile, search P2P network for remainder. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 52 / 62

The Core Idea Request first piece from Spotify servers. Meanwhile, search P2P network for remainder. Switch back and forth between Spotify servers and peers as needed. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 52 / 62

The Core Idea Request first piece from Spotify servers. Meanwhile, search P2P network for remainder. Switch back and forth between Spotify servers and peers as needed. Towards end of a track, start prefetching the next one. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 52 / 62

Main Problem in Using Spotify P2P Network Peer Discovery Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 53 / 62

Peer Discovery Sever-side tracker (BitTorrent style) Only remembers 20 peers per track. Returns 10 (online) peers to client on query. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 54 / 62

Peer Discovery Sever-side tracker (BitTorrent style) Only remembers 20 peers per track. Returns 10 (online) peers to client on query. Broadcast query in small (2 hops) neighborhood in overlay (Gnutella style) Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 54 / 62

Peer Discovery Sever-side tracker (BitTorrent style) Only remembers 20 peers per track. Returns 10 (online) peers to client on query. Broadcast query in small (2 hops) neighborhood in overlay (Gnutella style) LAN peer discovery Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 54 / 62

Downloading in P2P Ask for most urgent pieces first. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 55 / 62

Downloading in P2P Ask for most urgent pieces first. If a peer is slow, re-request from new peers. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 55 / 62

Downloading in P2P Ask for most urgent pieces first. If a peer is slow, re-request from new peers. When buffers are low, download from central server as well. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 55 / 62

Downloading in P2P Ask for most urgent pieces first. If a peer is slow, re-request from new peers. When buffers are low, download from central server as well. If buffers are very low, stop uploading. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 55 / 62

Spotify vs. BitTorrent One (well, three) P2P overlay for all tracks (not per-torrent). Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62

Spotify vs. BitTorrent One (well, three) P2P overlay for all tracks (not per-torrent). Does not inform peers about downloaded blocks. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62

Spotify vs. BitTorrent One (well, three) P2P overlay for all tracks (not per-torrent). Does not inform peers about downloaded blocks. Downloads blocks in order. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62

Spotify vs. BitTorrent One (well, three) P2P overlay for all tracks (not per-torrent). Does not inform peers about downloaded blocks. Downloads blocks in order. Does not enforce fairness (such as tit-for-tat). Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62

Spotify vs. BitTorrent One (well, three) P2P overlay for all tracks (not per-torrent). Does not inform peers about downloaded blocks. Downloads blocks in order. Does not enforce fairness (such as tit-for-tat). Informs peers about urgency of request. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62

Caching Player caches tracks it has played. Use 10% of free space (capped at 10GB) Least Recently Used policy for cache eviction. Over 50% of data comes from local cache. Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 57 / 62

Spotify Data Usage Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 58 / 62

Spotify Says Goodbye to P2P Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 59 / 62

Summary Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 60 / 62

Summary Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 61 / 62

Questions? Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 62 / 62