21 April, 2025

Design a Global Video Streaming Service (e.g., YouTube, Netflix)

 

Design a Global Video Streaming Service (e.g., YouTube, Netflix)

Question: Design a scalable and fault-tolerant video streaming platform that can:

  • Stream videos globally with low latency.

  • Allow users to upload videos.

  • Handle millions of users simultaneously.

Requirements (functional and non-functional)

functional requirement: video upload, video streaming, handle the network bandwidth, video for the different devices like mobile, smart tv, computer non-function : high availability, fault tolerance


1. High-Level Requirements

Functional Requirements:

  • Video Upload: Users can upload videos in various formats.

  • Video Streaming: Provide smooth playback with adaptive streaming for different network conditions.

  • Network Bandwidth Handling: Adjust video quality dynamically based on bandwidth.

  • Device Compatibility: Support multiple devices (e.g., mobile, smart TV, computer).

Non-Functional Requirements:

  • High Availability: The service should handle millions of concurrent viewers with minimal downtime.

  • Fault Tolerance: The system should recover gracefully from failures like server crashes or network issues.

2. High-Level Design

Here's the architectural breakdown:

  1. Frontend: Provides user interface for uploading, browsing, and watching videos.

  2. Backend Services:

    • Upload Service: Handles video uploads and metadata storage.

    • Processing Service: Transcodes videos into multiple resolutions and formats.

    • Streaming Service: Delivers videos to users with adaptive bitrate streaming.

  3. Content Delivery Network (CDN): Caches videos close to users for low-latency streaming.

  4. Database:

    • Metadata storage (e.g., title, description, resolution info).

    • User data (e.g., watch history, preferences).

  5. Storage: Distributed storage for original and transcoded videos.

  6. Load Balancer: Distributes requests across multiple servers to ensure availability.

3. Capacity Planning

Let’s estimate resource requirements for a system handling 10 million daily users:

Storage:

  • Assume 1 million uploads daily, average video size = 100 MB.

  • Original videos = 1 million x 100 MB = 100 TB/day.

  • Transcoded versions (3 resolutions) = 3 x 100 TB = 300 TB/day.

  • For 1 month of storage: 300 TB x 30 days = ~9 PB (Petabytes).

Traffic:

  • Assume 10 million users, each streaming an average of 1 hour/day.

  • Bitrate for 1080p video: 5 Mbps.

  • Total bandwidth required: 10 million x 5 Mbps = 50 Tbps.

  • A CDN can offload 80% of traffic, so backend bandwidth = 10 Tbps.

Processing:

  • Each video is transcoded into 3 resolutions.

  • Average transcoding time per video = 5 minutes.

  • Total processing required: 5 minutes x 1 million videos/day = ~83,333 hours/day.

  • With 100 servers handling 50 videos/hour, you’ll need ~1,667 servers for transcoding.

4. Detailed Design

Upload Workflow:

  1. User uploads video.

  2. Upload Service stores the video in temporary storage (e.g., S3 bucket).

  3. Metadata (e.g., title, uploader info) is stored in a relational database like PostgreSQL.

  4. Processing Service fetches the video, transcodes it into multiple resolutions (e.g., 1080p, 720p, 480p), and stores them in distributed storage (e.g., HDFS).

Streaming Workflow:

  1. User requests a video.

  2. The Streaming Service retrieves the video metadata.

  3. CDN serves the video, reducing load on the backend.

  4. Adaptive streaming adjusts resolution based on the user’s available bandwidth.

Device Compatibility:

  • Transcode videos into formats like H.264 or H.265 to support multiple devices.

  • Use HTML5 players for web and SDKs for smart TVs and mobile devices.

5. Handling Edge Cases

Video Uploads:

  • Large Files: Use chunked uploads to handle interruptions.

  • Invalid Formats: Validate video format during upload.

Streaming:

  • Low Bandwidth: Use adaptive bitrate streaming to lower resolution for slow connections.

  • Server Outages: Use replicated storage to serve videos from a different region.

High Traffic:

  • Use CDNs to cache popular videos geographically closer to users.

  • Auto-scale backend servers to handle traffic spikes.

6. Trade-Offs

1. Storage Cost vs. Quality:

  • Storing multiple resolutions increases costs but improves device compatibility.

  • You may decide to limit resolutions for infrequently accessed videos.

2. Caching vs. Latency:

  • CDNs reduce latency but introduce cache invalidation challenges for newly uploaded videos.

3. Consistency vs. Availability:

  • For highly available systems, some metadata (e.g., view counts) may be eventually consistent.

7. Final System Diagram

Here’s what the architecture looks like:

User -> CDN -> Load Balancer -> Streaming Service -> Video Storage
       -> Upload Service -> Processing Service -> Distributed Storage
       -> Metadata DB


No comments:

Post a Comment

Microservices vs Monolithic Architecture

 Microservices vs Monolithic Architecture Here’s a clear side-by-side comparison between Microservices and Monolithic architectures — fro...