盒子
盒子
文章目录
  1. What is the Distributed File System?
    1. GFS v.s. HDFS
  2. Why Google bothers to create this File System?
    1. Scenario Analysis
  3. Storage
    1. Where to store the big file?

Class 4 Google Distributed File System

以GFS为例探索分布式文件系统 Distrubted File System

  • Master Slave 的设计模式
  • 处理分布式系统总的failure 和 recovery 的问题
  • 如何做 replica,check sum 的检查
  • 了解 consistent hash 和 sharding的实际应用

What is the Distributed File System?

用堕胎机器去解决一台机器上不能够解决的问题,例如存储不够,QPS太大

Google Architecture 3 Parts

  • Google File System
  • Map Reduce
  • Bigtable

GFS v.s. HDFS

  • GFS is created by Google, is for internal use.
  • HDFS(Haddop Distributed File System) is an open-source framework, created by Yahoo. The same people who used to work for Google, and create the GFS.

Why Google bothers to create this File System?

Google need to save a great nunmber of content on the Internet.

Scenario Analysis

需求1:

  • 用户写入一个文件,用户读取一个文件
  • 要支持多大的文件?e.g. 1000T

需求2:

  • 多少台机器存储这些文件
  • 支持多少台机器? Google 2007 : 100,000K

Service
Client + Server

Mode 1:
client + Servers

Mode 2:
Client + Master + Servers

Storage

  1. Peer to Peer
  • Advantage
  • Disadvantage
  1. Master/ Slave

Use Master/Slave

  • single master, failure rate is 0.1%, reboot to solve the problem

Where to store the big file?

Memory ? Database? File System?

Interviewer: How to save a file(bigger than 100G) in one machine?

牵扯到操作系统的知识

支持一下
扫一扫,支持forsigner
  • 微信扫一扫
  • 支付宝扫一扫