Design Google Drive

We use cloud storage services daily to store our data, keep backups and share with other users. Cloud storage services like Google Drive, Dropbox, and iCloud let users store, access, and share files over the internet instead of keeping everything on a single local disk. In this blog, we will see how we can design such services. Similar to our previous blogs we will start with requirements, system entities, APIs, estimations, high level design and then deep dive the design to resolve any issues and optimise the system.

Requirements

Starting with functional requirements

1. User can upload a file from any device.
2. User can download the files to any device.
3. User can share his/her files with other users.
4. Files should get sync across all devices of users.

Now let’s see the non-functional requirements

1. System should be highly available.
2. Maximum file size that system should support is 10GB.
3. System should be secure and reliable, files should not get lost or corrupt.
4. System should have low latency and eventual consistency.

Capacity Estimations

Let’s assume the total user base of 50 million and 10 million daily active users. Assuming each user gets 10 GB of free space, so total storage requirements

50 million * 10 GB = 500 PB

Considering each user uploads 2 files per day on an average, so

queries per second (QPS) = (10 million * 2) / 86400 = 240

Entities and APIs

We will have simple 3 core entities for this system i.e. User, File and File Metadata. APIs for the system will be as following

1. POST /files → Upload a file
2. GET /files/{file_id} → Download a file
3. POST /files/{file_id}/share → Share the file with other users

High Level Design

Upload

Let’s start with the most important feature of this system and that is allowing users to upload their file and save them.

Get Adarsh Tyagi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To upload the files, we will need to take care of two things. First is to save the file content and other is to save the file metadata. Starting with file metadata, I think the most simple choice is to store this metadata in SQL database. The metadata will incude details like file id, user, size, type and SQL can take care of these things easily. But now where do we store the actual file content? First solution we can think of is to upload the file to our backend server, but the issue is that it will become a single point of failure, if server goes down, user will lose the files and also this solution is not scalable as the user base or file size grow.

Better approach will be to use the blog storage services like Amazon S3 or google cloud storage. Instead of storing files on our backend servers, we can use these services. These services are more reliable and also scalable. If our server goes down, we will not lose access to the files. To optimize this solution, what we can do is allow user to directly upload the file to the blob storage by using the presigned URLs. Once the file gets uploaded, blob storage service can send the notification to our backend so we can update and save the metadata.

Download

Now, second feature is to allow users to download their files. To handle this, we can allow users to download the files directly from blob storage and for this we can again use presigned URLs. This can also save the bandwidth in our backend. Again to optimize this feature, we can use CDN (content delivery networks). Using CDN, we can cache the file close to the user location and reduces the latency. The only challenge here will be to strategise what file to cache and for how long and for this to handle we can cache frequently accessed files.

File Sharing

Next we will handle the file sharing among users. Now to allow user to access the shared file of other user, we can create a shared users list in file metadata. Now when user will try to access the file, we will check whether that user is in shared users list of file metadata or not. But the issue here will be when user will open the application. So, when user will open the site or application, they would expect to see the shared file and to render those file will be expensive in terms of time as we would need to scan through the shared users list of each file and you can imagine how expensive that would be.

But we can solve this by normalizing the data. We can create a new database table that will map the user id to file id. This way when user will open the application, we can simply fetch the file ids by scaning this table based on user id and we can take advatnage of indexing based on user id.

Data Sync

Next feature that we all know is to sync the files across the devices. We will keep the copy of particular file on each client device and also in remote storage (cloud). So we will sync firstly from local to remote. When user will update the file on their device we will update the changes in remote storage. To do this we can use system event to track the changes locally and when it detect the change it can update the file on remote and for this it can use the upload API. Now in this, conflicts can occur when two users will try to update the file at the same time and to handle we can use the last write win strategy i.e. most recent change handled by the server will be stored.

Now next thing is to sync the local from remote servers i.e. each client will need to know when there will be change in remote server. For this we can use the websockets or server side events to update the client side for any changes occuring on remote server. For any change, the remote server can push the change event on client device and in case the websocket connection drops, we can use the polling as fallback option. In polling, the client device can perodically ask the remote server for any changes.

Design Deep Dive

Upload Large Files

Uploading the large file (in GB) will take large amount of time also, obviously based on user network. So we will need some method to let the user know how much has been uploaded and if they decide to resume their upload, we can allow them to do so. Also, uploading large files is scary because if at any point of time the network drops the user will have start the upload from scratch. So to handle this we can use chunking method. In chunking we will break the file in small parts or chunks and then upload them either one at a time or parallel depending on user’s network. Chunking needs to be done at client side only because if we will make our chunking step at our backend then this solution will not make sense because to chunk at our backend side we will need the file and to get the file we need to upload that to our server and again we are back to square one. So we will do the chunking at client side and break the file into pieces of 5–10 MB each. With chunks we can easily show the upload progress to the user.
Now we will store the status of chunk uploads and track which chunk has been uploaded and which haven’t. When user will resume the upload and start again, we can easily check which chunks needs to be uploaded. Now when any chunk will get uploaded, a request can be sent to the server with chunk status and a tag which can be used by backend server to verify this upload to S3 and then update the metadata.

Optimize the upload the download

Our chunking method will definitely optimize the upload when we will upload the chunks parallely. And similarly our CDN method will improve the downloading part but to more optimise the upload and download, we can use compression techniques. Before uploading we can compress the file on client side and then store the file as it is in S3 and then at downloading side, client can decompress the data after retrieving it from storage.

Data Security

It is very important part of the system, to keep the user data secure. First step we will definitely use is HTTPs to transfer the file securely from client to server. But on top of this we can also store the file in encrypted form in blob storage. So in case an external entity gets the access of the blob storage, they will not be able to see the actual content. Amazon S3 has this feature to store the file in encrypted form.

Architecture Diagram

Now let’s see the final desing architecture