User Uploads Introduction

This document series discusses how files are uploaded and managed by Instances. The articles outline why Insites use direct uploads and detail this method's advantages. The documents also discuss how direct uploads are implemented and demonstrate user uploads using real-world examples.

The usual way of doing things was to upload files to the application server, after which the server sends them to S3 (or any other cloud storage). Insites has moved away from this method for several important reasons: speed, cost, and security. Direct S3 uploads facilitate a faster and smoother user experience while decreasing infrastructure costs for service providers.

Advantages

Outlined below are a few benefits of utilising the direct upload approach:

• Less load on the application server

  Because files are sent directly to S3, the application server is not under load, no matter how many of those files are sent.

• Fewer middlemen

  When a user uploads files directly to S3 and skips the application server in the middle, the uploaded files are faster on the target location. In the traditional solution, files must be uploaded twice, once from the user to the application server and then from the server to S3. Usually, the second upload was placed in a background job (asynchronous) to queue uploads when there are many of them. No queues (that can be flooded) means that there is one less attack vector if someone tries to perform a DDoS attack on the infrastructure.

• Less bandwidth used

  Because files go straight to S3, Insites is not using bandwidth first to receive a file, just to upload it again to S3.

• No file processing

  Security holes in file uploads are often used to break into a web application. Eliminating the application server from this equation and separating those concerns means there is one less attack vector for the attacker to exploit.

• Bigger file size limits

  When files are processed and forwarded by the application server, it is beneficial to keep them as small as possible to mitigate attack vectors and queue sizes. Direct S3 upload is constrained only by AWS S3 limits, which are 5GB (gigabytes) for single file upload when using single-part upload and 5TB (terabytes) when uploading multi-part. For download, there is no limit - as long as you can upload a file, you can download it. You can read more about this on the AWS documentation site.

• Less moving parts

  To make a system more reliable, it's important to eliminate potential problems. For instance, a file could be uploaded to an application server but not stored in older systems. Such issues frequently happen in complex software, but the likelihood of these issues can be reduced by designing systems to use less complex code.

How it works

Now let's look at how things are glued together in Insites. To start, create a property either in OR in your . The property type is . You can use to configure whether the file should be private or public, max file size, generated versions, etc.

The diagram below gives you a high-level description of what happens in a successful file upload flow:

1. Application: Presign URL requested via GraphQL

   To be able to upload a file to S3, our server is creating a presigned URL. You, as a developer, need to get the URL using the mutation and point it to a record and property it will use. For example, you can define a record and property with type in :
   
   

2. Browser: POST file

   This is the stage where and - results from the mutation - will be consumed by the browser. The combination of URL, payload, and file from the user is a complete package that must be sent to S3. Your view can look like this:
   
   

3. S3: Return URL to file

   S3 will return status 204 and an XML response with the URL to the file on S3 if everything goes well. Otherwise, it will return status 4xx and the reason for the error in XML format.

4. Browser: POST URL to application

   Until now, Insites knows nothing about the file the user uploaded. This step is meant to change that. Send the URL to the uploaded file to the application server using any method you wish. Because is just an URL (string) property in a record, you update that property just like any other.

5. Application: Save URL to the database

   After getting the URL to a file from the browser, you usually want this URL to be saved in a record for future reference. Store it the same way as any other string - if the property type is , the system will do all the needed work behind the scenes:
   
   

   When you need the URL to the file, as always, you need to use GraphQL and query for the record containing the property to get it using , for example:
   
   

If you want your users to be able to upload images and automatically generate multiple versions of them (for example, a thumbnail), you can achieve it by using the option of property upload.