User Uploads Introduction

Last updated on August 09, 2024.

This document series discusses how files are uploaded and managed by Instances. The articles outline why Insites use direct uploads and detail this method's advantages. The documents also discuss how direct uploads are implemented and demonstrate user uploads using real-world examples.

The usual way of doing things was to upload files to the application server, after which the server sends them to S3 (or any other cloud storage). Insites has moved away from this method for several important reasons: speed, cost, and security. Direct S3 uploads facilitate a faster and smoother user experience while decreasing infrastructure costs for service providers.

Advantages

Outlined below are a few benefits of utilising the direct upload approach:

  • Less load on the application server

    Because files are sent directly to S3, the application server is not under load, no matter how many of those files are sent.

  • Fewer middlemen

    When a user uploads files directly to S3 and skips the application server in the middle, the uploaded files are faster on the target location. In the traditional solution, files must be uploaded twice, once from the user to the application server and then from the server to S3. Usually, the second upload was placed in a background job (asynchronous) to queue uploads when there are many of them. No queues (that can be flooded) means that there is one less attack vector if someone tries to perform a DDoS attack on the infrastructure.

  • Less bandwidth used

    Because files go straight to S3, Insites is not using bandwidth first to receive a file, just to upload it again to S3.

  • No file processing

    Security holes in file uploads are often used to break into a web application. Eliminating the application server from this equation and separating those concerns means there is one less attack vector for the attacker to exploit.

  • Bigger file size limits

    When files are processed and forwarded by the application server, it is beneficial to keep them as small as possible to mitigate attack vectors and queue sizes. Direct S3 upload is constrained only by AWS S3 limits, which are 5GB (gigabytes) for single file upload when using single-part upload and 5TB (terabytes) when uploading multi-part. For download, there is no limit - as long as you can upload a file, you can download it. You can read more about this on the AWS documentation site.

  • Less moving parts

    To make a system more reliable, it's important to eliminate potential problems. For instance, a file could be uploaded to an application server but not stored in older systems. Such issues frequently happen in complex software, but the likelihood of these issues can be reduced by designing systems to use less complex code.

How it works

Now let's look at how things are glued together in Insites. To start, create a property either in OR in your . The property type is . You can use to configure whether the file should be private or public, max file size, generated versions, etc.

The diagram below gives you a high-level description of what happens in a successful file upload flow:

user upload diagram
  1. Application: Presign URL requested via GraphQL

    To be able to upload a file to S3, our server is creating a presigned URL. You, as a developer, need to get the URL using the mutation and point it to a record and property it will use. For example, you can define a record and property with type in :

  2. Browser: POST file

    This is the stage where and - results from the mutation - will be consumed by the browser. The combination of URL, payload, and file from the user is a complete package that must be sent to S3. Your view can look like this:

  3. S3: Return URL to file

    S3 will return status 204 and an XML response with the URL to the file on S3 if everything goes well. Otherwise, it will return status 4xx and the reason for the error in XML format.

  4. Browser: POST URL to application

    Until now, Insites knows nothing about the file the user uploaded. This step is meant to change that. Send the URL to the uploaded file to the application server using any method you wish. Because is just an URL (string) property in a record, you update that property just like any other.

  5. Application: Save URL to the database

    After getting the URL to a file from the browser, you usually want this URL to be saved in a record for future reference. Store it the same way as any other string - if the property type is , the system will do all the needed work behind the scenes:

    When you need the URL to the file, as always, you need to use GraphQL and query for the record containing the property to get it using , for example:

If you want your users to be able to upload images and automatically generate multiple versions of them (for example, a thumbnail), you can achieve it by using the option of property upload.

Note

Insites is a multi-cloud solution, but this doesn't necessitate distinct implementations for each cloud provider. By following our recommended approach in step 2, using a loop to automate form generation for direct uploads, you can seamlessly replace S3 with another compatible service if needed.

Have a suggestion for this page?

Didn't quite find what you are looking for or have feedback on how we can make the content better then we would love to hear from you. Please provide us feedback and we will get back to you shortly.