Learn how to directly upload files from your Ruby web app to Amazon S3 storage using the Shrine gem in Part 2 of our series. Covers all the code and configuration files you need to upload efficiently as a background process.
Clever uploading to S3 from Ruby requires the use of a specific Ruby gem - Shrine. To use Amazon S3 with Shrine, in Part 1 - Setup & Configuration of our Store Your Files on S3 Using the Ruby Shrine Gem series, we installed and configured the gem in preparation for uploads. In today’s Part 2 - Direct File Uploads, we detail how to upload files directly from your web app, first to temp storage and then to a persistent location. Follow on for Part 3, Uploading Files from a Remote URL.
Direct file upload using a presigned URL
Our most desirable type of file uploading involves directly uploading files to AWS S3 from the frontend (FE) application (based on the URL provided - presigned - by the API), with the intermediate backend (BE) application taking care of finding an upload location on S3 and transferring the file from temporary to persistent storage.
A presigned URL is the location where the front-end app should upload the file; the URL is provided by the back-end app and contains the necessary authentication info so that the FE can upload the file directly to a temporary bucket without back-end being an intermediary.
The workflow for a direct file upload should look like:
1. FE application sends a request to the BE for a presign endpoint (provided by Shrine), which will be used to upload the file directly from the FE application to the S3.
2. BE application responds with JSON, which contains the data required to authenticate with S3 and upload a file.
3. FE application uploads the file by using the data obtained from the previous step.
4. FE generates JSON with the file cached on the S3 and sends it to the BE application.
5. BE application either creates or updates a record in the database and moves file between "cache" storage to "store" on S3.
Configuration for our Ruby application uploading files to Amazon S3 directly
To be able to complete this workflow, Shrine comes with the `presign_endpoint` plugin. This plugin should be enabled in your `config/initializers/shrine.rb`, by adding the following line of code:
1
2
3
# config/initializers/shrine.rb
# ...
Shrine.plugin :presign_endpointThe `presign_endpoint` plugin provides a Rack endpoint which generates presigned JSON data. Let's mount it in the routes:
1
2
3
4
# config/routes.rb
Rails.application.routes.draw do
mount Shrine.presign_endpoint(:cache) => "/presign"
endThe above will make a `GET` `/presign` endpoint available, which will return, in response, JSON as below:
1
2
3
4
5
6
7
8
9
10
11
12
{
"url": "https://bucket-name.s3.eu-central-1.amazonaws.com",
"fields": {
"key": "cache/2084783353bf6e5fdfe3420b8bff3a8c",
"policy": "eyJleHBpcmF0aW9uIjoiMjAxNy0xMC0xOVQwODoxMDowMFoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJpcm9uaW4tdG1zLXVwbG9hZP3hjRBnaW5nIn0seyJrZXkiOiJjYWNoZS8yMDg0NzgzMzUzYmY2ZTVmZGZlMzQyMGI4YmZmM2E4YyJ9LHsieC1hbXotY3JlZGVudGlhbCI6IkFLSUFKWk5BTFlBUUJBRE8zTzVBLzIwMTcxMDE5L2V1LWNlbnRyYWwtMS9zMy9hd3M0X3JlcXVlc3QifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1VZy1TSEEyNTYifSx7IngtYW26LWRhdGUiOiIyMDE3MTAxOVQwNzEwMDBaIn1dfQ==",
"x-amz-credential": "AKIAJZNAKLVM1ADO124A/20171019/eu-central-1/s3/aws4_request",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-date": "20171019T071000Z",
"x-amz-signature": "19c31571fb1031d64bed4c31eecc2bf5fe5737855z4378a640326d6061114714"
},
"headers": {}
}The above data allows us to upload a file directly to S3. However, there are a few problems with this.
Configuration for sending directly to development and testing storage
So, how can we create a mechanism to upload a file to the development environment, which will be consistent with sending direct to S3 for staging and production? The solution for that is to emulate the presign data, where the URL will point to our backend server.
To do that, two things are needed:
1. Enable the `upload_endpoint` plugin
2. Create `presign` endpoint for getting the upload URL
We should enable the upload_endpoint plugin conditionally in our uploader, so that it will available only for the specific uploader, instead of globally for all of them. It should be active only in development and test. Here’s the additional code to do just that:
1
2
3
4
5
# app/uploaders/attachment.rb
class AttachmentUploader < Shrine
plugin :determine_mime_type
plugin :upload_endpoint if Rails.env.development? || Rails.env.test?
endThe upload endpoint is similar to the presign endpoint, in that it provides a Rack endpoint. It accepts file uploads and forwards them to specified storage, so specific routing need to be mounted in the routes configuration.
We also create a custom `/presign` endpoint for `development` and `test` environments using `Shrine.presign_endpoint` which simply returns the URL of the upload endpoint.
1
2
3
4
5
6
7
8
9
10
11
12
13
# config/routes.rb
Rails.application.routes.draw do
if Rails.env.development? || Rails.env.test?
require 'ostruct'
presign_endpoint = Shrine.presign_endpoint(:cache, lambda do |id, _opts, req|
OpenStruct.new(url: "#{req.base_url}/attachments", key: "cache/#{id}")
end)
mount presign_endpoint => '/presign'
mount AttachmentUploader.upload_endpoint(:cache) => '/attachments'
else
mount Shrine.presign_endpoint(:cache) => '/presign'
end
endNow the `POST` `attachments` endpoint is available only in development and for test purposes.
Directly uploaded files are stored in a temporary location. When we assign the cached file to a database record, Shrine will copy it to a permanent location.
1
2
3
4
5
6
7
8
9
10
file_data = {
id: '2084783353bf6e5fdfe3420b8bff3a8c',
storage: 'cache',
metadata: {
filename: 'example.pdf',
size: 1024,
mime_type: 'application/pdf'
}
}.to_json
Attachment.create(file: file_data)Background file processing
Shrine, by default, copies files from temporary to permanent storage in the foreground - but it comes also with the `backgrounding` plugin, which very simply moves this process to the background. To use this functionality, just enable the `backgrounding` plugin, create background workers and then show Shrine which workers to use.
Of course, a backgrounding library needs to be configured before all this. For our purposes, we use Sidekiq as an adapter for ActiveJob.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# config/application.rb
# ...
class Application < Rails::Application
config.active_job.queue_adapter = :sidekiq
# ...
end
# config/initializers/shrine.rb
# ...
Shrine.plugin :backgrounding
# app/jobs/promote_worker.rb
class PromoteWorker < ActiveJob::Base
def perform(data)
Shrine::Attacher.promote(data)
end
end
# app/jobs/delete_worker.rb
class DeleteWorker < ActiveJob::Base
def perform(data)
Shrine::Attacher.delete(data)
end
end
# app/uploaders/attachment.rb
class AttachmentUploader < Shrine
# ...
Attacher.promote { |data| PromoteWorker.perform_later(data) }
Attacher.delete { |data| DeleteWorker.perform_later(data) }
endAnd that’s it! Now copying files between the cache and store, and deleting previously stored files will be queued and completed in the background.
Read on with Part 3, Uploading Files from a Remote URL to learn how to upload files to S3 from a remote URL, in the final installment of our Amazon S3 integration with Ruby series.
At iRonin, we strive to always configure the most efficient data flows and systems for our clients. If you would like us to take a look over your current project to see where we can make it more efficient, or have a new project you’d like developed, then make sure to get in touch. Whether it’s Ruby development using Amazon S3, web application development using Amazon S3, or something else entirely, we’d love to hear from you!
