How to upload files from a Phoenix app to Amazon S3 - a straightforward and ready-to-apply tutorial
Phoenix is known as a smart choice for a Elixir-based web framework, particularly for real-time applications. That’s why we wanted to take a closer look at how to upload files from a Phoenix app to the popular Amazon S3 storage. If you don’t have an Phoenix app you can play with already, don’t worry - we will start our tutorial from the very beginning, explaining how to create a basic app and configure it. Afterwards you will find out how to upload files from the app to Amazon S3. We'll use arc_ecto for handling file uploads and pushing them to S3 buckets. Let’s get stuck into it then!
Configuration
Creating a new Phoenix app takes just four steps:
1
2
3
4
mix local.hex
mix archive.install https://github.com/phoenixframework/archives/raw/master/phx_new.ez
mix phx.new file_uploader
cd file_uploaderIf you are creating the app for the first time, make sure your database config is correct - check the `config/{dev.exs|test.exs}` files and look for the `config :file_uploader, FileUploader.Repo` line to ensure it’s good to go.
Let's start by creating a resource that we can do CRUD (Create, Read, Update, Delete) operations on. We will use the `Picture` model (placed inside `Storage` context) in this tutorial:
1
mix phx.gen.html Storage Picture pictures title:string image:stringLet’s modify the migration created by the generator to include an `image` column:
1
2
3
4
5
6
7
8
9
10
11
# priv/repo/migrations/xxx_create_pictures.exs
defmodule FileUploader.Repo.Migrations.CreatePictures do
use Ecto.Migration
def change do
create table(:pictures) do
add :title, :string
add :image, :string
timestamps()
end
end
endThen add all routes for the `PictureController`:
1
2
3
4
5
6
7
8
# lib/file_uploader_web/router.ex
defmodule FileUploaderWeb.Router do
...
scope "/", FileUploaderWeb do
...
resources "/pictures", PictureController
end
endAnd now we can migrate the database:
1
mix ecto.migrateCreating the first uploader
Let's start by adding `Arc.Ecto` to our dependencies:
1
2
3
4
5
6
7
8
9
10
11
12
13
# mix.exs
def application do
[
mod: {FileUploader.Application, []},
extra_applications: [:logger, :arc_ecto, :runtime_tools]
]
end
def deps do
[
...
{:arc_ecto, "~> 0.7.0"}
]
endand then install them:
1
mix deps.getNow we can create the `Image` uploader for our `Picture` model:
1
2
3
4
5
6
# lib/file_uploader/storage/uploaders/image.ex
defmodule FileUploader.Image do
use Arc.Definition
use Arc.Ecto.Definition
@versions [:original]
endThen update our `Picture` model to use that uploader. Let’s also add extended basic validation to include images in the default `changeset`:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# lib/file_uploader/storage/picture.ex
defmodule FileUploader.Storage.Picture do
use Ecto.Schema
use Arc.Ecto.Schema
import Ecto.Changeset
alias FileUploader.Storage.Picture
schema "pictures" do
field :image, FileUploader.Image.Type
field :title, :string
timestamps()
end
@doc false
def changeset(%Picture{} = picture, attrs) do
picture
|> cast(attrs, [:title])
|> cast_attachments(attrs, [:image])
|> validate_required([:image, :title])
end
endIn our `development` and `test` environments we will probably want to store files locally to avoid calling S3:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# config/dev.exs
...
config :arc, storage: Arc.Storage.Local
# config/test.exs
...
config :arc, storage: Arc.Storage.Local
# config/prod.exs
...
config :arc,
storage: Arc.Storage.S3,
bucket: {:system, "S3_BUCKET"},
access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"},
s3: [
scheme: {:system, "S3_SCHEME"} || "https://",
host: {:system, "S3_HOST"} || "s3.amazonaws.com",
region: {:system, "S3_REGION"} || "us-east-1"
]We use environment variables for specifying S3 credentials and bucket settings to keep things safe for the `production` environments.
We also need to tell Phoenix to serve our files:
1
2
3
4
5
6
# lib/file_uploader_web/endpoint.ex
defmodule FileUploaderWeb.Endpoint do
...
plug Plug.Static, at: "/uploads", from: "uploads"
...
endCreating a file upload form
Let's add a file upload control to picture form in our application (remember to add `multipart: true` option):
1
2
3
4
5
6
7
8
9
10
11
# lib/file_uploader_web/templates/picture/form.html.eex
<%= form_for @changeset, @action, [multipart: true], fn f -> %>
...
<div class="form-group">
<%= label f, :image, class: "control-label" %>
<%= file_input f, :image, class: "form-control" %>
<%= error_tag f, :image %>
</div>
...
<%= submit "Submit", class: "btn btn-primary" %>
<% end %>and modify the `show` template to include the `image`:
1
2
3
4
5
6
7
8
9
10
# lib/file_uploader_web/templates/picture/show.html.eex
<h2>Show Picture</h2>
<ul>
...
<li>
<strong>Image</strong>
<img src="<%= FileUploader.Image.url({ @picture.image, @picture }) %>" width="300" />
</li>
</ul>
...Now it’s a good time to start our app to check if everything works as expected.
Run `mix phx.server` and open http://localhost:4000/pictures in your browser.
Uploading multiple files with the same name
By default, we won't be able to store multiple files with the same name because they will overwrite each other. To prevent this behavior, we will prepend a timestamp to the filename before saving it. Keep in mind this won’t prevent from simultaneously uploading files with the same name at a given timestamp, however for the purpose of this tutorial it is enough. For many incoming connections it is something you’ll need to think about.
Let's modify our `changeset` method inside the `FileUploader.Storage.Picture` module:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# lib/file_uploader/storage/picture.ex
defmodule FileUploader.Storage.Picture do
...
def changeset(%Picture{} = picture, attrs) do
attrs = add_timestamp(attrs)
picture
|> cast(attrs, [:title])
|> cast_attachments(attrs, [:image])
|> validate_required([:image, :title])
end
defp add_timestamp(%{"image" => %Plug.Upload{filename: name} = image} = attrs) do
image = %Plug.Upload{image | filename: prepend_timestamp(name)}
%{attrs | "image" => image}
end
defp add_timestamp(params), do: params
defp prepend_timestamp(name) do
"#{:os.system_time()}" <> name
end
endHere, we are doing some fancy pattern matching to extract a file name from the parameters structure and then append it back to the original `attrs` map.
Ensure files are cleaned up after running tests
To clean up the files created during test runs, let’s modify the `arc` configuration to store them in a separate directory, which we can then clean up safely without affecting existing files.
1
2
3
4
5
6
7
8
9
10
11
12
13
# lib/file_uploader/storage/uploaders/image.ex
defmodule FileUploader.Image do
use Arc.Definition
use Arc.Ecto.Definition
@versions [:original]
def storage_dir(_version, {_, _}) do
if Mix.env == :test do
"uploads/test"
else
"uploads"
end
end
endThe code above will ensure that files uploaded during tests are saved inside the `uploads/test` directory.
Let’s also create a simple helper module to clean up files after our tests are finished:
1
2
3
4
5
6
# test/support/file_test.ex
defmodule FileUploader.FileTests do
def remove_test_files do
File.rm_rf("uploads/test")
end
endNow we can include our `FileUploader.FileTests` module in `FileUploaderWeb.ConnCase` and `FileUploader.DataCase` modules during an `on_exit` callback:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# test/support/conn_case.ex
defmodule FileUploaderWeb.ConnCase do
...
setup tags do
on_exit fn ->
FileUploader.FileTests.remove_test_files
end
...
end
end
# test/support/data_case.ex
defmodule FileUploader.DataCase do
...
setup tags do
on_exit fn ->
FileUploader.FileTests.remove_test_files
end
...
end
...
endIf we run our tests (`mix test`) they will fail because of the presence validation of the image. Let’s fix them starting from the controller tests.
We need to provide some fixture file that can be used during tests - just put some `image.png` file inside the `test/fixtures` directory, then modify the `FileUploaderWeb.PictureControllerTest` module that was created by the generator to reference this file:
1
2
3
4
5
6
7
8
9
# test/file_uploader/web/controllers/picture_controller_test.exs
defmodule FileUploaderWeb.PictureControllerTest do
use FileUploaderWeb.ConnCase
alias FileUploader.Storage
@create_attrs %{title: "some title", image: %Plug.Upload{path: "test/fixtures/image.png", filename: "image.png"}}
@update_attrs %{title: "some updated title", image: %Plug.Upload{path: "test/fixtures/image.png", filename: "image.png"}}
@invalid_attrs %{title: nil}
...
endWe also need something similar for our `FileUploader.StorageTest` module:
1
2
3
4
5
6
7
8
9
10
11
defmodule FileUploader.StorageTest do
use FileUploader.DataCase
alias FileUploader.Storage
describe "pictures" do
alias FileUploader.Storage.Picture
@valid_attrs %{title: "some title", image: %Plug.Upload{path: "test/fixtures/image.png", filename: "image.png"}}
@update_attrs %{title: "some updated title", image: %Plug.Upload{path: "test/fixtures/image.png", filename: "image.png"}}
...
end
...
endIf we run our tests again (`mix test`) we should go back to green:
1
2
3
4
$ mix test
....................
Finished in 0.2 seconds
20 tests, 0 failuresAnd that’s us, over and out for our Phoenix file uploads guide - we hope you’ve enjoyed it! Full code for the tutorial can be found here.
If you’re struggling to complete your Phoenix app, looking to add features, or even want to build a brand new app then think of us at iRonin for all your Phoenix app development needs. We have world-class expertise and will gladly assist you - get in contact with us today.
