How to upload files from a Phoenix app to Amazon S3 - a straightforward and ready-to-apply tutorial

Phoenix is known as a smart choice for a Elixir-based web framework, particularly for real-time applications. That’s why we wanted to take a closer look at how to upload files from a Phoenix app to the popular Amazon S3 storage. If you don’t have an Phoenix app you can play with already, don’t worry - we will start our tutorial from the very beginning, explaining how to create a basic app and configure it. Afterwards you will find out how to upload files from the app to Amazon S3. We’ll use arc_ecto for handling file uploads and pushing them to S3 buckets. Let’s get stuck into it then!

Configuration

Creating a new Phoenix app takes just four steps:

mix local.hex
mix archive.install https://github.com/phoenixframework/archives/raw/master/phx_new.ez
mix phx.new file_uploader
cd file_uploader

If you are creating the app for the first time, make sure your database config is correct - check the config/{dev.exs|test.exs} files and look for the config :file_uploader, FileUploader.Repo line to ensure it’s good to go.

Let’s start by creating a resource that we can do CRUD (Create, Read, Update, Delete) operations on. We will use the Picture model (placed inside Storage context) in this tutorial:

mix phx.gen.html Storage Picture pictures title:string image:string

Let’s modify the migration created by the generator to include an image column:

# priv/repo/migrations/xxx_create_pictures.exs
defmodule FileUploader.Repo.Migrations.CreatePictures do
  use Ecto.Migration

  def change do
    create table(:pictures) do
      add :title, :string
      add :image, :string

      timestamps()
    end
  end
end

Then add all routes for the PictureController:

# lib/file_uploader_web/router.ex
defmodule FileUploaderWeb.Router do
  ...

  scope "/", FileUploaderWeb do
    ...

    resources "/pictures", PictureController
  end
end

And now we can migrate the database:

mix ecto.migrate

Creating the first uploader

Let’s start by adding Arc.Ecto to our dependencies:

# mix.exs

def application do
  [
    mod: {FileUploader.Application, []},
    extra_applications: [:logger, :arc_ecto, :runtime_tools]
  ]
end

def deps do
  [
    ...
    {:arc_ecto, "~> 0.7.0"}
  ]
end

and then install them:

mix deps.get

Now we can create the Image uploader for our Picture model:

# lib/file_uploader/storage/uploaders/image.ex
defmodule FileUploader.Image do
  use Arc.Definition
  use Arc.Ecto.Definition

  @versions [:original]
end

Then update our Picture model to use that uploader. Let’s also add extended basic validation to include images in the default changeset:

# lib/file_uploader/storage/picture.ex
defmodule FileUploader.Storage.Picture do
  use Ecto.Schema
  use Arc.Ecto.Schema

  import Ecto.Changeset
  alias FileUploader.Storage.Picture

  schema "pictures" do
    field :image, FileUploader.Image.Type
    field :title, :string

    timestamps()
  end

  @doc false
  def changeset(%Picture{} = picture, attrs) do
    picture
    |> cast(attrs, [:title])
    |> cast_attachments(attrs, [:image])
    |> validate_required([:image, :title])
  end
end

In our development and test environments we will probably want to store files locally to avoid calling S3:

# config/dev.exs
...
config :arc, storage: Arc.Storage.Local


# config/test.exs
...
config :arc, storage: Arc.Storage.Local


# config/prod.exs
...
config :arc,
  storage: Arc.Storage.S3,
  bucket: {:system, "S3_BUCKET"},
  access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
  secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"},
  s3: [
    scheme: {:system, "S3_SCHEME"} || "https://",
    host: {:system, "S3_HOST"} || "s3.amazonaws.com",
    region: {:system, "S3_REGION"} || "us-east-1"
  ]

We use environment variables for specifying S3 credentials and bucket settings to keep things safe for the production environments.

We also need to tell Phoenix to serve our files:

# lib/file_uploader_web/endpoint.ex

defmodule FileUploaderWeb.Endpoint do
  ...
  plug Plug.Static, at: "/uploads", from: "uploads"

  ...
end

Creating a file upload form

Let’s add a file upload control to picture form in our application (remember to add multipart: true option):

# lib/file_uploader_web/templates/picture/form.html.eex
<%= form_for @changeset, @action, [multipart: true], fn f -> %>
  ...

  <div class="form-group">
    <%= label f, :image, class: "control-label" %>
    <%= file_input f, :image, class: "form-control" %>
    <%= error_tag f, :image %>
  </div>

  ...
  <%= submit "Submit", class: "btn btn-primary" %>
<% end %>

and modify the show template to include the image:

# lib/file_uploader_web/templates/picture/show.html.eex
<h2>Show Picture</h2>

<ul>
  ...
  <li>
    <strong>Image</strong>
    <img src="<%= FileUploader.Image.url({ @picture.image, @picture }) %>" width="300" />
  </li>
</ul>

...

Now it’s a good time to start our app to check if everything works as expected.

Run mix phx.server and open http://localhost:4000/pictures in your browser.

Uploading multiple files with the same name

By default, we won’t be able to store multiple files with the same name because they will overwrite each other. To prevent this behavior, we will prepend a timestamp to the filename before saving it. Keep in mind this won’t prevent from simultaneously uploading files with the same name at a given timestamp, however for the purpose of this tutorial it is enough. For many incoming connections it is something you’ll need to think about.

Let’s modify our changeset method inside the FileUploader.Storage.Picture module:

# lib/file_uploader/storage/picture.ex
defmodule FileUploader.Storage.Picture do
  ...

  def changeset(%Picture{} = picture, attrs) do
    attrs = add_timestamp(attrs)
    picture
    |> cast(attrs, [:title])
    |> cast_attachments(attrs, [:image])
    |> validate_required([:image, :title])
  end

  defp add_timestamp(%{"image" => %Plug.Upload{filename: name} = image} = attrs) do
    image = %Plug.Upload{image | filename: prepend_timestamp(name)}
    %{attrs | "image" => image}
  end

  defp add_timestamp(params), do: params

  defp prepend_timestamp(name) do
    "#{:os.system_time()}" <> name
  end
end

Here, we are doing some fancy pattern matching to extract a file name from the parameters structure and then append it back to the original attrs map.

Ensure files are cleaned up after running tests

To clean up the files created during test runs, let’s modify the arc configuration to store them in a separate directory, which we can then clean up safely without affecting existing files.

# lib/file_uploader/storage/uploaders/image.ex
defmodule FileUploader.Image do
  use Arc.Definition
  use Arc.Ecto.Definition

  @versions [:original]

  def storage_dir(_version, {_, _}) do
    if Mix.env == :test do
      "uploads/test"
    else
      "uploads"
    end
  end
end

The code above will ensure that files uploaded during tests are saved inside the uploads/test directory.

Let’s also create a simple helper module to clean up files after our tests are finished:

# test/support/file_test.ex
defmodule FileUploader.FileTests do
  def remove_test_files do
    File.rm_rf("uploads/test")
  end
end

Now we can include our FileUploader.FileTests module in FileUploaderWeb.ConnCase and FileUploader.DataCase modules during an on_exit callback:

# test/support/conn_case.ex
defmodule FileUploaderWeb.ConnCase do
  ...

  setup tags do
    on_exit fn ->
      FileUploader.FileTests.remove_test_files
    end

    ...
  end
end


# test/support/data_case.ex
defmodule FileUploader.DataCase do
  ...

  setup tags do
    on_exit fn ->
      FileUploader.FileTests.remove_test_files
    end

    ...
  end

  ...
end

If we run our tests (mix test) they will fail because of the presence validation of the image. Let’s fix them starting from the controller tests.

We need to provide some fixture file that can be used during tests - just put some image.png file inside the test/fixtures directory, then modify the FileUploaderWeb.PictureControllerTest module that was created by the generator to reference this file:

# test/file_uploader/web/controllers/picture_controller_test.exs
defmodule FileUploaderWeb.PictureControllerTest do
  use FileUploaderWeb.ConnCase

  alias FileUploader.Storage

  @create_attrs %{title: "some title", image: %Plug.Upload{path: "test/fixtures/image.png", filename: "image.png"}}
  @update_attrs %{title: "some updated title", image: %Plug.Upload{path: "test/fixtures/image.png", filename: "image.png"}}
  @invalid_attrs %{title: nil}

  ...
end

We also need something similar for our FileUploader.StorageTest module:

defmodule FileUploader.StorageTest do
  use FileUploader.DataCase

  alias FileUploader.Storage

  describe "pictures" do
    alias FileUploader.Storage.Picture

    @valid_attrs %{title: "some title", image: %Plug.Upload{path: "test/fixtures/image.png", filename: "image.png"}}
    @update_attrs %{title: "some updated title", image: %Plug.Upload{path: "test/fixtures/image.png", filename: "image.png"}}

    ...
  end

  ...
end

If we run our tests again (mix test) we should go back to green:

$ mix test
....................

Finished in 0.2 seconds
20 tests, 0 failures

And that’s us, over and out for our Phoenix file uploads guide - we hope you’ve enjoyed it! Full code for the tutorial can be found here.

If you’re struggling to complete your Phoenix app, looking to add features, or even want to build a brand new app then think of us at iRonin for all your Phoenix app development needs. We have world-class expertise and will gladly assist you - get in contact with us today.