Reliable e-mail delivery in Elixir with Bamboo + Oban

In the dynamic landscape of web applications, effective communication with users is paramount. Sending bulk emails, such as newsletters or promotional messages, is a common practice for keeping users engaged. Elixir offers a powerful combination of tools for handling such tasks efficiently. In this blog post, we’ll explore how to leverage Bamboo and Oban libraries to send bulk emails seamlessly from your Elixir application.

Some considerations

• We should have a mailer module will be the entry-point for our mail delivery API. A user should be able to pass a list of emails and call a function like MyApp.Mailer.deliver_many(list_of_emails) to send the e-mails.
• The user shouldn’t be worried about the app being choked when delivering a large number of e-mails, i.e. mail delivery should be a background activity, it shouldn’t increase the load on our app significantly, and the app should still be able to serve its main purpose while still delivering e-mails reliably.
• In the real world, lots of things can happen - our app can crash due to some bugs, the machine/VM/datacenter running our app could go offline due to reasons like software bugs, hardware failures or for maintenance reasons. In such cases, we should be able to restart the e-mail delivery jobs once our app comes back up and it should be done in a reliable manner - only those e-mails which haven’t been delivered yet should be delivered.

About the libraries we’ll use

• Bamboo: A testable, composable, and adapter based Elixir email library for devs that love piping. Ships with adapters for several popular mail delivery services. It’s also quite easy to write your own delivery adapter if your platform isn’t yet supported.
• Oban: A robust job processing library which uses PostgreSQL or SQLite3 for storage and coordination, with primary goals of reliability, consistency and observability.
• Ecto: The Elixir toolkit for interacting with databases.

Setting up dependencies in your project

To install Bamboo and Oban, add them to your list of dependencies in mix.exs.

def deps do
  [
    {:bamboo, "~> 2.3.0"},
    {:oban, "~> 2.16"}
  ]
end

Run mix deps.get to fetch and install the dependencies.

Initial setup of the Mailer module

We’ll setup MyApp.Mailer module as the mailer module for our application. For setting up a mailer powered by Bamboo, all we have to do is the following:

defmodule MyApp.Mailer do
  use Bamboo.Mailer, otp_app: :my_app
end

Here’s the configuration we’ll add for our mailer module:

config :my_app, MyApp.Mailer,
  adapter: Bamboo.LocalAdapter

The above 3 lines of code and 2 lines of configuration are enough to create a mailer that can deliver e-mails (in development only; in production you’ll have to configure one of the Adapters for an e-mail delivery service or write your own adapter). However, since we need reliable e-mail delivery, our strategy will be to create an e-mail delivery job per e-mail and insert it into Oban’s job queue. Oban should take care of the rest.

Setting up Oban

Behind the scenes, Oban uses a database to store and retrieve the jobs that it has to perform. So, we’ll do the following:

• Setup the required database tables for Oban.
• Setup Oban to start in our application supervisor.
• Create an Oban job that will take an e-mail as a parameter and deliver it using the Mailer module we’ve written above.
• Insert jobs into Oban and watch it deliver e-mails.

Setting up the database for Oban

For Oban, I’ve created a job queue named events where mail delivery jobs will be put. It is powered by a PostgreSQL Ecto Repo called MyApp.Repo. Here’s the configuration:

# For Oban
config :my_app, Oban,
  repo: MyApp.Repo,
  plugins: [Oban.Plugins.Pruner],
  queues: [events: [limit: 200, dispatch_cooldown: 10]]

Here’s the configuration for the Ecto Repo that will power Oban:

config :my_app,
  ecto_repos: [MyApp.Repo]

# Configure your database
config :my_app, MyApp.Repo,
  username: "postgres",
  password: "postgres",
  hostname: "localhost",
  database: "my_app_dev",
  stacktrace: true,
  show_sensitive_data_on_connection_error: true,
  pool_size: 10

We’ll also create a migration using the following command from the terminal: mix ecto.gen.migration add_oban_jobs_table, which will create an empty ecto migration script in priv/repo/migrations. Change it to the following:

defmodule MyApp.Repo.Migrations.AddObanJobsTable do
  use Ecto.Migration

  def up do
    Oban.Migration.up(version: 11)
  end

  def down do
    Oban.Migration.down(version: 1)
  end
end

Once you’re done, make sure to create the necessary tables in your DB using the following command: mix ecto.create && mix ecto.migrate. Now let’s proceed to the next step.

Configuring Oban to start in our Application

Head over to application.ex file in your project and add the Repo and Oban items to the list of children in the application supervisor:

defmodule MyApp.Application do
  @impl true
  def start(_type, _args) do
    children = [
      # Start the Ecto repository
      MyApp.Repo,
      # Start Oban
      {Oban, Application.fetch_env!(:my_app, Oban)}
    ]
    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Creating the Oban Job for e-mail delivery

Creating an Oban Job is pretty simple.

defmodule MyApp.EmailJob do
  @moduledoc """
  Defines an email delivery job responsible for delivering a mail to a single recipient.
  The job is set to attempt 3 times.

  Powered by `Oban`
  """

  alias MyApp.Mailer
  import Bamboo.Email

  use Oban.Worker, queue: :events, max_attempts: 3, tags: ["user", "email"], unique: [period: 60]

  @impl true
  def perform(%Oban.Job{args: %{"to" => to, "subject" => subject, "body" => body}}) do
    email = new_email(
          to: to,
          from: "support@myapp.com",
          subject: subject,
          text_body: body
        )
    Mailer.deliver_now(email)
  end
end

Delivering e-mails with Oban Job

Suppose you have a list of emails to send. You can enqueue them as jobs in Oban:

emails_to_send = [
  %{"to" => "user1@example.com", "subject" => "Greetings", "body" => "Hello, User 1!"},
  %{"to" => "user2@example.com", "subject" => "Special Offer", "body" => "Check out our latest offers, User 2!"}
  # Add more emails as needed
]

Enum.each(emails_to_send, fn email ->
  email
  |> MyApp.EmailJob.new()
  |> Oban.insert()
end)

You can use Bamboo.SentEmail.all() function to view all the sent e-mails in development, if you are using Bamboo.LocalAdapter.

Adding bulk-delivery functionality to Mailer

Now that our proof-of-concept code works, lets add a function to our Mailer module to deliver a list of e-mails. We’ll make some optimizations along the way which are useful if you are inserting a large number of jobs into Oban.

defmodule MyApp.Mailer do
  @moduledoc """
  Defines additional functions for mail delivery and checking status of mail delivery requests.
  Powered by `Bamboo.Mailer`.
  """

  use Bamboo.Mailer, otp_app: :mk_be_elixir

  def deliver_many(emails, _opts \\ []) when is_list(emails) do
    for email <- emails do
      %{"to" => email.to, "subject" => email.subject, "body" => email.body}
      |> MyApp.EmailJob.new()
    end
    |> Stream.chunk_every(5000)
    |> Task.async_stream(fn jobs ->
      # Insert jobs in batches of 5k to avoid parameter limitations
      Oban.insert_all(fn _ ->
        jobs
      end)
    end)
    |> Stream.run()
  end
end

Conclusion

Congratulations! You now have a mailer that delivers bulk e-mails in the background.

By integrating Bamboo and Oban into your Elixir application, you can effortlessly send bulk emails without compromising performance. This combination provides a robust and scalable solution for managing background jobs and handling email delivery seamlessly. Experiment with different adapters, explore additional features, and tailor the implementation to suit your specific use case for an even more powerful and flexible solution.