.png)
At Mast, we want to make mortgage origination faster and simpler than ever. That means having a single source of truth for mortgage applications. Brokers and underwriters both can check Mast to get the latest updates on their cases, without having to switch between email, phone, and underwriting systems to piece together what's going on.
We shipped lender messaging in the original release of Mast in November 2021, but limited to lender users (usually underwriters) sending messages to brokers. If a broker wanted to reply, their reply would be directed to the lender's broker support inbox – meaning you'd end up with only one side of the conversation on Mast.
The solution is two-way messaging, where broker and lender can communicate in the same place. This is a common feature in lots of web apps, but something that's missing from a lot of mortgage origination software. Not only does this keep everything together in one place, it also removes the "you forgot to Cc me!" emails that are inevitable when you have a broker, broker admin user and multiple underwriters all trying to communicate on a case.
We run a majestic Rails monolith, so changes like this are pretty straightforward. Rails 6 introduced support for inbound email with action_mailbox, which we can use to power receiving email replies from brokers.
In a little under two weeks, we shipped secure two-way broker–lender messaging, baked right into the Mast platform. The recipient gets real-time notifications of new messages, and everything's stored encrypted at rest and in transit.
💌 Sendgrid to Postmark
Before writing a line of code, we needed a rock-solid email provider who we could rely on to send and receive emails. When we launched, we used Twilio's Sendgrid. Rails has an adapter that hooks into Sendgrid without much fuss, so setup was a cinch – but we quickly found issues with undeliverable emails, emails hitting IP blocks, and some providers rejecting our messages altogether (we couldn't deliver anything to @live.co.uk addresses). Sendgrid support wasn't helpful in getting our IP addresses whitelisted, except to suggest that we fork out for a dedicated IP.
After a bit of research we settled on Postmark – who believe there's more to deliverability than dedicated IPs! – and haven't looked back. Their support is stellar, and the product miles ahead of Sendgrid in terms of usability, reliability, and feature set.
Postmark also has excellent support for inbound emails.
📥 Receiving emails
First, we need to install Action Mailbox.
$ rails action_mailbox:install
$ rails db:migrate
Then update config/environments/production.rb to use Postmark as an action_mailbox ingress:
Rails.application.configure do
...
config.action_mailbox.ingress = :postmark
end
Rails then exposes an endpoint at rails/action_mailbox/postmark/inbound_emails, which when POSTed to with the correct username and password ( actionmailbox and whatever ENV["RAILS_INBOUND_EMAIL_PASSWORD"] is set to) will enqueue an email for processing.
Set this endpoint on Postmark, and make sure the box for "Include raw email content in JSON payload" is checked:
Inbound domain
Postmark gives us a unique inbound email address to use, but it's much nicer to have a custom domain we can send email to.
To achieve this, we set up MX records to point the domain (in our case, mail.usemast.com) to Postmark. Once that change propagates, <anything>@mail.usemast.com will hit our production Postmark server, which will make a POST to our production app and tell it that an email was received.
Testing locally
As an aside, the magic reverse proxy ngrok can be used to test a dummy ingress locally.
Set the RAILS_INBOUND_EMAIL_PASSWORD env var on your local machine to something meaningful. Then create a new server in Postmark, run ngrok http RAILS_PORT locally, and use https://actionmailbox:<password>@<ngrok URL>/rails/action_mailbox/postmark/inbound_emails as the inbound webhook in Postmark. Send an email in, and you'll see a request come in to your local machine.
Action Mailbox also sets up a conductor endpoint for testing inbound emails locally at /rails/conductor/action_mailbox/inbound_emails/new, although this only lets you send basic plain-text emails. As a result, you'll want to make sure you have a thorough test suite that tests a more diverse set of inbound emails, ideally sourced from real email clients.
✨ Processing emails
So far, so good: we can receive an email from the outside world. But this isn't much good if can't process those emails and do something with them.
For this, we need to create a mailbox, which should inherit from ApplicationMailbox and implement #process:
class MessagesMailbox < ApplicationMailbox
def process
# do your thing here
end
end
The mail variable is available in all methods, and is set to a Mail::Message created from the inbound email.
You can also use before_processing hooks much like you'd use a before_action hook in a controller – to take actions on the email before it gets processed. In our case this looks like:
- checking the
mail.fromarray to ensure it includes an active user, - checking
mail.toto ensure it references a valid mortgage application (as we insert the mortgage application's ID in the local part), and - checking that the user in question has write privileges on the mortgage application.
We then process the email body, stripping out HTML, getting the reply, and creating a new Message against the mortgage application. (The Message model already existed from our previous work on lender-to-broker messaging; now we're repurposing it for messages being sent the other way.) Once created, we trigger a notification – an email notification to the lender's broker support inbox, plus an in-app noticed notification to an underwriter if one is assigned to the case.
Testing this mailbox is straightforward: include the ActionMailbox::TestHelper module and use receive_inbound_email_from_mail or receive_inbound_email_from_fixture to simulate your system receiving an inbound email.
💥 Pitfalls
During testing, we encountered a few edge cases that broke our processing pipeline. (My advice would be to make no assumptions about the emails that you receive; mail clients do some weird and wonderful things when it comes to creating and formatting mail!)
Emails with no text. Some emails arrive without a text_part, only HTML. If your code depends on there being text, it'll break.
Splitting out email replies. One of the tricker things in email processing is pulling out just the meaningful reply content from the email, leaving behind all the quoted text and other gubbins (signatures etc). We implemented this by splitting the email body at several points – first if we saw a "--" on its own line as a signature delimiter, then if we saw "On <date>, <sender> wrote:", and finally with a "—Reply above this line—" divider that we inserted to the top of all our outbound emails. Postmark has its own way of doing this but we found it unreliable.
Emails with nonstandard "reply to" dividers. Some email clients, notably Outlook, use an <hr> tag to separate the quoted email, so ensuring you have a fallback (as outlined above) is essential.
Emails with invalid UTF-8 chars. Postmark handles these just fine, but our Rails app errored when trying to save these to the database. The solution was to ensure that any invalid UTF-8 bytes were discarded on save:
# Prefer the text-only part of the email if there is one
part_to_use = mail.text_part || mail.html_part || mail
body = part_to_use.body.decoded
body.encode(
"UTF-8", invalid: :replace, undef: :replace, replace: ""
).strip
Emails with a hateful amount of HTML. Some mail clients insist on adding masses of invisible HTML between every paragraph, which needs to be stripped out. One of our test fixtures, naughty_email.eml, looks like this:
<!doctype html>
<html>
<head>=20
<meta charset=3D"UTF-8">=20
</head>
<body>
<div class=3D"default-style">
<br>
</div>
<div class=3D"default-style">
Thanks, that's been noted.
</div>
<div class=3D"default-style">
<br>
</div>
<div class=3D"default-style">
Regards,
</div>
<div class=3D"default-style">
Broker
</div>
<blockquote type=3D"cite">
<div>
On 12/04/2022 15:26 Mast Support <support@usemast.com> wrote:
Catching errors
We use Sentry to catch errors in staging and production; this runs on our Sidekiq workers as well, so when an ActionMailbox::RoutingJob fails, we immediately get an error report.
It's possible that you want to tell your job runner to retry the routing job with an exponential backoff, at least at first. That way, you can see what went wrong, roll out a fix to your mail processing code, and the job will be re-run successfully with the new code.
Action Mailbox stores inbound emails for 30 days before deleting them; during this time you can download them, anonymise them, and add any tricky emails as fixtures to your test suite.
We also opted to remove the 30 day incineration with the following line in our production.rb config file:
config.action_mailbox.incinerate = false
We associate the received ActionMailbox::InboundEmail with the resulting Message record; if we need to do some debugging, we then know exactly which email triggered the creation of the Message. And in future, we can offer a 'view original email' option from our front-end.
🛤 The Rails way
Where Rails really shines is in building things that are a standard part of the modern web app. Receiving emails is pretty much a requirement – which is why Rails handles it as standard, stepping out of the way so you can build the business logic around what to do with those messages without having to reimplement everything else that needs to happen to receive emails.
By using Action Mailbox to do much of the heavy lifting, and getting rapid client feedback on each iteration of the messaging system, we were able to build and ship two-way messaging in two weeks. 🚀
We're changing the way lenders use technology in the mortgage application process. Join us!
