Do I Understand Emails Now ?
This blog is about how I tried to setup Frappe Mail locallyRecently, I went to Frappeverse 2024 - a Business focused conference organized by frappe.io every year as one of their Flagship Conferences. Whilst the conference was really fun for someone like me who is not interested in business the product keynotes were actually something really good on the 2nd day. Frappe Mail their newest product was in my hearing from the last 4-5 months and I was pretty excited to see if it was just a Frontend Wrapper to any existing service or if there was something interesting going on under the hood.
It is quite interesting to see how the architecture for it is planned for some newbie like me as the thought of digging deep into this process is what never came to my mind. That’s when I came to know a tad bit about everything how an email is sent and what the procedure in general is, which includes MTA’s (Mail Transfer Agents), MUA (Mail User Agents) other tools like RabbitMQ, Haraka MTA, and Frappe mail_agent tool and policies related to checks like DMARC and SPF.
Frappe Mail Basics
As Frappe mail was announced just 2-3 days ago, it obviously did not have any steps on how to do the actual configuration in any of Frappe’s internal stuff (I refer to doctypes here - checkout “What is a Doctype”). The only thing I have as a reference is the codebase which I used a bit to understand some whereabouts on how to configure mail locally. To understand how Frappe Mail is orchestrated, please check out the repository readme, it is very beautifully documented! (the only thing :p).
So, I started out by installing the app locally on my Frappe Bench and eventually the initial setup I was supposed to do was configuring a root domain name for the emails. With 0 clues of anything I started and decided to use mangeshm.xyz as the default domain and it automatically generated a user/email called “dmarc@mangeshm.xyz” which I am still confused about why it did. And I moreover also used this email as a postmaster to send status/failure messages (which never worked obviously).
After that came the mail domain configuration part where I have to connect the domain with frappe mail system using TXT (dmarc and spf) and MX records. It was pretty simple frappe mail itself gives you records you have to add and then you just have to verify the records.
Additionally, every email service has its own box called a Mailbox (it’s basically your email client’s front screen which includes a bunch of incoming and outgoing email conversations).
Frappe Mail uses RabbitMQ and Haraka MTA. RabbitMQ in short is basically a tool to push your emails into a queue and help in the delivery. And Frappe has implemented their own MTA called mail_agent which sits on the top and also runs haraka in a virtualenv. Haraka is again very old (the project was started in 2012) and well-maintained. I went ahead and installed mail-agent, and set it up (the setup here includes Haraka and RabbitMQ configuration ex: ports, username and passwords, etc).
With Haraka SMTP by default it is configured to listen on [::0]:25 but from what I have done myself and seen others do is listening on port 587 standardly. But fine, but the catch for me was whenever I was running mail-agent start (from Frappe’s mail agent) I was constantly getting this error where EADDRINUSE, Address already in use but after checking if the port was already occupied I got nothing and mail-agent used to try listening on port 25 and eventually stop after 30s. After searching a bit, I found a comment from Oct 4, 2024, in one of the Haraka issues and I found out that port 25 is privileged (I was dumb to not know this) and then I tried running it with sudo and it ran fine. Later, after talking with the dev, I redo the mail-agent setup on the root.
My shit-show with Frappe Mail in action
After I did all the setup it was time to send some emails. One noticeable and fun part here is that during all of this mess, I was trying to do, one key thing I forgot was “I can’t send emails from localhost to any email service like gmail.com or tutanota.com, etc” laughs harder. I continued trying to debug this by sending multiple emails and reading everything that I wasn’t supposed to.
I sent around something 50-60 emails doing multiple tries and all of them showed “Transferred” but what I needed to see as the status was “Bounced” because Frappe mail can only show the reason once the email bounced back. And after almost 7-8 hours the next day when I checked all of the emails had changed their statuses from “Transferred” to “Bounced”. And just like everyone else “I got the flow I wanted - error > fix > error > fix again” and I got nice reasons why it failed.
Unauthenticated sender email
The above description talks about DMARC Policies. DMARC Policies basically are some rules and regulations based on email authentication. It works with SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail) both of which help in finding spammers and malicious mediators between the sender and receiver. The last time my encountered DKIM was while setting up a bitwarden instance for FOSS United Foundation use during November 2023. I was trying to setup an email address on AWS and DKIM was failing every time. But only this time I was able to understand what DKIM actually does in the whole process.
So, getting back to 5.7.26 policy basically means that mangeshm.xyz is not authorized by the receiver (obviously because you can’t send emails from localhost to gmail.com). After digging up I found why and how it was identifying it. DMARC policies are of three levels “none/monitor”, “quarantine”, and “reject”. And as you can see below, the DMARC DNS record that frappe mail gave was in the highest security level “reject”.
In the whole record, v stands for DMARC version, p for policy tags, rua is basically the email address to which you want to send aggregate reports.
If I’m not wrong, I was getting the DMARC policy error because
- my domain isn’t verified (dmarc@mangeshm.xyz) which was setup locally (still not sure though)
- And probably the dumb part - it was running on localhost :skull:.
But someone can receive your email from localhost
And that someone is none other than a DKIM Validator Service.
Do you see that one “Sent” Green Status record in the above list? Apparently, while talking to GPT, I came to know about DKIM Validator. It’s again a very fun piece of software. DKIM Validator like tools help the user in verifying if their setup/configuration is perfect and ready to work.
How do these Validators work?
No concrete answer, but from what I can understand, it must be the following way.
As for DMARC checks, Alex (alex@unknownworld.com) is an unauthenticated email address (just like my above 5.7.26 DMARC case) and he sends an email to the validator (gobrr@validator.com). The validator receives it up and recursively does a DNS lookup on unknownworld.com to find a _dmarc record on it, if it’s found it verifies if the tags are right or not. And hence because p=reject is available in the record, I get that error message when the email bounces back? Must be fun to write a validator/inspection tool for DMARC and SPF too.
And For DKIM, maybe just verify the DKIM keys?
I am again not very confident about the above answer, if anyone knows or has good knowledge about the above, please do not hesitate to drop a text on telegram.
Anyways, after all the above things I did, I was happy that I was atleast able to send an email to a dkim validator. After which I realized I don’t have time and resource to setup an actual email server for myself.
But in all, I think its was also kind of easy to understand some of the above things because of Frappe mail being written in python which is very easy to read and figure out.
Do I understand any of the above properly now ?
I wouldn’t say I have understood all of the above concepts or technology very much in depth but with all the experimentation there’s a slight idea in my mind above all the above things. And it shouldn’t be very hard later in the future to debug all the issues I might get while setting up my own mail server. SMTP is really cool, I don’t know how simple it is. Maybe I’d understand SMTP well once I read the whole RFC5321. :)
And while I was trying to read about SMTP, I stumbled upon this 500-miles on wiby.me :p.
