I was reviewing some stats for Mailintel and was curious about what email service top apps use; being particularly interested in transactional email services. Is there a way I can go through my mailbox, check product emails and check the service for each? Sounds like a fun experiment to give a shot.
Connecting my mailbox
With my recent experiments with IMAP, connecting to my mailbox wasn’t difficult. I am using emailjs-imap-client, a JS IMAP client.
const ImapClient = require('emailjs-imap-client').default
;(async function () {
try {
const confOption = {
auth: {
user: process.env.EMAIL,
pass: process.env.PASS
},
logLevel: 'error'
}
if (process.env.PORT === '993'){ confOption.useSecureTransport = true }
const imap = new ImapClient(process.env.HOST, process.env.PORT, confOption)
await imap.connect()
} catch (e) {
console.log(e)
}
})()My environmental variables look something like this:
EMAIL="[email protected]"
PASS="mypassword"
PORT=993
HOST="imap.mail.yahoo.com"If you are connecting to a Yahoo mail, you will need to generate an app password. Using your real email password won’t work.
Gmail is a little more complex. To start with, you need to enable IMAP access your account. You can then go with any of the options:
- Allow less secure apps. But don’t forget to turn this off once you are done with the experiment. If you still can’t connect, you may need to allow app access to your account.
- Create an app password (more secured option).
Checking product emails
Getting product emails is a tricky one. How do you identify product emails from regular emails? How do you differentiate marketing from transactional emails? This can be done but the ideas I came up with were not worth the effort. In the end, I decided to pull all the emails instead. (This was also easier because I have a Yahoo mail dedicated to product signups, subscriptions and newsletters).
const ImapClient = require('emailjs-imap-client').default
;(async function () {
try {
// …connection code
await imap.connect()
const box = await imap.selectMailbox('INBOX')
// Reading 5k mails at once can choke the process so let's chunk into 50 mails per request. We are also assuming there are > 5k emails in the mailbox
let start = +box.exists - 5000
while (true) {
const messages = await imap.listMessages('INBOX', `${start + 1}:${start + 50}`, ['uid', 'body[]'])
if (!Array.isArray(messages)) { return }
// Do stuff with email here
start += 50
if (start >= +box.exists) { break }
}
} catch (e) {
console.log(e)
}
})()Checking the service provider
How do I know the email service used for the mail? I checked a couple of email headers and noticed a couple of places the provider details can be extracted from. Here is what an email header looks like:
At first glance at some headers, the
Message-IdMessage-ID: <[email protected]>
Message-ID: <010001703764a668-4ebb47b9-4454-4081-afc4-e6448cd22897-000000@email.amazonses.com>
Message-ID: <[email protected]>By looking at the host part of the
message-idmessage-idsreceivedMessage-ID: <[email protected]>
Received: by filter0225p1iad2.sendgrid.net with ~Message-Id: <[email protected]>
Received: from mail132-5.atl131.mandrillapp.com ~The
receivedreceived- For more than one received header, match
of the secondby *
header.received - If that doesn’t exist or does not have a host address or there is just one received header, then match the
of the first(EHLO *)
header.received
Now we can rewrite our script to use
receivedconst ImapClient = require('emailjs-imap-client').default
const simpleParser = require('mailparser').simpleParser
const PULL = 10000
;(async function () {
const sites = {}
try {
const confOption = {
auth: {
user: process.env.EMAIL,
pass: process.env.PASS
},
logLevel: 'error'
}
if (process.env.PORT === '993') { confOption.useSecureTransport = true }
const imap = new ImapClient(process.env.HOST, process.env.PORT, confOption)
await imap.connect()
const box = await imap.selectMailbox('INBOX')
const exists = +box.exists
if (exists <= PULL) {
console.log('You specified a pull number more than or equal to the number of emails in the mailbox')
return
}
let start = exists - PULL
while (true) {
const messages = await imap.listMessages('INBOX', `${start + 1}:${start + 50}`, ['uid', 'body[]'])
if (!Array.isArray(messages)) { return }
for (const message of messages) {
const mail = await simpleParser(message['body[]'])
const headers = mail.headers.get('received')
let esp
if (headers.length > 1) {
const header = headers[1]
const match = header.match(/by ([^\s]*)/)
if (match && match.length > 1 && match[1].indexOf('.') !== -1) {
esp = match[1].split('.').slice(-2).join('.')
}
}
if (!esp) {
const header = Array.isArray(headers) ? headers[0] : headers
const match = header.match(/EHLO ([^)]*)/)
if (match && match.length > 1 && match[1].indexOf('.') !== -1) {
esp = match[1].split('.').slice(-2).join('.')
}
}
if (!esp) {
// Todo: Track failed matches
continue
}
if (sites[esp]) {
sites[esp]++
} else {
sites[esp] = 1
}
}
start += 50
if (start >= exists) { break }
}
await imap.logout()
await imap.close()
} catch (e) {
console.log(e)
}
console.log(sites)
})()Results
This was the breakdown after stripping the results to the top ones.
Few notes:
- rsgsv.net, mcsv.net and mcdlv.net are from Mailchimp. As you know, Mailchimp provides marketing email service. Mandrillapp.com is the transactional email service for Mailchimp. It used to be a standalone service until it became deeply integrated into Mailchimp.
- spmta.com is for Sparkpost.
- marketo.org is for marketo.com, currently owned by Adobe. Provides marketing email service.
- Interesting to see intercom there. It's amazing the number of products that use Intercom.
- mailgun.net and mailgun.org belong to Mailgun, obviously. Mailgun also recently acquired Mailjet. Mailjet provides both marketing and transactional email service. It will take more deep-diving into the headers (or content) to figure if the mail was sent as a marketing or transactional email.
- Sendgrid, also like Mailjet, offers both marketing and transactional email service. It’s the most used from my experiment but it’s hard to know what fraction of that is marketing and what other is transactional. PS: they were recently acquired by Twilio.
Conclusion
I merged the same providers for a more accurate chart.
This is only a fun experiment and doesn’t show the true market share of the services. Since the data is based on the product emails in just my mailbox, it is flawed by selection bias.
However, it’s something you can run on your mailbox for the fun of it. You can also extend it to see what service your favourite product/app uses.
Remember, you may need to look at the full headers of some of your emails to come up with the best way to know the service provider.
(Originally published here)
