Hey there!

I'm trying to dive deeper into the web3 world and to keep up with its ecosystem news, I've decided to subscribe to some newsletters.

Mark Kop

@heymarkkop

@PatrickAlphaC Hey, Patrick! I'm loving your content! Do you have any web3/blockchain/smart contract newsletter to recommend?

13:55 PM - 17 Mar 2022

But the thing is, I'm not used to check my newsletter emails very often. I've tried making it a habit, but it just does not work for me.

What I'm used to, however, is checking announcement and groups channels on Telegram, so I figured out that it would be a good idea to forward those emails to my telegram chat.

To be honest, it ended up being way harder than I expected, but with some code and research I managed to accomplish this goal and want to share it with you.

🤖 Using Gmail Bot

This Telegram Gmail Bot was actually a good solution as it would allow me to filter new emails by labels and get them in a readable way in the telegram chat.

However I've experienced some heavy delay between commands and sometimes they didn't work at all.

I also didn't like giving it access to all my emails and rather forward them based on a filter.

📫 Forwarding Emails to Telegram with IFTTT

The first service I had in mind was IFTTT to get those emails forwarded to telegram directly.

IFTTT is an automation tool that lets you easily script actions that link together a wide variety of devices and services.

Still, I had some problems when using this approach

😑 GMail is not compatible with IFTTT

Since I'm using GMail to receive my newsletters, I would have to automatically forward them to IFTTT Email.

To be able to register a forward email address on GMail it's necessary to access a confirmation email in the target email provider.

But users can't access trigger@applet.ifttt.com emails, so the solution was forwarding those emails to another email service such as Outlook.

🧠 Raw HTML Emails are hard to read

Forwarding emails to another email service, then forwarding them again to IFTTT was awkward, but it worked.

But as you can imagine, those cool and colorful emails subscribers get in newsletters have a lot of HTML stuff instead of just plain text as normal emails.

And telegram just can't parse it. At least not with as raw content.

🎣 Forwarding Emails to Webhook with IFTTT

So the solution is coding a parser and then sending it to a telegram bot.

I already have a microservice application hosted on Heroku which I'm using for this system, but you can simply create and deploy an Express REST API and use it instead.

Now the IFTTT applet is getting an email and triggering a webhook - which is hosted by ourselves.

In the image above, you can see that I'm using a webhook.site url for the webhook url. You should use the one from your own application instead, but the webhook.site service is very useful for debugging the requests you expect receiving.

This IFTTT Applet works, but there are some caveats as well.

🔨 IFTTT breaks some lines "randomly"

It took me a while to figure out that IFTTT itself was adding a \n character in some lines. Probably because that line was too long.

While this was not a big problem, it was kinda sad to see some formatting breaking in the text message.

📜 IFTTT can't forward large emails

Now this was what made me to look for an alternative. The Week in Ethereum newsletter email is pretty big, and IFTTT just couldn't trigger the webhook when receiving it.

✅ Forwarding Emails to Webhook with Zoho

This is the current solution I'm using. I felt it was worth to mention the IFTTT approach in case people want to use it for other similar applications.

I've stumbled with this service on a reddit post and it worked perfectly for my needs. Not only that, but I could even stop using the Outlook email and forward emails from Gmail directly to Zoho Email.

📠 Parsing the HTML Email

Now that emails can be received successfully, they have to become readable and be sent to the telegram bot.

I will not enter the details on how to create a REST API application. I'll focus on which steps I did to parse the html and make it compatible to telegram's API.

Below I'm showing part of the code I'm using to parse the HTML.

const {decode} = require('html-entities')
const FilterHTML = require('filterhtml')

//...

async onWebhookTrigger({html}) {
    try {
        const parsedHtml = html
            .replace(/<p[^>]+>(.|\n)*?<\/p>/g, match => '\n' + match.replace(/\n/g, ''))
            .replace(/<li[^>]+>((.|\n)*?)<\/li>/g, (match, p1) => `\n* ${p1.replace(/\n/g, '')}`)
            .replace(/<a([^>]+)><\/a>/g, '')
            .replace(/<a([^>]+)>((.|\n)*?)<\/a>/g, match => match.replace(/\n/g, ''))
            .replace(/<span([^>]+)>((.|\n)*?)<\/span>/g, match => match.replace(/\n/g, ''))
            .replace(/<h1([^>]*)>((.|\n)*?)<\/h1>/g, (match, p1, p2) => `\n<b>${p2.replace(/\n/g, '')}</b>\n`)
            .replace(/<h2([^>]*)>((.|\n)*?)<\/h2>/g, (match, p1, p2) => `\n<b>${p2.replace(/\n/g, '')}</b>\n`)
            .replace(/<h3([^>]*)>((.|\n)*?)<\/h3>/g, (match, p1, p2) => `\n<b>${p2.replace(/\n/g, '')}</b>\n`)
            .replace(/<ul([^>]*)>((.|\n)*?)<\/ul>/g, (match, p1, p2) => `\n${p2}\n`)
            .replace(/<br>/g, '\n')

        const filteredHtml = FilterHTML.filter_html(parsedHtml, {
            'a': {
                'href': 'url'
            },
            'b': {},
            'strong': {},
            'i': {},
            'em': {},
            'u': {},
            's': {},
            'pre': {},
        })

        const decodedHtml = decode(filteredHtml)

        const text = decodedHtml
            .replace(/<b>\n*<\/b>/g, '')
            .replace(/<a([^>]+)><\/a>/g, '')
            .replace(/\n\s*\n\s*\n/g, '\n\n')
            .replace(/&nbsp;/g, ' ')
            .replace(/\n\n\s*/g, '\n\n')
            .replace(/<(([^<]*)@([^>]*))>/g, '')

        // Now you would send "text" to Telegram's Bot API
    } catch (error) {
        console.log(error)
    }
},

I'm first doing several replacements to remove some line breaks inside tags and to convert some tags to others.

I'm also using FilterHTML to remove tags that are not supported by telegram when using HTML parse mode.

Then I decode HTML entities with html-entities to restore special characters such as accents that are used in some Brazilian Portuguese newsletter emails I'm subscribed to.

Finally, I do a few more replacements to remove empty tags and multiple line breaks.

📩 Sending the parsed email to Telegram

As you can see in the code above, instead of triggering Telegram's API I've added a comment. That's because in my application, I'm calling another microservice that handles telegram integration. You can check it here.

In your code, you can simply call the sendMessage endpoint with your bot's token.

Something like this:

function sendMessage(chat_id, text) {
  return fetch(`https://api.telegram.org/bot${BOT_TOKEN}/sendMessage`, {
    method: "POST",
    headers: {
      'Accept': 'application/json',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      chat_id,
      text
    })
  });
}

You might need to search for other tutorials on how to create and use a Telegram Bot.

✨ Tips and Tricks

✂️ Sending multiple messages for big emails

Telegram has 4096 characters limit for text messages, and you can't simply split the message with a text.substring(0, 4096) like loop because you might end up leaving an HTML tag open. That throws an error when sending the message to Telegram's API.

// This won't work for HTML parsed messages
if (text.length >= 4096) {
    for (let i = 0; i < text.length; i+=4096) {
        await this.axios.post('/sendMessage', {
            ...options,
            text: text.substring(i, i+4096)
        })
    }
} else {
    await this.axios.post('/sendMessage', options)
}

The best solution that came to my mind was a recursive function to split the message in blocks using the last line break character.

async splitAndSend(text, options) {
    const subtext =  text.substring(0, 4096)
    const textSplittedByLineBreak = subtext.split('\n')
    if (textSplittedByLineBreak.length === 1) return
    textSplittedByLineBreak.splice(-1)
    const previousTextBlock = textSplittedByLineBreak.join('\n')
    if (!previousTextBlock) return
    await this.axios.post('/sendMessage', {
        ...options,
        text: previousTextBlock,
        disable_web_page_preview: true
    })
    const remainingText = text.substring(previousTextBlock.length, text.length)
    return this.splitAndSend(remainingText, options)
},

async sendTextToChatId(text, chatId, parseMode = 'html') {
    try {
        // ...

        if (text.length >= 4096) {
            await this.splitAndSend(text, options)
        } else {
            await this.axios.post('/sendMessage', options)
        }

        // ...
    } catch (error) {
        this.logger.error(error)
    }
},

🌐 Using Ngrok to test the webhook locally

Ngrok is a great tool to expose your localhost server securely.

You can use it to fire up your application locally and send emails to yourself so you can check how they are being received by your webhook.

🌙 Using Insomnia to test the webhook

You might want to use Insomnia to avoid the manual work of sending the same email over and over to your Zoho Mail and therefore triggering your webhook.

🏁 Conclusion

What I thought it was just using a Gmail-Telegram integration service ended up being the development of a webhook application.

I had a lot of fun studying these solutions and developing my own one, I hope someday non-developer users will have a more user friendly way to do this kind of integration.

Do you know any alternative? Let me know!