loading...
Cover image for Updating my Reddit Workflow with Pipedream

Updating my Reddit Workflow with Pipedream

raymondcamden profile image Raymond Camden Originally published at raymondcamden.com on ・3 min read

This was originally just going to be a tweet, but then I realized I wanted a bit more space to talk about it and figured I'd write it up as a post. And since this is my blog and I can do what I want to, you get to enjoy this little nugget of information.

Back almost exactly a month ago, I blogged about using Pipedream to build a Reddit email report. The idea was that I wanted a daily email of posts from my subscribed subreddits for the past 24 hours of content. The implementation was a bit complex. I used one workflow to handle "get a days worth of content from subreddit" as a general "API" and another workflow connected to my authentication. It handles getting my subscriptions, hitting the API, and then generating the email. Here's an example of how that looked:

Email report

This worked well, but after a while of actually getting the email, I noticed some problems. The email takes all of the posts from all of my subscriptions and sorts them together. I thought this made sense to me, but I noticed it made it harder to actually read the content. Sometimes I don't care about a subreddit and mentally it just felt weird going from the movies subreddit to the Acadiana one. Also, some subreddits get a huge amount of traffic in a day. The email was hard to read and just too long.

So I decided to fix that. I didn't want to edit my original workflow because I wanted it to still be a reference for the older post. Luckily Pipedream makes that simple. I turned off the CRON schedule on the workflow and just used the copy command.

Next, I edited the Node.js code step that combines and sorts my data. This:

for(let i=0;i<result.length;i++) {
    this.posts = this.posts.concat(result[i].data.result);
}

this.posts.sort((a,b) => {
    if(a.created_utc < b.created_utc) return 1;
    if(a.created_utc >b.created_utc) return -1;
    return 0;
});

Became this:

for(let i=0;i<result.length;i++) {
  this.posts = this.posts.concat(result[i].data.result.slice(0,10));
}

No more sorting together and I'm only getting the first ten entries from each subreddit. (Those posts should be date sorted already. Should be.)

Then I modified the step that formats the email. I added in code to notice when a new subreddit start and added HTML to make it more visibly separated.

let dotLeft = function(s, len) {
  if(s.length < len) return s;
  return s.substring(0, len)+'...';
};

let lastSub = '';

this.subject = `Daily Reddit Report`;
this.body = `
<h1>Daily New Posts for Your Reddit Subscriptions</h1>
<p/>
`;
steps.get_and_sort.posts.forEach(p => {
  // only show thumbnails when they aren't "self","default","nswf" - or more broadly, not a url
  if(p.thumbnail.indexOf('http') === -1) delete p.thumbnail;
  let text = '';
  if(p.is_self) text = dotLeft(p.selftext,200);
  if(p.subreddit !== lastSub) {
    this.body += `<hr/><h2>${p.subreddit}</h2>`;
    lastSub = p.subreddit;
  }
  this.body += `
<p>
${ p.thumbnail ? '<img src="'+p.thumbnail+'" align="left" style="margin-right:10px">':''}
<b>Title: ${p.title}</b><br/>
URL: <a href="${p.url}">${p.url}</a><br/>
Reddit URL: <a href="https://www.reddit.com${p.permalink}">https://www.reddit.com${p.permalink}</a><br/>
Author: ${p.author}<br/>
<br clear="left">
${text}
</p>  
  `
});

Basically - notice when the subreddit changes and add a horizontal rule and header. Also notice I modified my code on when to show images. This seems to work much better.

And that's it. My new workflow may be found here: https://pipedream.com/@raymondcamden/daily-reddit-posts-3-p_PAC9DV/edit?e=1c8nZNETuFxToNCya2eVCafjVAu I'm trying my best to make good use of the Readme feature to document what I've done.

Posted on Jan 18 '19 by:

raymondcamden profile

Raymond Camden

@raymondcamden

Lead Developer Evangelist at HERE Technologies, expert builder of cat demos, and best in my house at Just Dance.

Discussion

markdown guide