DEV Community

Feature update: Feed

Amy Lin on November 29, 2021

Hello, wonderful DEV community! My name is Amy, and I’m a product manager here at Forem. I started my journey here this past August so if we haven...
Collapse
afif profile image
Temani Afif

Yes! I was waiting for this.
Those "bad" posts gathering instant reactions and being visible for a long period to gather even more reactions to become even more visible .. are annoying. I have even seen people who intentionally create alt accounts to have that first boost and make sure their posts are visible on the top and if you check them, it's only promotional content.

I would also love if we can give more weight to the "rate quality" feature given to trusted users. I know we might have some abuse here as well but it would be insignificant compared to gaming the reactions especially that trusted users are manually selected by the DEV team.

Collapse
terabytetiger profile image
Tyler V. (he/him)

I would also love if we can give more weight to the "rate quality" feature given to trusted users.

I believe this is already the case - if I'm remembering correctly the Thumbs Up/Down by Trusted users are about 10x as valuable as a standard user's reaction. (This is based on a discussion in the connect mod chat so I can't go back to confirm this 😅)

Collapse
afif profile image
Temani Afif

Considering that a user can perform 3 reactions and that you can add reactions to your own post, 10x is still low. With 2 extra accounts you can get 9 reactions immediately so you are immune to the first vote.

Based on my experience, If I am late to downvote a bad post before 10/15 reactions then it's too late unless more trusted users do it before that post gather more reactions.

Once you have your first boost, you are on the top and more reactions will come easily so it become more difficult to fight. Bad posts can easily reach 100+ reactions in a few hours.

Thread Thread
amyatforem profile image
Amy Lin Author • Edited on

@afif This type of fine-tuning is exactly what @jeremyf and I have in mind as potential candidates for future experimentations. We believe this is just a start for many iterations to come. I am logging it here in an issue to make sure we don't miss it when we are ready for the next round of experimentation.

Thanks for your feedback!

Collapse
jeremyf profile image
Jeremy Friesen

As the architect of the feed update, there are two things I'm super curious about:

1) System performance of the new feed algorithm
2) Higher utility for the DEV community

In the case of system performance, if this new approach is more performant than the current approach, I’ll be super happy. Why? Because of how I architected the new feed algorithm. More on that in a bit, but it informs the higher utility for DEV.

The second point I’m curious about is the higher utility for DEV. In writing the weights for each “property” of a post, the weights were intuitive guesses. We needed something to propose, and those current weights are what we’re going with.

However, the weights are very configurable. To the point where we could envision allowing Forem creators to choose which “properties” to consider for their feed and the weights to apply to those properties.

So, as I see it, if the performance is better with the new feed algorithm, we’re going to then run experiments into tweaking the properties and weights of that algorithm.

In other words, if the new feed algorithm is viable from a performance stand-point, we will now have a more transparent and extensible approach that is adaptable for all Forem instances.

Collapse
inhuofficial profile image
InHuOfficial • Edited on

Hey Jeremy, quick heads up I think I am one of the ones (currently, not sure if you randomise at a certain time frame) using the new feed.

I think there is a tweak needed somewhere as my feed is pretty much in date order now like "latest" but with selected articles.

screen shot of feed with articles in release time order

I hope that is useful feedback (if you weren't already aware) as it is burying things of value.

Also to save you having to look it up I have no preferred tags selected, just in case it only affects people without any feed preferences.

Collapse
jeremyf profile image
Jeremy Friesen

Thank you for sharing this feedback. The "Relevant" feed in the experiment first finds the relevant posts, limits the results to the highest relevancy, then sorts by date.

Thread Thread
inhuofficial profile image
InHuOfficial • Edited on

No problem, I know how tough something like this can be to balance when there are so many variations of tag preferences etc.

Just a personal preference but relevancy first felt better (if that is the difference) and you might have slightly overcooked the pulling back of reactions (I could be entirely wrong, a sample size of one is not very useful I know!) as that seems to have 0 bearing now?

But I would imagine the end goal is to see which feed produces more balanced engagement and if this works then that is great!

Hope the experiments continue! Can't wait for the write up on the results! ❤

Thread Thread
jeremyf profile image
Jeremy Friesen

At present, the levers we are using are found here:

github.com/forem/forem/blob/main/a...

And the reactions are very limited in impact:

github.com/forem/forem/blob/main/a...

These "levers" are our first configuration. But we're planning on testing the configuration of those levers.

In addition, we're looking at the computational performance of the two feed options.

Thread Thread
inhuofficial profile image
InHuOfficial

Super, thanks for those links, I knew I have seen that somewhere in the codebase but couldn't find it again!

The bit I didn't get is does that config mean that reactions have a slightly negative impact of 0.9988 if there are 0,1,2 or 3 reactions and then fall back to 1 for more than 3 reactions?

So at this stage essentially reactions are completely pointless (as I can give myself 3 reactions) to let you fine tune the other parts of your experiment? (which makes sense, A / B test one thing at a time so you know which lever pulls the feed which way!)

It is super interesting as where my gut leads me is that reactions are one of the more important metrics but the way they are gathered / assessed is the bit that needs addressing (I mentioned only counting votes from accounts that meet a certain criteria such as certain age or total number of likes for example), but often you can think one thing and find it has a completely unintended side effect!

The way you have the levers set is smart, wish I was the one playing with them as I would cause havoc going to the extremes 🤣!

Thread Thread
jeremyf profile image
Jeremy Friesen

Reactions are such a challenging component to consider. They are very low effort, so a question is how much do we "weight" do we give for low effort tasks? A second variable class, as you hint at, is related to the person giving the reaction.

And yes, in this experiment, we've greatly limited the impact of reactions.

One goal I have is to find "lever" configurations for communities to adopt. I can envision a lever configuration that says "We want to see more posts from new members to hear what they have to say, but we want to ensure that old members reactions are more heavily considered." (That's an odd example but one that I hope the levers options can help make feasible).

Thread Thread
inhuofficial profile image
InHuOfficial • Edited on

Oh I love that last concept! The predefined preferences would be great for people running different instances of Forem!

The low effort part is a fair point! (I would love to see what would happen if you could only upvote 3/5 articles a day for example, that would be super interesting to see how people behaved when their votes were "valuable"!)

One idea could be time on page / scroll percentage to assess article quality / engagement? I know that is getting complex though (but one positive is it can nearly all be done client side with little performance impact)!

The only other idea I have to throw in the pot (and I have no idea if it is a good one) is to weight comments with a certain length / word count (so the "thank you" comments get ignored) more heavily.

Then the number of comments on an article that meet this criteria are then used as a weighting factor.

Yet again this is just based on my usage of the site where I look for posts with a lot of comments as they tend to either be really good or really controversial (and most of the DEV team know I am the one who loves to be involved in controversial stuff on here 🤣).

In all seriousness, looking at number of comments tend to lead me to some cracking content (or at least to engaged authors who respond), so it may be another lever to add after all of your planned experiments?

Anyway I have taken up plenty of your time I am just enjoying the complexity of the task you have undertaken as it is so nuanced!

Collapse
inhuofficial profile image
InHuOfficial

It will be interesting to see whether this works, I hope it does.

Being the cynic that I am I think this is the wrong move, however.

While reaction farming is a big problem every site faces, the reduced relevance of votes and comments will have a side effect,

Posts that are high quality, but from lesser known authors will not get the boost they deserve if people find their article useful and upvote it and comment to show their support.

The problem (as I see it) is the tag system and always has been as far as I can tell.

If you write an article just add the #javascript tag and you will double your views for example. Albeit a mod may come and remove the tag but I have gamed the system by adding a JS example that doesn't add much to the article just so I can include the tag and it worked.

That is where I would reduce the relevance.

As for tackling farming reactions, having a hidden system that decides whether or not to count a vote for the algorithm, based on account age or some other factor, would be great.

So you have a public likes counter and then you have a separate one for the algorithm that only counts votes and comments that are from accounts xxxx days older or with a total of yyyy votes.

Don't get me wrong, I have seen some big improvements in the feed since moderators seemed to have been more involved and some of the experiments you have run have been great.

The only other thing that would be interesting to run an experiment on would be "DEV picks".

Similar to the "Top 7 posts" weekly update but actually incorporated into the feed (increased visibility and longevity perhaps?) so there is a place we can see posts that have been hand picked by the dev team and the mods. These are likely to be very high quality and hopefully quite diverse in nature.

Anyway those are my 2 cents and I am always impressed by the work the team at FOREM and DEV do to try and always improve. ❤🦄

Collapse
amyatforem profile image
Amy Lin Author • Edited on

Just to make sure I understand you correctly (you know, I am still rather new here 😉), when tagging a post with JavaScript, you will double the views because there are a lot of people following this tag. Correct? I do see how people can abuse the tags and hence reduce the relevance here. This is going to be an interesting problem to solve.

I have also gathered your suggestions on feed experimentation here for review at a later time.

Thanks so much for the write-up! I love your candid feedback.

Collapse
inhuofficial profile image
InHuOfficial

Yes that is correct.

But the problem isn’t abusing the tags as such it is more that you have to abuse the tags if you want to get an article to show up enough to get traction (if that makes sense?)

I Love the steps the team are taking, I don’t envy you all as trying to solve biases and abuse in a feed is not an easy task! ❤️

Thread Thread
amyatforem profile image
Amy Lin Author

Ah. I see. This is one of the discovery problems we are facing for less known tags (i.e. with fewer followers), similar to new authors having a hard time getting their posts surfaced, albeit of high quality.

Thanks for the extra context. It helps me see what problems you are facing with the feed as an author. 🥰

Thread Thread
inhuofficial profile image
InHuOfficial • Edited on

All I have to say now is welcome to DEV and as your first 2 official posts / features these are great (despite the criticism! 🤣) and super positive steps!

I look forward to seeing what you tackle next / the outcome of this feed experiment! ❤

Collapse
madza profile image
Madza • Edited on

These are awesome news that the DEV team is working to fix the feed 👍
I've posted regularly for 2 years and really love the platform and the community 💖

Collapse
somedood profile image
Basti Ortiz

Some examples include your feed not reflecting tag preferences, too many listicles, and more recent posts rather than relevant posts... We realized we are unintentionally giving reactions too much weight when calculating the aforementioned score of a post.

It is worth noting, of course, that the extra weight (previously) put on reactions is not totally unfounded. After all, it is one of the main metrics for user engagement.

The "too-many-listicles" phenomenon only occurs simply because listicles are inherently designed to be bookmarked (i.e. one of the three reaction types that DEV considers). Hence, we end up in a situation where in addition to the hearts and unicorns a listicle receives, the extra bookmarks greatly amplify what would have otherwise been a regular article.

However, the same argument can be said about regular articles, tutorials, and visualizations. That is, they can also "game the Algorithm" by being designed to be bookmarked—intentionally or otherwise. But, what separates listicles from regular articles is that they are more easily automated than full articles, guides, etc. And that is exactly why the feed becomes bloated. To make a long story short, automation and bookmarks make a perfect storm.

Unfortunately, I have no recommendations for addressing this at the moment. It's really tricky because—as mentioned earlier—certain patterns are just inherently gameable.

Anyway, with that said, I just wanted to bring some additional perspective to the table. Looking forward to hearing about the proposed solutions! 🚀

Collapse
amyatforem profile image
Amy Lin Author

The "too-many-listicles" phenomenon only occurs simply because listicles are inherently designed to be bookmarked

This is a great insight! 🤔 We should test treating bookmarks differently from other reactions. I am logging it here for review.

Thanks for the great suggestion!

Collapse
wnbsmart profile image
Maroje Macola

It's great that you are trying to optimise this feature.

Since anyone could make alt accounts to give an initial boost to the post, these are my suggestions:

  • give more weight to the reactions from older accounts (e.g. older than month)
  • give more weight to the reactions from accounts which are more active (those who write posts should probably be the heaviest, then the ones who only comment posts, and finally, the ones who have no activity besides giving reactions)
  • if you collect how many posts has someone read (opened), those stats might also be included in the weighing algorithm (i.e. pretty much anything can be used for algorithm, sky is the limit)

Also, IMO, algorithm doesn't need to be publicly known, so it would be less abused. It's important that it works good and feedback is collected from users.

Collapse
jeremyf profile image
Jeremy Friesen

These are some great ideas, and one goal I have is to allow Forem creators to configure the feed "levers" to better match their community use case. And these "properties" that you're referencing may well be worth adding to the "levers" we make available.

Collapse
jeremyf profile image
Jeremy Friesen

A super quick update, we found a bug in the new strategy (see github.com/forem/forem/pull/15528 for details).

For the time being we've toggled off the new feed, until we can deploy the bug fix.

What did the bug look like? After interacting with DEV for a bit, you might have noticed your Relevant Feed went blank or became very sparse. That had to do with faulty logic (on my part) of limiting publication date ranges.

Collapse
thumbone profile image
Bernd Wechner

Love it. The idea of making the feed more relevant. I'm far less interested in popular than I am in relevant and interesting to me. Twitter, Facebook, Instagram are all for the popularity junkies.

I see some rather poor quality Dev articles on my feed at times but mostly just let them slide on by. Seeing less of those would please me.

Above all I don't need quantity either or volume in preference to quality. I don't mind a feed that delivers more slowly, the right stuff. I do read som top quality stuff here too. And I bookmark it for future reference. A lot in fact!

Collapse
jayjeckel profile image
Jay Jeckel

Good job to those that worked on this. It's always great to improve algorithms.

That said, I don't use the algorithm-driven feed; I use the latest feed that shows all the articles in the order they were posted, so instead of actually dealing with the low quality articles this seems to me like a way to sweep them under the proverbial rug where most users simply won't see them. So I have to ask, is anything being done to actually deal with low quality articles and those that post them?

Collapse
jeremyf profile image
Jeremy Friesen

Quick update, I've re-activated the alternate feed experiment.

Collapse
Sloan, the sloth mascot
Comment deleted
Collapse
jeremyf profile image
Jeremy Friesen

The "algorithm" of the alternate feed is to look at the "properties" of each article and give those properties weight.

Here's the relevant weighting, mostly with comments: github.com/forem/forem/blob/main/a...

Collapse
Sloan, the sloth mascot
Comment deleted
jeremyf profile image
Jeremy Friesen

I hope I didn't come off as aggressive. My intention was to highlight the implementation and how I hope it reflects people's choices over the more opaque current algorithm (which encodes and obfuscates people's choices).