Excuse my lack of knowledge on the subject, but what does it mean for a reducer to be "commutative" and "parallelizable"? And what do you mean by "merging function"?
Oh, now I understand your point about not thinking about the fact that iteration is being used! If they're not done in parallel, you don't get the previous value :).
If we don't care about the order of the incoming ids, and just want to get the sets of ids of each article, we could split the counting between multiple threads or even machines.
Something like this silly thing:
constposts=[{id:0,category:"fairy tales",title:"Gommunist Manifesto"},{id:1,category:"frontend",title:"All About That Sass"},{id:2,category:"backend",title:"Beam me up, Scotty: Apache Beam tips"},{id:3,category:"frontend",title:"Sanitizing HTML: Going antibacterial on XSS attacks"},{id:4,category:"frontend",title:"All About That Sass"},{id:5,category:"backend",title:"Beam me up, Scotty: Apache Beam tips"},{id:6,category:"frontend",title:"Sanitizing HTML: Going antibacterial on XSS attacks"},{id:7,category:"frontend",title:"All About That Sass"},{id:8,category:"backend",title:"Beam me up, Scotty: Apache Beam tips"},{id:9,category:"frontend",title:"Sanitizing HTML: Going antibacterial on XSS attacks"}]constidsByCategory=posts=>{constcategories=newMap()for(const{category,id}ofposts){constexisting=categories.get(category)if(!existing)categories.set(category,[id])elseexisting.push(id)}returncategories}constmergingFunction=([result,...results])=>{for(constotherofresults)for(const[category,ids]ofother){constexisting=result.get(category)if(!existing)result.set(category,ids)elseexisting.push(...ids)}returnresult}constparallel=posts=>{const{length}=postsconstresults=[]for(leti=0;i<length;i+=2)results.push(idsByCategory(posts.slice(i,i+2)))returnresults}constresults=parallel(posts)console.log(results)constcategoryPosts=mergingFunction(results)console.log(categoryPosts)
And by "commutative" I mean that if you pushed an array into a number you'd get an error, and that 'a'+'b' and 'b'+'a' gives you different strings.
Whereas integer addition without overflow is commutative: 1+2 gives the same result as 2+1 and const s = new Set; s.add(1); s.add(2) as well.
Excuse my lack of knowledge on the subject, but what does it mean for a reducer to be "commutative" and "parallelizable"? And what do you mean by "merging function"?
Oh, now I understand your point about not thinking about the fact that iteration is being used! If they're not done in parallel, you don't get the previous value :).
If we don't care about the order of the incoming
id
s, and just want to get the sets ofid
s of each article, we could split the counting between multiple threads or even machines.Something like this silly thing:
And by "commutative" I mean that if you pushed an array into a number you'd get an error, and that
'a'+'b'
and'b'+'a'
gives you different strings.Whereas integer addition without overflow is commutative:
1+2
gives the same result as2+1
andconst s = new Set; s.add(1); s.add(2)
as well.Oh, wow. You're right about calling it "silly". 😂
It's silly in the sense we have only 10 items instead of billions, they are in memory at once, and it doesn't actually spawn threads or workers.