This post is about the journey we took to improve the OpenSauced Dashboard, which provides valuable insights on open-source contributions.

Last October, we had an alpha launch of the new OpenSauced Insights dashboard. This launch was to coincide with the biggest open-source hackathon, Hacktoberfest. Our goal was to provide insights and reports on the number of contributions accepted, merged, and even spam. Since then, the dashboard has evolved beyond Hacktoberfest data and now includes GitHub avatars representing active and open GitHub Pull Requests in the last 30 days.

If you’d like to see your open source contributions connect your GitHub account to OpenSauced.

Challenges Faced

At the beginning, our team encountered challenges in sourcing and manipulating GitHub avatars to display a scatter plot e-chart on the dashboard. We struggled to find a suitable solution, especially when it came to handling a large number of images efficiently.

As part of this challenge, GitHub has an aggressive rate limit when you're not authenticated when requesting resources, like avatars from a URL. We needed to fetch the avatar, manipulate them, and then cache them. Because our product was meant to be public, we would attract a lot of usage, and it could cost real money quickly.

Solution: Leveraging nivo and Cloudinary

To overcome these challenges, the team turned to the nivo chart library for visualizations and Cloudinary for image manipulation and caching 271k images. We successfully integrated nivo charts into our dashboard, thanks to its rich set of data visualization components built on top of d3 and React.

By leveraging Cloudinary's image manipulation capabilities and caching strategy, we were able to round the avatars and seamlessly integrate them into the dashboard.

Today this works thanks to nivo and Cloudinary, but that journey included a lot of trials and testing for the perfect solution.

Nivo is an open-source library that provides a rich set of data visualization components built on top of the awesome d3 and React libraries.

yarn add @nivo/core @nivo/bar

Cloudinary is a hosted solution to manipulate and cache images for reuse.

Handling Image Processing with Apache E-charts

Before adopting nivo and Cloudinary, we initially used Apache E-charts (specifically a React wrapper called echarts-for-react) to handle image processing and loading. This approach proved extremely slow, and it was quite the process in figuring out a better solution while we had a real-time constraint.

The Journey to Finding Solutions

We faced a challenge being able to provide images on the page that were sourced directly from GitHub. The avatars needed could be cached and manipulated to match our rounded image design. The total number of displayed contributors during the event was around 150k, and today close to 300k contributors are represented as contributions in the most popular open source repositories on OpenSauced.

The e-charts for React solution gave us no access to the images when rendered, and provided limited options to edit the chart after it was displayed. We built our product in Figma designs first and were excited at the opportunity to have rounded images. Still, our e-chart library would only allow plain & squared images, and also any manipulation of the image was the challenge.

[design]

[reality]

There comes a time in every developer's career where you are presented a design that may be just out of reach or out of scope. The simplest request around the images broke our charts and required the entire team to brainstorm solutions.

You can see that brainstorming in our now closed issues.

Comment for #373

bdougie commented on Sep 16, 2022

Hey another thought here. What if we add cloudinary processing the images and pass those urls? This would give us the ability transform (border and round) the images, as well as cache them some where.

Alternatively, we could use imagemagick (cli tool for image transforms) + supabase storage to do the same for free.

@brandonroberts have you done anything like this before?

View on GitHub

Solution 1: JavaScript Approach

Our first attempt was manually rounding the images using a quick lib function I threw together.

function applyBorderRadius(imageElement) {
  imageElement.style.borderRadius = '50%';
}

// Get the image element
var image = document.getElementById('myImage');

// Apply the 50% border radius
applyBorderRadius(image);

It's always good to start with an approach, even if it is wrong.

This first solution needed to be corrected and didn't work because the images were limiting the e-chart with images on a scattered chart. We could not get direct access to the elements to manipulate after the fact.

We needed a way to manipulate the images before sending to the chart.

Solution 2: Imagemagick for Rounding Images

ImageMagick is a fun open-source platform for displaying, creating, converting, modifying, and editing images. I had some experience working with ImageMagick at a previous employer and quickly found a solution to round the images before sending to the chart.

But when I found a stackoverflow answer doing something similar, I opted to use that instead.

https://images.weserv.nl/?url=https://www.github.com/bdougie.png?size=60?v=4&h=300&w=300&fit=cover&mask=circle&maxage=7d

Link to this solution

This was working, but we still needed to solve the caching issue.

I really wanted to try out building a service to use ImageMagick + the new Supabase storage to do this, but I wasn't willing to maintain that solution and we only had a little time to explore more unique tools.

(If you want to build a service like this, find me, and I would love to be a beta tester.)

We needed a way to cache the images before sending to the chart and started looking at tools or services to make this easier.

Solution 3: Leveraging Cloudinary for Image Manipulation and Caching

To solve the caching issue and optimize image processing, we explored different tools and services.

Cloudinary offers image manipulation and a caching strategy. They also have a generous free tier--but for full transparency, I'll point out that our initial amount of data for Hacktoberfest was pushed up to the paid tier immediately.

https://res.cloudinary.com/bdougie/image/fetch/f_auto,q_auto/w_400,h_400,c_crop,r_400,g_auto/v1/https://avatars.githubusercontent.com/u/5713670

Link to Cloudinary solution

You can see the PR with the solution live as well.

feat: Leverage cloudinary for round images #467

bdougie posted on Sep 30, 2022

What type of PR is this? (check all applicable)

• [x] 🍕 Feature
• [ ] 🐛 Bug Fix
• [ ] 📝 Documentation Update
• [ ] 🎨 Style
• [ ] 🧑‍💻 Code Refactor
• [ ] 🔥 Performance Improvements
• [ ] ✅ Test
• [ ] 🤖 Build
• [ ] 🔁 CI
• [ ] 📦 Chore (Release)
• [ ] ⏩ Revert

Description

This PR is intentionally set up to replace cloudinary in the future. As Vortex has point out this solution will eventually cost real money. The trade off is that we will eventually make real money from this product and using that to pay for this, balance out.

Details

• Adds cloudinary url to edit and cache avatars on the fly.
• Only github.com urls can be edited. I am enforcing this on the cloudinary side.

Related Tickets & Documents

fixes #373

Mobile & Desktop Screenshots/Recordings

Added tests?

• [ ] 👍 yes
• [x] 🙅 no, because they aren't needed
• [ ] 🙋 no, because I need help

Added to documentation?

• [ ] 📜 README.md
• [ ] 📓 docs.opensauced.pizza
• [ ] 🍕 dev.to/opensauced
• [ ] 📕 storybook
• [x] 🙅 no documentation needed

[optional] Are there any post-deployment tasks we need to perform?

Complete this issue https://github.com/open-sauced/insights/issues/468

[optional] What gif best describes this PR or how it makes you feel?

View on GitHub

The solution was building a wrapper around using the GitHub avatar as the imageUrl.

// lib/utils/roundedImages

const roundedImage = (imageUrl: string, cloudName: string | undefined) => {
  return cloudName
    ? `https://res.cloudinary.com/${cloudName}/image/fetch/c_fill,g_face,h_300,w_300,bo_20px_solid_white,r_max/f_auto,e_shadow/${imageUrl}`
    : imageUrl;
};

// components/organisms/Dashboard/dashboard.tsx

import roundedImage from "lib/utils/roundedImages";

scatterChartData = prs.map(({ updated_at, linesCount, author_login }) => {
      const author_image = author_login.includes("[bot]") ? "octocat" : author_login;

      const data = {
        x: calcDaysFromToday(new Date(updated_at)),
        y: linesCount,
        contributor: author_login,
        image: roundedImage(`https://www.github.com/${author_image}.png?size=60`, process.env.NEXT_PUBLIC_CLOUD_NAME)
      };
      return data;
    });
  }

Link to code

The interaction with Cloudinary is all URL based, allowing us to pass the GitHub user id as an option on the fly. The same id is also used to recall the cached image, meaning we have had caching by default. We also did not need to build any new infrastructure for this.

With the initial 150k user profiles cached, we immediately needed to pay $90 per month for the pro tier, which we felt is reasonable and predictable for us.

We are still on the same Cloudinary plan today, mainly due to the inertia and the fact that we still need to be ready to build and maintain something not core to our product. So far, Cloudinary is caching 271k image transforms for us.

This scenario is a classic build vs buy scenario where we could build our own caching and image storage on S3, but the tech debt imposed on a solution like that was not our priority. The $90 was also not going to put us into debt either.

Migration to nivo Charts

In December, we made the switch from Apache E-charts to nivo charts due to its modern features and active community. The decision was driven by the outdated state of E-charts and the limited contributions it received. We plan to contribute upstream to nivo to improve its integration and interaction with images.

We have an active conversation in the nivo discussions about this and other feature enhancements. The maintainer has been really responsive, and we look forward to contributing upstream to support the project.

Support a popover like behavior for tooltips #2201

OgDev-01 posted on Dec 15, 2022

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Currently working on a project and making use of the scatterplot chart. But discovered some limitations with the tooltip provided

• Tooltip can't be hovered on

https://user-images.githubusercontent.com/62995161/207777072-97967a39-3b3c-4507-b978-2164cc2df77c.mov

Describe the solution you'd like A clear and concise description of what you want to happen.

A behavior similar to GitHub hovercard where we can hiver on the tooltip itself

https://user-images.githubusercontent.com/62995161/207776852-7d605c29-e78c-459f-b99f-f5a1b83e2fde.mov

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

View on GitHub

If you have any thoughts or comments about our approach or have alternative solutions to share, we welcome your input. Let's continue learning from each other and enhancing our open-source projects.