DEV Community

Cover image for The forges behind Netflix, LinkedIn and Co.
Victor Warno
Victor Warno

Posted on • Edited on

The forges behind Netflix, LinkedIn and Co.

When people at home bring up Spotify and Netflix, they often discuss the content these platforms provide. But inside the developer community we do not only discuss the newest season of House of Cards but rather the frameworks companies like this have contrived over time.

I was amused by this discrepancy and was wondering at which point a tech company turns from content-deliverer to content-handler. Take LinkedIn. The profiles, search queries and communication channels scaled over the years and LinkedIn turned their monolithic application infrastructure into microservices. In order to cope with these challenges, LinkedIn developed Kafka, a event-driven messaging system. They started off with several small queuing pipelines for the interaction between services, but then went on to develop a publisher-subscriber platform. Instead of maintaining and scaling those pipelines one-by-one.

Netflix met a similar problem: As they have tons of services, even in different zones, and only few edge services that receive requests, this ginormous amount of calls have to be distributed to these services. Load balancing algorithms back then did not deliver satisfying performances, so they came up with own ones and tied them up with Ribbon.

Infrastructural change is not the only catalysator for innovation. Even if given language frameworks do not meet your requirements, you could think of introducing your own. That's how and why Facebook gave birth to React.

And what about Spotify? While they did not contribute a huge technological framework, they demonstrate a way to incorporate the agile part of software development into the over-all structure of an organization. They decided to not only to use Agile Practices in small teams but on a big scale. So, Spotify introduced Squads, Chapters, Tribes and Guilds.

It is one thing to come up with these ideas. It is another to share them with others. All mentioned novelties have hit the open source field. The messaging system above is even just refered to as Apache Kafka, named after the Open Source License which Ribbon is also licensed under. Not to mention the role Facebook's React plays in the Frontend community. The trend in moving to Open Source and common knowledge is on-going. And these companies have a deep interest in giving access to their inventions.

What other companies do you expect to emerge as a pusher for new technologies and frameworks or mindsets? Or maybe you had a solid problem that you or your team solved with a self-invented toolset? And why Open Source?

Top comments (7)

Collapse
 
ben profile image
Ben Halpern

dev.to is working off of Basecamp's ideas a decade and a half ago. Of course, so are GitHub and many others.

Of course, we plan to become big centers of gravity for this kind of stuff ourselves, but we don't have the custom scaling issues just yet.

Robinhood strikes me as a rapidly growing startup that could make a big impact on open source because they probably have the momentum to benefit from community involvement. You never know how that fits culturally though.

This is currently their most popular repo on GitHub:

robinhood / ticker

An Android text view with scrolling text change animation

Migrating to version 2

There are some breaking API changes introduced in ticker 2.0. Please refer to the 2.0 migration doc.

What is Ticker?

Ticker is a simple Android UI component for displaying scrolling text. Think about how an odometer scrolls when going from one number to the next, that is similar to what Ticker does The Ticker handles smooth animations between strings and also string resizing (e.g. animate from "9999" to "10000").

You can specify how the animations proceed by defining an array of characters in order. Each character displayed by Ticker is controlled by this array which dictates how to animate from a starting character to a target character. For example, if you just use a basic ASCII character list, when animating from 'A' to 'Z', it will go from 'A' -> 'B' -> ... 'Z'. We will perform wrap-around animation when it's faster (e.g. 'Z' to…

In terms of companies contributing massively from day one and building around these ideas of shared good: Zeit.

zeit / hyper

A terminal built on web technologies

macOS CI Status Windows CI status Linux CI status Changelog #213 Join the community on Spectrum

For more details, head to: hyper.is

Usage

Download the latest release!

Linux

Arch and derivatives

Hyper is available in the AUR. Use an AUR package manager like aurman

aurman -S hyper

macOS

Use Homebrew Cask to download the app by running these commands:

brew update
brew cask install hyper

Windows

Use chocolatey to install the app by running the following command (package information can be found here):

choco install hyper

Note: The version available on Homebrew Cask, Chocolatey or the AUR may not be the latest. Please consider downloading it from here if that's the case.

Contribute

Regardless of the platform you are working on, you will need to have Yarn installed. If you have never installed Yarn before, you can find out how at: yarnpkg.com/en/docs/install.

  1. Install necessary packages:
  • Windows
    • Be sure to run yarn global add windows-build-tools to install windows-build-tools.
  • macOS
    • Once you have…

As a developer tooling company first, they may fall in a different camp, but either way, great stuff.

Collapse
 
rhymes profile image
rhymes • Edited

Didn't know about Robinhood at all, I see they released a stream processing library in Python, neat! They've also had half a billion dollars (!!!) in funding, that buys you a lot of time and space to dedicate full time resources to open source (as in having full time employees working mostly on those libraries and frameworks).

For some reason I didn't even know Next and Hyper were from Zeit.

I wonder what's the story behind all of these released tools. How they started, how they got here, why they decided to open source them and so on.

BTW what is it with fintech companies and Python? :-D

Collapse
 
ben profile image
Ben Halpern

I could have probably picked any company, but Robinhood popped into my mind as a software-based company with a lot of momentum but not so big the whole world has heard of them.

On Zeit: dev.to/changelog/213-zeit-hyperter...

Collapse
 
kaelscion profile image
kaelscion

As a Python dev I can say that it’s popular with fintech most likely because it is one of the big players in data science right now, but also has great support for building for the web and REST APIs with DRF, Flask, Respnder, etc. I’m not, by any means saying, that Python is the best at any of these things. However, it is a great all-rounder that allows companies to use the same stack to rapidly go from idea to prototype to launch on all fronts. SciKit, Numpy, Pandas, SQLAlchemy or Django ORM, and a plethora of web scraping and data aggregation tools to do you “money stuff”. The. Use the same language to build the API and/or web app to serve it all to customers.

A decent team can make this happen very quickly due to Python’s ease of use and you’ve got a decent chance of launching before somebody can steal your idea and launch before you which, I imagine, is a big deal where financial tech is involved. Not sure if those are the absolute reasons, but that’s why I personally believe it is so popular with data-driven and financial companies right now.

Thread Thread
 
rhymes profile image
rhymes

As a Python dev I can say that it’s popular with fintech most likely because it is one of the big players in data science right now

Eh eh I was half joking. A decade ago at my very first job they were doing data science in fintech, in Python (nobody knew it was called data science nor fintech :D). There's a long history of Python as a favorite tool in academia and science, and it is really easy to integrate with C and C++, and lots of numerical libraries where. Also, Boost Python was in vogue.

Not sure if those are the absolute reasons, but that’s why I personally believe it is so popular with data-driven and financial companies right now.

Yeah, rapid prototyping and the fact it can scale with the company (and be even replaced when there's need to) is one of the reasons why people like Python. The enourmous amount of tools is another.

Thread Thread
 
kaelscion profile image
kaelscion

Whoops! Sorry. Took that just a trifle too literally. Ma bad!!

Thread Thread
 
rhymes profile image
rhymes

Don't be sorry, I learned something 😂

Python is still my go to language