How Streaming Giant Netflix Uses Python: Reveals Its Libraries And Frameworks

Ever Wonder Which Language Is Used By Netflix?
Recent this week Netflix team with their tech blog updated the world that how they rely heavily on Python whether it’s content cycle, security automation, training machine learning models for its recommendation algorithm everything depends on Python. Netflix teams have been actively involved with Python development from past 5-7 years and we have seen their support through Pycon sponsorship, contributing to the opensource community, donating Netflix Originals posters to PyLadies Auction and even talking about the language on their blog actively from 2013.

Why All Of A Sudden It Became A Big Deal In 2019?
The answer relies on the number of users it actively handles currently – 140 million, yeah you read it right. These millions of people need fast uninterrupted streaming service, great personalized recommendation, security for their account, smooth experience during high traffic and more.

Here are few points Netflix mentioned regarding how they are using Python –

  • Open Connect – Netflix’s content delivery network, various software systems are needed to design, build, and operate this CDN infrastructure, and a significant number of them are written in Python.
  • Demand Engineering – Responsible for Regional Failovers, Traffic Distribution, Capacity Operations, and Fleet Efficiency of the Netflix cloud. 
  • Monitoring, alerting and auto-remediation – Responsible for building and operating the tools for operational insight, alerting, diagnostics, and auto-remediation. 
  • Information Security – Security automation, risk classification, auto-remediation, and vulnerability identification.
  • Personalization Algorithms – Recommendation algorithms, artwork personalization, marketing algorithms.
  • Machine Learning Infrastructure
  • Notebooks – Develop, debug, explore, and prototype different interactions with the Jupyter ecosystem.
  • Experimentation Platform – Scientists and engineers to analyze AB tests and other experiments. Contribution towards new innovations on three fronts, data, statistics, and visualizations.
  • Video Encoding and Media Cloud Engineering – Encoding (and re-encoding) the Netflix catalog, as well as leveraging machine learning for insights into that catalog.
  • Content Machine Learning, Science & Analytics

They have created their own ‘Python Universe‘. Netflix team uses pre-developed libraries, framework along with their in house developed libraries,

Here are few libraries mentions –

  • Numpy and Scipy – Numerical Analysis
  • Boto3 – AWS Infrastructure
  • rq – Asynchronous Workload
  • Flask – Handles RESTful API
  • Jupyter Notebook – Analyze Operational Data.
  • Numpy, Pandas, Matplotlib – Help analyze thousands of signals after an alert also to broader scientific stack in Python.
  • TensorFlow, Keras, and PyTorch – To learn Deep Neural Networks.
  • Metaflow – Used to leverage well parallelized and optimized Python code to fetch data at 10Gbps.
  • Plotly – Netflix’s visualizations library.
  • PyPika – Allows users to write reusable, parameterized SQL queries.

There are tons of other libraries that are connected directly or indirectly towards the development process which include Netflix in house developed pygenie, security_monkey, hubcommander, flamescope, lemur and more.

  • pygenie – Package provides a robust client for interacting with an existing Genie service.
  • security_monkey – Security Monkey monitors your AWS and GCP accounts for policy changes and alerts on insecure configurations.
  • hubcommander – A user-extendable Slack bot for GitHub organization management. flamescope.
  • lemur  – Lemur manages TLS certificate creation.

The way Netflix has developed themselves around Python during these years with active numbers is commendable.

Yo ucan read complete official blog here – Netflix at Python – 2019, Netflix at Python – 2013

UnwiredLearning.com 2019