ChiPy February 2020 __main__ Meeting


This month’s meeting is graciously sponsored by Lumere! Lumere is hiring! Check out their openings:

Our talks:

Cloud Data Warehouses with Python
by Michael McCarthy

The rapid growth of Python is, in part due, to its exceptional toolkit for Data Analysts, Scientists, and Engineers. Packages like Pandas, Scikit-Learn, PySpark, and Dask have become staples for teams looking to process data. However, when processing large amounts of data there are times when Python might not be the right solution for your task. In this conversation, we’ll learn about Cloud-based Data Warehouses, such as Google’s BigQuery, Amazon’s Redshift, and Snowflake. You’ll learn about the advantages of these platforms compared to in-memory processing in Python. We’ll also show examples of how you can use Apache Airflow to automate recurring tasks, turning your Data Warehouse into the cornerstone of your Data Science infrastructure.


Migrating from VMs to K8s – We did it, and so can you!
by Nick Petrovits

Join us as we describe our migration from a limiting cloud deployment on long-running VMs with shared infrastructure to a streamlined immutable infrastructure built on top of Docker and K8s. We’ll also discuss techniques to support local development during this transition. Many teams wish they could reap the widely known benefits of Kubernetes (K8s), but most struggle to migrate to a new infrastructure while simultaneously supporting two deployment models and avoiding impacts to the velocity of software development. In this talk, we describe the particular challenges we faced during our incremental migration from multiple long-running singleton EC2 instances to a containerized solution. We’ll highlight: – What challenges motivated us to transition to K8s? – Approaching an infrastructure migration incrementally to minimize impacts to local development and production deployments – Developing a solution to provide the same abstraction for local development that exists in production – Concurrently supporting multiple deployment models to reduce risk and simplify migration – Strategy variations for synchronous and asynchronous services – Networking challenges with Vagrant and Docker – Integrating K8s with a CI/CD pipeline – Tuning the environment


Experiences in the Guild
by Sam Mahisekar

If you have attended a few ChiPy events, chances are you have used the website. The ChiPy Web Guild is a group of volunteers that help maintain the site. In this talk, I will give a brief description of how the Web Guild works and touch on some aspects of the site. We will then go through an example of how team members were able to address a flaw in the code enhancing user experience. Finally, I will share some thoughts on what I learned and what the group might work on next.