Authors

Description

Experimenting with Modern Data Architectures and Python: From Ingestion to Dashboards. In this talk, we will explore how you can experiment with modern data architectures using Python as the primary language. From initial ingestion to real-time data serving, we'll cover the entire data lifecycle, showing attendees how we've gotten non-technical people outside of the IT field to build end-to-end solutions seamlessly. independent. Topics: Data Ingestion: We will start by discussing the different data ingestion methods we use, from structured sources to unstructured data. We will understand how we use tools such as Apache NiFi and Kafka, aided by Python, for efficient data ingestion in real time. Storage and Processing: Once we have the data ingested, we will see how we decide the storage and processing strategies. From traditional SQL databases to NoSQL solutions like MongoDB, and processing platforms like Apache Spark using PySpark, we'll discuss how to select the right technology for each use case. Visualization and Analysis: Data visualization is crucial to understanding the insights that can be extracted from it. We will explore how we integrate tools like Apache Superset, Grafana and Jupyter to create interactive visualizations with Python or SQL and how we also perform advanced data analysis. Automation and Orchestration: To maintain efficiency and consistency in our processes, we will discuss why it is essential that we learn to automate tasks using tools such as Python and Apache Airflow. Additionally, we will discuss how we orchestrate and coordinate complex workflows in a distributed data environment. Library and Script Development: Finally, we will explore how we use Python to develop libraries and scripts that simplify our daily tasks in data management and manipulation. From creating custom utilities to integrating with third-party APIs, we will see how Python can boost our capabilities in the field of data science and data engineering. Takeaway: This conference will provide attendees with knowledge about how tools can be used to build end-to-end data solutions using Python and a variety of modern technologies without having to go to a cloud solution or having to purchase additional servers. At the end, we will see how the use of these tools equips students to design, implement and maintain scalable and efficient data architectures in a variety of business contexts.