Notes

• ​If you can solve a problem with a simple heuristic, do that. Sometimes you don't need machine learning.​

• Data is not always useful and it doesn't matter how much of it you have.

  There’s no mathematical tool to tell you if your hypothesis is true; you can only see whether it is consistent with the data, and if the data is sparse or unclear, your conclusions are uncertain.

• ​By writing the data generating process first, and therefore knowing parameters, you can get more confidence that your model is well constructed.​

• ​Building a data pipeline in 2020 is like building a bridge in the 14th century: You do a lot of work that gets thrown away. Half the job is getting rid of the stuff you don't want. The folks who started it are dead by the time it's done.​

Links

• ​Some advice for young and aspiring Data Scientists​

• ​A Beginner’s Guide to Data Engineering — Part I​

• ​The Rise of the Data Engineer​

• ​Cookiecutter Data Science - Logical, reasonably standardized, but flexible project structure for doing and sharing data science work. (Code)

• ​Best way to organize research code? (2018)​

• ​Data Science Cheat Sheet​

• ​Our world in data​

• ​Free data science books​

• ​Pachyderm - Reproducible Data Science at Scale. (Web) (Pachyderm Hub)

• ​Data Science Cheat Sheets​

• ​Data Science in Visual Studio Code using Neuron, a new VS Code extension (2018)​

• ​Virgilio - Mentor for Data Science E-Learning.

• ​Awesome Data Science with Python - Curated list of Python resources for data science.

• ​Data Science (Cheat Sheets)​

• ​nteract - Interactive computing suite for you.

• ​120 Data Science Interview Questions​

• ​How To Become a Data Engineer​

• ​Ingestion Data Mapping Language​

• ​The reference implementation of IDML for the JVM​

• ​Pandas - Powerful Python data analysis toolkit.

• ​Programming Language Support for Data-intensive Applications meeting (2019)​

• ​Datasette - Tool for exploring and publishing data. (datasette-graphql) (Running Datasette on DigitalOcean App Platform) (Interesting ideas in Datasette)

• ​Weld - High-performance runtime for data analytics applications.

• ​Vaex - Out-of-Core DataFrames for Python, visualize and explore big tabular data at a billion rows per second.

• ​PyParis 2018 - Vaex: Out of Core Dataframes for Python​

• ​Maarten Breddels & Jovan Veljanoski- A new approach to DataFrames and pipelines - PyData London 2019 (GitHub)

• ​The Data Engineering Cookbook​

• ​Data science without borders - Wes McKinney (2017)​

• ​Curated list of awesome ETL frameworks, libraries, and software​

• ​Ibis - Python data analysis framework for Hadoop and SQL engines.

• ​Kyso - Data analytics knowledge hub.

• ​Feather - Fast, interoperable binary data frame storage for Python, R, and more powered by Apache Arrow.

• ​ROOT system - Provides a set of OO frameworks with all the functionality needed to handle and analyze large amounts of data in a very efficient way.

• ​Redash - Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data. (Web)

• ​Prefect - New workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine.

• ​Turn Python Scripts into Beautiful ML Tools (2019) (HN)

• ​Foundations of Data Science (2019) (HN)

• ​Gyana - No code desktop data science tool. (Article)

• ​Monument - High-productivity toolkit for predictions. AutoML for time series on any desktop, laptop or server.

• ​Numba - NumPy aware dynamic Python compiler using LLVM. (5 minute guide) (Web)

• ​What's your typical data pipeline in a small company? (2019)​

• ​dbt - Data build tool. Analytics engineering workflow.

• ​Apache Airflow - Platform to programmatically author, schedule, and monitor workflows. (Tutorial)

• ​Overview of Popular Open Source Big Data Technologies (2018)​

• ​Introducing Apache Arrow Flight: A Framework for Fast Data Transport (2019)​

• ​Understanding Apache Arrow Flight (2019)​

• ​New Developments in the Open Source Ecosystem: Apache Spark 3 0, Delta Lake, and Koalas (2019)​

• ​Materials and IPython notebooks for Python for Data Analysis, 2nd Edition book​

• ​Technical Notes On Using Data Science & Artificial Intelligence​

• ​Amazon Data Science Interview (2018)​

• ​Data Analysis and Prediction Algorithms with R (2019)​

• ​Elegant SciPy book​

• ​Data science blogs​

• ​Awesome Data Science​

• ​Things About Real-World Data Science Not Discussed In MOOCs and Thought Pieces (2018)​

• ​Apache Zeppelin - Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

• ​Apache Nifi - Easy to use, powerful, and reliable system to process and distribute data.

• ​Data Science: Principles and Practice Course materials (2018/19)​

• ​Modern Data Practice and the SQL Tradition (2019) (HN)

• ​Koalas - Pandas API on Apache Spark.

• ​Ask HN: What does your BI stack look like? (2019)​

• ​Easy Data Transform - Transform Your Data Without Programming. (HN)

• ​Prophet - Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.

• ​Dagster - Python library for building data applications: ETL, ML, Data Pipelines, and more. (Dagster: The Data Orchestrator) (Lobsters)

• ​CuPy - NumPy-like API accelerated with CUDA.

• ​An Introduction To Data Science On The Linux Command Line (2019) (HN)

• ​How to analyse 100 GB of data on your laptop with Python (2019)​

• ​Metaflow - Framework for real-life data science. (HN) (Code)

• ​SaturnCloud - Manage Data Science applications so Data Scientists don't have to do DevOps.

• ​Falcon - Interactive Visual Analysis for Big Data.

• ​160+ Data Science Interview Questions​

• ​Data Science Interview Questions (HN)

• ​Google Cloud DataLab - Interactive tools and developer experiences for Big Data on Google Cloud Platform.

• ​Great Expectations - Leading tool for validating, documenting, and profiling, your data to maintain quality and improve communication between teams. (Code)

• ​Path to a free self-taught education in Data Science​

• ​Common Workflow Language - Open standard for describing analysis workflows and tools. (HN)

• ​Apache Kudu - Completes Hadoop's storage layer to enable fast analytics on fast data.

• ​Time Series Forecasting Best Practices & Examples​

• ​Turing Way - Lightly opinionated guide to reproducible data science. (Code)

• ​What to do when you didn’t get any medal in a Kaggle competition? (2020)​

• ​Build a Career in Data Science book (2020)​

• ​Data Science Tutorials in Julia (Code)

• ​Data Science Resources​

• ​Master Data Analysis with Python​

• ​Introduction to Data Science book (2020) (Code)

• ​VisiData - Terminal spreadsheet multitool for discovering and arranging data. (Code) (HN)

• ​Sisu - Fastest Diagnostic Platform for Structured Data. (Introducing Sisu)

• ​Deepnote - Data science notebook for teams. (Docs) (Awesome Deepnote)

• ​Towards Data Science blog​

• ​Scaling Pandas: Dask vs Ray vs Modin vs Vaex vs RAPIDS (2020) (HN)

• ​A graphical analysis of women's tops sold on Goodwill's website (HN)

• ​1.1B Taxi Rides Using OmniSciDB and a MacBook Pro (2020) (HN)

• ​Data Science Interview Resources​

• ​Data Science Meets Devops: MLOps with Jupyter, Git, & Kubernetes (2020)​

• ​dplyr - Grammar of data manipulation, providing a consistent set of verbs that help you solve the most common data manipulation challenges. (Web)

• ​dstack.ai - Open-source tool for building data and ML applications using Python and R. (HN)

• ​Learn Python for Data Science - Collection of Jupyter Notebooks designed to learn Python for Data Science. (HN)

• ​Computational Causal Inference at Netflix (2020)​

• ​Practical Data Ethics (HN)

• ​New Google Data Science project​

• ​Python for Data Analytics course (Reddit)

• ​Modern Data Engineer Roadmap​

• ​How to share data with a statistician​

• ​Becoming 1% better at data science everyday​

• ​Jigsaw Labs - Learn Data Science part-time.

• ​Data Science Ontology - Knowledge base about data science.

• ​Narrator - Data modeling platform built on a single table. (HN)

• ​Data Engineering Project - Implementation of the data pipeline which consumes the latest news from RSS Feeds and makes them available for users via handy API.

• ​Hex Technologies - Turn your notebooks into collaborative, sharable data apps and stories. No more loose CSVs, chart screenshots, or stale decks.

• ​News Aggregator from Scratch in 2 Weeks (2020)​

• ​Awesome Scholarly Data Analysis​

• ​Nemo: Data discovery at Facebook (2020) (Tweet)

• ​Amundsen by Lyft - Open source data discovery and metadata engine.

• ​Google BigQuery: Node.js Client​

• ​The Modern Data Science Stack (2020)​

• ​Streamlit Sharing - Platform for deploying, managing, and sharing your apps. (HN)

• ​Awesome Data Engineering Learning Path (Code) (HN)

• ​Emerging Architectures for Modern Data Infrastructure (2020) (HN)

• ​PandasGUI - GUI for analyzing Pandas DataFrames.

• ​Holistics - Data Modeling & Self-Service BI Platform.

• ​Numerai data science tournament​