Notes

• ​If you can solve a problem with a simple heuristic, do that. Sometimes you don't need machine learning.​

• Data is not always useful and it doesn't matter how much of it you have.

  There’s no mathematical tool to tell you if your hypothesis is true; you can only see whether it is consistent with the data, and if the data is sparse or unclear, your conclusions are uncertain.

• ​By writing the data generating process first, and therefore knowing parameters, you can get more confidence that your model is well constructed.​

• ​Building a data pipeline in 2020 is like building a bridge in the 14th century: You do a lot of work that gets thrown away. Half the job is getting rid of the stuff you don't want. The folks who started it are dead by the time it's done.​

Links

• ​Some advice for young and aspiring Data Scientists​

• ​A Beginner’s Guide to Data Engineering — Part I​

• ​The Rise of the Data Engineer​

• ​Cookiecutter Data Science - Logical, reasonably standardized, but flexible project structure for doing and sharing data science work. (Code)

• ​Best way to organize research code? (2018)​

• ​Data Science Cheat Sheet​

• ​Our world in data​

• ​Free data science books​

• ​Pachyderm - Reproducible Data Science at Scale. (Web) (Pachyderm Hub)

• ​Data Science Cheat Sheets​

• ​Data Science in Visual Studio Code using Neuron, a new VS Code extension (2018)​

• ​Virgilio - Mentor for Data Science E-Learning.

• ​Awesome Data Science with Python - Curated list of Python resources for data science.

• ​Data Science (Cheat Sheets)​

• ​nteract - Interactive computing suite for you.

• ​120 Data Science Interview Questions​

• ​How To Become a Data Engineer​

• ​Ingestion Data Mapping Language​

• ​The reference implementation of IDML for the JVM​

• ​Pandas - Powerful Python data analysis toolkit. (Ongoing list of pandas quirks) (Manual) (HN)

• ​Programming Language Support for Data-intensive Applications meeting (2019)​

• ​Datasette - Open source multi-tool for exploring and publishing data. (Web) (datasette-graphql) (Running Datasette on DigitalOcean App Platform) (Interesting ideas in Datasette) (HN)

• ​Weld - High-performance runtime for data analytics applications.

• ​Vaex - Out-of-Core DataFrames for Python, visualize and explore big tabular data at a billion rows per second.

• ​PyParis 2018 - Vaex: Out of Core Dataframes for Python​

• ​Maarten Breddels & Jovan Veljanoski- A new approach to DataFrames and pipelines - PyData London 2019 (GitHub)

• ​The Data Engineering Cookbook​

• ​Data science without borders - Wes McKinney (2017)​

• ​Curated list of awesome ETL frameworks, libraries, and software​

• ​Ibis - Python data analysis framework for Hadoop and SQL engines.

• ​Kyso - Data analytics knowledge hub.

• ​Feather - Fast, interoperable binary data frame storage for Python, R, and more powered by Apache Arrow.

• ​ROOT system - Provides a set of OO frameworks with all the functionality needed to handle and analyze large amounts of data in a very efficient way.

• ​Redash - Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data. (Web)

• ​Prefect - New workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine.

• ​Turn Python Scripts into Beautiful ML Tools (2019) (HN)

• ​Foundations of Data Science (2019) (HN)

• ​Gyana - No code desktop data science tool. (Article)

• ​Monument - High-productivity toolkit for predictions. AutoML for time series on any desktop, laptop or server.

• ​Numba - NumPy aware dynamic Python compiler using LLVM. (5 minute guide) (Web)

• ​What's your typical data pipeline in a small company? (2019)​

• ​dbt - Data build tool. Analytics engineering workflow. (Code) (Docs)

• ​Introducing dbt + Materialize (2021) (HN)

• ​Apache Airflow - Platform to programmatically author, schedule, and monitor workflows. (Tutorial) (Kedro-Airflow - Makes it easy to deploy Kedro projects to Airflow.) (Airflow 2.0) (HN) (Introduction to Apache Airflow (2021))

• ​Overview of Popular Open Source Big Data Technologies (2018)​

• ​Introducing Apache Arrow Flight: A Framework for Fast Data Transport (2019)​

• ​Understanding Apache Arrow Flight (2019)​

• ​New Developments in the Open Source Ecosystem: Apache Spark 3 0, Delta Lake, and Koalas (2019)​

• ​Materials and IPython notebooks for Python for Data Analysis, 2nd Edition book​

• ​Technical Notes On Using Data Science & Artificial Intelligence​

• ​Amazon Data Science Interview (2018)​

• ​Data Analysis and Prediction Algorithms with R (2019)​

• ​Elegant SciPy book​

• ​Data science blogs​

• ​Awesome Data Science​

• ​Things About Real-World Data Science Not Discussed In MOOCs and Thought Pieces (2018)​

• ​Apache Zeppelin - Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

• ​Apache Nifi - Easy to use, powerful, and reliable system to process and distribute data.

• ​Data Science: Principles and Practice Course materials (2018/19)​

• ​Modern Data Practice and the SQL Tradition (2019) (HN)

• ​Koalas - Pandas API on Apache Spark.

• ​Ask HN: What does your BI stack look like? (2019)​

• ​Easy Data Transform - Transform Your Data Without Programming. (HN)

• ​Prophet - Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.

• ​Dagster - Python library for building data applications: ETL, ML, Data Pipelines, and more. (Dagster: The Data Orchestrator) (Lobsters)

• ​CuPy - NumPy-like API accelerated with CUDA.

• ​An Introduction To Data Science On The Linux Command Line (2019) (HN)

• ​How to analyse 100 GB of data on your laptop with Python (2019)​

• ​Metaflow - Framework for real-life data science. (HN) (Code) (Tools)

• ​SaturnCloud - Manage Data Science applications so Data Scientists don't have to do DevOps.

• ​Falcon - Interactive Visual Analysis for Big Data.

• ​160+ Data Science Interview Questions​

• ​Data Science Interview Questions (HN)

• ​Google Cloud DataLab - Interactive tools and developer experiences for Big Data on Google Cloud Platform.

• ​Great Expectations - Leading tool for validating, documenting, and profiling, your data to maintain quality and improve communication between teams. (Code)

• ​Path to a free self-taught education in Data Science​

• ​Common Workflow Language - Open standard for describing analysis workflows and tools. (HN)

• ​Apache Kudu - Completes Hadoop's storage layer to enable fast analytics on fast data.

• ​Time Series Forecasting Best Practices & Examples​

• ​Turing Way - Lightly opinionated guide to reproducible data science. (Code)

• ​What to do when you didn’t get any medal in a Kaggle competition? (2020)​

• ​Build a Career in Data Science book (2020)​

• ​Data Science Tutorials in Julia (Code)

• ​Data Science Resources​

• ​Master Data Analysis with Python​

• ​Introduction to Data Science book (2020) (Code)

• ​VisiData - Terminal spreadsheet multitool for discovering and arranging data. (Code) (HN)

• ​Sisu - Fastest Diagnostic Platform for Structured Data. (Introducing Sisu)

• ​Deepnote - Data science notebook for teams. (Docs) (Awesome Deepnote) (HN)

• ​Towards Data Science blog​

• ​Scaling Pandas: Dask vs Ray vs Modin vs Vaex vs RAPIDS (2020) (HN)

• ​A graphical analysis of women's tops sold on Goodwill's website (HN)

• ​1.1B Taxi Rides Using OmniSciDB and a MacBook Pro (2020) (HN)

• ​Data Science Interview Resources​

• ​Data Science Meets Devops: MLOps with Jupyter, Git, & Kubernetes (2020)​

• ​dplyr - Grammar of data manipulation, providing a consistent set of verbs that help you solve the most common data manipulation challenges. (Web)

• ​dstack.ai - Build data and ML applications using pure Python. (HN) (Code)

• ​Learn Python for Data Science - Collection of Jupyter Notebooks designed to learn Python for Data Science. (HN)

• ​Computational Causal Inference at Netflix (2020)​

• ​Practical Data Ethics (HN)

• ​New Google Data Science project​

• ​Python for Data Analytics course (Reddit)

• ​Modern Data Engineer Roadmap​

• ​How to share data with a statistician​

• ​Becoming 1% better at data science everyday​

• ​Jigsaw Labs - Learn Data Science part-time.

• ​Data Science Ontology - Knowledge base about data science.

• ​Narrator - Data modeling platform built on a single table. (HN)

• ​Data Engineering Project - Implementation of the data pipeline which consumes the latest news from RSS Feeds and makes them available for users via handy API.

• ​Hex Technologies - Turn your notebooks into collaborative, sharable data apps and stories. No more loose CSVs, chart screenshots, or stale decks.

• ​News Aggregator from Scratch in 2 Weeks (2020)​

• ​Awesome Scholarly Data Analysis​

• ​Nemo: Data discovery at Facebook (2020) (Tweet)

• ​Amundsen by Lyft - Open source data discovery and metadata engine.

• ​Google BigQuery: Node.js Client​

• ​The Modern Data Science Stack (2020)​

• ​Streamlit Sharing - Platform for deploying, managing, and sharing your apps. (HN)

• ​Awesome Data Engineering Learning Path (Code) (HN)

• ​Emerging Architectures for Modern Data Infrastructure (2020) (HN)

• ​PandasGUI - GUI for analyzing Pandas DataFrames.

• ​Holistics - Data Modeling & Self-Service BI Platform.

• ​Numerai data science tournament​

• ​Neptune.ai - Experiment tracking tool for you and your team. (GitHub)

• ​Neptune Python Client - Integrate your Python scripts with Neptune.

• ​AimStack - Version Control and Development Environment for AI. (Code) (GitHub)

• ​Synerise - Powerful ecosystem driven by Artificial Intelligence with real-time data orchestration created to drive business growth.

• ​Good Data Analysis​

• ​Data Science Learning Resources​

• ​Dataquest - Learn R, Python and SQL for Data Science.

• ​Awesome OSINT​

• ​Carpentries - Teach foundational coding and data science skills to researchers worldwide.

• ​Orchest - Web based tool for creating data science pipelines. (Code)

• ​Data Engineering Book - Accumulated knowledge and experience in the field of Data Engineering.

• ​Data Science Lifecycle Process - Set of prescriptive steps and best practices to enable data science teams to consistently deliver value.

• ​Data Science Lifecycle Base Repo - Template repository for data science projects using the Data Science Life Cycle Process.

• ​5th International Summer School on Data Science (2020) (Code)

• ​Scalable Data Science - Course sets in big data Using Apache Spark over databricks and their mathematical, statistical and computational foundations using SageMath. (Code)

• ​Datatap - Free user-friendly platform for visual data management. (Code)

• ​Data Carpentry - Develops and teaches workshops on the fundamental data skills needed to conduct research.

• ​Data Carpentry Lessons​

• ​Doing Symbolic Math with SymPy (2020) (HN)

• ​How To Become a Data Engineer (2021) (HN)

• ​We don't need data scientists, we need data engineers (2021) (HN)

• ​Airbyte - Open-Source Data Integration Pipelines To Your Warehouses. (Code) (HN)

• ​Data Together - Exploring Community-Driven Data Stewardship. (GitHub)

• ​Data Together Research - Research for tackling the general problem of data resilience & interactivity in all its forms.

• ​Apache Superset - Modern data exploration and visualization platform. (Code)

• ​Open-Source Data Science Masters (Web)

• ​Streamlit Data Science and ML Apps in Python​

• ​Storing and retrieving millions of ad impressions per second at Twitter (2021)​

• ​Elements of Data Science - Introduction to data science in Python, for people with no programming experience. (Code)

• ​Data Science on AWS - AI and Machine Learning with Kubeflow, Amazon EKS, and SageMaker. (Code)

• ​Tips for Shipping Data Products Fast (2021) (HN)

• ​Data Science Topics Notes​

• ​sq - Command line tool that provides jq-style access to structured data sources such as SQL databases, or document formats like CSV or Excel. (Web)

• ​Ask HN: What are the best data science bootcamps? (2021)​

• ​Data Science Learning Resources​

• ​Loading SQL data into Pandas without running out of memory (2021)​

• ​What is the best structured ds project you have seen? (2021)​

• ​Data Science Learning Resources (2021)​

• ​ZeroCostDL4Mic - Google Colab based no-cost toolbox to explore Deep-Learning in Microscopy.

• ​How do you manage Data Science experiments? (2021)​

• ​Querybook - Big Data Querying UI, combining collocated table metadata and a simple notebook interface. (Code)

• ​Principal Component Analysis explained​

• ​Dataproofer - Proofreader for your data.

• ​Automated Data Wrangling for Open Data (2021)​

• ​Data science interview questions with answers​

• ​Lightdash - Open Source BI for your whole team. (Code) (HN)

• ​Wildland - Open data management protocol.

• ​Data Science Fails​

• ​Building a data team at a mid-stage startup (2021) (HN)

• ​DataStation - Data IDE for Developers. (Code)

• ​Awesome data annotation​

• ​Data Movement in Netflix Studio via Data Mesh (2021)​

• ​Small Summaries for Big Data (2020)​

• ​What even is data mesh (2021)​

• ​Build 12 Data Science Apps with Python and Streamlit - Full Course (2021)​

• ​JetBrains DataSpell - IDE for Data Scientists.

• ​node-rapids - GPU-accelerated data science and visualization in node.

• ​R or Python for data analysis? (2021)​

• [Moving beyond “algorithmic bias is a data problem” (2021)](https://www.cell.com/patterns/fulltext/S2666-3899(21)00061-1) (HN)

• ​Clustering Algorithms with Python (2020) (HN)

• ​Data Science Cheatsheet 2.0​

• ​Lessons Learned from two years as a Data Scientist (2021) (HN)

• ​Free data science resources - Thematic list of high-quality data science resources.