Modern data science has become inseparable from machine learning. While data science alone can gather insights from data, machine learning is the secret to creating accurate and actionable predictions.
As a result, employers expect data scientists to understand both. Understanding where they intersect is essential to land a data science position.
Today, we’ll explore the fundamentals of data science and ML to help you start leveraging both in your projects.
Here’s what will be covered today:
This course is your comprehensive guide to getting your start as a data scientist. Find easy to follow, hands-on explanations in one place.
NumPy (Numerical Python) is a powerful, and extensively used library for storage and calculations. It’s designed for dealing with numerical data. It allows for data storage and calculations by providing data structures, algorithms, and other useful utilities.
For example, this library contains basic linear algebra functions, Fourier transforms, and advanced random number capabilities. It can also be used to load data to Python and export from it.
Here are some NumPy basics you should start to become familiar with:
NumPy arithmetic and statistics basics
Pandas is a library that you can’t avoid when working with Python on a data science project. It’s a powerful tool for data wrangling, a process required to prepare your data so that it can actually be consumed for analysis and model building.
Pandas contains a large variety of functions for data import, export, indexing, and data manipulation. It also provides handy data structures like DataFrames (series of columns and rows, and Series (1-dimensional arrays), and efficient methods for handling them.
For example, it allows you to reshape, merge, split, and aggregate data.
Here are some basic Pandas concepts you should start to become familiar with:
Pandas core components
Pandas DataFrame Operations
Matplotlib is widely used for data visualization exercises, like plotting histograms, line plots, and heat plots.
Seaborn is another great library based off Matplotlib. Seaborn helps create attractive and information rich graphics. Its goal is to make it easier to explore and understand data through easy-to-make visual representations.
Jupyter is a coding environment invaluable for developing and presenting data science projects.
Notebook documents can contain Python code and rich text elements like equations or figures. You can therefore combine visuals, mathematics, and text explanations to create a complete project and presentation in just a single document.
Jupyter Notebook can handle many other languages, like R, as well. Its intuitive workflows, ease of use, and zero-cost make it the tool at the heart of any data science project.
There are three basic components you need to train your machine learning systems: data, features, and algorithms. As a data scientist, it’s important to understand how your choices for each of these components affects your final predictive model.
Data can be collected both manually and automatically. For example, users’ personal details like age and gender, all their clicks, and purchase history are valuable data for an online store.
Do you recall “ReCaptcha” which forces you to “Select all the street signs”? That’s an example of some free manual data!
Data is not always images; it could be tables of data with many variables (features), text, sensor recordings, sound samples etc., depending on the problem at hand.
Features are often also called variables or parameters. These are essentially the factors for a machine to look at — the properties of the “object” in question, e.g., users’ age, stock price, area of the rental properties, number of words in a sentence, petal length, size of the cells.
Choosing meaningful features is very important, but it takes practice. Sometimes it’s difficult to tell which features to use, especially when working with large datasets.
Machine learning is based on general purpose algorithms.
For example, one kind of algorithm is classification. Classification allows you to put data into different groups.
The same classification algorithm used to recognize handwritten numbers could also be used to classify emails into spam and not-spam without changing a line of code! How is this possible?
Although the algorithm is the same, it’s fed different input data, so it comes up with different classification logic. Most of an ML system’s behavior comes from its training data rather than the starting algorithm.
However, this is not meant to imply that one algorithm can be used to solve all kinds of problems! The choice of the algorithm is important in determining the quality of the final machine learning model.
Below, we’ll see how different types of algorithms are better suited to some problems than others.
Master Python and Machine Learning before your next data science interview.
Educative’s text-based and hands-on courses help you become twice the data scientist in half the time.
In Supervised Learning, the training data provided as input to the algorithm includes the final solutions, called labels or classes because the algorithm learns by “looking” at the examples with correct answers.
In other words, the algorithm has a supervisor or a teacher who provides it with all the answers first, like whether it’s a cat in the picture or not. The machine uses these examples to learn one by one.
Another typical task, of a different type, would be to predict a target numeric value like housing prices given information about the home.
To train the system, you need to provide many correct examples of known housing prices, including both their features (number of bedrooms, location, etc.) and their labels.
Categorizing emails or recognizing which pictures have dogs are both classification-type supervised learning algorithms. Predicting housing prices is a different type, known as regression.
In regression the output is a continuous value or a decimal number like housing prices. In classification, the output is a binary label like “spam or not-spam”.
Basically, the type of algorithm you choose (classification or regression) depends on the type of output you want.
Most used Supervised Learning Algorithms:
In Unsupervised Learning the data has no labels; the goal of the algorithm is to find relationships in the data.
This system needs to learn without a teacher and finds relationships based on some hidden patterns in the data.
Segmentation like this is an example of what is known as clustering, classification with no predefined classes and based on some unknown features.
Most used Unsupervised Learning Algorithms:
Semi-supervised learning deals with partially labeled training data, usually a lot of unlabeled data with some labeled data.
Most semi-supervised learning algorithms are a combination of unsupervised and supervised algorithms.
Reinforcement Learning is a special and more advanced category where the learning system learns to act based on positive/negative reinforcement to different actions.
The learning system observes the environment to which it is exposed, it selects and performs actions, and gets rewards or penalties in return. Its goal is to choose actions which maximize the reward over time.
So, by trial and error, and based on past experience, the system learns the best strategy, called policy, on its own.
Now you know some of fundamental ML concepts, but how do data scientists apply them?
Let’s walk through the steps of a data science project to understand how you’ll use ML in the workplace.
Frame the problem and look at the big picture:
The first step is to understand the problem. You have to figure out the right questions to ask, how to frame them and know the assumptions based on domain knowledge.
Get the data permissions:
Do NOT forget about data privacy and compliance here, they are of paramount importance! Ask questions and engage with stakeholders, if needed.
Summarize the data:
Find the type of variables or map out the underlying data structure. This involves finding correlations among variables, identifying the most important variables, checking for missing values and mistakes in the data etc.
Visualize the data
Create a visual of the data to see the bigger picture. You’ll likely find new trends, anomalies, and outliers. Use data summarization and data visualization techniques to understand the story the data is telling you.
Create a simplistic model:
Linear or logistic regression are good starting points. Start with only the most important features (directly observed and reported features). This will allow you to gain a good familiarity with the problem at hand and also set the right direction for the next steps.
Narrow your features:
Prepare the data to extract the more intricate data patterns. Combine and modify existing features to create new features.
Explore and select a model:
Here, you’ll explore many different machine learning models and short-list the best ones based on comparative evaluation, e.g., compare RMSE or ROC-AUC scores for different models.
Customize the model
Fine-tune the parameters of your chosen model to fit the problem. Consider combining multiple models them for the best results.
Present your solution:
Share your findings in a simple, engaging, and visually appealing manner. Remember to tailor your presentation and language based on the technical level of your target audience. Give extra focus to what your findings mean and why that matters in the interests of the company.
From here, employers may ask you to create a system to address your findings or may ask you to form a proposal of next steps.
These 9 steps reflect the standard day-to-day machine learning work you can expect in the industry.
We’ve covered some of the basics of machine learning for data scientists, but there is still a lot more to learn and explore if you really want to get your career started, and you don’t have to go through it alone.
Industry expert and Microsoft Senior AI Engineer, Samia Khalid, has compiled her learnings into a comprehensive course, Grokking Data Science. This course lays out everything you’ll need in one place to get started and thrive in a data science career.
By the end of the course, you’ll have extensive hands-on practice with Python data science tools and real-life experience with a data science project.
Join a community of 500,000 monthly readers. A free, bi-monthly email with a roundup of Educative's top articles and coding tips.