Entry 52: Ensemble Methods: Bagging, Boosting, Stacking
There are multiple ways to reduce the bias and variance of Ensemble Learning, the three most common are bagging, boosting, and stacking. For more on bias and...
Posts by Tag
- machine learning 52
- graph 24
- neo4j 23
- supervised learning 20
- graph analytics 16
- model-eval 15
- pre-process 10
- regression 10
- trees 9
- process 8
- dataset titanic 8
- dataset breast cancer 7
- dataset planets 6
- dataset auto mpg 5
- overfitting 4
- create database 4
- nlp 4
- underfitting 3
- regularization 3
- dataset iris 3
- latex 2
- cat encoding 2
- cross-validation 2
- dataset horse colic 2
- regression models 2
- classification models 2
- thresholds 2
- dataset mnist 2
- dataset boston housing 2
- algorithms 2
- regression calculation 2
- ensembles 2
- aws 2
- sagemaker 2
- production pipeline 2
- frame problem 1
- dataset computer hardware 1
- dataset qsar fish toxicity 1
- dataset qsar aquatic toxicity 1
- dataset online news popularity 1
- dataset csm (conventional and social media movies) 1
- dataset mushrooms 1
- dataset mushroom 1
- dataset solar flare 1
- dataset nursery 1
- dataset chess 1
- pipeline 1
- bias 1
- variance 1
- load data 1
- dataset MNIST 1
- dataset click prediction 1
- dataset house_16H 1
- data size 1
- dataset forge 1
- dataset wave 1
- dataset california housing 1
- dataset bonanza 1
- datasets 1
- supervised learning datasets 1
- logistic regression 1
- graph visualization 1
- talks 1
- visualization 1
- data sensitivity 1
- n-grams 1
- entity resolution 1
- graph model 1
- measuring performance 1
- regex 1
- reading files 1
- ngrams 1
- text analysis 1
- boto3 1
- s3 1
- read from s3 1
- write to s3 1
- data cleaning 1
- data wrangling 1
machine learning
Entry 51: Ensemble Learning
Ensemble techniques began to appear in the 1990s according to page 192 of Applied Predictive Modeling. Most people think of Random Forests (basically a bunch...
Entry 50: Decision Tree Subtypes
The Complete Guide to Decision Trees sums up the similarities between Decision Tree Algorithms very succinctly:
Entry 49: Decision Trees Analysis and Interpretation
One of the major benefits of using Decision Trees is their interpretability. To take advantage of this benefit, you need to know how to pull out the informat...
Entry 48: Decision Tree Impurity Measures
Impurity seems like it should be a simple calculation. However, depending on prevalence of classes and quirks in the data, it’s usually not as straight forwa...
Entry 47: Pruning Decision Trees
If allowed to continue, a Decision Tree will continue to split the data until each leaf is pure. This causes two problems:
Entry 46: Overfitting, Underfitting, and Data Sensitivity
I wanted to point a co-worker to information about overfitting the other day. While I’ve discussed it in entries 17 and 30, I realized I haven’t covered it i...
Entry 45: Visualizing Decision Trees
A major benefit of tree-based models is how easy they are to visualize. This visualization aspect is also vital to discussing how trees work.
Entry 44: Decision Trees
In entries 35 through 42 we learned how to find linear patterns using Regression algorithms. But what if the patterns in the data aren’t linear? This is wher...
Entry 42: Logistic Regression
Page 143 of Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow states that Logistic Regression is “just like a Linear Regression model” excep...
Entry 41: Elastic Net
The notebook where I did my code for this entry can be found on my github page in the Entry 41 notebook.
Entry 40: Ridge Regression
Ridge Regression is a form of regression regularization using L2 regularization.
Entry 39: Lasso Regression
Least Absolute Shrinkage and Selection Operator (LASSO) Regression is a form of regression regularization using L1 regularization.
Entry 38: Regularization
Regularization is used to help address overfitting.
Entry 37b: Gradient Descent
The general idea of Gradient Descent is to iteratively minimize the errors (usually using the mean squared error) to arrive at better and better solutions. K...
Entry 37a: Normal Equation
The Normal Equation generates the weights that are used in Linear Regression. It calculates the theta array directly without having to iterate through differ...
Entry 36: Ordinary Least Squares (OLS)
Ordinary Leas Squares is usually the default method of Linear Regression and is the method used in the sklearn.linear_model.LinearRegression function. On pag...
Entry 35: Regression
Regression is used to predict on continuous data, for example housing prices. Logistic Regression is the subcategory that is the exception, as it does classi...
Entry 34: Supervised Learning Datasets
My anticipated fail from Entry 32 was correct, finding datasets and figuring out what kinds of patterns they have was a time suck. As such, “datasets” gets i...
Entry 33: Supervised Learning
Description
Entry 32: Modeling data
The challenge in this, the third series of entries, is to become familiar with the major categories of algorithms, how to implement them, determine the commo...
Entry 31: Training Size
At the very end of Entry 30 I mentioned that learning curves can be used to diagnose problems other than just high bias and high variance. Another easy probl...
Entry 30: Improve Performance - Learning Curves
Learning curves can help determine what avenues to pursue if a model isn’t up to expectations, or worse, is completely unusable. They can also be used to det...
Entry 29: Thresholds - Profit and cost
Sometimes considerations other than model performance need to be accounted for when choosing a threshold.
Entry 28: Cumulative gains and lift
Remember back in Entry 16 when I said I wasn’t planning to cover lift? Well, plans change.
Entry 27: Figuring out openml.org
Figuring out how to get the data from openml.org into the Entry 26e notebook (MNIST was surprisingly difficult. All the datasets are saved as arff files, an ...
Entry 26: Setting thresholds - precision, recall, and ROC
I don’t always want the default threshold for determining the classification (negative or positive) the way I did in Entry 24. As discussed in the precision ...
Entry 25: Baseline Models
As discussed in Entry 16, certain characteristics in the data can make a model look like it’s performing better than it is. One of these characteristics is c...
Entry 24: Scoring Classification Models - Implementation
Now that I’ve got a handle on the measurement options and equations for classification problems, it’s time to implement those measures on actual models.
Entry 23: Scoring Classification Models - Theory
Classification models present a different challenge than regression models. Because a numeric value isn’t returned, another way of measuring goodness of fit ...
Entry 22: Scoring Regression models - Implementation
Now that I’ve got a handle on the measurement options and equations, it’s time to implement those measures on actual models.
Entry 21: Scoring Regression Models - Theory
I need a way to measure a model’s performance. To do that, first I need to break it down by the type of prediction.
Entry 20: Scikit-Learn Pipeline
There are quite a few steps between pulling data and having a trained model. I need a good way to string it all together.
Entry 19: Implementing Cross-validation
In Entry 18 I finalized the decision to use a hybrid approach to validating models. Now I have to implement it.
Entry 18: Cross-validation
In Entry 17, I decided I want to use a hybrid cross-validation method. I intend to split out a hold-out set, then perform cross-validation on the training se...
Entry 17: Resampling
I have no intention of simply training a model and calling it quits. Once the model is trained, I need to now how well it did and what areas to concentrate o...
Entry 16: Model Evaluation
Max Kuhn and Kjell Johnson make the case in Applied Predictive Modeling that models can easily overemphasize patterns that are not reproducible.
Entry 15: Categorical Correlation/Collinearity
To ensure my process worked, I used it on multiple datasets. The code from notebook to notebook is mostly the same, just run on different data. Per usual, th...
Entry 14: Encoding Categorical Variables
The Problem
Entry 13: Categorical Preliminaries
Real world data often includes categorical variables like: Male or female Continent: North America, South America, Europe, Asia, Australia, Antarctica, ...
Entry 12: Missing Values
Wikipedia has a succinct definition of missing values: “missing data, or missing values, occur when no data value is stored for the variable in an observatio...
Entry 11: Consolidate Pre-processing Steps
Working on the machine learning process over 10 entries left my code fragmented with different versions of key functions scattered across multiple notebooks.
Entry 10: Reorder Pre-processing and Make Predictions
Using the pre-processing steps I worked through in entries 6-8, I can now predict mass while only changing the surface pressure.
Entry 9: Train Model
At the end of the Entry 8 noteboook I had a standardized dataset. Now it’s time to try training a model.
Entry 8: Centering and Scaling
By the end of Entry 7 I’d finalized the feature set to train a model and make predictions. One last step is needed before a prediction can be made.
Entry 7: Collinearity
In Entry 6 I looked at the correlation between the prediction features and the feature of interest (well, kinda. This problem question is a little weird in t...
Entry 6: Correlations
I need to find a way to automate some of the process outlined in Entry 5. My coworker Sabber suggested using correlation to help sort out the wheat from the ...
Entry 5: Explore the Data
Now that I have the dataset I put together in Entry 4, it’s time to see what’s in it.
Entry 4: Get the Data
In Entry 3 I defined my problem as: Holding all other factors constant, what mass is needed to retain an atmosphere on Mars? I need data to solve it.
Entry 3: Frame the Problem
In Entry 2 I decided to follow the ML Project Checklist from Hands on Machine Learning with Scikit-Learn & TensorFlow. The first step in that process is ...
Entry 2: Define the Process
In Entry 1, I decided to take the smallest problem I could find and solve just one aspect. Only after I’ve solved that single problem am I allowed to move on...
Entry 1: Impostor Syndrome
I am qualified. Quantitatively, I know this is true: I have a Master of Science degree in Data Science I have a job on a machine learning team at a larg...
graph
Entry G22: Mean Distance Between Connected Nodes
The apoc.path.expandingTree algorithm in Entry G19 revealed a gold mine of information. Once I had that table of results I knew that not only could I grab th...
Entry G21: Diameter
Having encountered the limit of connections between node pairs in Entry G19, I couldn’t resist taking a closer look. I’ve been trying to calculate the diamet...
Entry G20: Shortest Path
In an unweighted graph, the fewest steps between any node (A) to another node (B) is called the shortest path.
Entry G19: Neighborhood Node Counts
Now that we’ve established how to define an egocentric neighborhood and differentiated our people nodes, it’s time to start calculating metrics for the diffe...
Entry G18: Egocentric Networks
Now that we’ve gone through the global metrics and have a feel for the structure and composition of our data, it’s time to start running local metrics.
Entry G17: Add Villains
The use case I’m interested in is locating fraud within a graph. This is difficult to do when all the people nodes in your data are undifferentiated.
Entry G16: Components Comparison
The notebook that accompanies this entry is a cleaned up, concise version of the three notebooks I created for Entry G8, addressing just the graph components...
Entry G15: Global Density Comparison
The notebook that accompanies this entry is a cleaned up, concise version of the three notebooks I created for Entry G7, addressing only the number of possib...
Entry G14: Global Counts Comparison
The notebook that accompanies this entry is the cleaned up, concise version of the three notebooks that accompanied Entry G6, but limited to just the global ...
Entry G13: Weighted Degree Comparison
Like Entry G12 this is a redo of part of Entry G10. This entry addresses the weighted degrees. If you need a reminder as to what weighted relationships are s...
Entry G12: Degree Comparison
This is essentially a redo of the unweighted degree metrics from Entry G10. I ran the same metrics and queries from that entry, except I used the multigraph ...
Entry G11: Create the Marvel Multigraph Database
Creating a multigraph database was actually way easier than I expected.
Entry G10: Local Metrics
Now that I know what the larger graph looks like, I need metrics at the node level. The reason for these metrics is to be able to locate outlier nodes within...
Entry G9: Measuring Performance
As I may or may not have mentioned, the last time I tried running many of the calculations that I’ll be exploring in this series, I ran into timeout errors …...
Entry G8: Components
Now that I have a general feel for the graph database with counts and density, I want to look at components.
Entry G7: Density and Diameter
Density and diameter give you a feel for how strongly connected the full graph is: whether it’s dense or sparse. To get these measures, I need to calculate t...
Entry G6: Global Graph Counts
The first thing I want to do is get some global measures on the overall graph. I’m breaking these up into three categories: counts, density and diameter, and...
Entry G5.5: Analysis Metrics
I am 100% adding this entry retroactively. And yes, I dated it wrong so that it would show up in the right order in the Post list. That’s the nice thing abou...
Entry G5: Projecting Bimodal to Unimodal
I need to understand graph structure better and the repercussions of using the different model types. Specifically, I’m interested in memory use, processing ...
Entry G4: Modeling Relationships
Relationships are the lines that connect nodes. Neo4j usually describes nodes as nouns (a person, place, or thing - i.e. the subject) and relationships as ve...
Entry G3: Choosing a Graph Model (schema)
To harness the power of graph, data first has to be organized to fit the node/relationship format.
Entry G2: Create a Neo4j Database
To begin any data science project, I need a data to play with. For the graph project I’m going to start with the Marvel Universe Social Network available on ...
Entry G1: Connected Entities
The last post was the 52nd entry, meaning that I averaged one entry a week for an entire year. Now, just a little under a year into my chronicling journey, I...
Entry 43: NODES2020 Graph Visualization Talk
This is completely off topic for the machine learning algorithms series I’ve been working through, but it took up a lot of my time the last couple of weeks, ...
neo4j
Entry G22: Mean Distance Between Connected Nodes
The apoc.path.expandingTree algorithm in Entry G19 revealed a gold mine of information. Once I had that table of results I knew that not only could I grab th...
Entry G21: Diameter
Having encountered the limit of connections between node pairs in Entry G19, I couldn’t resist taking a closer look. I’ve been trying to calculate the diamet...
Entry G20: Shortest Path
In an unweighted graph, the fewest steps between any node (A) to another node (B) is called the shortest path.
Entry G19: Neighborhood Node Counts
Now that we’ve established how to define an egocentric neighborhood and differentiated our people nodes, it’s time to start calculating metrics for the diffe...
Entry G18: Egocentric Networks
Now that we’ve gone through the global metrics and have a feel for the structure and composition of our data, it’s time to start running local metrics.
Entry G17: Add Villains
The use case I’m interested in is locating fraud within a graph. This is difficult to do when all the people nodes in your data are undifferentiated.
Entry G16: Components Comparison
The notebook that accompanies this entry is a cleaned up, concise version of the three notebooks I created for Entry G8, addressing just the graph components...
Entry G15: Global Density Comparison
The notebook that accompanies this entry is a cleaned up, concise version of the three notebooks I created for Entry G7, addressing only the number of possib...
Entry G14: Global Counts Comparison
The notebook that accompanies this entry is the cleaned up, concise version of the three notebooks that accompanied Entry G6, but limited to just the global ...
Entry G13: Weighted Degree Comparison
Like Entry G12 this is a redo of part of Entry G10. This entry addresses the weighted degrees. If you need a reminder as to what weighted relationships are s...
Entry G12: Degree Comparison
This is essentially a redo of the unweighted degree metrics from Entry G10. I ran the same metrics and queries from that entry, except I used the multigraph ...
Entry G11: Create the Marvel Multigraph Database
Creating a multigraph database was actually way easier than I expected.
Entry G10: Local Metrics
Now that I know what the larger graph looks like, I need metrics at the node level. The reason for these metrics is to be able to locate outlier nodes within...
Entry G9: Measuring Performance
As I may or may not have mentioned, the last time I tried running many of the calculations that I’ll be exploring in this series, I ran into timeout errors …...
Entry G8: Components
Now that I have a general feel for the graph database with counts and density, I want to look at components.
Entry G7: Density and Diameter
Density and diameter give you a feel for how strongly connected the full graph is: whether it’s dense or sparse. To get these measures, I need to calculate t...
Entry G6: Global Graph Counts
The first thing I want to do is get some global measures on the overall graph. I’m breaking these up into three categories: counts, density and diameter, and...
Entry G5.5: Analysis Metrics
I am 100% adding this entry retroactively. And yes, I dated it wrong so that it would show up in the right order in the Post list. That’s the nice thing abou...
Entry G5: Projecting Bimodal to Unimodal
I need to understand graph structure better and the repercussions of using the different model types. Specifically, I’m interested in memory use, processing ...
Entry G4: Modeling Relationships
Relationships are the lines that connect nodes. Neo4j usually describes nodes as nouns (a person, place, or thing - i.e. the subject) and relationships as ve...
Entry G3: Choosing a Graph Model (schema)
To harness the power of graph, data first has to be organized to fit the node/relationship format.
Entry G2: Create a Neo4j Database
To begin any data science project, I need a data to play with. For the graph project I’m going to start with the Marvel Universe Social Network available on ...
Entry G1: Connected Entities
The last post was the 52nd entry, meaning that I averaged one entry a week for an entire year. Now, just a little under a year into my chronicling journey, I...
supervised learning
Entry 52: Ensemble Methods: Bagging, Boosting, Stacking
There are multiple ways to reduce the bias and variance of Ensemble Learning, the three most common are bagging, boosting, and stacking. For more on bias and...
Entry 51: Ensemble Learning
Ensemble techniques began to appear in the 1990s according to page 192 of Applied Predictive Modeling. Most people think of Random Forests (basically a bunch...
Entry 50: Decision Tree Subtypes
The Complete Guide to Decision Trees sums up the similarities between Decision Tree Algorithms very succinctly:
Entry 49: Decision Trees Analysis and Interpretation
One of the major benefits of using Decision Trees is their interpretability. To take advantage of this benefit, you need to know how to pull out the informat...
Entry 48: Decision Tree Impurity Measures
Impurity seems like it should be a simple calculation. However, depending on prevalence of classes and quirks in the data, it’s usually not as straight forwa...
Entry 47: Pruning Decision Trees
If allowed to continue, a Decision Tree will continue to split the data until each leaf is pure. This causes two problems:
Entry 46: Overfitting, Underfitting, and Data Sensitivity
I wanted to point a co-worker to information about overfitting the other day. While I’ve discussed it in entries 17 and 30, I realized I haven’t covered it i...
Entry 45: Visualizing Decision Trees
A major benefit of tree-based models is how easy they are to visualize. This visualization aspect is also vital to discussing how trees work.
Entry 44: Decision Trees
In entries 35 through 42 we learned how to find linear patterns using Regression algorithms. But what if the patterns in the data aren’t linear? This is wher...
Entry 43: NODES2020 Graph Visualization Talk
This is completely off topic for the machine learning algorithms series I’ve been working through, but it took up a lot of my time the last couple of weeks, ...
Entry 42: Logistic Regression
Page 143 of Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow states that Logistic Regression is “just like a Linear Regression model” excep...
Entry 41: Elastic Net
The notebook where I did my code for this entry can be found on my github page in the Entry 41 notebook.
Entry 40: Ridge Regression
Ridge Regression is a form of regression regularization using L2 regularization.
Entry 39: Lasso Regression
Least Absolute Shrinkage and Selection Operator (LASSO) Regression is a form of regression regularization using L1 regularization.
Entry 38: Regularization
Regularization is used to help address overfitting.
Entry 37b: Gradient Descent
The general idea of Gradient Descent is to iteratively minimize the errors (usually using the mean squared error) to arrive at better and better solutions. K...
Entry 37a: Normal Equation
The Normal Equation generates the weights that are used in Linear Regression. It calculates the theta array directly without having to iterate through differ...
Entry 36: Ordinary Least Squares (OLS)
Ordinary Leas Squares is usually the default method of Linear Regression and is the method used in the sklearn.linear_model.LinearRegression function. On pag...
Entry 35: Regression
Regression is used to predict on continuous data, for example housing prices. Logistic Regression is the subcategory that is the exception, as it does classi...
Entry 33: Supervised Learning
Description
graph analytics
Entry G22: Mean Distance Between Connected Nodes
The apoc.path.expandingTree algorithm in Entry G19 revealed a gold mine of information. Once I had that table of results I knew that not only could I grab th...
Entry G21: Diameter
Having encountered the limit of connections between node pairs in Entry G19, I couldn’t resist taking a closer look. I’ve been trying to calculate the diamet...
Entry G20: Shortest Path
In an unweighted graph, the fewest steps between any node (A) to another node (B) is called the shortest path.
Entry G19: Neighborhood Node Counts
Now that we’ve established how to define an egocentric neighborhood and differentiated our people nodes, it’s time to start calculating metrics for the diffe...
Entry G18: Egocentric Networks
Now that we’ve gone through the global metrics and have a feel for the structure and composition of our data, it’s time to start running local metrics.
Entry G16: Components Comparison
The notebook that accompanies this entry is a cleaned up, concise version of the three notebooks I created for Entry G8, addressing just the graph components...
Entry G15: Global Density Comparison
The notebook that accompanies this entry is a cleaned up, concise version of the three notebooks I created for Entry G7, addressing only the number of possib...
Entry G14: Global Counts Comparison
The notebook that accompanies this entry is the cleaned up, concise version of the three notebooks that accompanied Entry G6, but limited to just the global ...
Entry G13: Weighted Degree Comparison
Like Entry G12 this is a redo of part of Entry G10. This entry addresses the weighted degrees. If you need a reminder as to what weighted relationships are s...
Entry G12: Degree Comparison
This is essentially a redo of the unweighted degree metrics from Entry G10. I ran the same metrics and queries from that entry, except I used the multigraph ...
Entry G10: Local Metrics
Now that I know what the larger graph looks like, I need metrics at the node level. The reason for these metrics is to be able to locate outlier nodes within...
Entry G9: Measuring Performance
As I may or may not have mentioned, the last time I tried running many of the calculations that I’ll be exploring in this series, I ran into timeout errors …...
Entry G8: Components
Now that I have a general feel for the graph database with counts and density, I want to look at components.
Entry G7: Density and Diameter
Density and diameter give you a feel for how strongly connected the full graph is: whether it’s dense or sparse. To get these measures, I need to calculate t...
Entry G6: Global Graph Counts
The first thing I want to do is get some global measures on the overall graph. I’m breaking these up into three categories: counts, density and diameter, and...
Entry G5.5: Analysis Metrics
I am 100% adding this entry retroactively. And yes, I dated it wrong so that it would show up in the right order in the Post list. That’s the nice thing abou...
model-eval
Entry 31: Training Size
At the very end of Entry 30 I mentioned that learning curves can be used to diagnose problems other than just high bias and high variance. Another easy probl...
Entry 30: Improve Performance - Learning Curves
Learning curves can help determine what avenues to pursue if a model isn’t up to expectations, or worse, is completely unusable. They can also be used to det...
Entry 29: Thresholds - Profit and cost
Sometimes considerations other than model performance need to be accounted for when choosing a threshold.
Entry 28: Cumulative gains and lift
Remember back in Entry 16 when I said I wasn’t planning to cover lift? Well, plans change.
Entry 26: Setting thresholds - precision, recall, and ROC
I don’t always want the default threshold for determining the classification (negative or positive) the way I did in Entry 24. As discussed in the precision ...
Entry 25: Baseline Models
As discussed in Entry 16, certain characteristics in the data can make a model look like it’s performing better than it is. One of these characteristics is c...
Entry 24: Scoring Classification Models - Implementation
Now that I’ve got a handle on the measurement options and equations for classification problems, it’s time to implement those measures on actual models.
Entry 23: Scoring Classification Models - Theory
Classification models present a different challenge than regression models. Because a numeric value isn’t returned, another way of measuring goodness of fit ...
Entry 22: Scoring Regression models - Implementation
Now that I’ve got a handle on the measurement options and equations, it’s time to implement those measures on actual models.
Entry 21: Scoring Regression Models - Theory
I need a way to measure a model’s performance. To do that, first I need to break it down by the type of prediction.
Entry 20: Scikit-Learn Pipeline
There are quite a few steps between pulling data and having a trained model. I need a good way to string it all together.
Entry 19: Implementing Cross-validation
In Entry 18 I finalized the decision to use a hybrid approach to validating models. Now I have to implement it.
Entry 18: Cross-validation
In Entry 17, I decided I want to use a hybrid cross-validation method. I intend to split out a hold-out set, then perform cross-validation on the training se...
Entry 17: Resampling
I have no intention of simply training a model and calling it quits. Once the model is trained, I need to now how well it did and what areas to concentrate o...
Entry 16: Model Evaluation
Max Kuhn and Kjell Johnson make the case in Applied Predictive Modeling that models can easily overemphasize patterns that are not reproducible.
pre-process
Entry 15: Categorical Correlation/Collinearity
To ensure my process worked, I used it on multiple datasets. The code from notebook to notebook is mostly the same, just run on different data. Per usual, th...
Entry 14: Encoding Categorical Variables
The Problem
Entry 13: Categorical Preliminaries
Real world data often includes categorical variables like: Male or female Continent: North America, South America, Europe, Asia, Australia, Antarctica, ...
Entry 12: Missing Values
Wikipedia has a succinct definition of missing values: “missing data, or missing values, occur when no data value is stored for the variable in an observatio...
Entry 11: Consolidate Pre-processing Steps
Working on the machine learning process over 10 entries left my code fragmented with different versions of key functions scattered across multiple notebooks.
Entry 10: Reorder Pre-processing and Make Predictions
Using the pre-processing steps I worked through in entries 6-8, I can now predict mass while only changing the surface pressure.
Entry 9: Train Model
At the end of the Entry 8 noteboook I had a standardized dataset. Now it’s time to try training a model.
Entry 8: Centering and Scaling
By the end of Entry 7 I’d finalized the feature set to train a model and make predictions. One last step is needed before a prediction can be made.
Entry 7: Collinearity
In Entry 6 I looked at the correlation between the prediction features and the feature of interest (well, kinda. This problem question is a little weird in t...
Entry 6: Correlations
I need to find a way to automate some of the process outlined in Entry 5. My coworker Sabber suggested using correlation to help sort out the wheat from the ...
regression
Entry 43: NODES2020 Graph Visualization Talk
This is completely off topic for the machine learning algorithms series I’ve been working through, but it took up a lot of my time the last couple of weeks, ...
Entry 42: Logistic Regression
Page 143 of Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow states that Logistic Regression is “just like a Linear Regression model” excep...
Entry 41: Elastic Net
The notebook where I did my code for this entry can be found on my github page in the Entry 41 notebook.
Entry 40: Ridge Regression
Ridge Regression is a form of regression regularization using L2 regularization.
Entry 39: Lasso Regression
Least Absolute Shrinkage and Selection Operator (LASSO) Regression is a form of regression regularization using L1 regularization.
Entry 38: Regularization
Regularization is used to help address overfitting.
Entry 37b: Gradient Descent
The general idea of Gradient Descent is to iteratively minimize the errors (usually using the mean squared error) to arrive at better and better solutions. K...
Entry 37a: Normal Equation
The Normal Equation generates the weights that are used in Linear Regression. It calculates the theta array directly without having to iterate through differ...
Entry 36: Ordinary Least Squares (OLS)
Ordinary Leas Squares is usually the default method of Linear Regression and is the method used in the sklearn.linear_model.LinearRegression function. On pag...
Entry 35: Regression
Regression is used to predict on continuous data, for example housing prices. Logistic Regression is the subcategory that is the exception, as it does classi...
trees
Entry 52: Ensemble Methods: Bagging, Boosting, Stacking
There are multiple ways to reduce the bias and variance of Ensemble Learning, the three most common are bagging, boosting, and stacking. For more on bias and...
Entry 51: Ensemble Learning
Ensemble techniques began to appear in the 1990s according to page 192 of Applied Predictive Modeling. Most people think of Random Forests (basically a bunch...
Entry 50: Decision Tree Subtypes
The Complete Guide to Decision Trees sums up the similarities between Decision Tree Algorithms very succinctly:
Entry 49: Decision Trees Analysis and Interpretation
One of the major benefits of using Decision Trees is their interpretability. To take advantage of this benefit, you need to know how to pull out the informat...
Entry 48: Decision Tree Impurity Measures
Impurity seems like it should be a simple calculation. However, depending on prevalence of classes and quirks in the data, it’s usually not as straight forwa...
Entry 47: Pruning Decision Trees
If allowed to continue, a Decision Tree will continue to split the data until each leaf is pure. This causes two problems:
Entry 46: Overfitting, Underfitting, and Data Sensitivity
I wanted to point a co-worker to information about overfitting the other day. While I’ve discussed it in entries 17 and 30, I realized I haven’t covered it i...
Entry 45: Visualizing Decision Trees
A major benefit of tree-based models is how easy they are to visualize. This visualization aspect is also vital to discussing how trees work.
Entry 44: Decision Trees
In entries 35 through 42 we learned how to find linear patterns using Regression algorithms. But what if the patterns in the data aren’t linear? This is wher...
process
Entry 11: Consolidate Pre-processing Steps
Working on the machine learning process over 10 entries left my code fragmented with different versions of key functions scattered across multiple notebooks.
Entry 7: Collinearity
In Entry 6 I looked at the correlation between the prediction features and the feature of interest (well, kinda. This problem question is a little weird in t...
Entry 6: Correlations
I need to find a way to automate some of the process outlined in Entry 5. My coworker Sabber suggested using correlation to help sort out the wheat from the ...
Entry 5: Explore the Data
Now that I have the dataset I put together in Entry 4, it’s time to see what’s in it.
Entry 4: Get the Data
In Entry 3 I defined my problem as: Holding all other factors constant, what mass is needed to retain an atmosphere on Mars? I need data to solve it.
Entry 3: Frame the Problem
In Entry 2 I decided to follow the ML Project Checklist from Hands on Machine Learning with Scikit-Learn & TensorFlow. The first step in that process is ...
Entry 2: Define the Process
In Entry 1, I decided to take the smallest problem I could find and solve just one aspect. Only after I’ve solved that single problem am I allowed to move on...
Entry 1: Impostor Syndrome
I am qualified. Quantitatively, I know this is true: I have a Master of Science degree in Data Science I have a job on a machine learning team at a larg...
dataset titanic
Entry 49: Decision Trees Analysis and Interpretation
One of the major benefits of using Decision Trees is their interpretability. To take advantage of this benefit, you need to know how to pull out the informat...
Entry 47: Pruning Decision Trees
If allowed to continue, a Decision Tree will continue to split the data until each leaf is pure. This causes two problems:
Entry 44: Decision Trees
In entries 35 through 42 we learned how to find linear patterns using Regression algorithms. But what if the patterns in the data aren’t linear? This is wher...
Entry 29: Thresholds - Profit and cost
Sometimes considerations other than model performance need to be accounted for when choosing a threshold.
Entry 28: Cumulative gains and lift
Remember back in Entry 16 when I said I wasn’t planning to cover lift? Well, plans change.
Entry 27: Figuring out openml.org
Figuring out how to get the data from openml.org into the Entry 26e notebook (MNIST was surprisingly difficult. All the datasets are saved as arff files, an ...
Entry 26: Setting thresholds - precision, recall, and ROC
I don’t always want the default threshold for determining the classification (negative or positive) the way I did in Entry 24. As discussed in the precision ...
Entry 12: Missing Values
Wikipedia has a succinct definition of missing values: “missing data, or missing values, occur when no data value is stored for the variable in an observatio...
dataset breast cancer
Entry 49: Decision Trees Analysis and Interpretation
One of the major benefits of using Decision Trees is their interpretability. To take advantage of this benefit, you need to know how to pull out the informat...
Entry 47: Pruning Decision Trees
If allowed to continue, a Decision Tree will continue to split the data until each leaf is pure. This causes two problems:
Entry 44: Decision Trees
In entries 35 through 42 we learned how to find linear patterns using Regression algorithms. But what if the patterns in the data aren’t linear? This is wher...
Entry 33: Supervised Learning
Description
Entry 26: Setting thresholds - precision, recall, and ROC
I don’t always want the default threshold for determining the classification (negative or positive) the way I did in Entry 24. As discussed in the precision ...
Entry 25: Baseline Models
As discussed in Entry 16, certain characteristics in the data can make a model look like it’s performing better than it is. One of these characteristics is c...
Entry 24: Scoring Classification Models - Implementation
Now that I’ve got a handle on the measurement options and equations for classification problems, it’s time to implement those measures on actual models.
dataset planets
Entry 10: Reorder Pre-processing and Make Predictions
Using the pre-processing steps I worked through in entries 6-8, I can now predict mass while only changing the surface pressure.
Entry 9: Train Model
At the end of the Entry 8 noteboook I had a standardized dataset. Now it’s time to try training a model.
Entry 8: Centering and Scaling
By the end of Entry 7 I’d finalized the feature set to train a model and make predictions. One last step is needed before a prediction can be made.
Entry 7: Collinearity
In Entry 6 I looked at the correlation between the prediction features and the feature of interest (well, kinda. This problem question is a little weird in t...
Entry 6: Correlations
I need to find a way to automate some of the process outlined in Entry 5. My coworker Sabber suggested using correlation to help sort out the wheat from the ...
Entry 5: Explore the Data
Now that I have the dataset I put together in Entry 4, it’s time to see what’s in it.
dataset auto mpg
Entry 25: Baseline Models
As discussed in Entry 16, certain characteristics in the data can make a model look like it’s performing better than it is. One of these characteristics is c...
Entry 22: Scoring Regression models - Implementation
Now that I’ve got a handle on the measurement options and equations, it’s time to implement those measures on actual models.
Entry 20: Scikit-Learn Pipeline
There are quite a few steps between pulling data and having a trained model. I need a good way to string it all together.
Entry 19: Implementing Cross-validation
In Entry 18 I finalized the decision to use a hybrid approach to validating models. Now I have to implement it.
Entry 18: Cross-validation
In Entry 17, I decided I want to use a hybrid cross-validation method. I intend to split out a hold-out set, then perform cross-validation on the training se...
overfitting
Entry 46: Overfitting, Underfitting, and Data Sensitivity
I wanted to point a co-worker to information about overfitting the other day. While I’ve discussed it in entries 17 and 30, I realized I haven’t covered it i...
Entry 38: Regularization
Regularization is used to help address overfitting.
Entry 30: Improve Performance - Learning Curves
Learning curves can help determine what avenues to pursue if a model isn’t up to expectations, or worse, is completely unusable. They can also be used to det...
Entry 17: Resampling
I have no intention of simply training a model and calling it quits. Once the model is trained, I need to now how well it did and what areas to concentrate o...
create database
Entry G11: Create the Marvel Multigraph Database
Creating a multigraph database was actually way easier than I expected.
Entry G5: Projecting Bimodal to Unimodal
I need to understand graph structure better and the repercussions of using the different model types. Specifically, I’m interested in memory use, processing ...
Entry G4: Modeling Relationships
Relationships are the lines that connect nodes. Neo4j usually describes nodes as nouns (a person, place, or thing - i.e. the subject) and relationships as ve...
Entry G2: Create a Neo4j Database
To begin any data science project, I need a data to play with. For the graph project I’m going to start with the Marvel Universe Social Network available on ...
nlp
Entry NLP4: Frequencies and Comparison
In the previous entries in this series, I loaded all the files in a directory, processed the data, and transformed it into ngrams. Now it’s time to do math a...
Entry NLP3: Clean Data and Split into N-grams
In the first entry of this series, I figured out how to process the raw files. In the second entry, I figured out how to load all files in a directory (even ...
Entry NLP2: Load All Files in a Directory
In the previous entry, I figured out how to process individual files, removing many of the items on the “Remove lines/characters” list specified in the homew...
Entry NLP1: Corpus Cleaning with RegEx
Recently I’ve been working my way through one of the older versions of CSE 140 (now CSE 160) offered at the University of Washington. Homework 8 is a nice ex...
underfitting
Entry 46: Overfitting, Underfitting, and Data Sensitivity
I wanted to point a co-worker to information about overfitting the other day. While I’ve discussed it in entries 17 and 30, I realized I haven’t covered it i...
Entry 30: Improve Performance - Learning Curves
Learning curves can help determine what avenues to pursue if a model isn’t up to expectations, or worse, is completely unusable. They can also be used to det...
Entry 17: Resampling
I have no intention of simply training a model and calling it quits. Once the model is trained, I need to now how well it did and what areas to concentrate o...
regularization
Entry 40: Ridge Regression
Ridge Regression is a form of regression regularization using L2 regularization.
Entry 39: Lasso Regression
Least Absolute Shrinkage and Selection Operator (LASSO) Regression is a form of regression regularization using L1 regularization.
Entry 38: Regularization
Regularization is used to help address overfitting.
dataset iris
Entry 47: Pruning Decision Trees
If allowed to continue, a Decision Tree will continue to split the data until each leaf is pure. This causes two problems:
Entry 45: Visualizing Decision Trees
A major benefit of tree-based models is how easy they are to visualize. This visualization aspect is also vital to discussing how trees work.
Entry 44: Decision Trees
In entries 35 through 42 we learned how to find linear patterns using Regression algorithms. But what if the patterns in the data aren’t linear? This is wher...
latex
Entry 21: Scoring Regression Models - Theory
I need a way to measure a model’s performance. To do that, first I need to break it down by the type of prediction.
Entry 8: Centering and Scaling
By the end of Entry 7 I’d finalized the feature set to train a model and make predictions. One last step is needed before a prediction can be made.
cat encoding
Entry 14: Encoding Categorical Variables
The Problem
Entry 13: Categorical Preliminaries
Real world data often includes categorical variables like: Male or female Continent: North America, South America, Europe, Asia, Australia, Antarctica, ...
cross-validation
Entry 18: Cross-validation
In Entry 17, I decided I want to use a hybrid cross-validation method. I intend to split out a hold-out set, then perform cross-validation on the training se...
Entry 17: Resampling
I have no intention of simply training a model and calling it quits. Once the model is trained, I need to now how well it did and what areas to concentrate o...
dataset horse colic
Entry 26: Setting thresholds - precision, recall, and ROC
I don’t always want the default threshold for determining the classification (negative or positive) the way I did in Entry 24. As discussed in the precision ...
Entry 20: Scikit-Learn Pipeline
There are quite a few steps between pulling data and having a trained model. I need a good way to string it all together.
regression models
Entry 22: Scoring Regression models - Implementation
Now that I’ve got a handle on the measurement options and equations, it’s time to implement those measures on actual models.
Entry 21: Scoring Regression Models - Theory
I need a way to measure a model’s performance. To do that, first I need to break it down by the type of prediction.
classification models
Entry 24: Scoring Classification Models - Implementation
Now that I’ve got a handle on the measurement options and equations for classification problems, it’s time to implement those measures on actual models.
Entry 23: Scoring Classification Models - Theory
Classification models present a different challenge than regression models. Because a numeric value isn’t returned, another way of measuring goodness of fit ...
thresholds
Entry 29: Thresholds - Profit and cost
Sometimes considerations other than model performance need to be accounted for when choosing a threshold.
Entry 26: Setting thresholds - precision, recall, and ROC
I don’t always want the default threshold for determining the classification (negative or positive) the way I did in Entry 24. As discussed in the precision ...
dataset mnist
Entry 33: Supervised Learning
Description
Entry 26: Setting thresholds - precision, recall, and ROC
I don’t always want the default threshold for determining the classification (negative or positive) the way I did in Entry 24. As discussed in the precision ...
dataset boston housing
Entry 33: Supervised Learning
Description
Entry 30: Improve Performance - Learning Curves
Learning curves can help determine what avenues to pursue if a model isn’t up to expectations, or worse, is completely unusable. They can also be used to det...
algorithms
Entry 33: Supervised Learning
Description
Entry 32: Modeling data
The challenge in this, the third series of entries, is to become familiar with the major categories of algorithms, how to implement them, determine the commo...
regression calculation
Entry 37b: Gradient Descent
The general idea of Gradient Descent is to iteratively minimize the errors (usually using the mean squared error) to arrive at better and better solutions. K...
Entry 37a: Normal Equation
The Normal Equation generates the weights that are used in Linear Regression. It calculates the theta array directly without having to iterate through differ...
ensembles
Entry 52: Ensemble Methods: Bagging, Boosting, Stacking
There are multiple ways to reduce the bias and variance of Ensemble Learning, the three most common are bagging, boosting, and stacking. For more on bias and...
Entry 51: Ensemble Learning
Ensemble techniques began to appear in the 1990s according to page 192 of Applied Predictive Modeling. Most people think of Random Forests (basically a bunch...
aws
Entry SM02: Clean Data
Wrangling data into a usable form is a big part of any real world machine learning problem. When tackling these types of problems, several things are general...
Entry SM01: Using S3 from AWS’s SageMaker
There are a lot of considerations in moving from a local model used to train and predict on batch data to a production model. This series of posts explores h...
sagemaker
Entry SM02: Clean Data
Wrangling data into a usable form is a big part of any real world machine learning problem. When tackling these types of problems, several things are general...
Entry SM01: Using S3 from AWS’s SageMaker
There are a lot of considerations in moving from a local model used to train and predict on batch data to a production model. This series of posts explores h...
production pipeline
Entry SM02: Clean Data
Wrangling data into a usable form is a big part of any real world machine learning problem. When tackling these types of problems, several things are general...
Entry SM01: Using S3 from AWS’s SageMaker
There are a lot of considerations in moving from a local model used to train and predict on batch data to a production model. This series of posts explores h...
frame problem
Entry 3: Frame the Problem
In Entry 2 I decided to follow the ML Project Checklist from Hands on Machine Learning with Scikit-Learn & TensorFlow. The first step in that process is ...
dataset computer hardware
Entry 11: Consolidate Pre-processing Steps
Working on the machine learning process over 10 entries left my code fragmented with different versions of key functions scattered across multiple notebooks.
dataset qsar fish toxicity
Entry 11: Consolidate Pre-processing Steps
Working on the machine learning process over 10 entries left my code fragmented with different versions of key functions scattered across multiple notebooks.
dataset qsar aquatic toxicity
Entry 11: Consolidate Pre-processing Steps
Working on the machine learning process over 10 entries left my code fragmented with different versions of key functions scattered across multiple notebooks.
dataset online news popularity
Entry 11: Consolidate Pre-processing Steps
Working on the machine learning process over 10 entries left my code fragmented with different versions of key functions scattered across multiple notebooks.
dataset mushrooms
Entry 14: Encoding Categorical Variables
The Problem
dataset mushroom
Entry 15: Categorical Correlation/Collinearity
To ensure my process worked, I used it on multiple datasets. The code from notebook to notebook is mostly the same, just run on different data. Per usual, th...
dataset solar flare
Entry 15: Categorical Correlation/Collinearity
To ensure my process worked, I used it on multiple datasets. The code from notebook to notebook is mostly the same, just run on different data. Per usual, th...
dataset nursery
Entry 15: Categorical Correlation/Collinearity
To ensure my process worked, I used it on multiple datasets. The code from notebook to notebook is mostly the same, just run on different data. Per usual, th...
dataset chess
Entry 15: Categorical Correlation/Collinearity
To ensure my process worked, I used it on multiple datasets. The code from notebook to notebook is mostly the same, just run on different data. Per usual, th...
pipeline
Entry 20: Scikit-Learn Pipeline
There are quite a few steps between pulling data and having a trained model. I need a good way to string it all together.
bias
Entry 21: Scoring Regression Models - Theory
I need a way to measure a model’s performance. To do that, first I need to break it down by the type of prediction.
variance
Entry 21: Scoring Regression Models - Theory
I need a way to measure a model’s performance. To do that, first I need to break it down by the type of prediction.
load data
Entry 27: Figuring out openml.org
Figuring out how to get the data from openml.org into the Entry 26e notebook (MNIST was surprisingly difficult. All the datasets are saved as arff files, an ...
dataset MNIST
Entry 28: Cumulative gains and lift
Remember back in Entry 16 when I said I wasn’t planning to cover lift? Well, plans change.
dataset click prediction
Entry 28: Cumulative gains and lift
Remember back in Entry 16 when I said I wasn’t planning to cover lift? Well, plans change.
dataset house_16H
Entry 30: Improve Performance - Learning Curves
Learning curves can help determine what avenues to pursue if a model isn’t up to expectations, or worse, is completely unusable. They can also be used to det...
data size
Entry 31: Training Size
At the very end of Entry 30 I mentioned that learning curves can be used to diagnose problems other than just high bias and high variance. Another easy probl...
dataset forge
Entry 33: Supervised Learning
Description
dataset wave
Entry 33: Supervised Learning
Description
dataset california housing
Entry 33: Supervised Learning
Description
dataset bonanza
Entry 34: Supervised Learning Datasets
My anticipated fail from Entry 32 was correct, finding datasets and figuring out what kinds of patterns they have was a time suck. As such, “datasets” gets i...
datasets
Entry 34: Supervised Learning Datasets
My anticipated fail from Entry 32 was correct, finding datasets and figuring out what kinds of patterns they have was a time suck. As such, “datasets” gets i...
supervised learning datasets
Entry 34: Supervised Learning Datasets
My anticipated fail from Entry 32 was correct, finding datasets and figuring out what kinds of patterns they have was a time suck. As such, “datasets” gets i...
logistic regression
Entry 42: Logistic Regression
Page 143 of Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow states that Logistic Regression is “just like a Linear Regression model” excep...
graph visualization
Entry 43: NODES2020 Graph Visualization Talk
This is completely off topic for the machine learning algorithms series I’ve been working through, but it took up a lot of my time the last couple of weeks, ...
talks
Entry 43: NODES2020 Graph Visualization Talk
This is completely off topic for the machine learning algorithms series I’ve been working through, but it took up a lot of my time the last couple of weeks, ...
visualization
Entry 45: Visualizing Decision Trees
A major benefit of tree-based models is how easy they are to visualize. This visualization aspect is also vital to discussing how trees work.
data sensitivity
Entry 46: Overfitting, Underfitting, and Data Sensitivity
I wanted to point a co-worker to information about overfitting the other day. While I’ve discussed it in entries 17 and 30, I realized I haven’t covered it i...
n-grams
Entry 48: Decision Tree Impurity Measures
Impurity seems like it should be a simple calculation. However, depending on prevalence of classes and quirks in the data, it’s usually not as straight forwa...
entity resolution
Entry G1: Connected Entities
The last post was the 52nd entry, meaning that I averaged one entry a week for an entire year. Now, just a little under a year into my chronicling journey, I...
graph model
Entry G3: Choosing a Graph Model (schema)
To harness the power of graph, data first has to be organized to fit the node/relationship format.
measuring performance
Entry G9: Measuring Performance
As I may or may not have mentioned, the last time I tried running many of the calculations that I’ll be exploring in this series, I ran into timeout errors …...
regex
Entry NLP1: Corpus Cleaning with RegEx
Recently I’ve been working my way through one of the older versions of CSE 140 (now CSE 160) offered at the University of Washington. Homework 8 is a nice ex...
reading files
Entry NLP2: Load All Files in a Directory
In the previous entry, I figured out how to process individual files, removing many of the items on the “Remove lines/characters” list specified in the homew...
ngrams
Entry NLP3: Clean Data and Split into N-grams
In the first entry of this series, I figured out how to process the raw files. In the second entry, I figured out how to load all files in a directory (even ...
text analysis
Entry NLP4: Frequencies and Comparison
In the previous entries in this series, I loaded all the files in a directory, processed the data, and transformed it into ngrams. Now it’s time to do math a...
boto3
Entry SM01: Using S3 from AWS’s SageMaker
There are a lot of considerations in moving from a local model used to train and predict on batch data to a production model. This series of posts explores h...
s3
Entry SM01: Using S3 from AWS’s SageMaker
There are a lot of considerations in moving from a local model used to train and predict on batch data to a production model. This series of posts explores h...
read from s3
Entry SM01: Using S3 from AWS’s SageMaker
There are a lot of considerations in moving from a local model used to train and predict on batch data to a production model. This series of posts explores h...
write to s3
Entry SM01: Using S3 from AWS’s SageMaker
There are a lot of considerations in moving from a local model used to train and predict on batch data to a production model. This series of posts explores h...
data cleaning
Entry SM02: Clean Data
Wrangling data into a usable form is a big part of any real world machine learning problem. When tackling these types of problems, several things are general...
data wrangling
Entry SM02: Clean Data
Wrangling data into a usable form is a big part of any real world machine learning problem. When tackling these types of problems, several things are general...