Machine learning in Python with scikit-learn - How to evaluate a classifier in scikit-learn

In this video, you'll learn how to properly evaluate a classification model using a variety of common tools and metrics, as well as how to adjust the performance of a classifier to best match your business objectives. I'll start by demonstrating the weaknesses of classification accuracy as an evaluation metric. I'll then discuss the confusion matrix, the ROC curve and AUC, and metrics such as sensitivity, specificity, and precision. By the end of the video, you will have a solid foundation for intelligently evaluating your own classification model. Download the notebook: https://github.com/justmarkham/scikit-learn-videos == CONFUSION MATRIX RESOURCES == Simple guide to confusion matrix terminology: https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/ Intuitive sensitivity and specificity: https://www.youtube.com/watch?v=U4_3fditnWg The tradeoff between sensitivity and specificity: https://www.youtube.com/watch?v=vtYDyGGeQyo How to calculate "expected value" from a confusion matrix: https://github.com/podopie/DAT18NYC/blob/master/classes/13-expected_value_cost_benefit_analysis.ipynb Classification threshold graphic: https://media.amazonwebservices.com/blog/2015/ml_adjust_model_1.png == ROC/AUC RESOURCES == ROC Curves and Area Under the Curve: https://www.youtube.com/watch?v=OAl6eAyP-yo ROC visualization: http://www.navan.name/roc/ ROC Curves: https://www.youtube.com/watch?v=21Igj5Pr6u4 An introduction to ROC analysis: http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf Comparing different feature sets: http://research.microsoft.com/pubs/205472/aisec10-leontjeva.pdf Comparing different classifiers: http://www.cse.ust.hk/nevinZhangGroup/readings/yi/Bradley_PR97.pdf == OTHER RESOURCES == scikit-learn documentation on model evaluation: http://scikit-learn.org/stable/modules/model_evaluation.html Comparing model evaluation procedures and metrics: https://github.com/justmarkham/DAT8/blob/master/other/model_evaluation_comparison.md Counterfactual evaluation of machine learning models: https://www.youtube.com/watch?v=QWCSxAKR-h0 WANT TO GET BETTER AT MACHINE LEARNING? HERE ARE YOUR NEXT STEPS: 1) WATCH my scikit-learn video series: https://www.youtube.com/playlist?list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A 2) SUBSCRIBE for more videos: https://www.youtube.com/dataschool?sub_confirmation=1 3) JOIN "Data School Insiders" to access bonus content: https://www.patreon.com/dataschool 4) ENROLL in my Machine Learning course: https://www.dataschool.io/learn/ 5) LET'S CONNECT! - Newsletter: https://www.dataschool.io/subscribe/ - Twitter: https://twitter.com/justmarkham - Facebook: https://www.facebook.com/DataScienceSchool/ - LinkedIn: https://www.linkedin.com/in/justmarkham/