Skip to main content

Machine Learning - Model Evaluation Metrics


Confusion Matrix:




ROC (Receiver Operating Characteristics) and Area Under Curve (AUC)


ROC graphs are two-dimensional graphs in which tp rate (true positive rate or recall in above diagram) is plotted on the Y axis and fp rate is plotted on the X axis. An ROC graph depicts relative tradeoffs between benefits (true positives) and costs (false positives). More details: An introduction to ROC analysis by Tom Fawcett

For example, when you consider the results of a particular test in two populations, one population with a disease, the other population without the disease, you will rarely observe a perfect separation between the two groups. Indeed, the distribution of the test results will overlap, as shown in the following figure.


For every possible cut-off point or criterion value you select to discriminate between the two populations, there will be some cases with the disease correctly classified as positive (TP = True Positive fraction), but some cases with the disease will be classified negative (FN = False Negative fraction). On the other hand, some cases without the disease will be correctly classified as negative (TN = True Negative fraction), but some cases without the disease will be classified as positive (FP = False Positive fraction).

  • Sensitivity: probability that a test result will be positive when the disease is present (true positive rate, expressed as a percentage). 
  • Specificity: probability that a test result will be negative when the disease is not present (true negative rate, expressed as a percentage). 
  • Positive likelihood ratio: ratio between the probability of a positive test result given the presence of the disease and the probability of a positive test result given the absence of the disease, i.e. = True positive rate / False positive rate = Sensitivity / (1-Specificity)
  • Negative likelihood ratio: ratio between the probability of a negative test result given the presence of the disease and the probability of a negative test result given the absence of the disease, i.e. = False negative rate / True negative rate = (1-Sensitivity) / Specificity
  • Positive predictive value: probability that the disease is present when the test is positive (expressed as a percentage).
  • Negative predictive value: probability that the disease is not present when the test is negative (expressed as a percentage). 

In a Receiver Operating Characteristic (ROC) curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (100-Specificity) for different cut-off points. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. A test with perfect discrimination (no overlap in the two distributions) has a ROC curve that passes through the upper left corner (100% sensitivity, 100% specificity). Therefore the closer the ROC curve is to the upper left corner, the higher the overall accuracy of the test - see next diagram.


The most important metric are the following:



Comments

Post a Comment

Popular posts from this blog

IIMB PGSEM SOP Essays.

The IIMB PGSEM application for 2008 had a SOP section which required 5 short essays to be written. Here are the ones I had written: Statement of Purpose How do you see the PGSEM helping you in your goals? (150 words) My taking up the PGSEM course has twin objectives, namely, self-development and learning all aspects of setting up, managing a commercial/social enterprise. Having worked in the software industry for five years, I have closely seen the software development life-cycle. However, there are several aspects of business and the economy that are of interest to me and I find the time ripe to explore these in a formal way, through academics; specifically strategic management of a firm, innovation strategies, and the scope of strategic consulting. Getting ready to usher in acceleration in growth opportunities in my care
8 comments
Read more

Google BigQuery & Apache Hive

Google BIGQUERY is a fast, economical and fully-managed enterprise data warehouse for large-scale data analytics. Details of querying your custom table in BigQuery: https://cloud.google.com/bigquery/quickstart-web-ui The Apache Hive ™ data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage and queried using SQL syntax. Built on top of Apache Hadoop™, Hive provides the following features: Tools to enable easy access to data via SQL, thus enabling data warehousing tasks such as extract/transform/load (ETL), reporting, and data analysis. A mechanism to impose structure on a variety of data formats Access to files stored either directly in Apache HDFS™ or in other data storage systems such as Apache HBase™ Query execution via Apache Tez™, Apache Spark™, or MapReduce Procedural language with HPL-SQL Sub-second query retrieval via Hive LLAP, Apache YARN and Apache Slider. More details on getting started: 
Post a Comment
Read more

DNA newspaper plagiarizes my photographs!

The newspaper DNA (Daily News and Analysis - http://www.dnaindia.com/bangalore ) seems to have involved in not verifying its sources of photographs and having used my photographs (does this amount to plagiarism? I think it does) after it carried some of my pictures in the 'After Hrs' section of its newspaper on 31st January 2009, which I had taken at the IIMB Yamini 2009. It is good that they covered the event but they should have cited/verified the sources of the photographs. In all probability they or their sources just picked up the photos from my blog, with the belief that no one would notice anyways - seems they could not escape as luck would have it, I spotted them in the DNA paper on Saturday. It was early in the morning when as I flipped open the last page of the supplement that I was stunned to see my pics, which I was able to recognize immediately - however there were no credits anywhere in sight! Please check the photos below from the e-paper version on their website
23 comments
Read more