Otdfbt

How to tell your grandparent to not come to fetch you with their car?

Do simulator games use a realistic trajectory to get into orbit?

Confusion around using "des" in sentences

Generate a Graeco-Latin square

Where Mongol herds graze

How to return a security deposit to a tenant

Watts of filament extrusion

1980s live-action movie where individually-coloured nations on clouds fight

Is counterpoint still used today?

Does an ice chest packed full of frozen food need ice?

What is the reason for double NULL check of pointer for mutex lock

Why is one of Madera Municipal's runways labelled with only "R" on both sides?

Is it a problem if <h4>, <h5> and <h6> are smaller than regular text?

Second (easy access) account in case my bank screws up

Why would future John risk sending back a T-800 to save his younger self?

What language is software running on the ISS written in?

Should I avoid hard-packed crusher dust trails with my hybrid?

Is mleccha entirely different from Hindu classification?

At what point in time did Dumbledore ask Snape for this favor?

How is water heavier than petrol, even though its molecular weight is less than petrol?

What ways have you found to get edits from non-LaTeX users?

Are there any instruments that don't produce overtones?

Source that a married woman seduced by a “messianic figure” is still permitted to her husband

What makes Ada the language of choice for the ISS's safety-critical systems?

Interpretation of ROC AUC score

What is the right algorithm to detect segmentations of a line chart?Variance in cross validation score / model selectionOutlier detection by unsupervised algorithm: Fraud DetectionROC curve for different hyperparameters of `RandomForestClassifier`?different results with MEKA vs Scikit-learn!Interpretation of variable or feature importance in Random ForestGradient descent multidimensional linear regression - does learning rate affects concurrency?Decent ROC, but horrible Precision-Recall curveEvaluating the test setLooking for a classification (?) algorithm for linearly separable but unlabeled data points

2

1

$begingroup$

i tried to evaluate 6 models and after plotting , this what i get :

So i'm wondering , if those results are "Right" ?

Thank's in advance.

random-forest svm logistic-regression model-selection

share|improve this question

edited May 22 at 0:57

Esmailian

4,654422

asked May 21 at 14:01

DimiDimi

332

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Can you give some more information on your project? As Juan said, these AUC numbers are suspiciously good, and this chart alone is not very informative.
  $endgroup$
  – Upper_Case
  May 21 at 15:36
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I have to develop a model to predict whether a person will get hired as DS or not , and here is just some model evaluation , to see what's the best model to use for my task .
  $endgroup$
  – Dimi
  May 21 at 15:43
  

  
  
  
  

add a comment | 

2

1

$begingroup$

i tried to evaluate 6 models and after plotting , this what i get :

So i'm wondering , if those results are "Right" ?

Thank's in advance.

random-forest svm logistic-regression model-selection

share|improve this question

edited May 22 at 0:57

Esmailian

4,654422

asked May 21 at 14:01

DimiDimi

332

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Can you give some more information on your project? As Juan said, these AUC numbers are suspiciously good, and this chart alone is not very informative.
  $endgroup$
  – Upper_Case
  May 21 at 15:36
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I have to develop a model to predict whether a person will get hired as DS or not , and here is just some model evaluation , to see what's the best model to use for my task .
  $endgroup$
  – Dimi
  May 21 at 15:43
  

  
  
  
  

add a comment | 

$begingroup$

i tried to evaluate 6 models and after plotting , this what i get :

So i'm wondering , if those results are "Right" ?

Thank's in advance.

random-forest svm logistic-regression model-selection

share|improve this question

edited May 22 at 0:57

Esmailian

4,654422

asked May 21 at 14:01

DimiDimi

332

$endgroup$

share|improve this question

edited May 22 at 0:57

Esmailian

4,654422

asked May 21 at 14:01

DimiDimi

332

share|improve this question

edited May 22 at 0:57

Esmailian

4,654422

asked May 21 at 14:01

DimiDimi

332

edited May 22 at 0:57

Esmailian

4,654422

edited May 22 at 0:57

Esmailian

4,654422

asked May 21 at 14:01

DimiDimi

332

asked May 21 at 14:01

DimiDimi

332

2 Answers
2

active

oldest

votes

5

$begingroup$

Did you evaluate the results in the training set? Or in the test set?

Those results are outstandingly good! Suspiciously good.

I think you tried your results in the training set only, so your results reflect overfitting on your data, which means your model learned the set, it was not generalized (which means it is unapplicable to any other dataset you may encounter in the future, which is not useful).

For comparing ROC between methodologies you should model them being careful for overfitting and try them on a test dataset (a dataset which you never knew before, which can be obtained partitioning your dataset).

In that way your comparison is not measuring which model learns by memory your data.

share|improve this answer

answered May 21 at 14:58

Juan Esteban de la CalleJuan Esteban de la Calle

1,381324

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Well , I did split my data into Training and testing and this evaluation has been done only on the Training.
  $endgroup$
  – Dimi
  May 21 at 15:37
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Yes! That is what happens! You should assess your model in the dataset your model doesn't know (test dataset), so you can measure the generallity of it.
  $endgroup$
  – Juan Esteban de la Calle
  May 21 at 16:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Alright , thank's a lot
  $endgroup$
  – Dimi
  May 21 at 16:06
  

  
  
  
  

add a comment | 

2

$begingroup$

I'm not sure that AUC is the right value to use to compare these models. Have a look at this question for a bit more detail.

In any case, the AUC of your training data is not a very informative piece of information, and assessing the performance of your model on the training set isn't enough to determine how "right" your model(s) may be, no matter how you go about it. Such a comparison could be done on a test set, at the earliest, and better still, totally out-of-sample data (as final model training usually includes re-training on the full data set).

Finally, model performance (in application) will be defined by choosing a specific cut point for your predictors (and the corresponding true positive/false positive tradeoff) rather than by looking at the overall ability of your model to discern between outcomes across many possible cut points. Overall AUC may still be interesting, but applied model performance is a more precise question (this question has some good information on that).

share|improve this answer

edited May 21 at 16:00

answered May 21 at 15:57

Upper_CaseUpper_Case

28615

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Intresting , Thank you
  $endgroup$
  – Dimi
  May 21 at 15:58
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f52325%2finterpretation-of-roc-auc-score%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

5

$begingroup$

Did you evaluate the results in the training set? Or in the test set?

Those results are outstandingly good! Suspiciously good.

I think you tried your results in the training set only, so your results reflect overfitting on your data, which means your model learned the set, it was not generalized (which means it is unapplicable to any other dataset you may encounter in the future, which is not useful).

For comparing ROC between methodologies you should model them being careful for overfitting and try them on a test dataset (a dataset which you never knew before, which can be obtained partitioning your dataset).

In that way your comparison is not measuring which model learns by memory your data.

share|improve this answer

answered May 21 at 14:58

Juan Esteban de la CalleJuan Esteban de la Calle

1,381324

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Well , I did split my data into Training and testing and this evaluation has been done only on the Training.
  $endgroup$
  – Dimi
  May 21 at 15:37
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Yes! That is what happens! You should assess your model in the dataset your model doesn't know (test dataset), so you can measure the generallity of it.
  $endgroup$
  – Juan Esteban de la Calle
  May 21 at 16:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Alright , thank's a lot
  $endgroup$
  – Dimi
  May 21 at 16:06
  

  
  
  
  

add a comment | 

5

$begingroup$

Did you evaluate the results in the training set? Or in the test set?

Those results are outstandingly good! Suspiciously good.

I think you tried your results in the training set only, so your results reflect overfitting on your data, which means your model learned the set, it was not generalized (which means it is unapplicable to any other dataset you may encounter in the future, which is not useful).

For comparing ROC between methodologies you should model them being careful for overfitting and try them on a test dataset (a dataset which you never knew before, which can be obtained partitioning your dataset).

In that way your comparison is not measuring which model learns by memory your data.

share|improve this answer

answered May 21 at 14:58

Juan Esteban de la CalleJuan Esteban de la Calle

1,381324

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Well , I did split my data into Training and testing and this evaluation has been done only on the Training.
  $endgroup$
  – Dimi
  May 21 at 15:37
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Yes! That is what happens! You should assess your model in the dataset your model doesn't know (test dataset), so you can measure the generallity of it.
  $endgroup$
  – Juan Esteban de la Calle
  May 21 at 16:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Alright , thank's a lot
  $endgroup$
  – Dimi
  May 21 at 16:06
  

  
  
  
  

add a comment | 

$begingroup$

Did you evaluate the results in the training set? Or in the test set?

Those results are outstandingly good! Suspiciously good.

I think you tried your results in the training set only, so your results reflect overfitting on your data, which means your model learned the set, it was not generalized (which means it is unapplicable to any other dataset you may encounter in the future, which is not useful).

For comparing ROC between methodologies you should model them being careful for overfitting and try them on a test dataset (a dataset which you never knew before, which can be obtained partitioning your dataset).

In that way your comparison is not measuring which model learns by memory your data.

share|improve this answer

answered May 21 at 14:58

Juan Esteban de la CalleJuan Esteban de la Calle

1,381324

$endgroup$

share|improve this answer

answered May 21 at 14:58

Juan Esteban de la CalleJuan Esteban de la Calle

1,381324

answered May 21 at 14:58

Juan Esteban de la CalleJuan Esteban de la Calle

1,381324

answered May 21 at 14:58

Juan Esteban de la CalleJuan Esteban de la Calle

1,381324

2

$begingroup$

I'm not sure that AUC is the right value to use to compare these models. Have a look at this question for a bit more detail.

In any case, the AUC of your training data is not a very informative piece of information, and assessing the performance of your model on the training set isn't enough to determine how "right" your model(s) may be, no matter how you go about it. Such a comparison could be done on a test set, at the earliest, and better still, totally out-of-sample data (as final model training usually includes re-training on the full data set).

Finally, model performance (in application) will be defined by choosing a specific cut point for your predictors (and the corresponding true positive/false positive tradeoff) rather than by looking at the overall ability of your model to discern between outcomes across many possible cut points. Overall AUC may still be interesting, but applied model performance is a more precise question (this question has some good information on that).

share|improve this answer

edited May 21 at 16:00

answered May 21 at 15:57

Upper_CaseUpper_Case

28615

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Intresting , Thank you
  $endgroup$
  – Dimi
  May 21 at 15:58
  

  
  
  
  

add a comment | 

2

$begingroup$

I'm not sure that AUC is the right value to use to compare these models. Have a look at this question for a bit more detail.

In any case, the AUC of your training data is not a very informative piece of information, and assessing the performance of your model on the training set isn't enough to determine how "right" your model(s) may be, no matter how you go about it. Such a comparison could be done on a test set, at the earliest, and better still, totally out-of-sample data (as final model training usually includes re-training on the full data set).

Finally, model performance (in application) will be defined by choosing a specific cut point for your predictors (and the corresponding true positive/false positive tradeoff) rather than by looking at the overall ability of your model to discern between outcomes across many possible cut points. Overall AUC may still be interesting, but applied model performance is a more precise question (this question has some good information on that).

share|improve this answer

edited May 21 at 16:00

answered May 21 at 15:57

Upper_CaseUpper_Case

28615

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Intresting , Thank you
  $endgroup$
  – Dimi
  May 21 at 15:58
  

  
  
  
  

add a comment | 

$begingroup$

I'm not sure that AUC is the right value to use to compare these models. Have a look at this question for a bit more detail.

In any case, the AUC of your training data is not a very informative piece of information, and assessing the performance of your model on the training set isn't enough to determine how "right" your model(s) may be, no matter how you go about it. Such a comparison could be done on a test set, at the earliest, and better still, totally out-of-sample data (as final model training usually includes re-training on the full data set).

Finally, model performance (in application) will be defined by choosing a specific cut point for your predictors (and the corresponding true positive/false positive tradeoff) rather than by looking at the overall ability of your model to discern between outcomes across many possible cut points. Overall AUC may still be interesting, but applied model performance is a more precise question (this question has some good information on that).

share|improve this answer

edited May 21 at 16:00

answered May 21 at 15:57

Upper_CaseUpper_Case

28615

$endgroup$

share|improve this answer

edited May 21 at 16:00

answered May 21 at 15:57

Upper_CaseUpper_Case

28615

answered May 21 at 15:57

Upper_CaseUpper_Case

28615

answered May 21 at 15:57

Upper_CaseUpper_Case

28615