Otdfbt

different output for groups and groups USERNAME after adding a username to a group

"is" operation returns false with ndarray.data attribute, even though two array objects have same id

Identify 80s or 90s comics with ripped creatures (not dwarves)

How do I design a circuit to convert a 100 mV and 50 Hz sine wave to a square wave?

Is this wall load bearing? Blueprints and photos attached

What's the point in a preamp?

Did the new image of black hole confirm the general theory of relativity?

For what reasons would an animal species NOT cross a *horizontal* land bridge?

Why don't hard Brexiteers insist on a hard border to prevent illegal immigration after Brexit?

Is 'stolen' appropriate word?

How to determine omitted units in a publication

How do spell lists change if the party levels up without taking a long rest?

60's-70's movie: home appliances revolting against the owners

Did the UK government pay "millions and millions of dollars" to try to snag Julian Assange?

Are there continuous functions who are the same in an interval but differ in at least one other point?

Why are PDP-7-style microprogrammed instructions out of vogue?

What aspect of planet Earth must be changed to prevent the industrial revolution?

Windows 10: How to Lock (not sleep) laptop on lid close?

Match Roman Numerals

Mortgage adviser recommends a longer term than necessary combined with overpayments

Huge performance difference of the command find with and without using %M option to show permissions

Can a flute soloist sit?

What was the last x86 CPU that did not have the x87 floating-point unit built in?

What to do when moving next to a bird sanctuary with a loosely-domesticated cat?

How to handle columns with categorical data and many unique values

4

1

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

$endgroup$

add a comment | 

4

1

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

$endgroup$

add a comment | 

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

$endgroup$

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

394

asked Apr 8 at 11:04

dungeondungeon

394

asked Apr 8 at 11:04

dungeondungeon

394

1 Answer
1

active

oldest

votes

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5741314

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48875%2fhow-to-handle-columns-with-categorical-data-and-many-unique-values%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5741314

$endgroup$

add a comment | 

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5741314

$endgroup$

add a comment | 

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5741314

$endgroup$

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5741314

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5741314

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,5741314