Otdfbt

What does this quote in Small Gods refer to?

Exception propagation: When to catch exceptions?

Why do Thanos's punches not kill Captain America or at least cause some mortal injuries?

Ex-manager wants to stay in touch, I don't want to

How to select certain lines (n, n+4, n+8, n+12...) from the file?

Is a diamond sword feasible?

Is "now" UTC time in Solidity?

Control variables and other independent variables

Was the Highlands Ranch shooting the 115th mass shooting in the US in 2019

Cropping a message using array splits

What food production methods would allow a metropolis like New York to become self sufficient

How did Thanos not realise this had happened at the end of Endgame?

Are there variations of the regular runtimes of the Big-O-Notation?

Does the 500 feet falling cap apply per fall, or per turn?

How do I tell my supervisor that he is choosing poor replacements for me while I am on maternity leave?

Early arrival in Australia, early hotel check in not available

Is the homebrew weapon attack cantrip 'Arcane Strike' balanced?

Was there ever any real use for a 6800-based Apple I?

Guns in space with bullets that return?

LocalDate.plus Incorrect Answer

How do I get past a 3-year ban from overstay with VWP?

Pre-1993 comic in which Wolverine's claws were turned to rubber?

Why does the Earth follow an elliptical trajectory rather than a parabolic one?

Can I do brevets (long distance rides) on my hybrid bike? If yes, how to start?

How to set up the parameter of a spark job according to the available nodes?

How to get the best LINPACK result and conquer the Top500?Management of available file descriptors within a Hadoop clusterHow can set up a multi nodes job for a spark application through a batch file?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

0

How I can set the --num-executors, --executor-cores and --executor-memory through the slurm job in the following scenario:

Suppose I have 2 nodes with 32 cores and 128g each. The best configuration (I think) would be:

1. Let’s assign 4 cores per executors => --executor-cores = 4.


2. Leave 1 core per node for Hadoop/Yarn daemons => Num cores available per node = 32-1 = 31. So, Total available of cores in cluster = 31 x 2 = 62


3. Number of available executors = (total cores/num-cores-per-executor) = 62/4 = 15


4. Leaving 1 executor for ApplicationManager => --num-executors = 14
   


5. Number of executors per node = 15/2 = 7


6. Memory per executor = 127GB/7 = 18GB


7. Counting off heap overhead = 7% of 18GB = 1.26GB~2g. So, actual --executor-memory = 18 - 2 = 16GB
   

So using this (optimal) configuration how can I set up these parameters in an Slurm job?

hpc multi-threading mpio slurm

share|improve this question

edited May 1 at 23:27

Phillip -Zyan K Lee- Stockmann

804310

asked May 1 at 12:50

moudimoudi

12

add a comment | 

0

How I can set the --num-executors, --executor-cores and --executor-memory through the slurm job in the following scenario:

Suppose I have 2 nodes with 32 cores and 128g each. The best configuration (I think) would be:

1. Let’s assign 4 cores per executors => --executor-cores = 4.


2. Leave 1 core per node for Hadoop/Yarn daemons => Num cores available per node = 32-1 = 31. So, Total available of cores in cluster = 31 x 2 = 62


3. Number of available executors = (total cores/num-cores-per-executor) = 62/4 = 15


4. Leaving 1 executor for ApplicationManager => --num-executors = 14
   


5. Number of executors per node = 15/2 = 7


6. Memory per executor = 127GB/7 = 18GB


7. Counting off heap overhead = 7% of 18GB = 1.26GB~2g. So, actual --executor-memory = 18 - 2 = 16GB
   

So using this (optimal) configuration how can I set up these parameters in an Slurm job?

hpc multi-threading mpio slurm

share|improve this question

edited May 1 at 23:27

Phillip -Zyan K Lee- Stockmann

804310

asked May 1 at 12:50

moudimoudi

12

add a comment | 

How I can set the --num-executors, --executor-cores and --executor-memory through the slurm job in the following scenario:

Suppose I have 2 nodes with 32 cores and 128g each. The best configuration (I think) would be:

1. Let’s assign 4 cores per executors => --executor-cores = 4.


2. Leave 1 core per node for Hadoop/Yarn daemons => Num cores available per node = 32-1 = 31. So, Total available of cores in cluster = 31 x 2 = 62


3. Number of available executors = (total cores/num-cores-per-executor) = 62/4 = 15


4. Leaving 1 executor for ApplicationManager => --num-executors = 14
   


5. Number of executors per node = 15/2 = 7


6. Memory per executor = 127GB/7 = 18GB


7. Counting off heap overhead = 7% of 18GB = 1.26GB~2g. So, actual --executor-memory = 18 - 2 = 16GB
   

So using this (optimal) configuration how can I set up these parameters in an Slurm job?

hpc multi-threading mpio slurm

share|improve this question

edited May 1 at 23:27

Phillip -Zyan K Lee- Stockmann

804310

asked May 1 at 12:50

moudimoudi

12

share|improve this question

edited May 1 at 23:27

Phillip -Zyan K Lee- Stockmann

804310

asked May 1 at 12:50

moudimoudi

12

share|improve this question

edited May 1 at 23:27

Phillip -Zyan K Lee- Stockmann

804310

asked May 1 at 12:50

moudimoudi

12

edited May 1 at 23:27

Phillip -Zyan K Lee- Stockmann

804310

edited May 1 at 23:27

Phillip -Zyan K Lee- Stockmann

804310

asked May 1 at 12:50

moudimoudi

12

asked May 1 at 12:50

moudimoudi

12