How to share a EFS/s3 with code in a auto scaling group? Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Come Celebrate our 10 Year Anniversary!How can I automatically cycle a new image in an AWS Auto Scaling Group?Need a recommendation for shared storage on auto-scaling ec2 w/ scalrAuto-scaling EC2 Servers and Updating CodeShared File Systems between multiple AWS EC2 instancesAuto-heal an EC2 instance with an Auto Scaling Group?How do I push latest code into auto scaling groupRunning crontab on AWS auto scaling groupPHP Application with Web Root on Amazon EFS using up too much EFS bandwidthAmazon EFS as code repository for auto-scaled EC2sAuto-Scaling Group Simple Policy CPU Utilization

How does TikZ render an arc?

First paper to introduce the "principal-agent problem"

Why not use the yoke to control yaw, as well as pitch and roll?

Why is there so little support for joining EFTA in the British parliament?

The Nth Gryphon Number

How can I list files in reverse time order by a command and pass them as arguments to another command?

Sally's older brother

What is a more techy Technical Writer job title that isn't cutesy or confusing?

Did any compiler fully use 80-bit floating point?

Is there a verb for listening stealthily?

Are there any irrational/transcendental numbers for which the distribution of decimal digits is not uniform?

Noise in Eigenvalues plot

malloc in main() or malloc in another function: allocating memory for a struct and its members

One-one communication

Why does BitLocker not use RSA?

Did John Wesley plagiarize Matthew Henry...?

Found this skink in my tomato plant bucket. Is he trapped? Or could he leave if he wanted?

Table formatting with tabularx?

Keep at all times, the minus sign above aligned with minus sign below

Is it OK to use the testing sample to compare algorithms?

NIntegrate on a solution of a matrix ODE

Besides transaction validation, are there any other uses of the Script language in Bitcoin

Why complex landing gears are used instead of simple, reliable and light weight muscle wire or shape memory alloys?

Short story about astronauts fertilizing soil with their own bodies

How to share a EFS/s3 with code in a auto scaling group?

Announcing the arrival of Valued Associate #679: Cesar Manara

Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)

Come Celebrate our 10 Year Anniversary!How can I automatically cycle a new image in an AWS Auto Scaling Group?Need a recommendation for shared storage on auto-scaling ec2 w/ scalrAuto-scaling EC2 Servers and Updating CodeShared File Systems between multiple AWS EC2 instancesAuto-heal an EC2 instance with an Auto Scaling Group?How do I push latest code into auto scaling groupRunning crontab on AWS auto scaling groupPHP Application with Web Root on Amazon EFS using up too much EFS bandwidthAmazon EFS as code repository for auto-scaled EC2sAuto-Scaling Group Simple Policy CPU Utilization

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

the main problem is:

Infrastructure:

Autoscaling (min 1 - max 3)

elasticache (redis)

elasticsearch

I want to share a volume (EFS / S3) with the code of my application;
The size of it is about 1.3 Gb.

With EFS:

The first attempt I tried to mount with the permissions and uid, gid, umask, etc. and It works, but the EFS is really slow even with the performance mode and with the dedicated 10 Mb of data transfer.

Apache tries to read the content on that path (EFS) and the response is slow as hell.

mount -t nfs4 efs-amazonaws.com:/ /var/www/filesystem/custom/

With S3

It works faster than the EFS but the problem is that when apache reads the content on the bucket (mounted equal as the EFS), it fails to connect to the resources inside the bucket, example Mysql functions.

s3fs bucket-name /var/www/filesystem/custom/ -o allow_other,uid=33,gid=33,mp_umask=002

The other alternative that I have is:

1- Mount the S3 or EFS in other location of the server.

2- With Lsyncd replicate the changes on the s3 to the real path of the
server app.

What I need are alternatives of what can I do to share a volume in my autoscaling group . !!

Thanks!

asked Apr 15 at 22:01

Alanmunizrdz

262

New contributor

s3fs-fuse is not known for speed, so it's unexpected (at least to me) that it should be faster than EFS, unless EFS is saturated or there's some other problem. As I mentioned, below, please check your BurstCreditBalance and other CloudWatch metrics for your EFS filesystem and let us know what that looks like.

– Michael - sqlbot
Apr 16 at 17:22

add a comment |

the main problem is:

Infrastructure:

Autoscaling (min 1 - max 3)

elasticache (redis)

elasticsearch

I want to share a volume (EFS / S3) with the code of my application;
The size of it is about 1.3 Gb.

With EFS:

Apache tries to read the content on that path (EFS) and the response is slow as hell.

mount -t nfs4 efs-amazonaws.com:/ /var/www/filesystem/custom/

With S3

s3fs bucket-name /var/www/filesystem/custom/ -o allow_other,uid=33,gid=33,mp_umask=002

The other alternative that I have is:

1- Mount the S3 or EFS in other location of the server.

2- With Lsyncd replicate the changes on the s3 to the real path of the
server app.

What I need are alternatives of what can I do to share a volume in my autoscaling group . !!

Thanks!

asked Apr 15 at 22:01

Alanmunizrdz

262

New contributor

s3fs-fuse is not known for speed, so it's unexpected (at least to me) that it should be faster than EFS, unless EFS is saturated or there's some other problem. As I mentioned, below, please check your BurstCreditBalance and other CloudWatch metrics for your EFS filesystem and let us know what that looks like.

– Michael - sqlbot
Apr 16 at 17:22

add a comment |

the main problem is:

Infrastructure:

Autoscaling (min 1 - max 3)

elasticache (redis)

elasticsearch

I want to share a volume (EFS / S3) with the code of my application;
The size of it is about 1.3 Gb.

With EFS:

Apache tries to read the content on that path (EFS) and the response is slow as hell.

mount -t nfs4 efs-amazonaws.com:/ /var/www/filesystem/custom/

With S3

s3fs bucket-name /var/www/filesystem/custom/ -o allow_other,uid=33,gid=33,mp_umask=002

The other alternative that I have is:

1- Mount the S3 or EFS in other location of the server.

2- With Lsyncd replicate the changes on the s3 to the real path of the
server app.

What I need are alternatives of what can I do to share a volume in my autoscaling group . !!

Thanks!

asked Apr 15 at 22:01

Alanmunizrdz

262

New contributor

the main problem is:

Infrastructure:

Autoscaling (min 1 - max 3)

elasticache (redis)

elasticsearch

I want to share a volume (EFS / S3) with the code of my application;
The size of it is about 1.3 Gb.

With EFS:

Apache tries to read the content on that path (EFS) and the response is slow as hell.

mount -t nfs4 efs-amazonaws.com:/ /var/www/filesystem/custom/

With S3

s3fs bucket-name /var/www/filesystem/custom/ -o allow_other,uid=33,gid=33,mp_umask=002

The other alternative that I have is:

1- Mount the S3 or EFS in other location of the server.

2- With Lsyncd replicate the changes on the s3 to the real path of the
server app.

What I need are alternatives of what can I do to share a volume in my autoscaling group . !!

Thanks!

amazon-web-services amazon-ec2 amazon-s3 autoscaling amazon-efs

asked Apr 15 at 22:01

Alanmunizrdz

262

New contributor

asked Apr 15 at 22:01

Alanmunizrdz

262

New contributor

asked Apr 15 at 22:01

Alanmunizrdz

262

New contributor

asked Apr 15 at 22:01

Alanmunizrdz

262

asked Apr 15 at 22:01

Alanmunizrdz

262

New contributor

Alanmunizrdz is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

s3fs-fuse is not known for speed, so it's unexpected (at least to me) that it should be faster than EFS, unless EFS is saturated or there's some other problem. As I mentioned, below, please check your BurstCreditBalance and other CloudWatch metrics for your EFS filesystem and let us know what that looks like.

– Michael - sqlbot
Apr 16 at 17:22

add a comment |

s3fs-fuse is not known for speed, so it's unexpected (at least to me) that it should be faster than EFS, unless EFS is saturated or there's some other problem. As I mentioned, below, please check your BurstCreditBalance and other CloudWatch metrics for your EFS filesystem and let us know what that looks like.

– Michael - sqlbot
Apr 16 at 17:22

s3fs-fuse is not known for speed, so it's unexpected (at least to me) that it should be faster than EFS, unless EFS is saturated or there's some other problem. As I mentioned, below, please check your BurstCreditBalance and other CloudWatch metrics for your EFS filesystem and let us know what that looks like.

– Michael - sqlbot
Apr 16 at 17:22

add a comment |

1 Answer
1

active

oldest

votes

EFS performance depends on how much data you have on the volume. The more you store the higher is the performance. That’s probably why with just 1.3GB it’s slow.

You can however pay for EFS provisioned IOPS that will increase the performance for an extra cost.

Alternatively you can simply store a couple of big files (e.g. 10x 50GB) to increase the volume-size related performance.

Test both approaches and see how you go.

answered Apr 15 at 22:28

MLu

9,95722445

Hey, thanks 4 your answer, I have enabled the EFS provisioned IOPS with 10 Mb dedicated, the problem would be if I have to create "dummy files" just to make it work faster, that is not the kind of solution that I need :/

– Alanmunizrdz
Apr 15 at 22:34

@Alanmunizrdz as you are probably aware, the strategy of creating dummy files was just a workaround before the provisioned throughput was introduced for EFS. The suggestion here seems only to be try it, just to verify that it doesn't help -- it shouldn't. The impact of provisioned throughput of 10 MB/s should be equivalent to claiming 200 GB of storage. Please check your BurstCreditBalance in CloudWatch.

– Michael - sqlbot
Apr 16 at 17:09

@Alanmunizrdz also note that "performance mode" should probably have been called "distributed performance tradeoff mode" and would likely not be recommended for your use case. Whether it has a negative throughput impact at smaller scale is difficult to say, but it is not supposed to be used unless the PercentIOLimit metric on a General Purpose EFS system indicates saturation of whatever exactly that metric is measuring. (The metric, and the mode, appear to be related to internal index IO, rather than file IO.)

– Michael - sqlbot
Apr 16 at 17:15

@MLu a slightly pedantic note, EFS uses the term provisioned throughput rather than provisioned IOPS. The burst capacity appears to be IO-size agnostic, thus the number of IOs is less significant than the actual number of bytes being written and read.

– Michael - sqlbot
Apr 16 at 17:18

add a comment |

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "2"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

Alanmunizrdz is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f963192%2fhow-to-share-a-efs-s3-with-code-in-a-auto-scaling-group%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

EFS performance depends on how much data you have on the volume. The more you store the higher is the performance. That’s probably why with just 1.3GB it’s slow.

You can however pay for EFS provisioned IOPS that will increase the performance for an extra cost.

Alternatively you can simply store a couple of big files (e.g. 10x 50GB) to increase the volume-size related performance.

Test both approaches and see how you go.

answered Apr 15 at 22:28

MLu

9,95722445

Hey, thanks 4 your answer, I have enabled the EFS provisioned IOPS with 10 Mb dedicated, the problem would be if I have to create "dummy files" just to make it work faster, that is not the kind of solution that I need :/

– Alanmunizrdz
Apr 15 at 22:34

@Alanmunizrdz as you are probably aware, the strategy of creating dummy files was just a workaround before the provisioned throughput was introduced for EFS. The suggestion here seems only to be try it, just to verify that it doesn't help -- it shouldn't. The impact of provisioned throughput of 10 MB/s should be equivalent to claiming 200 GB of storage. Please check your BurstCreditBalance in CloudWatch.

– Michael - sqlbot
Apr 16 at 17:09

@Alanmunizrdz also note that "performance mode" should probably have been called "distributed performance tradeoff mode" and would likely not be recommended for your use case. Whether it has a negative throughput impact at smaller scale is difficult to say, but it is not supposed to be used unless the PercentIOLimit metric on a General Purpose EFS system indicates saturation of whatever exactly that metric is measuring. (The metric, and the mode, appear to be related to internal index IO, rather than file IO.)

– Michael - sqlbot
Apr 16 at 17:15

@MLu a slightly pedantic note, EFS uses the term provisioned throughput rather than provisioned IOPS. The burst capacity appears to be IO-size agnostic, thus the number of IOs is less significant than the actual number of bytes being written and read.

– Michael - sqlbot
Apr 16 at 17:18

add a comment |

EFS performance depends on how much data you have on the volume. The more you store the higher is the performance. That’s probably why with just 1.3GB it’s slow.

You can however pay for EFS provisioned IOPS that will increase the performance for an extra cost.

Alternatively you can simply store a couple of big files (e.g. 10x 50GB) to increase the volume-size related performance.

Test both approaches and see how you go.

answered Apr 15 at 22:28

MLu

9,95722445

Hey, thanks 4 your answer, I have enabled the EFS provisioned IOPS with 10 Mb dedicated, the problem would be if I have to create "dummy files" just to make it work faster, that is not the kind of solution that I need :/

– Alanmunizrdz
Apr 15 at 22:34

@Alanmunizrdz as you are probably aware, the strategy of creating dummy files was just a workaround before the provisioned throughput was introduced for EFS. The suggestion here seems only to be try it, just to verify that it doesn't help -- it shouldn't. The impact of provisioned throughput of 10 MB/s should be equivalent to claiming 200 GB of storage. Please check your BurstCreditBalance in CloudWatch.

– Michael - sqlbot
Apr 16 at 17:09

@Alanmunizrdz also note that "performance mode" should probably have been called "distributed performance tradeoff mode" and would likely not be recommended for your use case. Whether it has a negative throughput impact at smaller scale is difficult to say, but it is not supposed to be used unless the PercentIOLimit metric on a General Purpose EFS system indicates saturation of whatever exactly that metric is measuring. (The metric, and the mode, appear to be related to internal index IO, rather than file IO.)

– Michael - sqlbot
Apr 16 at 17:15

@MLu a slightly pedantic note, EFS uses the term provisioned throughput rather than provisioned IOPS. The burst capacity appears to be IO-size agnostic, thus the number of IOs is less significant than the actual number of bytes being written and read.

– Michael - sqlbot
Apr 16 at 17:18

add a comment |

EFS performance depends on how much data you have on the volume. The more you store the higher is the performance. That’s probably why with just 1.3GB it’s slow.

You can however pay for EFS provisioned IOPS that will increase the performance for an extra cost.

Alternatively you can simply store a couple of big files (e.g. 10x 50GB) to increase the volume-size related performance.

Test both approaches and see how you go.

answered Apr 15 at 22:28

MLu

9,95722445

EFS performance depends on how much data you have on the volume. The more you store the higher is the performance. That’s probably why with just 1.3GB it’s slow.

You can however pay for EFS provisioned IOPS that will increase the performance for an extra cost.

Alternatively you can simply store a couple of big files (e.g. 10x 50GB) to increase the volume-size related performance.

Test both approaches and see how you go.

answered Apr 15 at 22:28

MLu

9,95722445

answered Apr 15 at 22:28

MLu

9,95722445

answered Apr 15 at 22:28

MLu

9,95722445

answered Apr 15 at 22:28

MLu

9,95722445

Hey, thanks 4 your answer, I have enabled the EFS provisioned IOPS with 10 Mb dedicated, the problem would be if I have to create "dummy files" just to make it work faster, that is not the kind of solution that I need :/

– Alanmunizrdz
Apr 15 at 22:34

@Alanmunizrdz as you are probably aware, the strategy of creating dummy files was just a workaround before the provisioned throughput was introduced for EFS. The suggestion here seems only to be try it, just to verify that it doesn't help -- it shouldn't. The impact of provisioned throughput of 10 MB/s should be equivalent to claiming 200 GB of storage. Please check your BurstCreditBalance in CloudWatch.

– Michael - sqlbot
Apr 16 at 17:09

@Alanmunizrdz also note that "performance mode" should probably have been called "distributed performance tradeoff mode" and would likely not be recommended for your use case. Whether it has a negative throughput impact at smaller scale is difficult to say, but it is not supposed to be used unless the PercentIOLimit metric on a General Purpose EFS system indicates saturation of whatever exactly that metric is measuring. (The metric, and the mode, appear to be related to internal index IO, rather than file IO.)

– Michael - sqlbot
Apr 16 at 17:15

@MLu a slightly pedantic note, EFS uses the term provisioned throughput rather than provisioned IOPS. The burst capacity appears to be IO-size agnostic, thus the number of IOs is less significant than the actual number of bytes being written and read.

– Michael - sqlbot
Apr 16 at 17:18

add a comment |

Hey, thanks 4 your answer, I have enabled the EFS provisioned IOPS with 10 Mb dedicated, the problem would be if I have to create "dummy files" just to make it work faster, that is not the kind of solution that I need :/

– Alanmunizrdz
Apr 15 at 22:34

@Alanmunizrdz as you are probably aware, the strategy of creating dummy files was just a workaround before the provisioned throughput was introduced for EFS. The suggestion here seems only to be try it, just to verify that it doesn't help -- it shouldn't. The impact of provisioned throughput of 10 MB/s should be equivalent to claiming 200 GB of storage. Please check your BurstCreditBalance in CloudWatch.

– Michael - sqlbot
Apr 16 at 17:09

@Alanmunizrdz also note that "performance mode" should probably have been called "distributed performance tradeoff mode" and would likely not be recommended for your use case. Whether it has a negative throughput impact at smaller scale is difficult to say, but it is not supposed to be used unless the PercentIOLimit metric on a General Purpose EFS system indicates saturation of whatever exactly that metric is measuring. (The metric, and the mode, appear to be related to internal index IO, rather than file IO.)

– Michael - sqlbot
Apr 16 at 17:15

@MLu a slightly pedantic note, EFS uses the term provisioned throughput rather than provisioned IOPS. The burst capacity appears to be IO-size agnostic, thus the number of IOs is less significant than the actual number of bytes being written and read.

– Michael - sqlbot
Apr 16 at 17:18

Hey, thanks 4 your answer, I have enabled the EFS provisioned IOPS with 10 Mb dedicated, the problem would be if I have to create "dummy files" just to make it work faster, that is not the kind of solution that I need :/

– Alanmunizrdz
Apr 15 at 22:34

@Alanmunizrdz as you are probably aware, the strategy of creating dummy files was just a workaround before the provisioned throughput was introduced for EFS. The suggestion here seems only to be try it, just to verify that it doesn't help -- it shouldn't. The impact of provisioned throughput of 10 MB/s should be equivalent to claiming 200 GB of storage. Please check your BurstCreditBalance in CloudWatch.

– Michael - sqlbot
Apr 16 at 17:09

@Alanmunizrdz also note that "performance mode" should probably have been called "distributed performance tradeoff mode" and would likely not be recommended for your use case. Whether it has a negative throughput impact at smaller scale is difficult to say, but it is not supposed to be used unless the PercentIOLimit metric on a General Purpose EFS system indicates saturation of whatever exactly that metric is measuring. (The metric, and the mode, appear to be related to internal index IO, rather than file IO.)

– Michael - sqlbot
Apr 16 at 17:15

@MLu a slightly pedantic note, EFS uses the term provisioned throughput rather than provisioned IOPS. The burst capacity appears to be IO-size agnostic, thus the number of IOs is less significant than the actual number of bytes being written and read.

– Michael - sqlbot
Apr 16 at 17:18

add a comment |

Alanmunizrdz is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Alanmunizrdz is a new contributor. Be nice, and check out our Code of Conduct.

Thanks for contributing an answer to Server Fault!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Otdfbt

1 Answer
1

Your Answer

Post as a guest

1 Answer
1

1 Answer
1

Post as a guest

Popular posts from this blog

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Post as a guest

1 Answer 1

1 Answer 1

Sign up or log in

Post as a guest

Post as a guest

Sign up or log in

Post as a guest

Sign up or log in

Post as a guest

Sign up or log in

Post as a guest

Popular posts from this blog

1 Answer
1

1 Answer
1

1 Answer
1