Computing elements of a 1000 x 60 matrix exhausts RAMWhat is a Mathematica packed array?Paging RAM in case of memory shortage issueHeavy duty operations, RAM, and ReadyboostEfficient calculation of diagonal matrix elementsHow to know each variable used how much RAMReplacing elements of a matrixHow to clear RAM memory in a running code?How does this code behave with more than 32GiB RAM?Replace diagonal elements in sparse matrixSolution list from “Solve” too large for my RAM spaceLimit the amount of RAM Mathematica may access?

Their answer is discrete, mine is continuous. They baited me into the wrong answer. I have a P Exam question

C SIGINT signal in Linux

How do I write "Show, Don't Tell" as an Asperger?

Can't login after removing Flatpak

How is TD(0) method helpful? What good does it do?

Are there cubesats in GEO?

Building a road to escape Earth's gravity by making a pyramid on Antartica

In this example, which path would a monster affected by the Dissonant Whispers spell take?

Can you `= delete` a templated function on a second declaration?

Did Darth Vader wear the same suit for 20+ years?

Why don’t airliners have temporary liveries?

How to make thick Asian sauces?

Does an ice chest packed full of frozen food need ice? 18 day Grand Canyon trip

Incremental Ranges!

Secure offsite backup, even in the case of hacker root access

How can drunken, homicidal elves successfully conduct a wild hunt?

What are the words for people who cause trouble believing they know better?

How could a government be implemented in a virtual reality?

PhD student with mental health issues and bad performance

What can plausibly explain many of my very long and low-tech bridges?

Sharing one invocation list between multiple events on the same object in C#

PC video game involving floating islands doing aerial combat

How is it possible that Gollum speaks Westron?

Why is c4 bad when playing the London against a King's Indian?



Computing elements of a 1000 x 60 matrix exhausts RAM


What is a Mathematica packed array?Paging RAM in case of memory shortage issueHeavy duty operations, RAM, and ReadyboostEfficient calculation of diagonal matrix elementsHow to know each variable used how much RAMReplacing elements of a matrixHow to clear RAM memory in a running code?How does this code behave with more than 32GiB RAM?Replace diagonal elements in sparse matrixSolution list from “Solve” too large for my RAM spaceLimit the amount of RAM Mathematica may access?













5












$begingroup$


I am trying to compute a 1000 x 60 matrix or list of lists (and ideally this should go up to 1000 x 500 or 1000 x 1000).



Each element is the result of a FindRoot operation, so I make my list by doing



Table[Flatten[h /. FindRoot[h == F[h, b, g], b, 1, 1000, g, 1, 60)


but 16GB of RAM are filled up. I think I should be able to hold list of lists much bigger than that, so probably using Table with FindRoot is causing Mathematica to store a lot of undeeded stuff in memory.



Here is the code:



ι[m_, n_] := Binomial[n, n*(1 - m)/2]*2^(-n);
f[m_, h_, b_, g_, n_] := (h*m + g/2*m^2) +
1/(n*b)*Log[ι[m, n]];
μ[m_, h_, b_, g_, n_] :=
Exp[b*n*f[m, h, b, g, n] + b*n*(-h + g/2)]/
Sum[Exp[b*n*f[x, h, b, g, n] + b*n*(-h + g/2)], x, -1 + 2/n,
1 - 2/n, 2/n];
moment[h_, x_, b_, g_, n_] := Sum[m^x*μ[m, h, b, g, n], m, -1 + 2/n, 1 - 2/n, 2/n];
var[h_, b_, g_, n_] := moment[h, 2, b, g, n] - moment[h, 1, b, g, n]^2;
cov[h_, b_, g_, n_] := moment[h, 3, b, g, n] - moment[h, 1, b, g,n]*moment[h, 2, b, g, n];
F[h_,b_,g_,n_]:= -d*b*(cov[h, b, gg, n] +
2 var[h, b, gg, n]);
n = 100;
d = 0.9;

glist = Table[g, g, 0.4, 1, 0.01];
blist = Table[b, b, 1.1, 10.1, 0.01];

heatdata = Table[
Flatten[h /.
FindRoot[
h == F[h,b,g,n], h, -0.01]][[1]]
, b, blist, g, glist];









share|improve this question











$endgroup$











  • $begingroup$
    Please show a complete minimal example that reproduces the problem.
    $endgroup$
    – Szabolcs
    May 19 at 8:31






  • 1




    $begingroup$
    If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
    $endgroup$
    – Szabolcs
    May 19 at 8:34






  • 1




    $begingroup$
    One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
    $endgroup$
    – Szabolcs
    May 19 at 8:35










  • $begingroup$
    Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
    $endgroup$
    – Three Diag
    May 19 at 8:40















5












$begingroup$


I am trying to compute a 1000 x 60 matrix or list of lists (and ideally this should go up to 1000 x 500 or 1000 x 1000).



Each element is the result of a FindRoot operation, so I make my list by doing



Table[Flatten[h /. FindRoot[h == F[h, b, g], b, 1, 1000, g, 1, 60)


but 16GB of RAM are filled up. I think I should be able to hold list of lists much bigger than that, so probably using Table with FindRoot is causing Mathematica to store a lot of undeeded stuff in memory.



Here is the code:



ι[m_, n_] := Binomial[n, n*(1 - m)/2]*2^(-n);
f[m_, h_, b_, g_, n_] := (h*m + g/2*m^2) +
1/(n*b)*Log[ι[m, n]];
μ[m_, h_, b_, g_, n_] :=
Exp[b*n*f[m, h, b, g, n] + b*n*(-h + g/2)]/
Sum[Exp[b*n*f[x, h, b, g, n] + b*n*(-h + g/2)], x, -1 + 2/n,
1 - 2/n, 2/n];
moment[h_, x_, b_, g_, n_] := Sum[m^x*μ[m, h, b, g, n], m, -1 + 2/n, 1 - 2/n, 2/n];
var[h_, b_, g_, n_] := moment[h, 2, b, g, n] - moment[h, 1, b, g, n]^2;
cov[h_, b_, g_, n_] := moment[h, 3, b, g, n] - moment[h, 1, b, g,n]*moment[h, 2, b, g, n];
F[h_,b_,g_,n_]:= -d*b*(cov[h, b, gg, n] +
2 var[h, b, gg, n]);
n = 100;
d = 0.9;

glist = Table[g, g, 0.4, 1, 0.01];
blist = Table[b, b, 1.1, 10.1, 0.01];

heatdata = Table[
Flatten[h /.
FindRoot[
h == F[h,b,g,n], h, -0.01]][[1]]
, b, blist, g, glist];









share|improve this question











$endgroup$











  • $begingroup$
    Please show a complete minimal example that reproduces the problem.
    $endgroup$
    – Szabolcs
    May 19 at 8:31






  • 1




    $begingroup$
    If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
    $endgroup$
    – Szabolcs
    May 19 at 8:34






  • 1




    $begingroup$
    One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
    $endgroup$
    – Szabolcs
    May 19 at 8:35










  • $begingroup$
    Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
    $endgroup$
    – Three Diag
    May 19 at 8:40













5












5








5


1



$begingroup$


I am trying to compute a 1000 x 60 matrix or list of lists (and ideally this should go up to 1000 x 500 or 1000 x 1000).



Each element is the result of a FindRoot operation, so I make my list by doing



Table[Flatten[h /. FindRoot[h == F[h, b, g], b, 1, 1000, g, 1, 60)


but 16GB of RAM are filled up. I think I should be able to hold list of lists much bigger than that, so probably using Table with FindRoot is causing Mathematica to store a lot of undeeded stuff in memory.



Here is the code:



ι[m_, n_] := Binomial[n, n*(1 - m)/2]*2^(-n);
f[m_, h_, b_, g_, n_] := (h*m + g/2*m^2) +
1/(n*b)*Log[ι[m, n]];
μ[m_, h_, b_, g_, n_] :=
Exp[b*n*f[m, h, b, g, n] + b*n*(-h + g/2)]/
Sum[Exp[b*n*f[x, h, b, g, n] + b*n*(-h + g/2)], x, -1 + 2/n,
1 - 2/n, 2/n];
moment[h_, x_, b_, g_, n_] := Sum[m^x*μ[m, h, b, g, n], m, -1 + 2/n, 1 - 2/n, 2/n];
var[h_, b_, g_, n_] := moment[h, 2, b, g, n] - moment[h, 1, b, g, n]^2;
cov[h_, b_, g_, n_] := moment[h, 3, b, g, n] - moment[h, 1, b, g,n]*moment[h, 2, b, g, n];
F[h_,b_,g_,n_]:= -d*b*(cov[h, b, gg, n] +
2 var[h, b, gg, n]);
n = 100;
d = 0.9;

glist = Table[g, g, 0.4, 1, 0.01];
blist = Table[b, b, 1.1, 10.1, 0.01];

heatdata = Table[
Flatten[h /.
FindRoot[
h == F[h,b,g,n], h, -0.01]][[1]]
, b, blist, g, glist];









share|improve this question











$endgroup$




I am trying to compute a 1000 x 60 matrix or list of lists (and ideally this should go up to 1000 x 500 or 1000 x 1000).



Each element is the result of a FindRoot operation, so I make my list by doing



Table[Flatten[h /. FindRoot[h == F[h, b, g], b, 1, 1000, g, 1, 60)


but 16GB of RAM are filled up. I think I should be able to hold list of lists much bigger than that, so probably using Table with FindRoot is causing Mathematica to store a lot of undeeded stuff in memory.



Here is the code:



ι[m_, n_] := Binomial[n, n*(1 - m)/2]*2^(-n);
f[m_, h_, b_, g_, n_] := (h*m + g/2*m^2) +
1/(n*b)*Log[ι[m, n]];
μ[m_, h_, b_, g_, n_] :=
Exp[b*n*f[m, h, b, g, n] + b*n*(-h + g/2)]/
Sum[Exp[b*n*f[x, h, b, g, n] + b*n*(-h + g/2)], x, -1 + 2/n,
1 - 2/n, 2/n];
moment[h_, x_, b_, g_, n_] := Sum[m^x*μ[m, h, b, g, n], m, -1 + 2/n, 1 - 2/n, 2/n];
var[h_, b_, g_, n_] := moment[h, 2, b, g, n] - moment[h, 1, b, g, n]^2;
cov[h_, b_, g_, n_] := moment[h, 3, b, g, n] - moment[h, 1, b, g,n]*moment[h, 2, b, g, n];
F[h_,b_,g_,n_]:= -d*b*(cov[h, b, gg, n] +
2 var[h, b, gg, n]);
n = 100;
d = 0.9;

glist = Table[g, g, 0.4, 1, 0.01];
blist = Table[b, b, 1.1, 10.1, 0.01];

heatdata = Table[
Flatten[h /.
FindRoot[
h == F[h,b,g,n], h, -0.01]][[1]]
, b, blist, g, glist];






performance-tuning memory






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited May 20 at 0:11









m_goldberg

90.4k873203




90.4k873203










asked May 19 at 8:24









Three DiagThree Diag

346111




346111











  • $begingroup$
    Please show a complete minimal example that reproduces the problem.
    $endgroup$
    – Szabolcs
    May 19 at 8:31






  • 1




    $begingroup$
    If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
    $endgroup$
    – Szabolcs
    May 19 at 8:34






  • 1




    $begingroup$
    One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
    $endgroup$
    – Szabolcs
    May 19 at 8:35










  • $begingroup$
    Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
    $endgroup$
    – Three Diag
    May 19 at 8:40
















  • $begingroup$
    Please show a complete minimal example that reproduces the problem.
    $endgroup$
    – Szabolcs
    May 19 at 8:31






  • 1




    $begingroup$
    If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
    $endgroup$
    – Szabolcs
    May 19 at 8:34






  • 1




    $begingroup$
    One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
    $endgroup$
    – Szabolcs
    May 19 at 8:35










  • $begingroup$
    Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
    $endgroup$
    – Three Diag
    May 19 at 8:40















$begingroup$
Please show a complete minimal example that reproduces the problem.
$endgroup$
– Szabolcs
May 19 at 8:31




$begingroup$
Please show a complete minimal example that reproduces the problem.
$endgroup$
– Szabolcs
May 19 at 8:31




1




1




$begingroup$
If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
$endgroup$
– Szabolcs
May 19 at 8:34




$begingroup$
If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
$endgroup$
– Szabolcs
May 19 at 8:34




1




1




$begingroup$
One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
$endgroup$
– Szabolcs
May 19 at 8:35




$begingroup$
One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
$endgroup$
– Szabolcs
May 19 at 8:35












$begingroup$
Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
$endgroup$
– Three Diag
May 19 at 8:40




$begingroup$
Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
$endgroup$
– Three Diag
May 19 at 8:40










1 Answer
1






active

oldest

votes


















20












$begingroup$

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.






share|improve this answer











$endgroup$












  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    May 19 at 23:07










  • $begingroup$
    You're welcome!
    $endgroup$
    – Henrik Schumacher
    May 20 at 6:40











Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198651%2fcomputing-elements-of-a-1000-x-60-matrix-exhausts-ram%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









20












$begingroup$

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.






share|improve this answer











$endgroup$












  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    May 19 at 23:07










  • $begingroup$
    You're welcome!
    $endgroup$
    – Henrik Schumacher
    May 20 at 6:40















20












$begingroup$

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.






share|improve this answer











$endgroup$












  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    May 19 at 23:07










  • $begingroup$
    You're welcome!
    $endgroup$
    – Henrik Schumacher
    May 20 at 6:40













20












20








20





$begingroup$

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.






share|improve this answer











$endgroup$



Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.







share|improve this answer














share|improve this answer



share|improve this answer








edited May 19 at 17:44

























answered May 19 at 10:03









Henrik SchumacherHenrik Schumacher

63.5k589177




63.5k589177











  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    May 19 at 23:07










  • $begingroup$
    You're welcome!
    $endgroup$
    – Henrik Schumacher
    May 20 at 6:40
















  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    May 19 at 23:07










  • $begingroup$
    You're welcome!
    $endgroup$
    – Henrik Schumacher
    May 20 at 6:40















$begingroup$
Thanks for this very helpful and informative answer!
$endgroup$
– Three Diag
May 19 at 23:07




$begingroup$
Thanks for this very helpful and informative answer!
$endgroup$
– Three Diag
May 19 at 23:07












$begingroup$
You're welcome!
$endgroup$
– Henrik Schumacher
May 20 at 6:40




$begingroup$
You're welcome!
$endgroup$
– Henrik Schumacher
May 20 at 6:40

















draft saved

draft discarded
















































Thanks for contributing an answer to Mathematica Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

Use MathJax to format equations. MathJax reference.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198651%2fcomputing-elements-of-a-1000-x-60-matrix-exhausts-ram%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Club Baloncesto Breogán Índice Historia | Pavillón | Nome | O Breogán na cultura popular | Xogadores | Adestradores | Presidentes | Palmarés | Historial | Líderes | Notas | Véxase tamén | Menú de navegacióncbbreogan.galCadroGuía oficial da ACB 2009-10, páxina 201Guía oficial ACB 1992, páxina 183. Editorial DB.É de 6.500 espectadores sentados axeitándose á última normativa"Estudiantes Junior, entre as mellores canteiras"o orixinalHemeroteca El Mundo Deportivo, 16 setembro de 1970, páxina 12Historia do BreogánAlfredo Pérez, o último canoneiroHistoria C.B. BreogánHemeroteca de El Mundo DeportivoJimmy Wright, norteamericano do Breogán deixará Lugo por ameazas de morteResultados de Breogán en 1986-87Resultados de Breogán en 1990-91Ficha de Velimir Perasović en acb.comResultados de Breogán en 1994-95Breogán arrasa al Barça. "El Mundo Deportivo", 27 de setembro de 1999, páxina 58CB Breogán - FC BarcelonaA FEB invita a participar nunha nova Liga EuropeaCharlie Bell na prensa estatalMáximos anotadores 2005Tempada 2005-06 : Tódolos Xogadores da Xornada""Non quero pensar nunha man negra, mais pregúntome que está a pasar""o orixinalRaúl López, orgulloso dos xogadores, presume da boa saúde económica do BreogánJulio González confirma que cesa como presidente del BreogánHomenaxe a Lisardo GómezA tempada do rexurdimento celesteEntrevista a Lisardo GómezEl COB dinamita el Pazo para forzar el quinto (69-73)Cafés Candelas, patrocinador del CB Breogán"Suso Lázare, novo presidente do Breogán"o orixinalCafés Candelas Breogán firma el mayor triunfo de la historiaEl Breogán realizará 17 homenajes por su cincuenta aniversario"O Breogán honra ao seu fundador e primeiro presidente"o orixinalMiguel Giao recibiu a homenaxe do PazoHomenaxe aos primeiros gladiadores celestesO home que nos amosa como ver o Breo co corazónTita Franco será homenaxeada polos #50anosdeBreoJulio Vila recibirá unha homenaxe in memoriam polos #50anosdeBreo"O Breogán homenaxeará aos seus aboados máis veteráns"Pechada ovación a «Capi» Sanmartín e Ricardo «Corazón de González»Homenaxe por décadas de informaciónPaco García volve ao Pazo con motivo do 50 aniversario"Resultados y clasificaciones""O Cafés Candelas Breogán, campión da Copa Princesa""O Cafés Candelas Breogán, equipo ACB"C.B. Breogán"Proxecto social"o orixinal"Centros asociados"o orixinalFicha en imdb.comMario Camus trata la recuperación del amor en 'La vieja música', su última película"Páxina web oficial""Club Baloncesto Breogán""C. B. Breogán S.A.D."eehttp://www.fegaba.com

Vilaño, A Laracha Índice Patrimonio | Lugares e parroquias | Véxase tamén | Menú de navegación43°14′52″N 8°36′03″O / 43.24775, -8.60070

Cegueira Índice Epidemioloxía | Deficiencia visual | Tipos de cegueira | Principais causas de cegueira | Tratamento | Técnicas de adaptación e axudas | Vida dos cegos | Primeiros auxilios | Crenzas respecto das persoas cegas | Crenzas das persoas cegas | O neno deficiente visual | Aspectos psicolóxicos da cegueira | Notas | Véxase tamén | Menú de navegación54.054.154.436928256blindnessDicionario da Real Academia GalegaPortal das Palabras"International Standards: Visual Standards — Aspects and Ranges of Vision Loss with Emphasis on Population Surveys.""Visual impairment and blindness""Presentan un plan para previr a cegueira"o orixinalACCDV Associació Catalana de Cecs i Disminuïts Visuals - PMFTrachoma"Effect of gene therapy on visual function in Leber's congenital amaurosis"1844137110.1056/NEJMoa0802268Cans guía - os mellores amigos dos cegosArquivadoEscola de cans guía para cegos en Mortágua, PortugalArquivado"Tecnología para ciegos y deficientes visuales. Recopilación de recursos gratuitos en la Red""Colorino""‘COL.diesis’, escuchar los sonidos del color""COL.diesis: Transforming Colour into Melody and Implementing the Result in a Colour Sensor Device"o orixinal"Sistema de desarrollo de sinestesia color-sonido para invidentes utilizando un protocolo de audio""Enseñanza táctil - geometría y color. Juegos didácticos para niños ciegos y videntes""Sistema Constanz"L'ocupació laboral dels cecs a l'Estat espanyol està pràcticament equiparada a la de les persones amb visió, entrevista amb Pedro ZuritaONCE (Organización Nacional de Cegos de España)Prevención da cegueiraDescrición de deficiencias visuais (Disc@pnet)Braillín, un boneco atractivo para calquera neno, con ou sen discapacidade, que permite familiarizarse co sistema de escritura e lectura brailleAxudas Técnicas36838ID00897494007150-90057129528256DOID:1432HP:0000618D001766C10.597.751.941.162C97109C0155020