Why does a table with a defined constant in its index compute 10X slower?Why does array packing in Table behave like this?Why is building a table of function values so much slower than just plotting the function?Issue with very large lists in MathematicaWhy does `Table` returns values but `Plot` doesn't plot them?How to speed up computation of medoids?Why is CompilationTarget -> C slower than directly writing with C?ParallelTable much slower than Table on RandomReal with arbitrary precisionPart does not exist in table construction, but I don't explicitly index that highwhy Findroot slower with jacobian?Why does Table[] slow down exponentially with increasing length?
What is the purpose of std::forward()'s rvalue reference overload?
Why is there a need to modify system call tables in Linux?
Looking after a wayward brother in mother's will
Why does my electric oven present the option of 40A and 50A breakers?
Can you dispel the Slow effect of a Stone Golem?
Applicants clearly not having the skills they advertise
Term for checking piece whose opponent daren't capture it
Are there regional foods in Westeros?
How to detach yourself from a character you're going to kill?
What if you don't bring your credit card or debit for incidentals?
What caused the tendency for conservatives to not support climate change regulations?
Is having a hidden directory under /etc safe?
Strange math syntax in old basic listing
Is the capacitor drawn or wired wrongly?
The oldest tradition stopped before it got back to him
What does it mean by "d-ism of Leibniz" and "dotage of Newton" in simple English?
Site-specific value for an appsetting in a multisite solution
How can a single Member of the House block a Congressional bill?
The original word for a wild boar
What is the most important characteristic of New Weird as a genre?
How can I stop my presentation being derailed by audience questions?
Creating Fictional Slavic Place Names
Humans meet a distant alien species. How do they standardize? - Units of Measure
Can a rogue effectively triple their speed by combining Dash and Ready?
Why does a table with a defined constant in its index compute 10X slower?
Why does array packing in Table behave like this?Why is building a table of function values so much slower than just plotting the function?Issue with very large lists in MathematicaWhy does `Table` returns values but `Plot` doesn't plot them?How to speed up computation of medoids?Why is CompilationTarget -> C slower than directly writing with C?ParallelTable much slower than Table on RandomReal with arbitrary precisionPart does not exist in table construction, but I don't explicitly index that highwhy Findroot slower with jacobian?Why does Table[] slow down exponentially with increasing length?
$begingroup$
I need to do some iterative summations. Here is a minimum working example:
data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]
(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]
And here are the outputs:
0.5625, 5.49936*10^6
9.28125, 5.49936*10^6
For some reason, using m=10 makes it much slower. I will need to do a bunch of m's, so this is the bottom of a larger nest.
What is a faster way to do this?
Late Edit:
Bonus question: How to optimize this one as well:
Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]
list-manipulation performance-tuning table
$endgroup$
add a comment |
$begingroup$
I need to do some iterative summations. Here is a minimum working example:
data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]
(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]
And here are the outputs:
0.5625, 5.49936*10^6
9.28125, 5.49936*10^6
For some reason, using m=10 makes it much slower. I will need to do a bunch of m's, so this is the bottom of a larger nest.
What is a faster way to do this?
Late Edit:
Bonus question: How to optimize this one as well:
Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]
list-manipulation performance-tuning table
$endgroup$
add a comment |
$begingroup$
I need to do some iterative summations. Here is a minimum working example:
data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]
(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]
And here are the outputs:
0.5625, 5.49936*10^6
9.28125, 5.49936*10^6
For some reason, using m=10 makes it much slower. I will need to do a bunch of m's, so this is the bottom of a larger nest.
What is a faster way to do this?
Late Edit:
Bonus question: How to optimize this one as well:
Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]
list-manipulation performance-tuning table
$endgroup$
I need to do some iterative summations. Here is a minimum working example:
data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]
(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]
And here are the outputs:
0.5625, 5.49936*10^6
9.28125, 5.49936*10^6
For some reason, using m=10 makes it much slower. I will need to do a bunch of m's, so this is the bottom of a larger nest.
What is a faster way to do this?
Late Edit:
Bonus question: How to optimize this one as well:
Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]
list-manipulation performance-tuning table
list-manipulation performance-tuning table
edited May 16 at 19:53
Carl Woll
81.2k3105209
81.2k3105209
asked May 16 at 17:07
axsvl77axsvl77
425316
425316
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
The problem lies mostly in the inner Table:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.
You can help the JIT compiler by using With:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
$endgroup$
add a comment |
$begingroup$
You can use ListCorrelate:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
$endgroup$
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198514%2fwhy-does-a-table-with-a-defined-constant-in-its-index-compute-10x-slower%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The problem lies mostly in the inner Table:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.
You can help the JIT compiler by using With:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
$endgroup$
add a comment |
$begingroup$
The problem lies mostly in the inner Table:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.
You can help the JIT compiler by using With:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
$endgroup$
add a comment |
$begingroup$
The problem lies mostly in the inner Table:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.
You can help the JIT compiler by using With:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
$endgroup$
The problem lies mostly in the inner Table:
Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]
m = 10;
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]
0.366407, 5.50276*10^6
8.01738, 5.50276*10^6
I think the reason is this:
Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.
You can help the JIT compiler by using With:
With[m = 10,
Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
]
0.369601, 5.5049*10^6
Addendum:
By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):
a = With[m = 10,
Total[
Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
Length[data] - 5*m]]
]; // RepeatedTiming // First
b = Total@ListCorrelate[ConstantArray[1., m + 1],
data[[;; -50 + m - 1]]]; // RepeatedTiming // First
c = Plus[
Range[1., m].data[[1 ;; m]],
(m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
]; // RepeatedTiming // First
a == b == c
0.28
0.017
0.0018
True
edited May 16 at 19:12
answered May 16 at 17:15
Henrik SchumacherHenrik Schumacher
63.2k587176
63.2k587176
add a comment |
add a comment |
$begingroup$
You can use ListCorrelate:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
$endgroup$
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
$begingroup$
You can use ListCorrelate:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
$endgroup$
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
$begingroup$
You can use ListCorrelate:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
$endgroup$
You can use ListCorrelate:
m=10;
Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming
0.017725, 5.50044*10^6
Bonus question
For the bonus question:
data = RandomReal[1, 10^5];
Your version:
With[m = 10,
Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
] //AbsoluteTiming
0.448739, 3.11778*10^7
Using ListCorrelate again:
m = 10;
#.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming
0.018401, 3.11778*10^7
edited May 16 at 20:04
answered May 16 at 17:21
Carl WollCarl Woll
81.2k3105209
81.2k3105209
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
$begingroup$
Carl, nice use of the dot-product to overcome the internal compiling issues that would prevent implementing the mapping of a slotted set! I was attempting a solution with this same thought process, but was unable to overcome the change of #^2 to #1^2 during my attempts at a solution. Your use of ListCorrelate will be helpful in the future, thank you!
$endgroup$
– CA Trevillian
May 17 at 14:21
add a comment |
Thanks for contributing an answer to Mathematica Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198514%2fwhy-does-a-table-with-a-defined-constant-in-its-index-compute-10x-slower%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown