parallel pool not recognizing global variables
I have a MATLAB program that I want to run in parallel so that it runs faster. However, when I do that parallel workers seem not to be able to access global variables created beforehand. Here is what my code looks like:
createData % a .m file that creates a global variable (Var)
parfor i:j
processData() % a function that is dependent on some global variables
end
However, I get an error message undefined function or variable Var
. I've already included a call for global
variables global Var
inside the function processData()
but this is not working either. Is there any way of making global
variables visible within the parallel loop?
This is not the same question as here as I declared global variables outside of the parfor
loop and want to access them within the loop with out the need to modify or update the its value across workers of the parallel loop.
matlab parallel-processing global-variables parfor
add a comment |
I have a MATLAB program that I want to run in parallel so that it runs faster. However, when I do that parallel workers seem not to be able to access global variables created beforehand. Here is what my code looks like:
createData % a .m file that creates a global variable (Var)
parfor i:j
processData() % a function that is dependent on some global variables
end
However, I get an error message undefined function or variable Var
. I've already included a call for global
variables global Var
inside the function processData()
but this is not working either. Is there any way of making global
variables visible within the parallel loop?
This is not the same question as here as I declared global variables outside of the parfor
loop and want to access them within the loop with out the need to modify or update the its value across workers of the parallel loop.
matlab parallel-processing global-variables parfor
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting aglobal
outside aparfor
(or in the body of the loop) will not work. All that will work is setting up a function inside theparfor
setting theglobal
. Not ideal, but all you've got.global
is generally bad anyways, might as well avoid it completely.
– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
add a comment |
I have a MATLAB program that I want to run in parallel so that it runs faster. However, when I do that parallel workers seem not to be able to access global variables created beforehand. Here is what my code looks like:
createData % a .m file that creates a global variable (Var)
parfor i:j
processData() % a function that is dependent on some global variables
end
However, I get an error message undefined function or variable Var
. I've already included a call for global
variables global Var
inside the function processData()
but this is not working either. Is there any way of making global
variables visible within the parallel loop?
This is not the same question as here as I declared global variables outside of the parfor
loop and want to access them within the loop with out the need to modify or update the its value across workers of the parallel loop.
matlab parallel-processing global-variables parfor
I have a MATLAB program that I want to run in parallel so that it runs faster. However, when I do that parallel workers seem not to be able to access global variables created beforehand. Here is what my code looks like:
createData % a .m file that creates a global variable (Var)
parfor i:j
processData() % a function that is dependent on some global variables
end
However, I get an error message undefined function or variable Var
. I've already included a call for global
variables global Var
inside the function processData()
but this is not working either. Is there any way of making global
variables visible within the parallel loop?
This is not the same question as here as I declared global variables outside of the parfor
loop and want to access them within the loop with out the need to modify or update the its value across workers of the parallel loop.
matlab parallel-processing global-variables parfor
matlab parallel-processing global-variables parfor
edited Nov 21 '18 at 6:40
Adriaan
12.9k63162
12.9k63162
asked Nov 20 '18 at 20:54
Abdallah AtefAbdallah Atef
4731110
4731110
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting aglobal
outside aparfor
(or in the body of the loop) will not work. All that will work is setting up a function inside theparfor
setting theglobal
. Not ideal, but all you've got.global
is generally bad anyways, might as well avoid it completely.
– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
add a comment |
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting aglobal
outside aparfor
(or in the body of the loop) will not work. All that will work is setting up a function inside theparfor
setting theglobal
. Not ideal, but all you've got.global
is generally bad anyways, might as well avoid it completely.
– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "
GLOBAL
data is hard to use inside PARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting a global
outside a parfor
(or in the body of the loop) will not work. All that will work is setting up a function inside the parfor
setting the global
. Not ideal, but all you've got. global
is generally bad anyways, might as well avoid it completely.– Adriaan
Nov 20 '18 at 22:31
Sorry, rereading the 2nd answer, read the first sentence: "
GLOBAL
data is hard to use inside PARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting a global
outside a parfor
(or in the body of the loop) will not work. All that will work is setting up a function inside the parfor
setting the global
. Not ideal, but all you've got. global
is generally bad anyways, might as well avoid it completely.– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
add a comment |
2 Answers
2
active
oldest
votes
The simplest advice is: don't use global
for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor
is smart enough to send Var
to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor
loops. In that case, I would suggest using parallel.pool.Constant
. How you use that depends on the cost of creating Var
compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
One more question @Edric, is it possible for theparallel.pool.Constant
to be accessible by the client. The situation I'm having is the the functionprocessData()
is called several times in the model. Some of which are within normalfor
loop (unparallelizable part of the model). The function is also called within aparfor
loop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constant
doesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
add a comment |
Citing the author of the parallel toolbox:
GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global
variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global
variables, run that on each worker, then run your own, global
-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
global
variables things likeh
,a
etc (this of course makes for bad reading also when the variable is notglobal
)
And finally an article outlining most of the reasons using global
variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401372%2fparallel-pool-not-recognizing-global-variables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
The simplest advice is: don't use global
for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor
is smart enough to send Var
to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor
loops. In that case, I would suggest using parallel.pool.Constant
. How you use that depends on the cost of creating Var
compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
One more question @Edric, is it possible for theparallel.pool.Constant
to be accessible by the client. The situation I'm having is the the functionprocessData()
is called several times in the model. Some of which are within normalfor
loop (unparallelizable part of the model). The function is also called within aparfor
loop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constant
doesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
add a comment |
The simplest advice is: don't use global
for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor
is smart enough to send Var
to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor
loops. In that case, I would suggest using parallel.pool.Constant
. How you use that depends on the cost of creating Var
compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
One more question @Edric, is it possible for theparallel.pool.Constant
to be accessible by the client. The situation I'm having is the the functionprocessData()
is called several times in the model. Some of which are within normalfor
loop (unparallelizable part of the model). The function is also called within aparfor
loop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constant
doesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
add a comment |
The simplest advice is: don't use global
for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor
is smart enough to send Var
to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor
loops. In that case, I would suggest using parallel.pool.Constant
. How you use that depends on the cost of creating Var
compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
The simplest advice is: don't use global
for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor
is smart enough to send Var
to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor
loops. In that case, I would suggest using parallel.pool.Constant
. How you use that depends on the cost of creating Var
compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
answered Nov 21 '18 at 9:01
EdricEdric
19.5k23336
19.5k23336
One more question @Edric, is it possible for theparallel.pool.Constant
to be accessible by the client. The situation I'm having is the the functionprocessData()
is called several times in the model. Some of which are within normalfor
loop (unparallelizable part of the model). The function is also called within aparfor
loop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constant
doesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
add a comment |
One more question @Edric, is it possible for theparallel.pool.Constant
to be accessible by the client. The situation I'm having is the the functionprocessData()
is called several times in the model. Some of which are within normalfor
loop (unparallelizable part of the model). The function is also called within aparfor
loop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constant
doesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
One more question @Edric, is it possible for the
parallel.pool.Constant
to be accessible by the client. The situation I'm having is the the function processData()
is called several times in the model. Some of which are within normal for
loop (unparallelizable part of the model). The function is also called within a parfor
loop. So, I want that constant to be available to both workers and client as well. Thanks in advance– Abdallah Atef
Nov 21 '18 at 22:03
One more question @Edric, is it possible for the
parallel.pool.Constant
to be accessible by the client. The situation I'm having is the the function processData()
is called several times in the model. Some of which are within normal for
loop (unparallelizable part of the model). The function is also called within a parfor
loop. So, I want that constant to be available to both workers and client as well. Thanks in advance– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by the
parallel.pool.Constant
doesn't exist at the client.– Edric
Nov 22 '18 at 6:23
Unfortunately, right now, the value held by the
parallel.pool.Constant
doesn't exist at the client.– Edric
Nov 22 '18 at 6:23
add a comment |
Citing the author of the parallel toolbox:
GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global
variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global
variables, run that on each worker, then run your own, global
-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
global
variables things likeh
,a
etc (this of course makes for bad reading also when the variable is notglobal
)
And finally an article outlining most of the reasons using global
variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
add a comment |
Citing the author of the parallel toolbox:
GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global
variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global
variables, run that on each worker, then run your own, global
-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
global
variables things likeh
,a
etc (this of course makes for bad reading also when the variable is notglobal
)
And finally an article outlining most of the reasons using global
variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
add a comment |
Citing the author of the parallel toolbox:
GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global
variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global
variables, run that on each worker, then run your own, global
-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
global
variables things likeh
,a
etc (this of course makes for bad reading also when the variable is notglobal
)
And finally an article outlining most of the reasons using global
variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
Citing the author of the parallel toolbox:
GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global
variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global
variables, run that on each worker, then run your own, global
-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
global
variables things likeh
,a
etc (this of course makes for bad reading also when the variable is notglobal
)
And finally an article outlining most of the reasons using global
variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
edited Nov 21 '18 at 8:40
answered Nov 21 '18 at 6:30
AdriaanAdriaan
12.9k63162
12.9k63162
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401372%2fparallel-pool-not-recognizing-global-variables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "
GLOBAL
data is hard to use insidePARFOR
because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting aglobal
outside aparfor
(or in the body of the loop) will not work. All that will work is setting up a function inside theparfor
setting theglobal
. Not ideal, but all you've got.global
is generally bad anyways, might as well avoid it completely.– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45