parallel pool not recognizing global variables
I have a MATLAB program that I want to run in parallel so that it runs faster. However, when I do that parallel workers seem not to be able to access global variables created beforehand. Here is what my code looks like:
createData % a .m file that creates a global variable (Var)
parfor i:j
processData() % a function that is dependent on some global variables
end
However, I get an error message undefined function or variable Var. I've already included a call for global variables global Var inside the function processData() but this is not working either. Is there any way of making global variables visible within the parallel loop?
This is not the same question as here as I declared global variables outside of the parfor loop and want to access them within the loop with out the need to modify or update the its value across workers of the parallel loop.
matlab parallel-processing global-variables parfor
add a comment |
I have a MATLAB program that I want to run in parallel so that it runs faster. However, when I do that parallel workers seem not to be able to access global variables created beforehand. Here is what my code looks like:
createData % a .m file that creates a global variable (Var)
parfor i:j
processData() % a function that is dependent on some global variables
end
However, I get an error message undefined function or variable Var. I've already included a call for global variables global Var inside the function processData() but this is not working either. Is there any way of making global variables visible within the parallel loop?
This is not the same question as here as I declared global variables outside of the parfor loop and want to access them within the loop with out the need to modify or update the its value across workers of the parallel loop.
matlab parallel-processing global-variables parfor
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting aglobaloutside aparfor(or in the body of the loop) will not work. All that will work is setting up a function inside theparforsetting theglobal. Not ideal, but all you've got.globalis generally bad anyways, might as well avoid it completely.
– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
add a comment |
I have a MATLAB program that I want to run in parallel so that it runs faster. However, when I do that parallel workers seem not to be able to access global variables created beforehand. Here is what my code looks like:
createData % a .m file that creates a global variable (Var)
parfor i:j
processData() % a function that is dependent on some global variables
end
However, I get an error message undefined function or variable Var. I've already included a call for global variables global Var inside the function processData() but this is not working either. Is there any way of making global variables visible within the parallel loop?
This is not the same question as here as I declared global variables outside of the parfor loop and want to access them within the loop with out the need to modify or update the its value across workers of the parallel loop.
matlab parallel-processing global-variables parfor
I have a MATLAB program that I want to run in parallel so that it runs faster. However, when I do that parallel workers seem not to be able to access global variables created beforehand. Here is what my code looks like:
createData % a .m file that creates a global variable (Var)
parfor i:j
processData() % a function that is dependent on some global variables
end
However, I get an error message undefined function or variable Var. I've already included a call for global variables global Var inside the function processData() but this is not working either. Is there any way of making global variables visible within the parallel loop?
This is not the same question as here as I declared global variables outside of the parfor loop and want to access them within the loop with out the need to modify or update the its value across workers of the parallel loop.
matlab parallel-processing global-variables parfor
matlab parallel-processing global-variables parfor
edited Nov 21 '18 at 6:40
Adriaan
12.9k63162
12.9k63162
asked Nov 20 '18 at 20:54
Abdallah AtefAbdallah Atef
4731110
4731110
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting aglobaloutside aparfor(or in the body of the loop) will not work. All that will work is setting up a function inside theparforsetting theglobal. Not ideal, but all you've got.globalis generally bad anyways, might as well avoid it completely.
– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
add a comment |
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting aglobaloutside aparfor(or in the body of the loop) will not work. All that will work is setting up a function inside theparforsetting theglobal. Not ideal, but all you've got.globalis generally bad anyways, might as well avoid it completely.
– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "
GLOBAL data is hard to use inside PARFOR because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting a global outside a parfor (or in the body of the loop) will not work. All that will work is setting up a function inside the parfor setting the global. Not ideal, but all you've got. global is generally bad anyways, might as well avoid it completely.– Adriaan
Nov 20 '18 at 22:31
Sorry, rereading the 2nd answer, read the first sentence: "
GLOBAL data is hard to use inside PARFOR because each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting a global outside a parfor (or in the body of the loop) will not work. All that will work is setting up a function inside the parfor setting the global. Not ideal, but all you've got. global is generally bad anyways, might as well avoid it completely.– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45
add a comment |
2 Answers
2
active
oldest
votes
The simplest advice is: don't use global for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor is smart enough to send Var to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor loops. In that case, I would suggest using parallel.pool.Constant. How you use that depends on the cost of creating Var compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
One more question @Edric, is it possible for theparallel.pool.Constantto be accessible by the client. The situation I'm having is the the functionprocessData()is called several times in the model. Some of which are within normalforloop (unparallelizable part of the model). The function is also called within aparforloop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constantdoesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
add a comment |
Citing the author of the parallel toolbox:
GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global variables, run that on each worker, then run your own, global-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
globalvariables things likeh,aetc (this of course makes for bad reading also when the variable is notglobal)
And finally an article outlining most of the reasons using global variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401372%2fparallel-pool-not-recognizing-global-variables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
The simplest advice is: don't use global for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor is smart enough to send Var to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor loops. In that case, I would suggest using parallel.pool.Constant. How you use that depends on the cost of creating Var compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
One more question @Edric, is it possible for theparallel.pool.Constantto be accessible by the client. The situation I'm having is the the functionprocessData()is called several times in the model. Some of which are within normalforloop (unparallelizable part of the model). The function is also called within aparforloop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constantdoesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
add a comment |
The simplest advice is: don't use global for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor is smart enough to send Var to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor loops. In that case, I would suggest using parallel.pool.Constant. How you use that depends on the cost of creating Var compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
One more question @Edric, is it possible for theparallel.pool.Constantto be accessible by the client. The situation I'm having is the the functionprocessData()is called several times in the model. Some of which are within normalforloop (unparallelizable part of the model). The function is also called within aparforloop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constantdoesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
add a comment |
The simplest advice is: don't use global for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor is smart enough to send Var to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor loops. In that case, I would suggest using parallel.pool.Constant. How you use that depends on the cost of creating Var compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
The simplest advice is: don't use global for the myriad reasons already described/linked here. Ideally, you would restructure your code like so:
Var = createData(); % returns 'Var' rather than creating a global 'Var'
parfor idx = ...
% simply use 'Var' inside the parfor loop.
out(idx) = processData(Var, ...);
end
Note that parfor is smart enough to send Var to each worker exactly once for the above loop. However, it isn't smart enough not to send it across multiple times if you have multiple parfor loops. In that case, I would suggest using parallel.pool.Constant. How you use that depends on the cost of creating Var compared to its size. If it is small, but expensive to create - that implies you're best off creating it only once at the client and sending it to the workers, like this:
cVar = parallel.pool.Constant(Var);
If it is large, but relatively quick to construct, you could consider getting the workers each to construct their own copy independently, like this:
cVar = parallel.pool.Constant(@createData); % invokes 'createData' on each worker
answered Nov 21 '18 at 9:01
EdricEdric
19.5k23336
19.5k23336
One more question @Edric, is it possible for theparallel.pool.Constantto be accessible by the client. The situation I'm having is the the functionprocessData()is called several times in the model. Some of which are within normalforloop (unparallelizable part of the model). The function is also called within aparforloop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constantdoesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
add a comment |
One more question @Edric, is it possible for theparallel.pool.Constantto be accessible by the client. The situation I'm having is the the functionprocessData()is called several times in the model. Some of which are within normalforloop (unparallelizable part of the model). The function is also called within aparforloop. So, I want that constant to be available to both workers and client as well. Thanks in advance
– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by theparallel.pool.Constantdoesn't exist at the client.
– Edric
Nov 22 '18 at 6:23
One more question @Edric, is it possible for the
parallel.pool.Constant to be accessible by the client. The situation I'm having is the the function processData() is called several times in the model. Some of which are within normal for loop (unparallelizable part of the model). The function is also called within a parfor loop. So, I want that constant to be available to both workers and client as well. Thanks in advance– Abdallah Atef
Nov 21 '18 at 22:03
One more question @Edric, is it possible for the
parallel.pool.Constant to be accessible by the client. The situation I'm having is the the function processData() is called several times in the model. Some of which are within normal for loop (unparallelizable part of the model). The function is also called within a parfor loop. So, I want that constant to be available to both workers and client as well. Thanks in advance– Abdallah Atef
Nov 21 '18 at 22:03
Unfortunately, right now, the value held by the
parallel.pool.Constant doesn't exist at the client.– Edric
Nov 22 '18 at 6:23
Unfortunately, right now, the value held by the
parallel.pool.Constant doesn't exist at the client.– Edric
Nov 22 '18 at 6:23
add a comment |
Citing the author of the parallel toolbox:
GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global variables, run that on each worker, then run your own, global-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
globalvariables things likeh,aetc (this of course makes for bad reading also when the variable is notglobal)
And finally an article outlining most of the reasons using global variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
add a comment |
Citing the author of the parallel toolbox:
GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global variables, run that on each worker, then run your own, global-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
globalvariables things likeh,aetc (this of course makes for bad reading also when the variable is notglobal)
And finally an article outlining most of the reasons using global variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
add a comment |
Citing the author of the parallel toolbox:
GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global variables, run that on each worker, then run your own, global-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
globalvariables things likeh,aetc (this of course makes for bad reading also when the variable is notglobal)
And finally an article outlining most of the reasons using global variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
Citing the author of the parallel toolbox:
GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers.
Emphasis mine. So the only way to get a global variable on a worker (which is a bad idea for reasons mentioned in the linked post) is to write a function which sets up the global variables, run that on each worker, then run your own, global-dependent function.
Citing another comment of mine to illustrate why this is a bad idea:
One of the pitfalls in terms of good practise is that you can suddenly overwrite a variable which is used inside a function in other functions. Therefore it can be difficult to keep track of changes and going back and forth between functions might cause unexpected behaviour because of that. This happens especially often if you call your
globalvariables things likeh,aetc (this of course makes for bad reading also when the variable is notglobal)
And finally an article outlining most of the reasons using global variables is generally a bad idea.
Bottom line: what you want is not possible, and generally thought to be bad practise.
edited Nov 21 '18 at 8:40
answered Nov 21 '18 at 6:30
AdriaanAdriaan
12.9k63162
12.9k63162
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401372%2fparallel-pool-not-recognizing-global-variables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I think this question is different from the link you provided @Adriaan. I've declared the global variable out side the parfor loop and what I want to do is to access the variable within the loop not modify it or update its value across workers
– Abdallah Atef
Nov 20 '18 at 22:21
Sorry, rereading the 2nd answer, read the first sentence: "
GLOBALdata is hard to use insidePARFORbecause each worker is a separate MATLAB process, and global variables are not synchronised from the client (or any other process) to the workers." emphasis mine. Ergo: the author of the toolbox tells you that setting aglobaloutside aparfor(or in the body of the loop) will not work. All that will work is setting up a function inside theparforsetting theglobal. Not ideal, but all you've got.globalis generally bad anyways, might as well avoid it completely.– Adriaan
Nov 20 '18 at 22:31
Thanks @Adriaan. I'll try to reformulate with more detail so that I can get more focused answer specific to my situation.
– Abdallah Atef
Nov 20 '18 at 22:45