Error submitting automl jobs to batch AI cluster
up vote
0
down vote
favorite
I'm trying to get the automl digit classification sample running on my Batch AI cluster. This cell appears to run correctly:
from azureml.core.compute import BatchAiCompute
from azureml.core.compute import ComputeTarget
# Choose a name for your cluster.
batchai_cluster_name = "nc12-cluster-04"
found = False
# Check if this compute target already exists in the workspace.
for ct_name, ct in ws.compute_targets().items():
if (ct.name == batchai_cluster_name and ct.type == 'BatchAI'):
found = True
print('Found existing compute target: {0}'.format(ct.name))
compute_target = ct
break
if not found:
print('Creating a new compute target...')
provisioning_config = BatchAiCompute.provisioning_configuration(vm_size = "STANDARD_NC12", # for GPU, use "STANDARD_NC6"
#vm_priority = 'lowpriority', # optional
autoscale_enabled = True,
cluster_min_nodes = 0,
cluster_max_nodes = 4)
# Create the cluster.
compute_target = ComputeTarget.create(ws, batchai_cluster_name, provisioning_config)
# Can poll for a minimum number of nodes and for a specific timeout.
# If no min_node_count is provided, it will use the scale settings for the cluster.
compute_target.wait_for_completion(show_output = True, min_node_count = None, timeout_in_minutes = 20)
# For a more detailed view of current Batch AI cluster status, use the 'status' property.
and I get the expected output:
Found existing compute target: nc12-cluster-04
But when I try and submit my job like this:
from azureml.core.experiment import Experiment
experiment = Experiment(ws, experiment_name)
remote_run = experiment.submit(automl_config, show_output = False)
I get this error:
~/anaconda3_501/lib/python3.6/site-packages/azureml/_restclient/operations/jasmine_operations.py in post_remote_snapshot_run(self, subscription_id, resource_group_name, workspace_name, project_name, parent_run_id, json_definition, snapshot_id, custom_headers, raw, **operation_config)
237 if response.status_code not in [200]:
--> 238 raise HttpOperationError(self._deserialize, response)
239
HttpOperationError: Operation returned an invalid status code "ErrorMessage: 'BatchAI cluster nc12-cluster-04f7543302 does not exist'. Possible cause: Remote dsvm"
Notice how the cluster name now has a suffix: nc12-cluster-04f7543302 that isn't in the compute_target passed in:
vars(compute_target)
{
...
'name': 'nc12-cluster-04',
'provisioning_errors': None,
'provisioning_state': 'Succeeded',
...
}
Any ideas?
automl azure-notebooks
add a comment |
up vote
0
down vote
favorite
I'm trying to get the automl digit classification sample running on my Batch AI cluster. This cell appears to run correctly:
from azureml.core.compute import BatchAiCompute
from azureml.core.compute import ComputeTarget
# Choose a name for your cluster.
batchai_cluster_name = "nc12-cluster-04"
found = False
# Check if this compute target already exists in the workspace.
for ct_name, ct in ws.compute_targets().items():
if (ct.name == batchai_cluster_name and ct.type == 'BatchAI'):
found = True
print('Found existing compute target: {0}'.format(ct.name))
compute_target = ct
break
if not found:
print('Creating a new compute target...')
provisioning_config = BatchAiCompute.provisioning_configuration(vm_size = "STANDARD_NC12", # for GPU, use "STANDARD_NC6"
#vm_priority = 'lowpriority', # optional
autoscale_enabled = True,
cluster_min_nodes = 0,
cluster_max_nodes = 4)
# Create the cluster.
compute_target = ComputeTarget.create(ws, batchai_cluster_name, provisioning_config)
# Can poll for a minimum number of nodes and for a specific timeout.
# If no min_node_count is provided, it will use the scale settings for the cluster.
compute_target.wait_for_completion(show_output = True, min_node_count = None, timeout_in_minutes = 20)
# For a more detailed view of current Batch AI cluster status, use the 'status' property.
and I get the expected output:
Found existing compute target: nc12-cluster-04
But when I try and submit my job like this:
from azureml.core.experiment import Experiment
experiment = Experiment(ws, experiment_name)
remote_run = experiment.submit(automl_config, show_output = False)
I get this error:
~/anaconda3_501/lib/python3.6/site-packages/azureml/_restclient/operations/jasmine_operations.py in post_remote_snapshot_run(self, subscription_id, resource_group_name, workspace_name, project_name, parent_run_id, json_definition, snapshot_id, custom_headers, raw, **operation_config)
237 if response.status_code not in [200]:
--> 238 raise HttpOperationError(self._deserialize, response)
239
HttpOperationError: Operation returned an invalid status code "ErrorMessage: 'BatchAI cluster nc12-cluster-04f7543302 does not exist'. Possible cause: Remote dsvm"
Notice how the cluster name now has a suffix: nc12-cluster-04f7543302 that isn't in the compute_target passed in:
vars(compute_target)
{
...
'name': 'nc12-cluster-04',
'provisioning_errors': None,
'provisioning_state': 'Succeeded',
...
}
Any ideas?
automl azure-notebooks
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm trying to get the automl digit classification sample running on my Batch AI cluster. This cell appears to run correctly:
from azureml.core.compute import BatchAiCompute
from azureml.core.compute import ComputeTarget
# Choose a name for your cluster.
batchai_cluster_name = "nc12-cluster-04"
found = False
# Check if this compute target already exists in the workspace.
for ct_name, ct in ws.compute_targets().items():
if (ct.name == batchai_cluster_name and ct.type == 'BatchAI'):
found = True
print('Found existing compute target: {0}'.format(ct.name))
compute_target = ct
break
if not found:
print('Creating a new compute target...')
provisioning_config = BatchAiCompute.provisioning_configuration(vm_size = "STANDARD_NC12", # for GPU, use "STANDARD_NC6"
#vm_priority = 'lowpriority', # optional
autoscale_enabled = True,
cluster_min_nodes = 0,
cluster_max_nodes = 4)
# Create the cluster.
compute_target = ComputeTarget.create(ws, batchai_cluster_name, provisioning_config)
# Can poll for a minimum number of nodes and for a specific timeout.
# If no min_node_count is provided, it will use the scale settings for the cluster.
compute_target.wait_for_completion(show_output = True, min_node_count = None, timeout_in_minutes = 20)
# For a more detailed view of current Batch AI cluster status, use the 'status' property.
and I get the expected output:
Found existing compute target: nc12-cluster-04
But when I try and submit my job like this:
from azureml.core.experiment import Experiment
experiment = Experiment(ws, experiment_name)
remote_run = experiment.submit(automl_config, show_output = False)
I get this error:
~/anaconda3_501/lib/python3.6/site-packages/azureml/_restclient/operations/jasmine_operations.py in post_remote_snapshot_run(self, subscription_id, resource_group_name, workspace_name, project_name, parent_run_id, json_definition, snapshot_id, custom_headers, raw, **operation_config)
237 if response.status_code not in [200]:
--> 238 raise HttpOperationError(self._deserialize, response)
239
HttpOperationError: Operation returned an invalid status code "ErrorMessage: 'BatchAI cluster nc12-cluster-04f7543302 does not exist'. Possible cause: Remote dsvm"
Notice how the cluster name now has a suffix: nc12-cluster-04f7543302 that isn't in the compute_target passed in:
vars(compute_target)
{
...
'name': 'nc12-cluster-04',
'provisioning_errors': None,
'provisioning_state': 'Succeeded',
...
}
Any ideas?
automl azure-notebooks
I'm trying to get the automl digit classification sample running on my Batch AI cluster. This cell appears to run correctly:
from azureml.core.compute import BatchAiCompute
from azureml.core.compute import ComputeTarget
# Choose a name for your cluster.
batchai_cluster_name = "nc12-cluster-04"
found = False
# Check if this compute target already exists in the workspace.
for ct_name, ct in ws.compute_targets().items():
if (ct.name == batchai_cluster_name and ct.type == 'BatchAI'):
found = True
print('Found existing compute target: {0}'.format(ct.name))
compute_target = ct
break
if not found:
print('Creating a new compute target...')
provisioning_config = BatchAiCompute.provisioning_configuration(vm_size = "STANDARD_NC12", # for GPU, use "STANDARD_NC6"
#vm_priority = 'lowpriority', # optional
autoscale_enabled = True,
cluster_min_nodes = 0,
cluster_max_nodes = 4)
# Create the cluster.
compute_target = ComputeTarget.create(ws, batchai_cluster_name, provisioning_config)
# Can poll for a minimum number of nodes and for a specific timeout.
# If no min_node_count is provided, it will use the scale settings for the cluster.
compute_target.wait_for_completion(show_output = True, min_node_count = None, timeout_in_minutes = 20)
# For a more detailed view of current Batch AI cluster status, use the 'status' property.
and I get the expected output:
Found existing compute target: nc12-cluster-04
But when I try and submit my job like this:
from azureml.core.experiment import Experiment
experiment = Experiment(ws, experiment_name)
remote_run = experiment.submit(automl_config, show_output = False)
I get this error:
~/anaconda3_501/lib/python3.6/site-packages/azureml/_restclient/operations/jasmine_operations.py in post_remote_snapshot_run(self, subscription_id, resource_group_name, workspace_name, project_name, parent_run_id, json_definition, snapshot_id, custom_headers, raw, **operation_config)
237 if response.status_code not in [200]:
--> 238 raise HttpOperationError(self._deserialize, response)
239
HttpOperationError: Operation returned an invalid status code "ErrorMessage: 'BatchAI cluster nc12-cluster-04f7543302 does not exist'. Possible cause: Remote dsvm"
Notice how the cluster name now has a suffix: nc12-cluster-04f7543302 that isn't in the compute_target passed in:
vars(compute_target)
{
...
'name': 'nc12-cluster-04',
'provisioning_errors': None,
'provisioning_state': 'Succeeded',
...
}
Any ideas?
automl azure-notebooks
automl azure-notebooks
asked Nov 12 at 15:45
Dave Bending
13626
13626
add a comment |
add a comment |
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53265559%2ferror-submitting-automl-jobs-to-batch-ai-cluster%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53265559%2ferror-submitting-automl-jobs-to-batch-ai-cluster%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown