how to run a big docker image on the Google Cloud Platform?
I would like to run a quite big docker image (~6 GB). I can create the docker image from a config file using Google Cloud Platform cloudshell
gcloud builds submit --timeout=36000 --tag gcr.io/docker-ml-dl-xxxx/docker-anaconda-env-ml-dl
This works perfectly fine and I can see the buidl is succesfull
https://console.cloud.google.com/cloud-build/
I can also see my image in the Registry Container:
https://console.cloud.google.com/gcr/images/docker-ml-dl-xxxxx
so far so good. The issue is when I try to run this image from cloudshell:
xxxxx@cloudshell:~ (docker-ml-dl-xxxxx)$ docker run gcr.io/docker-ml-dl-xxxxx/docker-anaconda-env-ml-dl
Unable to find image 'gcr.io/docker-ml-dl-xxxx/docker-anaconda-env-ml-dl:latest' locally
latest: Pulling from docker-ml-dl-xxxx/docker-anaconda-env-ml-dl
993c50d47469: Pull complete
c71c2bfd82ad: Pull complete
05fbbe050330: Pull complete
5586ce1e5329: Pull complete
1faf1ec50c57: Pull complete
fda25b84aec7: Pull complete
b5b4ca70f42c: Extracting [=======================> ] 708MB/1.522GB
0088935a1845: Download complete
36f80eb6aa84: Download complete
b08b38d2d4a3: Download complete
5ae3364fe2cf: Download complete
25da48fc753b: Downloading [==================================================>] 5.857GB/5.857GB
302cfeb76ade: Download complete
1f6d69ed4c84: Download complete
58c798a01f92: Download complete
docker: write /var/lib/docker/tmp/GetImageBlob997013344: no space left on device.
See 'docker run --help'.
Ok so my docker image is to big to be run from cloudshell.
Is this correct ?
What are the other/best option ? (to be 100% I can run the docker image on my Mac)
- creating a custom VM
- with 10 GB storage
- install all software needed on this VM: docker gcloud ...
I need to devellop and run Machine Learning and Deep Learning code (this is the exploration phase, not the deployment phase with kubernetes).
Is this the best work on the cloud ?
docker google-cloud-platform gcloud
edited Nov 21 '18 at 14:50
desertnaut
20.3k74379
asked Nov 21 '18 at 14:43
Dr. Fabien TarradeDr. Fabien Tarrade
1972519
add a comment |
I would like to run a quite big docker image (~6 GB). I can create the docker image from a config file using Google Cloud Platform cloudshell
gcloud builds submit --timeout=36000 --tag gcr.io/docker-ml-dl-xxxx/docker-anaconda-env-ml-dl
This works perfectly fine and I can see the buidl is succesfull
https://console.cloud.google.com/cloud-build/
I can also see my image in the Registry Container:
https://console.cloud.google.com/gcr/images/docker-ml-dl-xxxxx
so far so good. The issue is when I try to run this image from cloudshell:
xxxxx@cloudshell:~ (docker-ml-dl-xxxxx)$ docker run gcr.io/docker-ml-dl-xxxxx/docker-anaconda-env-ml-dl
Unable to find image 'gcr.io/docker-ml-dl-xxxx/docker-anaconda-env-ml-dl:latest' locally
latest: Pulling from docker-ml-dl-xxxx/docker-anaconda-env-ml-dl
993c50d47469: Pull complete
c71c2bfd82ad: Pull complete
05fbbe050330: Pull complete
5586ce1e5329: Pull complete
1faf1ec50c57: Pull complete
fda25b84aec7: Pull complete
b5b4ca70f42c: Extracting [=======================> ] 708MB/1.522GB
0088935a1845: Download complete
36f80eb6aa84: Download complete
b08b38d2d4a3: Download complete
5ae3364fe2cf: Download complete
25da48fc753b: Downloading [==================================================>] 5.857GB/5.857GB
302cfeb76ade: Download complete
1f6d69ed4c84: Download complete
58c798a01f92: Download complete
docker: write /var/lib/docker/tmp/GetImageBlob997013344: no space left on device.
See 'docker run --help'.
Ok so my docker image is to big to be run from cloudshell.
Is this correct ?
What are the other/best option ? (to be 100% I can run the docker image on my Mac)
- creating a custom VM
- with 10 GB storage
- install all software needed on this VM: docker gcloud ...
I need to devellop and run Machine Learning and Deep Learning code (this is the exploration phase, not the deployment phase with kubernetes).
Is this the best work on the cloud ?
docker google-cloud-platform gcloud
edited Nov 21 '18 at 14:50
desertnaut
20.3k74379
asked Nov 21 '18 at 14:43
Dr. Fabien TarradeDr. Fabien Tarrade
1972519
6GB is ridiculously huge for a docker image. Does it really need to be that big?
– SiHa
Nov 21 '18 at 14:47
Question has nothing to do withmachine-learning- kindly do not spam the tag (removed).
– desertnaut
Nov 21 '18 at 14:50
yes, this is huge. I have a acaconda env with a lot of packages and config file (spacy, nltk). Clearly this need to be optimize but this is not the question for the moment. Yes, this is not a ML topic but I want to know the best way to do exploration work on ML on the cloud. Focus is in ML, not sowtare dev on a 300 MB RedHat docker image. This why i added the tag but this was not a good. Thanks for the clean-up.
– Dr. Fabien Tarrade
Nov 21 '18 at 15:06
Cloud Shell provides you with 5 GB of total disk space. Why are you trying to run a container in Cloud Shell?
– John Hanley
Nov 21 '18 at 19:15
What is your proposal to develop a DL NLP code that need an Anaconda python env including a lot of ML packages like Tensorflow Keras, SHAP, LIME (and seems to be few GB, don't asked me why and I am looking into it as well) and for which the data are stored in BigQuery (200 GB). I am not talking about running the code in production. I am new on the Cloud. I know how I will do that on an Hadoop Data Lake. Looking forward for best pratice and proposal.
– Dr. Fabien Tarrade
Nov 21 '18 at 19:41
add a comment |
I would like to run a quite big docker image (~6 GB). I can create the docker image from a config file using Google Cloud Platform cloudshell
gcloud builds submit --timeout=36000 --tag gcr.io/docker-ml-dl-xxxx/docker-anaconda-env-ml-dl
This works perfectly fine and I can see the buidl is succesfull
https://console.cloud.google.com/cloud-build/
I can also see my image in the Registry Container:
https://console.cloud.google.com/gcr/images/docker-ml-dl-xxxxx
so far so good. The issue is when I try to run this image from cloudshell:
xxxxx@cloudshell:~ (docker-ml-dl-xxxxx)$ docker run gcr.io/docker-ml-dl-xxxxx/docker-anaconda-env-ml-dl
Unable to find image 'gcr.io/docker-ml-dl-xxxx/docker-anaconda-env-ml-dl:latest' locally
latest: Pulling from docker-ml-dl-xxxx/docker-anaconda-env-ml-dl
993c50d47469: Pull complete
c71c2bfd82ad: Pull complete
05fbbe050330: Pull complete
5586ce1e5329: Pull complete
1faf1ec50c57: Pull complete
fda25b84aec7: Pull complete
b5b4ca70f42c: Extracting [=======================> ] 708MB/1.522GB
0088935a1845: Download complete
36f80eb6aa84: Download complete
b08b38d2d4a3: Download complete
5ae3364fe2cf: Download complete
25da48fc753b: Downloading [==================================================>] 5.857GB/5.857GB
302cfeb76ade: Download complete
1f6d69ed4c84: Download complete
58c798a01f92: Download complete
docker: write /var/lib/docker/tmp/GetImageBlob997013344: no space left on device.
See 'docker run --help'.
Ok so my docker image is to big to be run from cloudshell.
Is this correct ?
What are the other/best option ? (to be 100% I can run the docker image on my Mac)
- creating a custom VM
- with 10 GB storage
- install all software needed on this VM: docker gcloud ...
I need to devellop and run Machine Learning and Deep Learning code (this is the exploration phase, not the deployment phase with kubernetes).
Is this the best work on the cloud ?
docker google-cloud-platform gcloud
edited Nov 21 '18 at 14:50
desertnaut
20.3k74379
asked Nov 21 '18 at 14:43
Dr. Fabien TarradeDr. Fabien Tarrade
1972519
I would like to run a quite big docker image (~6 GB). I can create the docker image from a config file using Google Cloud Platform cloudshell
gcloud builds submit --timeout=36000 --tag gcr.io/docker-ml-dl-xxxx/docker-anaconda-env-ml-dl
This works perfectly fine and I can see the buidl is succesfull
https://console.cloud.google.com/cloud-build/
I can also see my image in the Registry Container:
https://console.cloud.google.com/gcr/images/docker-ml-dl-xxxxx
so far so good. The issue is when I try to run this image from cloudshell:
xxxxx@cloudshell:~ (docker-ml-dl-xxxxx)$ docker run gcr.io/docker-ml-dl-xxxxx/docker-anaconda-env-ml-dl
Unable to find image 'gcr.io/docker-ml-dl-xxxx/docker-anaconda-env-ml-dl:latest' locally
latest: Pulling from docker-ml-dl-xxxx/docker-anaconda-env-ml-dl
993c50d47469: Pull complete
c71c2bfd82ad: Pull complete
05fbbe050330: Pull complete
5586ce1e5329: Pull complete
1faf1ec50c57: Pull complete
fda25b84aec7: Pull complete
b5b4ca70f42c: Extracting [=======================> ] 708MB/1.522GB
0088935a1845: Download complete
36f80eb6aa84: Download complete
b08b38d2d4a3: Download complete
5ae3364fe2cf: Download complete
25da48fc753b: Downloading [==================================================>] 5.857GB/5.857GB
302cfeb76ade: Download complete
1f6d69ed4c84: Download complete
58c798a01f92: Download complete
docker: write /var/lib/docker/tmp/GetImageBlob997013344: no space left on device.
See 'docker run --help'.
Ok so my docker image is to big to be run from cloudshell.
Is this correct ?
What are the other/best option ? (to be 100% I can run the docker image on my Mac)
- creating a custom VM
- with 10 GB storage
- install all software needed on this VM: docker gcloud ...
I need to devellop and run Machine Learning and Deep Learning code (this is the exploration phase, not the deployment phase with kubernetes).
Is this the best work on the cloud ?
docker google-cloud-platform gcloud
docker google-cloud-platform gcloud
edited Nov 21 '18 at 14:50
desertnaut
20.3k74379
asked Nov 21 '18 at 14:43
Dr. Fabien TarradeDr. Fabien Tarrade
1972519
edited Nov 21 '18 at 14:50
desertnaut
20.3k74379
asked Nov 21 '18 at 14:43
Dr. Fabien TarradeDr. Fabien Tarrade
1972519
edited Nov 21 '18 at 14:50
desertnaut
20.3k74379
edited Nov 21 '18 at 14:50
desertnaut
20.3k74379
edited Nov 21 '18 at 14:50
desertnaut
20.3k74379
20.3k74379
asked Nov 21 '18 at 14:43
Dr. Fabien TarradeDr. Fabien Tarrade
1972519
asked Nov 21 '18 at 14:43
Dr. Fabien TarradeDr. Fabien Tarrade
1972519
asked Nov 21 '18 at 14:43
Dr. Fabien TarradeDr. Fabien Tarrade
1972519
1972519
6GB is ridiculously huge for a docker image. Does it really need to be that big?
– SiHa
Nov 21 '18 at 14:47
Question has nothing to do withmachine-learning- kindly do not spam the tag (removed).
– desertnaut
Nov 21 '18 at 14:50
yes, this is huge. I have a acaconda env with a lot of packages and config file (spacy, nltk). Clearly this need to be optimize but this is not the question for the moment. Yes, this is not a ML topic but I want to know the best way to do exploration work on ML on the cloud. Focus is in ML, not sowtare dev on a 300 MB RedHat docker image. This why i added the tag but this was not a good. Thanks for the clean-up.
– Dr. Fabien Tarrade
Nov 21 '18 at 15:06
Cloud Shell provides you with 5 GB of total disk space. Why are you trying to run a container in Cloud Shell?
– John Hanley
Nov 21 '18 at 19:15
What is your proposal to develop a DL NLP code that need an Anaconda python env including a lot of ML packages like Tensorflow Keras, SHAP, LIME (and seems to be few GB, don't asked me why and I am looking into it as well) and for which the data are stored in BigQuery (200 GB). I am not talking about running the code in production. I am new on the Cloud. I know how I will do that on an Hadoop Data Lake. Looking forward for best pratice and proposal.
– Dr. Fabien Tarrade
Nov 21 '18 at 19:41
add a comment |
6GB is ridiculously huge for a docker image. Does it really need to be that big?
– SiHa
Nov 21 '18 at 14:47
Question has nothing to do withmachine-learning- kindly do not spam the tag (removed).
– desertnaut
Nov 21 '18 at 14:50
yes, this is huge. I have a acaconda env with a lot of packages and config file (spacy, nltk). Clearly this need to be optimize but this is not the question for the moment. Yes, this is not a ML topic but I want to know the best way to do exploration work on ML on the cloud. Focus is in ML, not sowtare dev on a 300 MB RedHat docker image. This why i added the tag but this was not a good. Thanks for the clean-up.
– Dr. Fabien Tarrade
Nov 21 '18 at 15:06
Cloud Shell provides you with 5 GB of total disk space. Why are you trying to run a container in Cloud Shell?
– John Hanley
Nov 21 '18 at 19:15
What is your proposal to develop a DL NLP code that need an Anaconda python env including a lot of ML packages like Tensorflow Keras, SHAP, LIME (and seems to be few GB, don't asked me why and I am looking into it as well) and for which the data are stored in BigQuery (200 GB). I am not talking about running the code in production. I am new on the Cloud. I know how I will do that on an Hadoop Data Lake. Looking forward for best pratice and proposal.
– Dr. Fabien Tarrade
Nov 21 '18 at 19:41
6GB is ridiculously huge for a docker image. Does it really need to be that big?
– SiHa
Nov 21 '18 at 14:47
6GB is ridiculously huge for a docker image. Does it really need to be that big?
– SiHa
Nov 21 '18 at 14:47
Question has nothing to do with
machine-learning - kindly do not spam the tag (removed).– desertnaut
Nov 21 '18 at 14:50
Question has nothing to do with
machine-learning - kindly do not spam the tag (removed).– desertnaut
Nov 21 '18 at 14:50
yes, this is huge. I have a acaconda env with a lot of packages and config file (spacy, nltk). Clearly this need to be optimize but this is not the question for the moment. Yes, this is not a ML topic but I want to know the best way to do exploration work on ML on the cloud. Focus is in ML, not sowtare dev on a 300 MB RedHat docker image. This why i added the tag but this was not a good. Thanks for the clean-up.
– Dr. Fabien Tarrade
Nov 21 '18 at 15:06
yes, this is huge. I have a acaconda env with a lot of packages and config file (spacy, nltk). Clearly this need to be optimize but this is not the question for the moment. Yes, this is not a ML topic but I want to know the best way to do exploration work on ML on the cloud. Focus is in ML, not sowtare dev on a 300 MB RedHat docker image. This why i added the tag but this was not a good. Thanks for the clean-up.
– Dr. Fabien Tarrade
Nov 21 '18 at 15:06
Cloud Shell provides you with 5 GB of total disk space. Why are you trying to run a container in Cloud Shell?
– John Hanley
Nov 21 '18 at 19:15
Cloud Shell provides you with 5 GB of total disk space. Why are you trying to run a container in Cloud Shell?
– John Hanley
Nov 21 '18 at 19:15
What is your proposal to develop a DL NLP code that need an Anaconda python env including a lot of ML packages like Tensorflow Keras, SHAP, LIME (and seems to be few GB, don't asked me why and I am looking into it as well) and for which the data are stored in BigQuery (200 GB). I am not talking about running the code in production. I am new on the Cloud. I know how I will do that on an Hadoop Data Lake. Looking forward for best pratice and proposal.
– Dr. Fabien Tarrade
Nov 21 '18 at 19:41
What is your proposal to develop a DL NLP code that need an Anaconda python env including a lot of ML packages like Tensorflow Keras, SHAP, LIME (and seems to be few GB, don't asked me why and I am looking into it as well) and for which the data are stored in BigQuery (200 GB). I am not talking about running the code in production. I am new on the Cloud. I know how I will do that on an Hadoop Data Lake. Looking forward for best pratice and proposal.
– Dr. Fabien Tarrade
Nov 21 '18 at 19:41
add a comment |
1 Answer
1
active
oldest
votes
The docker image is too big to run on Cloud Shell. You might run it on Kubernetes or Compute Engine instead, but since you're still in the early stages and you've already said you can run the tools you need locally, then this might not be necessary for your needs. Looking into the future, when you're more concerned with performance, you might want to consider a solution such as Cloud ML Engine or BigQuery ML.
answered Nov 27 '18 at 11:01
Christopher PChristopher P
58827
Thanks, this is what I did by using a Compute Engine. You will have to check that even when adding a 20 GB disk in your custom VM that you have still enought disk space in "/mnt/stateful_partition" for your Docker image. There are some other useful option like chaning the restart policy to "Never" or "Allocate a buffer for STDIN".
– Dr. Fabien Tarrade
Nov 29 '18 at 20:37
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53414523%2fhow-to-run-a-big-docker-image-on-the-google-cloud-platform%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The docker image is too big to run on Cloud Shell. You might run it on Kubernetes or Compute Engine instead, but since you're still in the early stages and you've already said you can run the tools you need locally, then this might not be necessary for your needs. Looking into the future, when you're more concerned with performance, you might want to consider a solution such as Cloud ML Engine or BigQuery ML.
answered Nov 27 '18 at 11:01
Christopher PChristopher P
58827
Thanks, this is what I did by using a Compute Engine. You will have to check that even when adding a 20 GB disk in your custom VM that you have still enought disk space in "/mnt/stateful_partition" for your Docker image. There are some other useful option like chaning the restart policy to "Never" or "Allocate a buffer for STDIN".
– Dr. Fabien Tarrade
Nov 29 '18 at 20:37
add a comment |
The docker image is too big to run on Cloud Shell. You might run it on Kubernetes or Compute Engine instead, but since you're still in the early stages and you've already said you can run the tools you need locally, then this might not be necessary for your needs. Looking into the future, when you're more concerned with performance, you might want to consider a solution such as Cloud ML Engine or BigQuery ML.
answered Nov 27 '18 at 11:01
Christopher PChristopher P
58827
Thanks, this is what I did by using a Compute Engine. You will have to check that even when adding a 20 GB disk in your custom VM that you have still enought disk space in "/mnt/stateful_partition" for your Docker image. There are some other useful option like chaning the restart policy to "Never" or "Allocate a buffer for STDIN".
– Dr. Fabien Tarrade
Nov 29 '18 at 20:37
add a comment |
The docker image is too big to run on Cloud Shell. You might run it on Kubernetes or Compute Engine instead, but since you're still in the early stages and you've already said you can run the tools you need locally, then this might not be necessary for your needs. Looking into the future, when you're more concerned with performance, you might want to consider a solution such as Cloud ML Engine or BigQuery ML.
answered Nov 27 '18 at 11:01
Christopher PChristopher P
58827
The docker image is too big to run on Cloud Shell. You might run it on Kubernetes or Compute Engine instead, but since you're still in the early stages and you've already said you can run the tools you need locally, then this might not be necessary for your needs. Looking into the future, when you're more concerned with performance, you might want to consider a solution such as Cloud ML Engine or BigQuery ML.
answered Nov 27 '18 at 11:01
Christopher PChristopher P
58827
answered Nov 27 '18 at 11:01
Christopher PChristopher P
58827
answered Nov 27 '18 at 11:01
Christopher PChristopher P
58827
answered Nov 27 '18 at 11:01
Christopher PChristopher P
58827
58827
Thanks, this is what I did by using a Compute Engine. You will have to check that even when adding a 20 GB disk in your custom VM that you have still enought disk space in "/mnt/stateful_partition" for your Docker image. There are some other useful option like chaning the restart policy to "Never" or "Allocate a buffer for STDIN".
– Dr. Fabien Tarrade
Nov 29 '18 at 20:37
add a comment |
Thanks, this is what I did by using a Compute Engine. You will have to check that even when adding a 20 GB disk in your custom VM that you have still enought disk space in "/mnt/stateful_partition" for your Docker image. There are some other useful option like chaning the restart policy to "Never" or "Allocate a buffer for STDIN".
– Dr. Fabien Tarrade
Nov 29 '18 at 20:37
Thanks, this is what I did by using a Compute Engine. You will have to check that even when adding a 20 GB disk in your custom VM that you have still enought disk space in "/mnt/stateful_partition" for your Docker image. There are some other useful option like chaning the restart policy to "Never" or "Allocate a buffer for STDIN".
– Dr. Fabien Tarrade
Nov 29 '18 at 20:37
Thanks, this is what I did by using a Compute Engine. You will have to check that even when adding a 20 GB disk in your custom VM that you have still enought disk space in "/mnt/stateful_partition" for your Docker image. There are some other useful option like chaning the restart policy to "Never" or "Allocate a buffer for STDIN".
– Dr. Fabien Tarrade
Nov 29 '18 at 20:37
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53414523%2fhow-to-run-a-big-docker-image-on-the-google-cloud-platform%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
6GB is ridiculously huge for a docker image. Does it really need to be that big?
– SiHa
Nov 21 '18 at 14:47
Question has nothing to do with
machine-learning- kindly do not spam the tag (removed).– desertnaut
Nov 21 '18 at 14:50
yes, this is huge. I have a acaconda env with a lot of packages and config file (spacy, nltk). Clearly this need to be optimize but this is not the question for the moment. Yes, this is not a ML topic but I want to know the best way to do exploration work on ML on the cloud. Focus is in ML, not sowtare dev on a 300 MB RedHat docker image. This why i added the tag but this was not a good. Thanks for the clean-up.
– Dr. Fabien Tarrade
Nov 21 '18 at 15:06
Cloud Shell provides you with 5 GB of total disk space. Why are you trying to run a container in Cloud Shell?
– John Hanley
Nov 21 '18 at 19:15
What is your proposal to develop a DL NLP code that need an Anaconda python env including a lot of ML packages like Tensorflow Keras, SHAP, LIME (and seems to be few GB, don't asked me why and I am looking into it as well) and for which the data are stored in BigQuery (200 GB). I am not talking about running the code in production. I am new on the Cloud. I know how I will do that on an Hadoop Data Lake. Looking forward for best pratice and proposal.
– Dr. Fabien Tarrade
Nov 21 '18 at 19:41