Image preprocessing with Dataflow
Task: I am to run an ETL job that will Extract TIFF images from GCS, Transform those images to text with a combination of open source computer vision tools such as OpenCV + Tesseract and ultimately Load the data into BigQuery
Problem: I am trying to use Dataflow to perform the ETL job because I have millions of images (each image is a separate file/blob) and I want to scale to hundreds of machines. However, I am running into some problems with Dataflow (which will be described in greater detail below) regarding the best means to download the images.
Questions: Ultimately I am trying to determine:
1) Is dataflow the best solution to do this? Alternatives that I have considered are running a multithreaded job on a large machine. Are there any other alternatives that I should be considering?
2) If dataflow is the best solution, then how should I specifically handle downloading the millions of images (so that I can run them through a transformation)?
Technical challenges:
The following post Recommended solution recommends using beam.io.gcp.gcsio.GcsIO().open(filepath, 'r') in a DoFn to download images from GCS.
I've attempted going down this path, using beam.io.gcp.gcsio.GcsIO().open(filepath, 'r'), however, I am having trouble open the images. That issue is described here: IO.BufferReader issue.
When using the DirectRunner I can download the image files using this client api from google.cloud import storage and I can open and pre-process the images no problem. However, when using the dataflow runner, I am having dependency issues, AttributeError: 'module' object has no attribute 'storage'.
That being said, If Dataflow is the best solution, what is the best method to download and process millions of images?
python image-processing google-cloud-dataflow apache-beam
edited Nov 21 '18 at 19:24
oakinlaja
1816
asked Nov 20 '18 at 15:12
Ryan StackRyan Stack
361213
add a comment |
Task: I am to run an ETL job that will Extract TIFF images from GCS, Transform those images to text with a combination of open source computer vision tools such as OpenCV + Tesseract and ultimately Load the data into BigQuery
Problem: I am trying to use Dataflow to perform the ETL job because I have millions of images (each image is a separate file/blob) and I want to scale to hundreds of machines. However, I am running into some problems with Dataflow (which will be described in greater detail below) regarding the best means to download the images.
Questions: Ultimately I am trying to determine:
1) Is dataflow the best solution to do this? Alternatives that I have considered are running a multithreaded job on a large machine. Are there any other alternatives that I should be considering?
2) If dataflow is the best solution, then how should I specifically handle downloading the millions of images (so that I can run them through a transformation)?
Technical challenges:
The following post Recommended solution recommends using beam.io.gcp.gcsio.GcsIO().open(filepath, 'r') in a DoFn to download images from GCS.
I've attempted going down this path, using beam.io.gcp.gcsio.GcsIO().open(filepath, 'r'), however, I am having trouble open the images. That issue is described here: IO.BufferReader issue.
When using the DirectRunner I can download the image files using this client api from google.cloud import storage and I can open and pre-process the images no problem. However, when using the dataflow runner, I am having dependency issues, AttributeError: 'module' object has no attribute 'storage'.
That being said, If Dataflow is the best solution, what is the best method to download and process millions of images?
python image-processing google-cloud-dataflow apache-beam
edited Nov 21 '18 at 19:24
oakinlaja
1816
asked Nov 20 '18 at 15:12
Ryan StackRyan Stack
361213
add a comment |
Task: I am to run an ETL job that will Extract TIFF images from GCS, Transform those images to text with a combination of open source computer vision tools such as OpenCV + Tesseract and ultimately Load the data into BigQuery
Problem: I am trying to use Dataflow to perform the ETL job because I have millions of images (each image is a separate file/blob) and I want to scale to hundreds of machines. However, I am running into some problems with Dataflow (which will be described in greater detail below) regarding the best means to download the images.
Questions: Ultimately I am trying to determine:
1) Is dataflow the best solution to do this? Alternatives that I have considered are running a multithreaded job on a large machine. Are there any other alternatives that I should be considering?
2) If dataflow is the best solution, then how should I specifically handle downloading the millions of images (so that I can run them through a transformation)?
Technical challenges:
The following post Recommended solution recommends using beam.io.gcp.gcsio.GcsIO().open(filepath, 'r') in a DoFn to download images from GCS.
I've attempted going down this path, using beam.io.gcp.gcsio.GcsIO().open(filepath, 'r'), however, I am having trouble open the images. That issue is described here: IO.BufferReader issue.
When using the DirectRunner I can download the image files using this client api from google.cloud import storage and I can open and pre-process the images no problem. However, when using the dataflow runner, I am having dependency issues, AttributeError: 'module' object has no attribute 'storage'.
That being said, If Dataflow is the best solution, what is the best method to download and process millions of images?
python image-processing google-cloud-dataflow apache-beam
edited Nov 21 '18 at 19:24
oakinlaja
1816
asked Nov 20 '18 at 15:12
Ryan StackRyan Stack
361213
Task: I am to run an ETL job that will Extract TIFF images from GCS, Transform those images to text with a combination of open source computer vision tools such as OpenCV + Tesseract and ultimately Load the data into BigQuery
Problem: I am trying to use Dataflow to perform the ETL job because I have millions of images (each image is a separate file/blob) and I want to scale to hundreds of machines. However, I am running into some problems with Dataflow (which will be described in greater detail below) regarding the best means to download the images.
Questions: Ultimately I am trying to determine:
1) Is dataflow the best solution to do this? Alternatives that I have considered are running a multithreaded job on a large machine. Are there any other alternatives that I should be considering?
2) If dataflow is the best solution, then how should I specifically handle downloading the millions of images (so that I can run them through a transformation)?
Technical challenges:
The following post Recommended solution recommends using beam.io.gcp.gcsio.GcsIO().open(filepath, 'r') in a DoFn to download images from GCS.
I've attempted going down this path, using beam.io.gcp.gcsio.GcsIO().open(filepath, 'r'), however, I am having trouble open the images. That issue is described here: IO.BufferReader issue.
When using the DirectRunner I can download the image files using this client api from google.cloud import storage and I can open and pre-process the images no problem. However, when using the dataflow runner, I am having dependency issues, AttributeError: 'module' object has no attribute 'storage'.
That being said, If Dataflow is the best solution, what is the best method to download and process millions of images?
python image-processing google-cloud-dataflow apache-beam
python image-processing google-cloud-dataflow apache-beam
edited Nov 21 '18 at 19:24
oakinlaja
1816
asked Nov 20 '18 at 15:12
Ryan StackRyan Stack
361213
edited Nov 21 '18 at 19:24
oakinlaja
1816
asked Nov 20 '18 at 15:12
Ryan StackRyan Stack
361213
edited Nov 21 '18 at 19:24
oakinlaja
1816
edited Nov 21 '18 at 19:24
oakinlaja
1816
edited Nov 21 '18 at 19:24
oakinlaja
1816
1816
asked Nov 20 '18 at 15:12
Ryan StackRyan Stack
361213
asked Nov 20 '18 at 15:12
Ryan StackRyan Stack
361213
asked Nov 20 '18 at 15:12
Ryan StackRyan Stack
361213
361213
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53395999%2fimage-preprocessing-with-dataflow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53395999%2fimage-preprocessing-with-dataflow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
