Question about high level architecture required to process and visualize fitness app data (From Apple Health...
I'm working on a project where I am tasked to use google cloud services to process and visualize fitness data. For example, I have exported some apple health data from my watch, and it is in .xml format. From a high level, I envision this .xml file starting off in object storage, and being converted to .csv through a cloud function (triggered by the creation of the .xml object in storage) and stored again in object storage (different bucket). Then I see these .csv files being processed by a DataFlow pipeline, which will reformat the data to the template schema that I would like the data to be organized with. This pipeline will output the resultant .csv to BigQuery, which will then be designated as a data source for Data Studio. I will then configure Data Studio to produce some simple reports that compare the health data to recommended values. I would like for this report to be accessible as a .pdf in object storage potentially as well. Am I on the right track, or am I missing some key services to accomplish this?
Also, I'm new to posting on StackOverflow, so if this question is against the rules or not welcome, please let me know.
Any feedback is greatly appreciated, as I have not been able to bounce these ideas off of other experienced cloud architects/developers.
google-cloud-platform
edited Nov 21 '18 at 20:08
Mikhail Berlyant
62.6k43774
asked Nov 21 '18 at 19:17
Clay BarcusClay Barcus
62
add a comment |
I'm working on a project where I am tasked to use google cloud services to process and visualize fitness data. For example, I have exported some apple health data from my watch, and it is in .xml format. From a high level, I envision this .xml file starting off in object storage, and being converted to .csv through a cloud function (triggered by the creation of the .xml object in storage) and stored again in object storage (different bucket). Then I see these .csv files being processed by a DataFlow pipeline, which will reformat the data to the template schema that I would like the data to be organized with. This pipeline will output the resultant .csv to BigQuery, which will then be designated as a data source for Data Studio. I will then configure Data Studio to produce some simple reports that compare the health data to recommended values. I would like for this report to be accessible as a .pdf in object storage potentially as well. Am I on the right track, or am I missing some key services to accomplish this?
Also, I'm new to posting on StackOverflow, so if this question is against the rules or not welcome, please let me know.
Any feedback is greatly appreciated, as I have not been able to bounce these ideas off of other experienced cloud architects/developers.
google-cloud-platform
edited Nov 21 '18 at 20:08
Mikhail Berlyant
62.6k43774
asked Nov 21 '18 at 19:17
Clay BarcusClay Barcus
62
add a comment |
I'm working on a project where I am tasked to use google cloud services to process and visualize fitness data. For example, I have exported some apple health data from my watch, and it is in .xml format. From a high level, I envision this .xml file starting off in object storage, and being converted to .csv through a cloud function (triggered by the creation of the .xml object in storage) and stored again in object storage (different bucket). Then I see these .csv files being processed by a DataFlow pipeline, which will reformat the data to the template schema that I would like the data to be organized with. This pipeline will output the resultant .csv to BigQuery, which will then be designated as a data source for Data Studio. I will then configure Data Studio to produce some simple reports that compare the health data to recommended values. I would like for this report to be accessible as a .pdf in object storage potentially as well. Am I on the right track, or am I missing some key services to accomplish this?
Also, I'm new to posting on StackOverflow, so if this question is against the rules or not welcome, please let me know.
Any feedback is greatly appreciated, as I have not been able to bounce these ideas off of other experienced cloud architects/developers.
google-cloud-platform
edited Nov 21 '18 at 20:08
Mikhail Berlyant
62.6k43774
asked Nov 21 '18 at 19:17
Clay BarcusClay Barcus
62
I'm working on a project where I am tasked to use google cloud services to process and visualize fitness data. For example, I have exported some apple health data from my watch, and it is in .xml format. From a high level, I envision this .xml file starting off in object storage, and being converted to .csv through a cloud function (triggered by the creation of the .xml object in storage) and stored again in object storage (different bucket). Then I see these .csv files being processed by a DataFlow pipeline, which will reformat the data to the template schema that I would like the data to be organized with. This pipeline will output the resultant .csv to BigQuery, which will then be designated as a data source for Data Studio. I will then configure Data Studio to produce some simple reports that compare the health data to recommended values. I would like for this report to be accessible as a .pdf in object storage potentially as well. Am I on the right track, or am I missing some key services to accomplish this?
Also, I'm new to posting on StackOverflow, so if this question is against the rules or not welcome, please let me know.
Any feedback is greatly appreciated, as I have not been able to bounce these ideas off of other experienced cloud architects/developers.
google-cloud-platform
google-cloud-platform
edited Nov 21 '18 at 20:08
Mikhail Berlyant
62.6k43774
asked Nov 21 '18 at 19:17
Clay BarcusClay Barcus
62
edited Nov 21 '18 at 20:08
Mikhail Berlyant
62.6k43774
asked Nov 21 '18 at 19:17
Clay BarcusClay Barcus
62
edited Nov 21 '18 at 20:08
Mikhail Berlyant
62.6k43774
edited Nov 21 '18 at 20:08
Mikhail Berlyant
62.6k43774
edited Nov 21 '18 at 20:08
Mikhail Berlyant
62.6k43774
62.6k43774
asked Nov 21 '18 at 19:17
Clay BarcusClay Barcus
62
asked Nov 21 '18 at 19:17
Clay BarcusClay Barcus
62
asked Nov 21 '18 at 19:17
Clay BarcusClay Barcus
62
62
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
This question is currently off-topics by the rule of StackOverflow, as it does not contain any problems to resolve. See point 4-5.
As a high-level advice, I do not see why it should not be possible based on the services you mentioned but you would need to implement it and try it on your side and evaluate the features of each service in your workflow.
In terms of solution or architecture advice, those are generally paid services and you would most likely find little help here for those unless you have a specific problem to solve with said services. You might find some help on the internet as well. ie.Cloud Solutions, Built it on GCP, etc
You might find this interesting to review as well as it mimics your solution. Hope this helps.
answered Nov 22 '18 at 1:08
AlexAlex
463
Ok, thanks for pointing that out. I will make sure there is an actual problem to resolve moving forward.
– Clay Barcus
Nov 22 '18 at 19:53
Also, Thank you for the resources! They are very helpful and honestly they answer my very broad/general questions
– Clay Barcus
Nov 22 '18 at 19:54
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53419116%2fquestion-about-high-level-architecture-required-to-process-and-visualize-fitness%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This question is currently off-topics by the rule of StackOverflow, as it does not contain any problems to resolve. See point 4-5.
As a high-level advice, I do not see why it should not be possible based on the services you mentioned but you would need to implement it and try it on your side and evaluate the features of each service in your workflow.
In terms of solution or architecture advice, those are generally paid services and you would most likely find little help here for those unless you have a specific problem to solve with said services. You might find some help on the internet as well. ie.Cloud Solutions, Built it on GCP, etc
You might find this interesting to review as well as it mimics your solution. Hope this helps.
answered Nov 22 '18 at 1:08
AlexAlex
463
Ok, thanks for pointing that out. I will make sure there is an actual problem to resolve moving forward.
– Clay Barcus
Nov 22 '18 at 19:53
Also, Thank you for the resources! They are very helpful and honestly they answer my very broad/general questions
– Clay Barcus
Nov 22 '18 at 19:54
add a comment |
This question is currently off-topics by the rule of StackOverflow, as it does not contain any problems to resolve. See point 4-5.
As a high-level advice, I do not see why it should not be possible based on the services you mentioned but you would need to implement it and try it on your side and evaluate the features of each service in your workflow.
In terms of solution or architecture advice, those are generally paid services and you would most likely find little help here for those unless you have a specific problem to solve with said services. You might find some help on the internet as well. ie.Cloud Solutions, Built it on GCP, etc
You might find this interesting to review as well as it mimics your solution. Hope this helps.
answered Nov 22 '18 at 1:08
AlexAlex
463
Ok, thanks for pointing that out. I will make sure there is an actual problem to resolve moving forward.
– Clay Barcus
Nov 22 '18 at 19:53
Also, Thank you for the resources! They are very helpful and honestly they answer my very broad/general questions
– Clay Barcus
Nov 22 '18 at 19:54
add a comment |
This question is currently off-topics by the rule of StackOverflow, as it does not contain any problems to resolve. See point 4-5.
As a high-level advice, I do not see why it should not be possible based on the services you mentioned but you would need to implement it and try it on your side and evaluate the features of each service in your workflow.
In terms of solution or architecture advice, those are generally paid services and you would most likely find little help here for those unless you have a specific problem to solve with said services. You might find some help on the internet as well. ie.Cloud Solutions, Built it on GCP, etc
You might find this interesting to review as well as it mimics your solution. Hope this helps.
answered Nov 22 '18 at 1:08
AlexAlex
463
This question is currently off-topics by the rule of StackOverflow, as it does not contain any problems to resolve. See point 4-5.
As a high-level advice, I do not see why it should not be possible based on the services you mentioned but you would need to implement it and try it on your side and evaluate the features of each service in your workflow.
In terms of solution or architecture advice, those are generally paid services and you would most likely find little help here for those unless you have a specific problem to solve with said services. You might find some help on the internet as well. ie.Cloud Solutions, Built it on GCP, etc
You might find this interesting to review as well as it mimics your solution. Hope this helps.
answered Nov 22 '18 at 1:08
AlexAlex
463
answered Nov 22 '18 at 1:08
AlexAlex
463
answered Nov 22 '18 at 1:08
AlexAlex
463
answered Nov 22 '18 at 1:08
AlexAlex
463
463
Ok, thanks for pointing that out. I will make sure there is an actual problem to resolve moving forward.
– Clay Barcus
Nov 22 '18 at 19:53
Also, Thank you for the resources! They are very helpful and honestly they answer my very broad/general questions
– Clay Barcus
Nov 22 '18 at 19:54
add a comment |
Ok, thanks for pointing that out. I will make sure there is an actual problem to resolve moving forward.
– Clay Barcus
Nov 22 '18 at 19:53
Also, Thank you for the resources! They are very helpful and honestly they answer my very broad/general questions
– Clay Barcus
Nov 22 '18 at 19:54
Ok, thanks for pointing that out. I will make sure there is an actual problem to resolve moving forward.
– Clay Barcus
Nov 22 '18 at 19:53
Ok, thanks for pointing that out. I will make sure there is an actual problem to resolve moving forward.
– Clay Barcus
Nov 22 '18 at 19:53
Also, Thank you for the resources! They are very helpful and honestly they answer my very broad/general questions
– Clay Barcus
Nov 22 '18 at 19:54
Also, Thank you for the resources! They are very helpful and honestly they answer my very broad/general questions
– Clay Barcus
Nov 22 '18 at 19:54
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53419116%2fquestion-about-high-level-architecture-required-to-process-and-visualize-fitness%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
