How to Optimize Google Big Query Bytes Billed
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I have recently discovered Google Big Query and it's open datasets. Upon performing the following query on the 311_service requests table in the new_york dataset, the cloud console reports the bytes billed to be 130 MB.
SQL Query:
SELECT unique_key FROM `bigquery-public-data.new_york.311_service_requests` LIMIT 10
Query Returns:
+------+-------------+
| Rows | unique_key |
+------+-------------+
| 1 | 37911459 |
| 2 | 38162601 |
| 3 | 32560181 |
| 4 | 38259076 |
| 5 | 36034528 |
| 6 | 36975822 |
| 7 | 38028455 |
| 8 | 37993135 |
| 9 | 37988664 |
| 10 | 35382611 |
+------+-------------+
For a query returning such a small amount of data, why is the bytes billed valued at 130 MB?
Is there a way to optimize this? Should the results of a query be stored in another database for later retrieval?
sql google-bigquery
asked Nov 21 '18 at 22:13
craydencrayden
4181621
add a comment |
I have recently discovered Google Big Query and it's open datasets. Upon performing the following query on the 311_service requests table in the new_york dataset, the cloud console reports the bytes billed to be 130 MB.
SQL Query:
SELECT unique_key FROM `bigquery-public-data.new_york.311_service_requests` LIMIT 10
Query Returns:
+------+-------------+
| Rows | unique_key |
+------+-------------+
| 1 | 37911459 |
| 2 | 38162601 |
| 3 | 32560181 |
| 4 | 38259076 |
| 5 | 36034528 |
| 6 | 36975822 |
| 7 | 38028455 |
| 8 | 37993135 |
| 9 | 37988664 |
| 10 | 35382611 |
+------+-------------+
For a query returning such a small amount of data, why is the bytes billed valued at 130 MB?
Is there a way to optimize this? Should the results of a query be stored in another database for later retrieval?
sql google-bigquery
asked Nov 21 '18 at 22:13
craydencrayden
4181621
add a comment |
I have recently discovered Google Big Query and it's open datasets. Upon performing the following query on the 311_service requests table in the new_york dataset, the cloud console reports the bytes billed to be 130 MB.
SQL Query:
SELECT unique_key FROM `bigquery-public-data.new_york.311_service_requests` LIMIT 10
Query Returns:
+------+-------------+
| Rows | unique_key |
+------+-------------+
| 1 | 37911459 |
| 2 | 38162601 |
| 3 | 32560181 |
| 4 | 38259076 |
| 5 | 36034528 |
| 6 | 36975822 |
| 7 | 38028455 |
| 8 | 37993135 |
| 9 | 37988664 |
| 10 | 35382611 |
+------+-------------+
For a query returning such a small amount of data, why is the bytes billed valued at 130 MB?
Is there a way to optimize this? Should the results of a query be stored in another database for later retrieval?
sql google-bigquery
asked Nov 21 '18 at 22:13
craydencrayden
4181621
I have recently discovered Google Big Query and it's open datasets. Upon performing the following query on the 311_service requests table in the new_york dataset, the cloud console reports the bytes billed to be 130 MB.
SQL Query:
SELECT unique_key FROM `bigquery-public-data.new_york.311_service_requests` LIMIT 10
Query Returns:
+------+-------------+
| Rows | unique_key |
+------+-------------+
| 1 | 37911459 |
| 2 | 38162601 |
| 3 | 32560181 |
| 4 | 38259076 |
| 5 | 36034528 |
| 6 | 36975822 |
| 7 | 38028455 |
| 8 | 37993135 |
| 9 | 37988664 |
| 10 | 35382611 |
+------+-------------+
For a query returning such a small amount of data, why is the bytes billed valued at 130 MB?
Is there a way to optimize this? Should the results of a query be stored in another database for later retrieval?
sql google-bigquery
sql google-bigquery
asked Nov 21 '18 at 22:13
craydencrayden
4181621
asked Nov 21 '18 at 22:13
craydencrayden
4181621
asked Nov 21 '18 at 22:13
craydencrayden
4181621
asked Nov 21 '18 at 22:13
craydencrayden
4181621
asked Nov 21 '18 at 22:13
craydencrayden
4181621
4181621
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
why is the bytes billed valued at 130 MB?
Query pricing refers to the cost of running your SQL commands and user-defined functions. BigQuery charges for queries by using one metric: the number of bytes processed (also referred to as bytes read). You are charged for the number of bytes processed whether the data is stored in BigQuery or in an external data source such as Cloud Storage, Google Drive, or Cloud Bigtable.
When you run a query, you're charged according to the total data processed in the columns you select, even if you set an explicit LIMIT on the results. The total bytes per column is calculated based on the types of data in the column. For more information about how we calculate your data size, see Data size calculation.
Query pricing is based on your usage pattern: a monthly flat rate for queries or pricing based on interactive queries. Enterprise customers generally prefer flat-rate pricing for queries because that model offers consistent month-to-month costs. On-demand (or interactive) pricing offers flexibility and is based solely on usage.
You can see more at https://cloud.google.com/bigquery/pricing
So, in your case 130MB is the size of the respective unique_key column
Should the results of a query be stored in another database for later retrieval?
sure
You can do so to manage cost for consecutive processing of that small data w/o touching the original one
Have in mind - this will invoke storage price for you - see same above mentioned link for details
answered Nov 21 '18 at 22:21
Mikhail BerlyantMikhail Berlyant
62.9k43874
Thanks Mikhail. Are there any other querying techniques that would result in lower costs?
– crayden
Nov 21 '18 at 23:03
Sure - you can check BigQuery Best Practices: Controlling Costs and in specific use case you will get into - just post new question and we will help you
– Mikhail Berlyant
Nov 21 '18 at 23:16
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421181%2fhow-to-optimize-google-big-query-bytes-billed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
why is the bytes billed valued at 130 MB?
Query pricing refers to the cost of running your SQL commands and user-defined functions. BigQuery charges for queries by using one metric: the number of bytes processed (also referred to as bytes read). You are charged for the number of bytes processed whether the data is stored in BigQuery or in an external data source such as Cloud Storage, Google Drive, or Cloud Bigtable.
When you run a query, you're charged according to the total data processed in the columns you select, even if you set an explicit LIMIT on the results. The total bytes per column is calculated based on the types of data in the column. For more information about how we calculate your data size, see Data size calculation.
Query pricing is based on your usage pattern: a monthly flat rate for queries or pricing based on interactive queries. Enterprise customers generally prefer flat-rate pricing for queries because that model offers consistent month-to-month costs. On-demand (or interactive) pricing offers flexibility and is based solely on usage.
You can see more at https://cloud.google.com/bigquery/pricing
So, in your case 130MB is the size of the respective unique_key column
Should the results of a query be stored in another database for later retrieval?
sure
You can do so to manage cost for consecutive processing of that small data w/o touching the original one
Have in mind - this will invoke storage price for you - see same above mentioned link for details
answered Nov 21 '18 at 22:21
Mikhail BerlyantMikhail Berlyant
62.9k43874
Thanks Mikhail. Are there any other querying techniques that would result in lower costs?
– crayden
Nov 21 '18 at 23:03
Sure - you can check BigQuery Best Practices: Controlling Costs and in specific use case you will get into - just post new question and we will help you
– Mikhail Berlyant
Nov 21 '18 at 23:16
add a comment |
why is the bytes billed valued at 130 MB?
Query pricing refers to the cost of running your SQL commands and user-defined functions. BigQuery charges for queries by using one metric: the number of bytes processed (also referred to as bytes read). You are charged for the number of bytes processed whether the data is stored in BigQuery or in an external data source such as Cloud Storage, Google Drive, or Cloud Bigtable.
When you run a query, you're charged according to the total data processed in the columns you select, even if you set an explicit LIMIT on the results. The total bytes per column is calculated based on the types of data in the column. For more information about how we calculate your data size, see Data size calculation.
Query pricing is based on your usage pattern: a monthly flat rate for queries or pricing based on interactive queries. Enterprise customers generally prefer flat-rate pricing for queries because that model offers consistent month-to-month costs. On-demand (or interactive) pricing offers flexibility and is based solely on usage.
You can see more at https://cloud.google.com/bigquery/pricing
So, in your case 130MB is the size of the respective unique_key column
Should the results of a query be stored in another database for later retrieval?
sure
You can do so to manage cost for consecutive processing of that small data w/o touching the original one
Have in mind - this will invoke storage price for you - see same above mentioned link for details
answered Nov 21 '18 at 22:21
Mikhail BerlyantMikhail Berlyant
62.9k43874
Thanks Mikhail. Are there any other querying techniques that would result in lower costs?
– crayden
Nov 21 '18 at 23:03
Sure - you can check BigQuery Best Practices: Controlling Costs and in specific use case you will get into - just post new question and we will help you
– Mikhail Berlyant
Nov 21 '18 at 23:16
add a comment |
why is the bytes billed valued at 130 MB?
Query pricing refers to the cost of running your SQL commands and user-defined functions. BigQuery charges for queries by using one metric: the number of bytes processed (also referred to as bytes read). You are charged for the number of bytes processed whether the data is stored in BigQuery or in an external data source such as Cloud Storage, Google Drive, or Cloud Bigtable.
When you run a query, you're charged according to the total data processed in the columns you select, even if you set an explicit LIMIT on the results. The total bytes per column is calculated based on the types of data in the column. For more information about how we calculate your data size, see Data size calculation.
Query pricing is based on your usage pattern: a monthly flat rate for queries or pricing based on interactive queries. Enterprise customers generally prefer flat-rate pricing for queries because that model offers consistent month-to-month costs. On-demand (or interactive) pricing offers flexibility and is based solely on usage.
You can see more at https://cloud.google.com/bigquery/pricing
So, in your case 130MB is the size of the respective unique_key column
Should the results of a query be stored in another database for later retrieval?
sure
You can do so to manage cost for consecutive processing of that small data w/o touching the original one
Have in mind - this will invoke storage price for you - see same above mentioned link for details
answered Nov 21 '18 at 22:21
Mikhail BerlyantMikhail Berlyant
62.9k43874
why is the bytes billed valued at 130 MB?
Query pricing refers to the cost of running your SQL commands and user-defined functions. BigQuery charges for queries by using one metric: the number of bytes processed (also referred to as bytes read). You are charged for the number of bytes processed whether the data is stored in BigQuery or in an external data source such as Cloud Storage, Google Drive, or Cloud Bigtable.
When you run a query, you're charged according to the total data processed in the columns you select, even if you set an explicit LIMIT on the results. The total bytes per column is calculated based on the types of data in the column. For more information about how we calculate your data size, see Data size calculation.
Query pricing is based on your usage pattern: a monthly flat rate for queries or pricing based on interactive queries. Enterprise customers generally prefer flat-rate pricing for queries because that model offers consistent month-to-month costs. On-demand (or interactive) pricing offers flexibility and is based solely on usage.
You can see more at https://cloud.google.com/bigquery/pricing
So, in your case 130MB is the size of the respective unique_key column
Should the results of a query be stored in another database for later retrieval?
sure
You can do so to manage cost for consecutive processing of that small data w/o touching the original one
Have in mind - this will invoke storage price for you - see same above mentioned link for details
answered Nov 21 '18 at 22:21
Mikhail BerlyantMikhail Berlyant
62.9k43874
answered Nov 21 '18 at 22:21
Mikhail BerlyantMikhail Berlyant
62.9k43874
answered Nov 21 '18 at 22:21
Mikhail BerlyantMikhail Berlyant
62.9k43874
answered Nov 21 '18 at 22:21
Mikhail BerlyantMikhail Berlyant
62.9k43874
62.9k43874
Thanks Mikhail. Are there any other querying techniques that would result in lower costs?
– crayden
Nov 21 '18 at 23:03
Sure - you can check BigQuery Best Practices: Controlling Costs and in specific use case you will get into - just post new question and we will help you
– Mikhail Berlyant
Nov 21 '18 at 23:16
add a comment |
Thanks Mikhail. Are there any other querying techniques that would result in lower costs?
– crayden
Nov 21 '18 at 23:03
Sure - you can check BigQuery Best Practices: Controlling Costs and in specific use case you will get into - just post new question and we will help you
– Mikhail Berlyant
Nov 21 '18 at 23:16
Thanks Mikhail. Are there any other querying techniques that would result in lower costs?
– crayden
Nov 21 '18 at 23:03
Thanks Mikhail. Are there any other querying techniques that would result in lower costs?
– crayden
Nov 21 '18 at 23:03
Sure - you can check BigQuery Best Practices: Controlling Costs and in specific use case you will get into - just post new question and we will help you
– Mikhail Berlyant
Nov 21 '18 at 23:16
Sure - you can check BigQuery Best Practices: Controlling Costs and in specific use case you will get into - just post new question and we will help you
– Mikhail Berlyant
Nov 21 '18 at 23:16
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421181%2fhow-to-optimize-google-big-query-bytes-billed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
