Passing tensorflow records one by one
up vote
0
down vote
favorite
The tensorflow record size is very huge 100 GB storing 1000 mini-batches.
But I will need 10000 mini-batches.
If a tensorflow record is created while the program is running replacing the old tensorflow record, how can I place the newly created record in the queue?
Do I need to have all the tensorflow records before running the tensorflow code or can I pass one by one, creating each when the last one is over.
tensorflow record
asked Nov 9 at 12:33
bemma
11
add a comment |
up vote
0
down vote
favorite
The tensorflow record size is very huge 100 GB storing 1000 mini-batches.
But I will need 10000 mini-batches.
If a tensorflow record is created while the program is running replacing the old tensorflow record, how can I place the newly created record in the queue?
Do I need to have all the tensorflow records before running the tensorflow code or can I pass one by one, creating each when the last one is over.
tensorflow record
asked Nov 9 at 12:33
bemma
11
You could have an initializable iterator and use a placeholder as records file name. Then you can generate one file, initialize iterator, train with that, generate second file, initialize, train, etc.
– jdehesa
Nov 9 at 13:45
Thank you very much @jdehesa. It is very helpful.
– bemma
Nov 9 at 14:34
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
The tensorflow record size is very huge 100 GB storing 1000 mini-batches.
But I will need 10000 mini-batches.
If a tensorflow record is created while the program is running replacing the old tensorflow record, how can I place the newly created record in the queue?
Do I need to have all the tensorflow records before running the tensorflow code or can I pass one by one, creating each when the last one is over.
tensorflow record
asked Nov 9 at 12:33
bemma
11
The tensorflow record size is very huge 100 GB storing 1000 mini-batches.
But I will need 10000 mini-batches.
If a tensorflow record is created while the program is running replacing the old tensorflow record, how can I place the newly created record in the queue?
Do I need to have all the tensorflow records before running the tensorflow code or can I pass one by one, creating each when the last one is over.
tensorflow record
tensorflow record
asked Nov 9 at 12:33
bemma
11
asked Nov 9 at 12:33
bemma
11
asked Nov 9 at 12:33
bemma
11
asked Nov 9 at 12:33
bemma
11
asked Nov 9 at 12:33
bemma
11
11
You could have an initializable iterator and use a placeholder as records file name. Then you can generate one file, initialize iterator, train with that, generate second file, initialize, train, etc.
– jdehesa
Nov 9 at 13:45
Thank you very much @jdehesa. It is very helpful.
– bemma
Nov 9 at 14:34
add a comment |
You could have an initializable iterator and use a placeholder as records file name. Then you can generate one file, initialize iterator, train with that, generate second file, initialize, train, etc.
– jdehesa
Nov 9 at 13:45
Thank you very much @jdehesa. It is very helpful.
– bemma
Nov 9 at 14:34
You could have an initializable iterator and use a placeholder as records file name. Then you can generate one file, initialize iterator, train with that, generate second file, initialize, train, etc.
– jdehesa
Nov 9 at 13:45
You could have an initializable iterator and use a placeholder as records file name. Then you can generate one file, initialize iterator, train with that, generate second file, initialize, train, etc.
– jdehesa
Nov 9 at 13:45
Thank you very much @jdehesa. It is very helpful.
– bemma
Nov 9 at 14:34
Thank you very much @jdehesa. It is very helpful.
– bemma
Nov 9 at 14:34
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53225804%2fpassing-tensorflow-records-one-by-one%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You could have an initializable iterator and use a placeholder as records file name. Then you can generate one file, initialize iterator, train with that, generate second file, initialize, train, etc.
– jdehesa
Nov 9 at 13:45
Thank you very much @jdehesa. It is very helpful.
– bemma
Nov 9 at 14:34