Why does memory leak after large file download in Ruby?
I have some Ruby that is downloading a large XML file (~ 380MB).
I'm using on_body to download it in chunks:
to = File.open('/path', 'wb')
http = HTTPI::Request.new(options)
http.url = uri
http.on_body { |chunk| to.write(chunk) }
HTTPI.get(http, :net_http)
to.close
on_body, according to its documentation (https://www.rubydoc.info/gems/httpi/HTTPI/Request#on_body-instance_method) will read the response in chunks, to prevent filling up memory.
This seems to work, in practice my memory usage jumps up about ~178 MB after downloading the file, so some garbage collection appears to happen while the file is being streamed, or else I would expect it to jump by the full file size.
However, afterwords, that 178MB just hangs around forever.
I've narrowed it down to being due to the file download. If I read the same XML file from disk, memory has only increased by about 4MB after I'm finished processing the file.
ruby performance memory memory-leaks
asked Nov 15 '18 at 16:47
Zachary WrightZachary Wright
14.7k93546
|
show 2 more comments
I have some Ruby that is downloading a large XML file (~ 380MB).
I'm using on_body to download it in chunks:
to = File.open('/path', 'wb')
http = HTTPI::Request.new(options)
http.url = uri
http.on_body { |chunk| to.write(chunk) }
HTTPI.get(http, :net_http)
to.close
on_body, according to its documentation (https://www.rubydoc.info/gems/httpi/HTTPI/Request#on_body-instance_method) will read the response in chunks, to prevent filling up memory.
This seems to work, in practice my memory usage jumps up about ~178 MB after downloading the file, so some garbage collection appears to happen while the file is being streamed, or else I would expect it to jump by the full file size.
However, afterwords, that 178MB just hangs around forever.
I've narrowed it down to being due to the file download. If I read the same XML file from disk, memory has only increased by about 4MB after I'm finished processing the file.
ruby performance memory memory-leaks
asked Nov 15 '18 at 16:47
Zachary WrightZachary Wright
14.7k93546
because you didn't close the file ??? try addingto.close
– engineersmnky
Nov 15 '18 at 17:09
@engineersmnky I should have included that in the code sample - I do close it. (I've added it to the sample now)
– Zachary Wright
Nov 15 '18 at 17:17
For what it's worth, closing vs not closing doesn't have any impact on the memory usage.
– Zachary Wright
Nov 15 '18 at 17:24
Are you sure it hangs around forever maybe GC just hasn't gotten to it yet
– engineersmnky
Nov 15 '18 at 17:25
1
Garbage collection frees memory for re-use by Ruby. Releasing memory to the OS is something else. sitepoint.com/ruby-uses-memory explains it better than I can.
– steenslag
Nov 15 '18 at 19:56
|
show 2 more comments
I have some Ruby that is downloading a large XML file (~ 380MB).
I'm using on_body to download it in chunks:
to = File.open('/path', 'wb')
http = HTTPI::Request.new(options)
http.url = uri
http.on_body { |chunk| to.write(chunk) }
HTTPI.get(http, :net_http)
to.close
on_body, according to its documentation (https://www.rubydoc.info/gems/httpi/HTTPI/Request#on_body-instance_method) will read the response in chunks, to prevent filling up memory.
This seems to work, in practice my memory usage jumps up about ~178 MB after downloading the file, so some garbage collection appears to happen while the file is being streamed, or else I would expect it to jump by the full file size.
However, afterwords, that 178MB just hangs around forever.
I've narrowed it down to being due to the file download. If I read the same XML file from disk, memory has only increased by about 4MB after I'm finished processing the file.
ruby performance memory memory-leaks
asked Nov 15 '18 at 16:47
Zachary WrightZachary Wright
14.7k93546
I have some Ruby that is downloading a large XML file (~ 380MB).
I'm using on_body to download it in chunks:
to = File.open('/path', 'wb')
http = HTTPI::Request.new(options)
http.url = uri
http.on_body { |chunk| to.write(chunk) }
HTTPI.get(http, :net_http)
to.close
on_body, according to its documentation (https://www.rubydoc.info/gems/httpi/HTTPI/Request#on_body-instance_method) will read the response in chunks, to prevent filling up memory.
This seems to work, in practice my memory usage jumps up about ~178 MB after downloading the file, so some garbage collection appears to happen while the file is being streamed, or else I would expect it to jump by the full file size.
However, afterwords, that 178MB just hangs around forever.
I've narrowed it down to being due to the file download. If I read the same XML file from disk, memory has only increased by about 4MB after I'm finished processing the file.
ruby performance memory memory-leaks
ruby performance memory memory-leaks
asked Nov 15 '18 at 16:47
Zachary WrightZachary Wright
14.7k93546
asked Nov 15 '18 at 16:47
Zachary WrightZachary Wright
14.7k93546
edited Nov 15 '18 at 18:31
Zachary Wright
asked Nov 15 '18 at 16:47
Zachary WrightZachary Wright
14.7k93546
asked Nov 15 '18 at 16:47
Zachary WrightZachary Wright
14.7k93546
asked Nov 15 '18 at 16:47
Zachary WrightZachary Wright
14.7k93546
14.7k93546
because you didn't close the file ??? try addingto.close
– engineersmnky
Nov 15 '18 at 17:09
@engineersmnky I should have included that in the code sample - I do close it. (I've added it to the sample now)
– Zachary Wright
Nov 15 '18 at 17:17
For what it's worth, closing vs not closing doesn't have any impact on the memory usage.
– Zachary Wright
Nov 15 '18 at 17:24
Are you sure it hangs around forever maybe GC just hasn't gotten to it yet
– engineersmnky
Nov 15 '18 at 17:25
1
Garbage collection frees memory for re-use by Ruby. Releasing memory to the OS is something else. sitepoint.com/ruby-uses-memory explains it better than I can.
– steenslag
Nov 15 '18 at 19:56
|
show 2 more comments
because you didn't close the file ??? try addingto.close
– engineersmnky
Nov 15 '18 at 17:09
@engineersmnky I should have included that in the code sample - I do close it. (I've added it to the sample now)
– Zachary Wright
Nov 15 '18 at 17:17
For what it's worth, closing vs not closing doesn't have any impact on the memory usage.
– Zachary Wright
Nov 15 '18 at 17:24
Are you sure it hangs around forever maybe GC just hasn't gotten to it yet
– engineersmnky
Nov 15 '18 at 17:25
1
Garbage collection frees memory for re-use by Ruby. Releasing memory to the OS is something else. sitepoint.com/ruby-uses-memory explains it better than I can.
– steenslag
Nov 15 '18 at 19:56
because you didn't close the file ??? try adding
to.close– engineersmnky
Nov 15 '18 at 17:09
because you didn't close the file ??? try adding
to.close– engineersmnky
Nov 15 '18 at 17:09
@engineersmnky I should have included that in the code sample - I do close it. (I've added it to the sample now)
– Zachary Wright
Nov 15 '18 at 17:17
@engineersmnky I should have included that in the code sample - I do close it. (I've added it to the sample now)
– Zachary Wright
Nov 15 '18 at 17:17
For what it's worth, closing vs not closing doesn't have any impact on the memory usage.
– Zachary Wright
Nov 15 '18 at 17:24
For what it's worth, closing vs not closing doesn't have any impact on the memory usage.
– Zachary Wright
Nov 15 '18 at 17:24
Are you sure it hangs around forever maybe GC just hasn't gotten to it yet
– engineersmnky
Nov 15 '18 at 17:25
Are you sure it hangs around forever maybe GC just hasn't gotten to it yet
– engineersmnky
Nov 15 '18 at 17:25
1
1
Garbage collection frees memory for re-use by Ruby. Releasing memory to the OS is something else. sitepoint.com/ruby-uses-memory explains it better than I can.
– steenslag
Nov 15 '18 at 19:56
Garbage collection frees memory for re-use by Ruby. Releasing memory to the OS is something else. sitepoint.com/ruby-uses-memory explains it better than I can.
– steenslag
Nov 15 '18 at 19:56
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53324201%2fwhy-does-memory-leak-after-large-file-download-in-ruby%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53324201%2fwhy-does-memory-leak-after-large-file-download-in-ruby%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
because you didn't close the file ??? try adding
to.close– engineersmnky
Nov 15 '18 at 17:09
@engineersmnky I should have included that in the code sample - I do close it. (I've added it to the sample now)
– Zachary Wright
Nov 15 '18 at 17:17
For what it's worth, closing vs not closing doesn't have any impact on the memory usage.
– Zachary Wright
Nov 15 '18 at 17:24
Are you sure it hangs around forever maybe GC just hasn't gotten to it yet
– engineersmnky
Nov 15 '18 at 17:25
1
Garbage collection frees memory for re-use by Ruby. Releasing memory to the OS is something else. sitepoint.com/ruby-uses-memory explains it better than I can.
– steenslag
Nov 15 '18 at 19:56