Sending large files over sockets on separate computers

I have created a basic client-server program that transfers file from server to the client. Since the file is about 500MB, the server transfers the file as chunks of bytes to the client over the DataOutputStream object. While this logic works okay when both client and server are running on the same computer, it doesn't work when the two programs are running on separate computers(Both computers are on the same network and I have disabled the firewall for both)

When running on separate computers the problem is that a few bytes get transferred

Server Logic:

byte byteArr = new byte[1024];

while((c=fileInputStream.read(byteArr, 0, 1024) != -1))

{

    dataOutputStream.writeBoolean(true);

    dataOutputStream.flush();

    dataOutputStream.write(byteArr, 0, 1024);

    dataOutputStream.flush();

}

 /*When running on different computers, after a few hundred iterations 

   it just stops looping the following lines are never executed*/

dataOutputStream.writeBoolean(false);

System.out.println("Transfer complete");

Client Logic

byte byteArr = new byte[1024];

while(dataInputStream.readBoolean())

{

     dataInputStream.read(byteArr, 0, 1024);

     fileOutputStream.write(byteArr, 0, 1024);

}

asked Nov 1 '18 at 14:36

Danny

You only need flush once finished. You don't need the Boolean flag but you need to always ensure you only write as much data as you read at both ends (and only read on the client as much data as the server wrote)

– Peter Lawrey
Nov 1 '18 at 16:13

add a comment |

When running on separate computers the problem is that a few bytes get transferred

Server Logic:

byte byteArr = new byte[1024];

while((c=fileInputStream.read(byteArr, 0, 1024) != -1))

{

    dataOutputStream.writeBoolean(true);

    dataOutputStream.flush();

    dataOutputStream.write(byteArr, 0, 1024);

    dataOutputStream.flush();

}

 /*When running on different computers, after a few hundred iterations 

   it just stops looping the following lines are never executed*/

dataOutputStream.writeBoolean(false);

System.out.println("Transfer complete");

Client Logic

byte byteArr = new byte[1024];

while(dataInputStream.readBoolean())

{

     dataInputStream.read(byteArr, 0, 1024);

     fileOutputStream.write(byteArr, 0, 1024);

}

asked Nov 1 '18 at 14:36

Danny

You only need flush once finished. You don't need the Boolean flag but you need to always ensure you only write as much data as you read at both ends (and only read on the client as much data as the server wrote)

– Peter Lawrey
Nov 1 '18 at 16:13

add a comment |

When running on separate computers the problem is that a few bytes get transferred

Server Logic:

byte byteArr = new byte[1024];

while((c=fileInputStream.read(byteArr, 0, 1024) != -1))

{

    dataOutputStream.writeBoolean(true);

    dataOutputStream.flush();

    dataOutputStream.write(byteArr, 0, 1024);

    dataOutputStream.flush();

}

 /*When running on different computers, after a few hundred iterations 

   it just stops looping the following lines are never executed*/

dataOutputStream.writeBoolean(false);

System.out.println("Transfer complete");

Client Logic

byte byteArr = new byte[1024];

while(dataInputStream.readBoolean())

{

     dataInputStream.read(byteArr, 0, 1024);

     fileOutputStream.write(byteArr, 0, 1024);

}

asked Nov 1 '18 at 14:36

Danny

When running on separate computers the problem is that a few bytes get transferred

Server Logic:

byte byteArr = new byte[1024];

while((c=fileInputStream.read(byteArr, 0, 1024) != -1))

{

    dataOutputStream.writeBoolean(true);

    dataOutputStream.flush();

    dataOutputStream.write(byteArr, 0, 1024);

    dataOutputStream.flush();

}

 /*When running on different computers, after a few hundred iterations 

   it just stops looping the following lines are never executed*/

dataOutputStream.writeBoolean(false);

System.out.println("Transfer complete");

Client Logic

byte byteArr = new byte[1024];

while(dataInputStream.readBoolean())

{

     dataInputStream.read(byteArr, 0, 1024);

     fileOutputStream.write(byteArr, 0, 1024);

}

java sockets client-server

asked Nov 1 '18 at 14:36

Danny

asked Nov 1 '18 at 14:36

Danny

asked Nov 1 '18 at 14:36

Danny

asked Nov 1 '18 at 14:36

Danny

asked Nov 1 '18 at 14:36

Danny

You only need flush once finished. You don't need the Boolean flag but you need to always ensure you only write as much data as you read at both ends (and only read on the client as much data as the server wrote)

– Peter Lawrey
Nov 1 '18 at 16:13

add a comment |

You only need flush once finished. You don't need the Boolean flag but you need to always ensure you only write as much data as you read at both ends (and only read on the client as much data as the server wrote)

– Peter Lawrey
Nov 1 '18 at 16:13

You only need flush once finished. You don't need the Boolean flag but you need to always ensure you only write as much data as you read at both ends (and only read on the client as much data as the server wrote)

– Peter Lawrey
Nov 1 '18 at 16:13

add a comment |

1 Answer
1

active

oldest

votes

A read(buf, 0, 1024) call is not guaranteed to read exactly 1024 bytes. This causes bugs in both pieces of code:

the server incorrectly assumes that each chunk read from the file is always exactly 1024 bytes long and sends the whole buffer to the client.

the client might not read the whole chunk in a single iteration. It will then treat the first byte of the remainder as a boolean and get out of sync with the server.

To resolve this, you could:

send the file size (if known) before sending the file,
then just keep reading until you've read that many bytes.

or send c (the chunk size) instead of a single boolean,
then use dataInputStream.readFully() to make sure that
many bytes will be read.

answered Nov 1 '18 at 15:00

Tadeusz Sznuk

30113

Ok I'll try the second approach since the file size may be too big...although how is it working smoothly when both programs are running on the same pc?

– Danny
Nov 1 '18 at 15:07

While read is not guaranteed to read the full amount, it might do so - and in a local setting it probably does.

– Tadeusz Sznuk
Nov 1 '18 at 15:09

Ok..so the readFully call should be readFully(buff, 0, (c-1)) right?

– Danny
Nov 1 '18 at 15:20

I think it should be readFully(buff, 0, c). The last argument is the number of bytes to read (not the index of the last byte).

– Tadeusz Sznuk
Nov 1 '18 at 15:32

Yup...it worked...just transferred a 379mb file...thank you so much for your help!!

– Danny
Nov 1 '18 at 16:08

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53103463%2fsending-large-files-over-sockets-on-separate-computers%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

A read(buf, 0, 1024) call is not guaranteed to read exactly 1024 bytes. This causes bugs in both pieces of code:

the server incorrectly assumes that each chunk read from the file is always exactly 1024 bytes long and sends the whole buffer to the client.

the client might not read the whole chunk in a single iteration. It will then treat the first byte of the remainder as a boolean and get out of sync with the server.

To resolve this, you could:

send the file size (if known) before sending the file,
then just keep reading until you've read that many bytes.

or send c (the chunk size) instead of a single boolean,
then use dataInputStream.readFully() to make sure that
many bytes will be read.

answered Nov 1 '18 at 15:00

Tadeusz Sznuk

30113

Ok I'll try the second approach since the file size may be too big...although how is it working smoothly when both programs are running on the same pc?

– Danny
Nov 1 '18 at 15:07

While read is not guaranteed to read the full amount, it might do so - and in a local setting it probably does.

– Tadeusz Sznuk
Nov 1 '18 at 15:09

Ok..so the readFully call should be readFully(buff, 0, (c-1)) right?

– Danny
Nov 1 '18 at 15:20

I think it should be readFully(buff, 0, c). The last argument is the number of bytes to read (not the index of the last byte).

– Tadeusz Sznuk
Nov 1 '18 at 15:32

Yup...it worked...just transferred a 379mb file...thank you so much for your help!!

– Danny
Nov 1 '18 at 16:08

add a comment |

A read(buf, 0, 1024) call is not guaranteed to read exactly 1024 bytes. This causes bugs in both pieces of code:

the server incorrectly assumes that each chunk read from the file is always exactly 1024 bytes long and sends the whole buffer to the client.

the client might not read the whole chunk in a single iteration. It will then treat the first byte of the remainder as a boolean and get out of sync with the server.

To resolve this, you could:

send the file size (if known) before sending the file,
then just keep reading until you've read that many bytes.

or send c (the chunk size) instead of a single boolean,
then use dataInputStream.readFully() to make sure that
many bytes will be read.

answered Nov 1 '18 at 15:00

Tadeusz Sznuk

30113

Ok I'll try the second approach since the file size may be too big...although how is it working smoothly when both programs are running on the same pc?

– Danny
Nov 1 '18 at 15:07

While read is not guaranteed to read the full amount, it might do so - and in a local setting it probably does.

– Tadeusz Sznuk
Nov 1 '18 at 15:09

Ok..so the readFully call should be readFully(buff, 0, (c-1)) right?

– Danny
Nov 1 '18 at 15:20

I think it should be readFully(buff, 0, c). The last argument is the number of bytes to read (not the index of the last byte).

– Tadeusz Sznuk
Nov 1 '18 at 15:32

Yup...it worked...just transferred a 379mb file...thank you so much for your help!!

– Danny
Nov 1 '18 at 16:08

add a comment |

A read(buf, 0, 1024) call is not guaranteed to read exactly 1024 bytes. This causes bugs in both pieces of code:

the server incorrectly assumes that each chunk read from the file is always exactly 1024 bytes long and sends the whole buffer to the client.

the client might not read the whole chunk in a single iteration. It will then treat the first byte of the remainder as a boolean and get out of sync with the server.

To resolve this, you could:

send the file size (if known) before sending the file,
then just keep reading until you've read that many bytes.

or send c (the chunk size) instead of a single boolean,
then use dataInputStream.readFully() to make sure that
many bytes will be read.

answered Nov 1 '18 at 15:00

Tadeusz Sznuk

30113

A read(buf, 0, 1024) call is not guaranteed to read exactly 1024 bytes. This causes bugs in both pieces of code:

the server incorrectly assumes that each chunk read from the file is always exactly 1024 bytes long and sends the whole buffer to the client.

the client might not read the whole chunk in a single iteration. It will then treat the first byte of the remainder as a boolean and get out of sync with the server.

To resolve this, you could:

send the file size (if known) before sending the file,
then just keep reading until you've read that many bytes.

or send c (the chunk size) instead of a single boolean,
then use dataInputStream.readFully() to make sure that
many bytes will be read.

answered Nov 1 '18 at 15:00

Tadeusz Sznuk

30113

answered Nov 1 '18 at 15:00

Tadeusz Sznuk

30113

answered Nov 1 '18 at 15:00

Tadeusz Sznuk

30113

answered Nov 1 '18 at 15:00

Tadeusz Sznuk

30113

Ok I'll try the second approach since the file size may be too big...although how is it working smoothly when both programs are running on the same pc?

– Danny
Nov 1 '18 at 15:07

While read is not guaranteed to read the full amount, it might do so - and in a local setting it probably does.

– Tadeusz Sznuk
Nov 1 '18 at 15:09

Ok..so the readFully call should be readFully(buff, 0, (c-1)) right?

– Danny
Nov 1 '18 at 15:20

I think it should be readFully(buff, 0, c). The last argument is the number of bytes to read (not the index of the last byte).

– Tadeusz Sznuk
Nov 1 '18 at 15:32

Yup...it worked...just transferred a 379mb file...thank you so much for your help!!

– Danny
Nov 1 '18 at 16:08

add a comment |

Ok I'll try the second approach since the file size may be too big...although how is it working smoothly when both programs are running on the same pc?

– Danny
Nov 1 '18 at 15:07

While read is not guaranteed to read the full amount, it might do so - and in a local setting it probably does.

– Tadeusz Sznuk
Nov 1 '18 at 15:09

Ok..so the readFully call should be readFully(buff, 0, (c-1)) right?

– Danny
Nov 1 '18 at 15:20

I think it should be readFully(buff, 0, c). The last argument is the number of bytes to read (not the index of the last byte).

– Tadeusz Sznuk
Nov 1 '18 at 15:32

Yup...it worked...just transferred a 379mb file...thank you so much for your help!!

– Danny
Nov 1 '18 at 16:08

Ok I'll try the second approach since the file size may be too big...although how is it working smoothly when both programs are running on the same pc?

– Danny
Nov 1 '18 at 15:07

While read is not guaranteed to read the full amount, it might do so - and in a local setting it probably does.

– Tadeusz Sznuk
Nov 1 '18 at 15:09

Ok..so the readFully call should be readFully(buff, 0, (c-1)) right?

– Danny
Nov 1 '18 at 15:20

I think it should be readFully(buff, 0, c). The last argument is the number of bytes to read (not the index of the last byte).

– Tadeusz Sznuk
Nov 1 '18 at 15:32

Yup...it worked...just transferred a 379mb file...thank you so much for your help!!

– Danny
Nov 1 '18 at 16:08

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Agfdhyk