Slow Resilio Sync between two Docker containers on same host
I'm testing out Resilio (formerly BitTorrent) Sync using two docker containers on the same host, and I've noticed that syncing files is very slow when they are added via bind mounts.
First Resilio container:
docker run -d --name resilio1 -p 81:8888 -p 55555:55555 -v /c/Users/Test/resilio/resilio1/config:/mnt/sync -v /c/Users/Test/resilio/resilio1/data:/mnt/mounted_folders/data --restart on-failure resilio/sync
Second Resilio container (I've updated the configuration to listen on port 55556 to avoid a conflict with resilio1):
docker run -d --name resilio2 -p 82:8888 -p 55556:55556 -v /c/Users/Test/resilio/resilio2/config:/mnt/sync -v /c/Users/Test/resilio/resilio2/data:/mnt/mounted_folders/data --restart on-failure resilio/sync
It takes about 10 minutes for even small files to sync when they are added to the host's directories that are bind mounted into the containers.
Conversely, if I docker exec into the containers and create files or folders the syncing happens immediately.
Is there something I'm missing here?
docker bittorrent-sync
asked Oct 21 '18 at 4:11
atmatm
606817
add a comment |
I'm testing out Resilio (formerly BitTorrent) Sync using two docker containers on the same host, and I've noticed that syncing files is very slow when they are added via bind mounts.
First Resilio container:
docker run -d --name resilio1 -p 81:8888 -p 55555:55555 -v /c/Users/Test/resilio/resilio1/config:/mnt/sync -v /c/Users/Test/resilio/resilio1/data:/mnt/mounted_folders/data --restart on-failure resilio/sync
Second Resilio container (I've updated the configuration to listen on port 55556 to avoid a conflict with resilio1):
docker run -d --name resilio2 -p 82:8888 -p 55556:55556 -v /c/Users/Test/resilio/resilio2/config:/mnt/sync -v /c/Users/Test/resilio/resilio2/data:/mnt/mounted_folders/data --restart on-failure resilio/sync
It takes about 10 minutes for even small files to sync when they are added to the host's directories that are bind mounted into the containers.
Conversely, if I docker exec into the containers and create files or folders the syncing happens immediately.
Is there something I'm missing here?
docker bittorrent-sync
asked Oct 21 '18 at 4:11
atmatm
606817
If I remember correctly, BitTorrent Sync cannot always rely on file system events for synchronization, therefore it scans the file system regularly (defaults to 10 minutes, maybe?) to detect new and changed files. I don't have the time to research this, but hopefully my recollection may help you resolve this. Perhaps file system events are not available or triggered correctly?
– Jochem Kuijpers
Oct 31 '18 at 11:45
add a comment |
I'm testing out Resilio (formerly BitTorrent) Sync using two docker containers on the same host, and I've noticed that syncing files is very slow when they are added via bind mounts.
First Resilio container:
docker run -d --name resilio1 -p 81:8888 -p 55555:55555 -v /c/Users/Test/resilio/resilio1/config:/mnt/sync -v /c/Users/Test/resilio/resilio1/data:/mnt/mounted_folders/data --restart on-failure resilio/sync
Second Resilio container (I've updated the configuration to listen on port 55556 to avoid a conflict with resilio1):
docker run -d --name resilio2 -p 82:8888 -p 55556:55556 -v /c/Users/Test/resilio/resilio2/config:/mnt/sync -v /c/Users/Test/resilio/resilio2/data:/mnt/mounted_folders/data --restart on-failure resilio/sync
It takes about 10 minutes for even small files to sync when they are added to the host's directories that are bind mounted into the containers.
Conversely, if I docker exec into the containers and create files or folders the syncing happens immediately.
Is there something I'm missing here?
docker bittorrent-sync
asked Oct 21 '18 at 4:11
atmatm
606817
I'm testing out Resilio (formerly BitTorrent) Sync using two docker containers on the same host, and I've noticed that syncing files is very slow when they are added via bind mounts.
First Resilio container:
docker run -d --name resilio1 -p 81:8888 -p 55555:55555 -v /c/Users/Test/resilio/resilio1/config:/mnt/sync -v /c/Users/Test/resilio/resilio1/data:/mnt/mounted_folders/data --restart on-failure resilio/sync
Second Resilio container (I've updated the configuration to listen on port 55556 to avoid a conflict with resilio1):
docker run -d --name resilio2 -p 82:8888 -p 55556:55556 -v /c/Users/Test/resilio/resilio2/config:/mnt/sync -v /c/Users/Test/resilio/resilio2/data:/mnt/mounted_folders/data --restart on-failure resilio/sync
It takes about 10 minutes for even small files to sync when they are added to the host's directories that are bind mounted into the containers.
Conversely, if I docker exec into the containers and create files or folders the syncing happens immediately.
Is there something I'm missing here?
docker bittorrent-sync
docker bittorrent-sync
asked Oct 21 '18 at 4:11
atmatm
606817
asked Oct 21 '18 at 4:11
atmatm
606817
edited Nov 20 '18 at 2:50
atm
asked Oct 21 '18 at 4:11
atmatm
606817
asked Oct 21 '18 at 4:11
atmatm
606817
asked Oct 21 '18 at 4:11
atmatm
606817
606817
If I remember correctly, BitTorrent Sync cannot always rely on file system events for synchronization, therefore it scans the file system regularly (defaults to 10 minutes, maybe?) to detect new and changed files. I don't have the time to research this, but hopefully my recollection may help you resolve this. Perhaps file system events are not available or triggered correctly?
– Jochem Kuijpers
Oct 31 '18 at 11:45
add a comment |
If I remember correctly, BitTorrent Sync cannot always rely on file system events for synchronization, therefore it scans the file system regularly (defaults to 10 minutes, maybe?) to detect new and changed files. I don't have the time to research this, but hopefully my recollection may help you resolve this. Perhaps file system events are not available or triggered correctly?
– Jochem Kuijpers
Oct 31 '18 at 11:45
If I remember correctly, BitTorrent Sync cannot always rely on file system events for synchronization, therefore it scans the file system regularly (defaults to 10 minutes, maybe?) to detect new and changed files. I don't have the time to research this, but hopefully my recollection may help you resolve this. Perhaps file system events are not available or triggered correctly?
– Jochem Kuijpers
Oct 31 '18 at 11:45
If I remember correctly, BitTorrent Sync cannot always rely on file system events for synchronization, therefore it scans the file system regularly (defaults to 10 minutes, maybe?) to detect new and changed files. I don't have the time to research this, but hopefully my recollection may help you resolve this. Perhaps file system events are not available or triggered correctly?
– Jochem Kuijpers
Oct 31 '18 at 11:45
add a comment |
1 Answer
1
active
oldest
votes
Resilio Sync uses two mechanisms to track changes in a file system:
- File system notifications Resilio Sync subscribes to (to obtain info about changes in a file system immediately). These notifications are just an OS kernel mechanism, for example Inotify API is used on Linux.
- Periodic re-scan of all folders added in Sync (occurs every 10 mins by default and can be configured in settings)
As you may know Docker on Windows works using a Linux VM and relies on SMB/CIFS support in Linux kernel. As Jochem Kuijpers mentioned the issue is related to a file system notifications which don't work properly in case of Windows and mounted volumes and this is a known issue.
The workaround is to use some 3rd party tool that tracks changes in mounted windows volumes and notifies Docker containers about it.
You can use docker-windows-volume-watcher for example. Install is pretty simple (the script needs to be installed on Windows):
pip install docker-windows-volume-watcher
Usage is simple too:
docker-volume-watcher <container_name> C:pathtomounteddirectory
As the article says:
The script will inspect all running containers and start notifying
containers about changes in mounted directories. The script will also
listen container start/stop events and update the list of watched
directories.
answered Nov 1 '18 at 16:08
Artsiom PraneuskiArtsiom Praneuski
1,047412
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f52912061%2fslow-resilio-sync-between-two-docker-containers-on-same-host%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Resilio Sync uses two mechanisms to track changes in a file system:
- File system notifications Resilio Sync subscribes to (to obtain info about changes in a file system immediately). These notifications are just an OS kernel mechanism, for example Inotify API is used on Linux.
- Periodic re-scan of all folders added in Sync (occurs every 10 mins by default and can be configured in settings)
As you may know Docker on Windows works using a Linux VM and relies on SMB/CIFS support in Linux kernel. As Jochem Kuijpers mentioned the issue is related to a file system notifications which don't work properly in case of Windows and mounted volumes and this is a known issue.
The workaround is to use some 3rd party tool that tracks changes in mounted windows volumes and notifies Docker containers about it.
You can use docker-windows-volume-watcher for example. Install is pretty simple (the script needs to be installed on Windows):
pip install docker-windows-volume-watcher
Usage is simple too:
docker-volume-watcher <container_name> C:pathtomounteddirectory
As the article says:
The script will inspect all running containers and start notifying
containers about changes in mounted directories. The script will also
listen container start/stop events and update the list of watched
directories.
answered Nov 1 '18 at 16:08
Artsiom PraneuskiArtsiom Praneuski
1,047412
add a comment |
Resilio Sync uses two mechanisms to track changes in a file system:
- File system notifications Resilio Sync subscribes to (to obtain info about changes in a file system immediately). These notifications are just an OS kernel mechanism, for example Inotify API is used on Linux.
- Periodic re-scan of all folders added in Sync (occurs every 10 mins by default and can be configured in settings)
As you may know Docker on Windows works using a Linux VM and relies on SMB/CIFS support in Linux kernel. As Jochem Kuijpers mentioned the issue is related to a file system notifications which don't work properly in case of Windows and mounted volumes and this is a known issue.
The workaround is to use some 3rd party tool that tracks changes in mounted windows volumes and notifies Docker containers about it.
You can use docker-windows-volume-watcher for example. Install is pretty simple (the script needs to be installed on Windows):
pip install docker-windows-volume-watcher
Usage is simple too:
docker-volume-watcher <container_name> C:pathtomounteddirectory
As the article says:
The script will inspect all running containers and start notifying
containers about changes in mounted directories. The script will also
listen container start/stop events and update the list of watched
directories.
answered Nov 1 '18 at 16:08
Artsiom PraneuskiArtsiom Praneuski
1,047412
add a comment |
Resilio Sync uses two mechanisms to track changes in a file system:
- File system notifications Resilio Sync subscribes to (to obtain info about changes in a file system immediately). These notifications are just an OS kernel mechanism, for example Inotify API is used on Linux.
- Periodic re-scan of all folders added in Sync (occurs every 10 mins by default and can be configured in settings)
As you may know Docker on Windows works using a Linux VM and relies on SMB/CIFS support in Linux kernel. As Jochem Kuijpers mentioned the issue is related to a file system notifications which don't work properly in case of Windows and mounted volumes and this is a known issue.
The workaround is to use some 3rd party tool that tracks changes in mounted windows volumes and notifies Docker containers about it.
You can use docker-windows-volume-watcher for example. Install is pretty simple (the script needs to be installed on Windows):
pip install docker-windows-volume-watcher
Usage is simple too:
docker-volume-watcher <container_name> C:pathtomounteddirectory
As the article says:
The script will inspect all running containers and start notifying
containers about changes in mounted directories. The script will also
listen container start/stop events and update the list of watched
directories.
answered Nov 1 '18 at 16:08
Artsiom PraneuskiArtsiom Praneuski
1,047412
Resilio Sync uses two mechanisms to track changes in a file system:
- File system notifications Resilio Sync subscribes to (to obtain info about changes in a file system immediately). These notifications are just an OS kernel mechanism, for example Inotify API is used on Linux.
- Periodic re-scan of all folders added in Sync (occurs every 10 mins by default and can be configured in settings)
As you may know Docker on Windows works using a Linux VM and relies on SMB/CIFS support in Linux kernel. As Jochem Kuijpers mentioned the issue is related to a file system notifications which don't work properly in case of Windows and mounted volumes and this is a known issue.
The workaround is to use some 3rd party tool that tracks changes in mounted windows volumes and notifies Docker containers about it.
You can use docker-windows-volume-watcher for example. Install is pretty simple (the script needs to be installed on Windows):
pip install docker-windows-volume-watcher
Usage is simple too:
docker-volume-watcher <container_name> C:pathtomounteddirectory
As the article says:
The script will inspect all running containers and start notifying
containers about changes in mounted directories. The script will also
listen container start/stop events and update the list of watched
directories.
answered Nov 1 '18 at 16:08
Artsiom PraneuskiArtsiom Praneuski
1,047412
edited Nov 13 '18 at 8:33
answered Nov 1 '18 at 16:08
Artsiom PraneuskiArtsiom Praneuski
1,047412
answered Nov 1 '18 at 16:08
Artsiom PraneuskiArtsiom Praneuski
1,047412
answered Nov 1 '18 at 16:08
Artsiom PraneuskiArtsiom Praneuski
1,047412
1,047412
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f52912061%2fslow-resilio-sync-between-two-docker-containers-on-same-host%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If I remember correctly, BitTorrent Sync cannot always rely on file system events for synchronization, therefore it scans the file system regularly (defaults to 10 minutes, maybe?) to detect new and changed files. I don't have the time to research this, but hopefully my recollection may help you resolve this. Perhaps file system events are not available or triggered correctly?
– Jochem Kuijpers
Oct 31 '18 at 11:45