Cancelling a task cancels a future the task was waiting for. How does it work?
I have an awaitable object implementing a request/reply transaction. If the transaction times out, it will be retried few times before giving up and raising an exception.
Now assume it always times out, because that is the case I have problem with.
When a task starts this operation and then gets cancelled, the retries will continue. This is not what I want. I want to cancel the operation entirely.
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
import asyncio
RETRIES = 2
TIMEOUT = 1.0
class ClientRPC:
def __init__(self):
self._reply = None
self._retries = RETRIES
def __await__(self):
self.start()
return self._reply.__await__()
def start(self):
loop = asyncio.get_event_loop()
if self._reply is None:
self._reply = loop.create_future()
loop.call_later(TIMEOUT, self.handle_timeout)
# send a request
print("REQUEST")
def handle_timeout(self):
print("TIMEOUT")
print("future", repr(self._reply._state))
if self._retries > 0:
self._retries -= 1
self.start()
else:
self._reply.set_exception(RuntimeError("Timeout!"))
def handle_reply(self, reply):
# unused in this example
pass
async def client():
transaction = ClientRPC()
try:
reply = await transaction
except asyncio.CancelledError:
print("--CANCELLED--")
async def test():
loop = asyncio.get_event_loop()
task = loop.create_task(client())
await asyncio.sleep(1.5)
task.cancel()
await asyncio.sleep(3)
asyncio.run(test()) # python 3.7+
Output (traceback omitted):
REQUEST
TIMEOUT
future 'PENDING'
REQUEST
--CANCELLED--
TIMEOUT
future 'CANCELLED' <-- why?
REQUEST
TIMEOUT
future 'CANCELLED'
Exception in callback ClientRPC.handle_timeout()
handle:
asyncio.base_futures.InvalidStateError: invalid state
python python-asyncio
add a comment |
I have an awaitable object implementing a request/reply transaction. If the transaction times out, it will be retried few times before giving up and raising an exception.
Now assume it always times out, because that is the case I have problem with.
When a task starts this operation and then gets cancelled, the retries will continue. This is not what I want. I want to cancel the operation entirely.
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
import asyncio
RETRIES = 2
TIMEOUT = 1.0
class ClientRPC:
def __init__(self):
self._reply = None
self._retries = RETRIES
def __await__(self):
self.start()
return self._reply.__await__()
def start(self):
loop = asyncio.get_event_loop()
if self._reply is None:
self._reply = loop.create_future()
loop.call_later(TIMEOUT, self.handle_timeout)
# send a request
print("REQUEST")
def handle_timeout(self):
print("TIMEOUT")
print("future", repr(self._reply._state))
if self._retries > 0:
self._retries -= 1
self.start()
else:
self._reply.set_exception(RuntimeError("Timeout!"))
def handle_reply(self, reply):
# unused in this example
pass
async def client():
transaction = ClientRPC()
try:
reply = await transaction
except asyncio.CancelledError:
print("--CANCELLED--")
async def test():
loop = asyncio.get_event_loop()
task = loop.create_task(client())
await asyncio.sleep(1.5)
task.cancel()
await asyncio.sleep(3)
asyncio.run(test()) # python 3.7+
Output (traceback omitted):
REQUEST
TIMEOUT
future 'PENDING'
REQUEST
--CANCELLED--
TIMEOUT
future 'CANCELLED' <-- why?
REQUEST
TIMEOUT
future 'CANCELLED'
Exception in callback ClientRPC.handle_timeout()
handle:
asyncio.base_futures.InvalidStateError: invalid state
python python-asyncio
add a comment |
I have an awaitable object implementing a request/reply transaction. If the transaction times out, it will be retried few times before giving up and raising an exception.
Now assume it always times out, because that is the case I have problem with.
When a task starts this operation and then gets cancelled, the retries will continue. This is not what I want. I want to cancel the operation entirely.
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
import asyncio
RETRIES = 2
TIMEOUT = 1.0
class ClientRPC:
def __init__(self):
self._reply = None
self._retries = RETRIES
def __await__(self):
self.start()
return self._reply.__await__()
def start(self):
loop = asyncio.get_event_loop()
if self._reply is None:
self._reply = loop.create_future()
loop.call_later(TIMEOUT, self.handle_timeout)
# send a request
print("REQUEST")
def handle_timeout(self):
print("TIMEOUT")
print("future", repr(self._reply._state))
if self._retries > 0:
self._retries -= 1
self.start()
else:
self._reply.set_exception(RuntimeError("Timeout!"))
def handle_reply(self, reply):
# unused in this example
pass
async def client():
transaction = ClientRPC()
try:
reply = await transaction
except asyncio.CancelledError:
print("--CANCELLED--")
async def test():
loop = asyncio.get_event_loop()
task = loop.create_task(client())
await asyncio.sleep(1.5)
task.cancel()
await asyncio.sleep(3)
asyncio.run(test()) # python 3.7+
Output (traceback omitted):
REQUEST
TIMEOUT
future 'PENDING'
REQUEST
--CANCELLED--
TIMEOUT
future 'CANCELLED' <-- why?
REQUEST
TIMEOUT
future 'CANCELLED'
Exception in callback ClientRPC.handle_timeout()
handle:
asyncio.base_futures.InvalidStateError: invalid state
python python-asyncio
I have an awaitable object implementing a request/reply transaction. If the transaction times out, it will be retried few times before giving up and raising an exception.
Now assume it always times out, because that is the case I have problem with.
When a task starts this operation and then gets cancelled, the retries will continue. This is not what I want. I want to cancel the operation entirely.
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
import asyncio
RETRIES = 2
TIMEOUT = 1.0
class ClientRPC:
def __init__(self):
self._reply = None
self._retries = RETRIES
def __await__(self):
self.start()
return self._reply.__await__()
def start(self):
loop = asyncio.get_event_loop()
if self._reply is None:
self._reply = loop.create_future()
loop.call_later(TIMEOUT, self.handle_timeout)
# send a request
print("REQUEST")
def handle_timeout(self):
print("TIMEOUT")
print("future", repr(self._reply._state))
if self._retries > 0:
self._retries -= 1
self.start()
else:
self._reply.set_exception(RuntimeError("Timeout!"))
def handle_reply(self, reply):
# unused in this example
pass
async def client():
transaction = ClientRPC()
try:
reply = await transaction
except asyncio.CancelledError:
print("--CANCELLED--")
async def test():
loop = asyncio.get_event_loop()
task = loop.create_task(client())
await asyncio.sleep(1.5)
task.cancel()
await asyncio.sleep(3)
asyncio.run(test()) # python 3.7+
Output (traceback omitted):
REQUEST
TIMEOUT
future 'PENDING'
REQUEST
--CANCELLED--
TIMEOUT
future 'CANCELLED' <-- why?
REQUEST
TIMEOUT
future 'CANCELLED'
Exception in callback ClientRPC.handle_timeout()
handle:
asyncio.base_futures.InvalidStateError: invalid state
python python-asyncio
python python-asyncio
asked Nov 13 at 8:51
VPfB
4,16211128
4,16211128
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
Yes, if the task awaits a future, that future will be cancelled. That future can be another task, so the cancellation will spread to the bottom-most future that is awaited. The implementation makes sure of that, but the documentation doesn't make it explicit.
I would go on to rely on this behavior, for two reasons:
it is impossible to change it at this point without a major breakage in backward compatibility. Developers have already rejected smaller changes because they would break existing code.
there is no other way to implement this that wouldn't lead to resource leaks. If the task you are cancelling is awaiting a future, what do you do except cancel it? If you just let it run in the background, you are potentially keeping it around forever, because the future might never exit on its own. If this were "fixed" by just dropping it from the scheduler (again, without cancellation), the future would never get a chance to clean up the resources it acquired, which would certainly cause a resource leak.
Thus it is safe to rely on canceling being propagated downward, with the exception of futures shielded with asyncio.shield()
, which is reserved for futures that are meant to remain running in the background and have their own lifetime management.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53277091%2fcancelling-a-task-cancels-a-future-the-task-was-waiting-for-how-does-it-work%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
Yes, if the task awaits a future, that future will be cancelled. That future can be another task, so the cancellation will spread to the bottom-most future that is awaited. The implementation makes sure of that, but the documentation doesn't make it explicit.
I would go on to rely on this behavior, for two reasons:
it is impossible to change it at this point without a major breakage in backward compatibility. Developers have already rejected smaller changes because they would break existing code.
there is no other way to implement this that wouldn't lead to resource leaks. If the task you are cancelling is awaiting a future, what do you do except cancel it? If you just let it run in the background, you are potentially keeping it around forever, because the future might never exit on its own. If this were "fixed" by just dropping it from the scheduler (again, without cancellation), the future would never get a chance to clean up the resources it acquired, which would certainly cause a resource leak.
Thus it is safe to rely on canceling being propagated downward, with the exception of futures shielded with asyncio.shield()
, which is reserved for futures that are meant to remain running in the background and have their own lifetime management.
add a comment |
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
Yes, if the task awaits a future, that future will be cancelled. That future can be another task, so the cancellation will spread to the bottom-most future that is awaited. The implementation makes sure of that, but the documentation doesn't make it explicit.
I would go on to rely on this behavior, for two reasons:
it is impossible to change it at this point without a major breakage in backward compatibility. Developers have already rejected smaller changes because they would break existing code.
there is no other way to implement this that wouldn't lead to resource leaks. If the task you are cancelling is awaiting a future, what do you do except cancel it? If you just let it run in the background, you are potentially keeping it around forever, because the future might never exit on its own. If this were "fixed" by just dropping it from the scheduler (again, without cancellation), the future would never get a chance to clean up the resources it acquired, which would certainly cause a resource leak.
Thus it is safe to rely on canceling being propagated downward, with the exception of futures shielded with asyncio.shield()
, which is reserved for futures that are meant to remain running in the background and have their own lifetime management.
add a comment |
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
Yes, if the task awaits a future, that future will be cancelled. That future can be another task, so the cancellation will spread to the bottom-most future that is awaited. The implementation makes sure of that, but the documentation doesn't make it explicit.
I would go on to rely on this behavior, for two reasons:
it is impossible to change it at this point without a major breakage in backward compatibility. Developers have already rejected smaller changes because they would break existing code.
there is no other way to implement this that wouldn't lead to resource leaks. If the task you are cancelling is awaiting a future, what do you do except cancel it? If you just let it run in the background, you are potentially keeping it around forever, because the future might never exit on its own. If this were "fixed" by just dropping it from the scheduler (again, without cancellation), the future would never get a chance to clean up the resources it acquired, which would certainly cause a resource leak.
Thus it is safe to rely on canceling being propagated downward, with the exception of futures shielded with asyncio.shield()
, which is reserved for futures that are meant to remain running in the background and have their own lifetime management.
I prepared a MCVE and noticed, that a future that the task is waiting for gets cancelled when the task is cancelled. This suits me fine, it could be a base for a solution, but I do not understand why that future gets cancelled and if I can rely on it.
Yes, if the task awaits a future, that future will be cancelled. That future can be another task, so the cancellation will spread to the bottom-most future that is awaited. The implementation makes sure of that, but the documentation doesn't make it explicit.
I would go on to rely on this behavior, for two reasons:
it is impossible to change it at this point without a major breakage in backward compatibility. Developers have already rejected smaller changes because they would break existing code.
there is no other way to implement this that wouldn't lead to resource leaks. If the task you are cancelling is awaiting a future, what do you do except cancel it? If you just let it run in the background, you are potentially keeping it around forever, because the future might never exit on its own. If this were "fixed" by just dropping it from the scheduler (again, without cancellation), the future would never get a chance to clean up the resources it acquired, which would certainly cause a resource leak.
Thus it is safe to rely on canceling being propagated downward, with the exception of futures shielded with asyncio.shield()
, which is reserved for futures that are meant to remain running in the background and have their own lifetime management.
edited Nov 13 at 14:04
answered Nov 13 at 12:41
user4815162342
60.4k490141
60.4k490141
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53277091%2fcancelling-a-task-cancels-a-future-the-task-was-waiting-for-how-does-it-work%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown