processing a dictionary in multithreading is locked
up vote
0
down vote
favorite
I'm trying to process a dictionary in a multiprocess scenario, but the process is locked and function process_data is never fired.
if __name__ == "__main__":
manager = Manager()
directoary_names = manager.dict(lock=True)
time_list = manager.list()
with lock:
ts = datetime.datetime.strptime('2018-08-21', '%Y-%m-%d')
end_ts = datetime.datetime.strptime('2018-8-21', '%Y-%m-%d')
while ts <= end_ts:
print(ts)
ts += datetime.timedelta(minutes=15)
path = os.path.join(
r'C:\ahmed\SpeedFT-meter1\peakdata\' + ts.strftime("%Y-%m-%d") + r'\peakdata_' + ts.strftime(
"%Y%m%d_%H%M") + r'.bz2')
time_list.append(ts)
directoary_names[path] = time_list.pop()
modified_dic = directoary_names
p = Pool(4)
t = p.map(process_data, modified_dic)
p.close()
p.join()
python multithreading
asked Nov 8 at 11:19
andre ahmed
257
|
show 2 more comments
up vote
0
down vote
favorite
I'm trying to process a dictionary in a multiprocess scenario, but the process is locked and function process_data is never fired.
if __name__ == "__main__":
manager = Manager()
directoary_names = manager.dict(lock=True)
time_list = manager.list()
with lock:
ts = datetime.datetime.strptime('2018-08-21', '%Y-%m-%d')
end_ts = datetime.datetime.strptime('2018-8-21', '%Y-%m-%d')
while ts <= end_ts:
print(ts)
ts += datetime.timedelta(minutes=15)
path = os.path.join(
r'C:\ahmed\SpeedFT-meter1\peakdata\' + ts.strftime("%Y-%m-%d") + r'\peakdata_' + ts.strftime(
"%Y%m%d_%H%M") + r'.bz2')
time_list.append(ts)
directoary_names[path] = time_list.pop()
modified_dic = directoary_names
p = Pool(4)
t = p.map(process_data, modified_dic)
p.close()
p.join()
python multithreading
asked Nov 8 at 11:19
andre ahmed
257
where is theprocess_datafunction?
– Netwave
Nov 8 at 11:25
@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41
is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45
yes, using Pool
– andre ahmed
Nov 8 at 11:46
so, if you are using multiprocesing then the IO is not taking place in your current process. try to makeprocess_datato return the values instead of printing them
– Netwave
Nov 8 at 11:49
|
show 2 more comments
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm trying to process a dictionary in a multiprocess scenario, but the process is locked and function process_data is never fired.
if __name__ == "__main__":
manager = Manager()
directoary_names = manager.dict(lock=True)
time_list = manager.list()
with lock:
ts = datetime.datetime.strptime('2018-08-21', '%Y-%m-%d')
end_ts = datetime.datetime.strptime('2018-8-21', '%Y-%m-%d')
while ts <= end_ts:
print(ts)
ts += datetime.timedelta(minutes=15)
path = os.path.join(
r'C:\ahmed\SpeedFT-meter1\peakdata\' + ts.strftime("%Y-%m-%d") + r'\peakdata_' + ts.strftime(
"%Y%m%d_%H%M") + r'.bz2')
time_list.append(ts)
directoary_names[path] = time_list.pop()
modified_dic = directoary_names
p = Pool(4)
t = p.map(process_data, modified_dic)
p.close()
p.join()
python multithreading
asked Nov 8 at 11:19
andre ahmed
257
I'm trying to process a dictionary in a multiprocess scenario, but the process is locked and function process_data is never fired.
if __name__ == "__main__":
manager = Manager()
directoary_names = manager.dict(lock=True)
time_list = manager.list()
with lock:
ts = datetime.datetime.strptime('2018-08-21', '%Y-%m-%d')
end_ts = datetime.datetime.strptime('2018-8-21', '%Y-%m-%d')
while ts <= end_ts:
print(ts)
ts += datetime.timedelta(minutes=15)
path = os.path.join(
r'C:\ahmed\SpeedFT-meter1\peakdata\' + ts.strftime("%Y-%m-%d") + r'\peakdata_' + ts.strftime(
"%Y%m%d_%H%M") + r'.bz2')
time_list.append(ts)
directoary_names[path] = time_list.pop()
modified_dic = directoary_names
p = Pool(4)
t = p.map(process_data, modified_dic)
p.close()
p.join()
python multithreading
python multithreading
asked Nov 8 at 11:19
andre ahmed
257
asked Nov 8 at 11:19
andre ahmed
257
asked Nov 8 at 11:19
andre ahmed
257
asked Nov 8 at 11:19
andre ahmed
257
asked Nov 8 at 11:19
andre ahmed
257
257
where is theprocess_datafunction?
– Netwave
Nov 8 at 11:25
@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41
is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45
yes, using Pool
– andre ahmed
Nov 8 at 11:46
so, if you are using multiprocesing then the IO is not taking place in your current process. try to makeprocess_datato return the values instead of printing them
– Netwave
Nov 8 at 11:49
|
show 2 more comments
where is theprocess_datafunction?
– Netwave
Nov 8 at 11:25
@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41
is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45
yes, using Pool
– andre ahmed
Nov 8 at 11:46
so, if you are using multiprocesing then the IO is not taking place in your current process. try to makeprocess_datato return the values instead of printing them
– Netwave
Nov 8 at 11:49
where is the
process_data function?– Netwave
Nov 8 at 11:25
where is the
process_data function?– Netwave
Nov 8 at 11:25
@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41
@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41
is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45
is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45
yes, using Pool
– andre ahmed
Nov 8 at 11:46
yes, using Pool
– andre ahmed
Nov 8 at 11:46
so, if you are using multiprocesing then the IO is not taking place in your current process. try to make
process_data to return the values instead of printing them– Netwave
Nov 8 at 11:49
so, if you are using multiprocesing then the IO is not taking place in your current process. try to make
process_data to return the values instead of printing them– Netwave
Nov 8 at 11:49
|
show 2 more comments
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53206711%2fprocessing-a-dictionary-in-multithreading-is-locked%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
where is the
process_datafunction?– Netwave
Nov 8 at 11:25
@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41
is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45
yes, using Pool
– andre ahmed
Nov 8 at 11:46
so, if you are using multiprocesing then the IO is not taking place in your current process. try to make
process_datato return the values instead of printing them– Netwave
Nov 8 at 11:49