processing a dictionary in multithreading is locked

up vote
0
down vote

favorite

I'm trying to process a dictionary in a multiprocess scenario, but the process is locked and function process_data is never fired.

if __name__ == "__main__":

    manager = Manager()

    directoary_names = manager.dict(lock=True)

    time_list = manager.list()

    with lock:

        ts = datetime.datetime.strptime('2018-08-21', '%Y-%m-%d')

        end_ts = datetime.datetime.strptime('2018-8-21', '%Y-%m-%d')

        while ts <= end_ts:

            print(ts)

            ts += datetime.timedelta(minutes=15)

            path = os.path.join(

                r'C:\ahmed\SpeedFT-meter1\peakdata\' + ts.strftime("%Y-%m-%d") + r'\peakdata_' + ts.strftime(

                    "%Y%m%d_%H%M") + r'.bz2')

            time_list.append(ts)

            directoary_names[path] = time_list.pop()



    modified_dic = directoary_names

    p = Pool(4)

    t = p.map(process_data, modified_dic)

    p.close()

    p.join()

asked Nov 8 at 11:19

andre ahmed

257

where is the process_data function?
– Netwave
Nov 8 at 11:25

@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41

is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45

yes, using Pool
– andre ahmed
Nov 8 at 11:46

so, if you are using multiprocesing then the IO is not taking place in your current process. try to make process_data to return the values instead of printing them
– Netwave
Nov 8 at 11:49

|
show 2 more comments

up vote
0
down vote

favorite

I'm trying to process a dictionary in a multiprocess scenario, but the process is locked and function process_data is never fired.

if __name__ == "__main__":

    manager = Manager()

    directoary_names = manager.dict(lock=True)

    time_list = manager.list()

    with lock:

        ts = datetime.datetime.strptime('2018-08-21', '%Y-%m-%d')

        end_ts = datetime.datetime.strptime('2018-8-21', '%Y-%m-%d')

        while ts <= end_ts:

            print(ts)

            ts += datetime.timedelta(minutes=15)

            path = os.path.join(

                r'C:\ahmed\SpeedFT-meter1\peakdata\' + ts.strftime("%Y-%m-%d") + r'\peakdata_' + ts.strftime(

                    "%Y%m%d_%H%M") + r'.bz2')

            time_list.append(ts)

            directoary_names[path] = time_list.pop()



    modified_dic = directoary_names

    p = Pool(4)

    t = p.map(process_data, modified_dic)

    p.close()

    p.join()

asked Nov 8 at 11:19

andre ahmed

257

where is the process_data function?
– Netwave
Nov 8 at 11:25

@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41

is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45

yes, using Pool
– andre ahmed
Nov 8 at 11:46

so, if you are using multiprocesing then the IO is not taking place in your current process. try to make process_data to return the values instead of printing them
– Netwave
Nov 8 at 11:49

|
show 2 more comments

up vote
0
down vote

favorite

I'm trying to process a dictionary in a multiprocess scenario, but the process is locked and function process_data is never fired.

if __name__ == "__main__":

    manager = Manager()

    directoary_names = manager.dict(lock=True)

    time_list = manager.list()

    with lock:

        ts = datetime.datetime.strptime('2018-08-21', '%Y-%m-%d')

        end_ts = datetime.datetime.strptime('2018-8-21', '%Y-%m-%d')

        while ts <= end_ts:

            print(ts)

            ts += datetime.timedelta(minutes=15)

            path = os.path.join(

                r'C:\ahmed\SpeedFT-meter1\peakdata\' + ts.strftime("%Y-%m-%d") + r'\peakdata_' + ts.strftime(

                    "%Y%m%d_%H%M") + r'.bz2')

            time_list.append(ts)

            directoary_names[path] = time_list.pop()



    modified_dic = directoary_names

    p = Pool(4)

    t = p.map(process_data, modified_dic)

    p.close()

    p.join()

asked Nov 8 at 11:19

andre ahmed

257

I'm trying to process a dictionary in a multiprocess scenario, but the process is locked and function process_data is never fired.

if __name__ == "__main__":

    manager = Manager()

    directoary_names = manager.dict(lock=True)

    time_list = manager.list()

    with lock:

        ts = datetime.datetime.strptime('2018-08-21', '%Y-%m-%d')

        end_ts = datetime.datetime.strptime('2018-8-21', '%Y-%m-%d')

        while ts <= end_ts:

            print(ts)

            ts += datetime.timedelta(minutes=15)

            path = os.path.join(

                r'C:\ahmed\SpeedFT-meter1\peakdata\' + ts.strftime("%Y-%m-%d") + r'\peakdata_' + ts.strftime(

                    "%Y%m%d_%H%M") + r'.bz2')

            time_list.append(ts)

            directoary_names[path] = time_list.pop()



    modified_dic = directoary_names

    p = Pool(4)

    t = p.map(process_data, modified_dic)

    p.close()

    p.join()

python multithreading

asked Nov 8 at 11:19

andre ahmed

257

asked Nov 8 at 11:19

andre ahmed

257

asked Nov 8 at 11:19

andre ahmed

257

asked Nov 8 at 11:19

andre ahmed

257

asked Nov 8 at 11:19

andre ahmed

257

where is the process_data function?
– Netwave
Nov 8 at 11:25

@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41

is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45

yes, using Pool
– andre ahmed
Nov 8 at 11:46

so, if you are using multiprocesing then the IO is not taking place in your current process. try to make process_data to return the values instead of printing them
– Netwave
Nov 8 at 11:49

|
show 2 more comments

where is the process_data function?
– Netwave
Nov 8 at 11:25

@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41

is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45

yes, using Pool
– andre ahmed
Nov 8 at 11:46

so, if you are using multiprocesing then the IO is not taking place in your current process. try to make process_data to return the values instead of printing them
– Netwave
Nov 8 at 11:49

where is the process_data function?
– Netwave
Nov 8 at 11:25

@Netwave it just does this print(directoary_names.values())
– andre ahmed
Nov 8 at 11:41

is it multiprocesing what you are using?
– Netwave
Nov 8 at 11:45

yes, using Pool
– andre ahmed
Nov 8 at 11:46

so, if you are using multiprocesing then the IO is not taking place in your current process. try to make process_data to return the values instead of printing them
– Netwave
Nov 8 at 11:49

|
show 2 more comments

active

oldest

votes

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53206711%2fprocessing-a-dictionary-in-multithreading-is-locked%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

active

oldest

votes

draft saved

draft discarded

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Agfdhyk