Python (multiprocessing): How to pass a dictionary as the argument of a worker process initializer function?












1















I'm using a function to initialize the worker processes of a process pool, and this function has a single argument, which is a dictionary. When the process pool is created and the function is called to initialize each worker process I get an error regarding the wrong number of arguments:



TypeError: _init_worker() takes 1 positional argument but 2 were given


The process initializer function being used:



def _init_worker(shared_arrays):



    _global_shared_arrays = shared_arrays


The initializer is being called in the normal way for each worker process:



with multiprocessing.Pool(processes=_NUMBER_OF_WORKER_PROCESSES,

                          initializer=_init_worker, initargs=(arrays_dict)) as pool:


I think this has something to do with how dictionaries are passed as the argument, in that the above error always lists the number of items in the dictionary as the number of positional arguments that were passed, as if what's being passed is the keys of the dictionary rather than the dictionary itself. When I step into the code in the debugger this is exactly what's going on, i.e. if there's a single item in the dictionary argument then only the key is passed through to the initializer function, rather than the dictionary itself. If there are multiple items in the dictionary used as the argument being passed to the initializer function then the above error message is displayed, reporting the number of items in the dictionary as the number of positional arguments given, so it's somehow passing the keys of the dictionary as arguments rather than the dictionary itself.



Can anyone see what I'm doing wrong here? Thanks in advance for your suggestions.






python python-multiprocessing







share|improve this question


















  • 2





    add a comma to make initargs to be a tuple, initargs=(arrays_dict,)

    – Jacky
    Nov 19 '18 at 14:59
















1















I'm using a function to initialize the worker processes of a process pool, and this function has a single argument, which is a dictionary. When the process pool is created and the function is called to initialize each worker process I get an error regarding the wrong number of arguments:



TypeError: _init_worker() takes 1 positional argument but 2 were given


The process initializer function being used:



def _init_worker(shared_arrays):



    _global_shared_arrays = shared_arrays


The initializer is being called in the normal way for each worker process:



with multiprocessing.Pool(processes=_NUMBER_OF_WORKER_PROCESSES,

                          initializer=_init_worker, initargs=(arrays_dict)) as pool:


I think this has something to do with how dictionaries are passed as the argument, in that the above error always lists the number of items in the dictionary as the number of positional arguments that were passed, as if what's being passed is the keys of the dictionary rather than the dictionary itself. When I step into the code in the debugger this is exactly what's going on, i.e. if there's a single item in the dictionary argument then only the key is passed through to the initializer function, rather than the dictionary itself. If there are multiple items in the dictionary used as the argument being passed to the initializer function then the above error message is displayed, reporting the number of items in the dictionary as the number of positional arguments given, so it's somehow passing the keys of the dictionary as arguments rather than the dictionary itself.



Can anyone see what I'm doing wrong here? Thanks in advance for your suggestions.






python python-multiprocessing







share|improve this question


















  • 2





    add a comma to make initargs to be a tuple, initargs=(arrays_dict,)

    – Jacky
    Nov 19 '18 at 14:59














1












1








1








I'm using a function to initialize the worker processes of a process pool, and this function has a single argument, which is a dictionary. When the process pool is created and the function is called to initialize each worker process I get an error regarding the wrong number of arguments:



TypeError: _init_worker() takes 1 positional argument but 2 were given


The process initializer function being used:



def _init_worker(shared_arrays):



    _global_shared_arrays = shared_arrays


The initializer is being called in the normal way for each worker process:



with multiprocessing.Pool(processes=_NUMBER_OF_WORKER_PROCESSES,

                          initializer=_init_worker, initargs=(arrays_dict)) as pool:


I think this has something to do with how dictionaries are passed as the argument, in that the above error always lists the number of items in the dictionary as the number of positional arguments that were passed, as if what's being passed is the keys of the dictionary rather than the dictionary itself. When I step into the code in the debugger this is exactly what's going on, i.e. if there's a single item in the dictionary argument then only the key is passed through to the initializer function, rather than the dictionary itself. If there are multiple items in the dictionary used as the argument being passed to the initializer function then the above error message is displayed, reporting the number of items in the dictionary as the number of positional arguments given, so it's somehow passing the keys of the dictionary as arguments rather than the dictionary itself.



Can anyone see what I'm doing wrong here? Thanks in advance for your suggestions.






python python-multiprocessing







share|improve this question














I'm using a function to initialize the worker processes of a process pool, and this function has a single argument, which is a dictionary. When the process pool is created and the function is called to initialize each worker process I get an error regarding the wrong number of arguments:



TypeError: _init_worker() takes 1 positional argument but 2 were given


The process initializer function being used:



def _init_worker(shared_arrays):



    _global_shared_arrays = shared_arrays


The initializer is being called in the normal way for each worker process:



with multiprocessing.Pool(processes=_NUMBER_OF_WORKER_PROCESSES,

                          initializer=_init_worker, initargs=(arrays_dict)) as pool:


I think this has something to do with how dictionaries are passed as the argument, in that the above error always lists the number of items in the dictionary as the number of positional arguments that were passed, as if what's being passed is the keys of the dictionary rather than the dictionary itself. When I step into the code in the debugger this is exactly what's going on, i.e. if there's a single item in the dictionary argument then only the key is passed through to the initializer function, rather than the dictionary itself. If there are multiple items in the dictionary used as the argument being passed to the initializer function then the above error message is displayed, reporting the number of items in the dictionary as the number of positional arguments given, so it's somehow passing the keys of the dictionary as arguments rather than the dictionary itself.



Can anyone see what I'm doing wrong here? Thanks in advance for your suggestions.






python python-multiprocessing




python python-multiprocessing






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 19 '18 at 14:44









James AdamsJames Adams

3,313125283




3,313125283








  • 2





    add a comma to make initargs to be a tuple, initargs=(arrays_dict,)

    – Jacky
    Nov 19 '18 at 14:59














  • 2





    add a comma to make initargs to be a tuple, initargs=(arrays_dict,)

    – Jacky
    Nov 19 '18 at 14:59








2




2





add a comma to make initargs to be a tuple, initargs=(arrays_dict,)

– Jacky
Nov 19 '18 at 14:59





add a comma to make initargs to be a tuple, initargs=(arrays_dict,)

– Jacky
Nov 19 '18 at 14:59












2 Answers
2






active

oldest

votes


















1














if you look at the documentation here



you will see the following: 



If initializer is not None then each worker process will   

call initializer(*initargs) when it starts.


as you can see the the args for the initializer function are are being unpacked by the * operator.

So your custom init function should be ready to accept more than one argument in case you pass it a dict with more than one element or else it will fail.

Something like this: def _init_worker(*shared_arrays)






share|improve this answer


























  • Thank you for this clarification. I remedied the issue by surrounding the dictionary argument with brackets, it's now being unpacked as expected. As suggested by @Jacky in the comment above, using a comma after the argument also works, as this turns the argument into a tuple.

    – James Adams
    Nov 19 '18 at 15:06





















0














initargs will be unpacked, so you must pass a tuple, like



initargs=(arrays_dict,)





share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53377028%2fpython-multiprocessing-how-to-pass-a-dictionary-as-the-argument-of-a-worker-p%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1














    if you look at the documentation here



    you will see the following: 



    If initializer is not None then each worker process will   
    
call initializer(*initargs) when it starts.


    as you can see the the args for the initializer function are are being unpacked by the * operator.

    So your custom init function should be ready to accept more than one argument in case you pass it a dict with more than one element or else it will fail.

    Something like this: def _init_worker(*shared_arrays)






    share|improve this answer


























    • Thank you for this clarification. I remedied the issue by surrounding the dictionary argument with brackets, it's now being unpacked as expected. As suggested by @Jacky in the comment above, using a comma after the argument also works, as this turns the argument into a tuple.

      – James Adams
      Nov 19 '18 at 15:06


















    1














    if you look at the documentation here



    you will see the following: 



    If initializer is not None then each worker process will   
    
call initializer(*initargs) when it starts.


    as you can see the the args for the initializer function are are being unpacked by the * operator.

    So your custom init function should be ready to accept more than one argument in case you pass it a dict with more than one element or else it will fail.

    Something like this: def _init_worker(*shared_arrays)






    share|improve this answer


























    • Thank you for this clarification. I remedied the issue by surrounding the dictionary argument with brackets, it's now being unpacked as expected. As suggested by @Jacky in the comment above, using a comma after the argument also works, as this turns the argument into a tuple.

      – James Adams
      Nov 19 '18 at 15:06
















    1












    1








    1







    if you look at the documentation here



    you will see the following: 



    If initializer is not None then each worker process will   
    
call initializer(*initargs) when it starts.


    as you can see the the args for the initializer function are are being unpacked by the * operator.

    So your custom init function should be ready to accept more than one argument in case you pass it a dict with more than one element or else it will fail.

    Something like this: def _init_worker(*shared_arrays)






    share|improve this answer















    if you look at the documentation here



    you will see the following: 



    If initializer is not None then each worker process will   
    
call initializer(*initargs) when it starts.


    as you can see the the args for the initializer function are are being unpacked by the * operator.

    So your custom init function should be ready to accept more than one argument in case you pass it a dict with more than one element or else it will fail.

    Something like this: def _init_worker(*shared_arrays)







    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited Nov 19 '18 at 15:02

























    answered Nov 19 '18 at 14:56









    eladm26eladm26

    387314




    387314













    • Thank you for this clarification. I remedied the issue by surrounding the dictionary argument with brackets, it's now being unpacked as expected. As suggested by @Jacky in the comment above, using a comma after the argument also works, as this turns the argument into a tuple.

      – James Adams
      Nov 19 '18 at 15:06





















    • Thank you for this clarification. I remedied the issue by surrounding the dictionary argument with brackets, it's now being unpacked as expected. As suggested by @Jacky in the comment above, using a comma after the argument also works, as this turns the argument into a tuple.

      – James Adams
      Nov 19 '18 at 15:06



















    Thank you for this clarification. I remedied the issue by surrounding the dictionary argument with brackets, it's now being unpacked as expected. As suggested by @Jacky in the comment above, using a comma after the argument also works, as this turns the argument into a tuple.

    – James Adams
    Nov 19 '18 at 15:06







    Thank you for this clarification. I remedied the issue by surrounding the dictionary argument with brackets, it's now being unpacked as expected. As suggested by @Jacky in the comment above, using a comma after the argument also works, as this turns the argument into a tuple.

    – James Adams
    Nov 19 '18 at 15:06















    0














    initargs will be unpacked, so you must pass a tuple, like



    initargs=(arrays_dict,)





    share|improve this answer




























      0














      initargs will be unpacked, so you must pass a tuple, like



      initargs=(arrays_dict,)





      share|improve this answer


























        0












        0








        0







        initargs will be unpacked, so you must pass a tuple, like



        initargs=(arrays_dict,)





        share|improve this answer













        initargs will be unpacked, so you must pass a tuple, like



        initargs=(arrays_dict,)






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 19 '18 at 15:00









        pxepxe

        1207




        1207






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53377028%2fpython-multiprocessing-how-to-pass-a-dictionary-as-the-argument-of-a-worker-p%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            Guess what letter conforming each word

            Image
            6 $begingroup$ This is an entry in the Fortnightly Topic Challenge #41: Short and Sweet. 1 letter only could be mean anything . With an additional letter in front , could be used as medicine component and something you definitely learn or at least heard once. Add another letter in front , you would possibly hear that many times . Add an additional letter in front , you would found an independent country . With an additional letter, in the end , you would found something related to apple . Add another letter in the end , you would found something bad . Hint: Each letter is unique . word knowledge letters chemistry share | improve this question ...
            Read more

            Run scheduled task as local user group (not BUILTIN)

            Image
            0 I am trying to run a scheduled task with User Group (not a single user). This is working fine when I in the field "when running a task, use the following user account:" insert BUILTIN User Group (for example: Administrators or Users). But when I try it with the created Local User Group (example: TestGroup) it is not working. The Users in the TestGroup are the same as in the Administrators group. What could be the problem? Are there some restrictions to the Created User Group compared to BUILTINUsers? scheduled-tasks taskscheduler windowstaskschedule share | improve this question asked Nov 15 '18 at 15:36 Far...
            Read more

            Port of Spain

            Image
            For ports in the country of Spain, see List of ports in Spain. City in City of Port of Spain, Trinidad and Tobago Port of Spain City City of Port of Spain Top to bottom (left to right): Skyline of Port of Spain from Gulf of Paria; National Academy for the Performing Arts, Independence Square, Eric Williams Plaza; MovieTowne, Port of Port of Spain Motto(s):  "We Gather Strength As We Go Along" Port of Spain Coordinates: 10°40′N 61°31′W  /  10.667°N 61.517°W  / 10.667; -61.517 Coordinates: 10°40′N 61°31′W  /  10.667°N 61.517°W  / 10.667; -61.517 Country   Trinidad and Tobago Jurisdiction City of Port of Spain Settled 1560 Incorporated (town) 1853 Incorporated (city) 1914 Government  • Mayor Joel Martinez  • Governing body City Corporation Area [1]  • City 12 km 2 (5 sq mi) Elevation [2] 66 m (217 ft) Population (2011) [1]  • City 37,074  • Rank 3rd  • Density 3,090/km 2 (8,000/sq mi)...
            Read more