Using KuduContext in pyspark











up vote
0
down vote

favorite












I would like to use kudu with pyspark.
While I can use it with:



sc.read.format('org.apache.kudu.spark.kudu').option('kudu.master',"hdp1:7051").option('kudu.table',"impala::test.z_kudu_tab").load() 


I cannot find a way to import KuduContext.
I'm working in a jupyter notebook, and importing it with:



os.environ["PYSPARK_SUBMIT_ARGS"] = "--driver-memory 2g --packages com.ibm.spss.hive.serde2.xml:hivexmlserde:1.0.5.3 --packages org.apache.kudu:kudu-spark2_2.11:1.7.0 pyspark-shell"


My not working code:



kudu_Context = KuduContext("es2-hdp1:7051", sc)


Dies with error:



NameError: name 'KuduContext' is not defined




I've also tried:



kudu_context = sc._jvm.org.apache.kudu.spark.kudu.KuduContext("hdp1:7051", sc.sparkContext)


which dies with error:



AttributeError: 'SparkContext' object has no attribute '_get_object_id'









share|improve this question




























    up vote
    0
    down vote

    favorite












    I would like to use kudu with pyspark.
    While I can use it with:



    sc.read.format('org.apache.kudu.spark.kudu').option('kudu.master',"hdp1:7051").option('kudu.table',"impala::test.z_kudu_tab").load() 


    I cannot find a way to import KuduContext.
    I'm working in a jupyter notebook, and importing it with:



    os.environ["PYSPARK_SUBMIT_ARGS"] = "--driver-memory 2g --packages com.ibm.spss.hive.serde2.xml:hivexmlserde:1.0.5.3 --packages org.apache.kudu:kudu-spark2_2.11:1.7.0 pyspark-shell"


    My not working code:



    kudu_Context = KuduContext("es2-hdp1:7051", sc)


    Dies with error:



    NameError: name 'KuduContext' is not defined




    I've also tried:



    kudu_context = sc._jvm.org.apache.kudu.spark.kudu.KuduContext("hdp1:7051", sc.sparkContext)


    which dies with error:



    AttributeError: 'SparkContext' object has no attribute '_get_object_id'









    share|improve this question


























      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I would like to use kudu with pyspark.
      While I can use it with:



      sc.read.format('org.apache.kudu.spark.kudu').option('kudu.master',"hdp1:7051").option('kudu.table',"impala::test.z_kudu_tab").load() 


      I cannot find a way to import KuduContext.
      I'm working in a jupyter notebook, and importing it with:



      os.environ["PYSPARK_SUBMIT_ARGS"] = "--driver-memory 2g --packages com.ibm.spss.hive.serde2.xml:hivexmlserde:1.0.5.3 --packages org.apache.kudu:kudu-spark2_2.11:1.7.0 pyspark-shell"


      My not working code:



      kudu_Context = KuduContext("es2-hdp1:7051", sc)


      Dies with error:



      NameError: name 'KuduContext' is not defined




      I've also tried:



      kudu_context = sc._jvm.org.apache.kudu.spark.kudu.KuduContext("hdp1:7051", sc.sparkContext)


      which dies with error:



      AttributeError: 'SparkContext' object has no attribute '_get_object_id'









      share|improve this question















      I would like to use kudu with pyspark.
      While I can use it with:



      sc.read.format('org.apache.kudu.spark.kudu').option('kudu.master',"hdp1:7051").option('kudu.table',"impala::test.z_kudu_tab").load() 


      I cannot find a way to import KuduContext.
      I'm working in a jupyter notebook, and importing it with:



      os.environ["PYSPARK_SUBMIT_ARGS"] = "--driver-memory 2g --packages com.ibm.spss.hive.serde2.xml:hivexmlserde:1.0.5.3 --packages org.apache.kudu:kudu-spark2_2.11:1.7.0 pyspark-shell"


      My not working code:



      kudu_Context = KuduContext("es2-hdp1:7051", sc)


      Dies with error:



      NameError: name 'KuduContext' is not defined




      I've also tried:



      kudu_context = sc._jvm.org.apache.kudu.spark.kudu.KuduContext("hdp1:7051", sc.sparkContext)


      which dies with error:



      AttributeError: 'SparkContext' object has no attribute '_get_object_id'






      pyspark apache-kudu






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 8 at 11:50

























      asked Nov 8 at 11:20









      Federico Ponzi

      1,22332243




      1,22332243





























          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














           

          draft saved


          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53206722%2fusing-kuducontext-in-pyspark%23new-answer', 'question_page');
          }
          );

          Post as a guest





































          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















           

          draft saved


          draft discarded



















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53206722%2fusing-kuducontext-in-pyspark%23new-answer', 'question_page');
          }
          );

          Post as a guest




















































































          Popular posts from this blog

          Guess what letter conforming each word

          Port of Spain

          Run scheduled task as local user group (not BUILTIN)