Spark RDD Windowing using pyspark





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}







-1















There is a Spark RDD, called rdd1. It has(key, value) pair and I have a list, whose elements are a tuple(key1,key2).



I want to get a rdd2, with rows `((key1,key2), (value of key1 in rdd1, value of key2 in rdd1)).



Can somebody help me?



rdd1:



key1, value1,
key2, value2,
key3, value3


array: [(key1,key2),(key2,key3)]



Result:



(key1,key2),value1,value2
(key2,key3),value2,value3


I have tried



spark.parallize(array).map(lambda x:)









share|improve this question































    -1















    There is a Spark RDD, called rdd1. It has(key, value) pair and I have a list, whose elements are a tuple(key1,key2).



    I want to get a rdd2, with rows `((key1,key2), (value of key1 in rdd1, value of key2 in rdd1)).



    Can somebody help me?



    rdd1:



    key1, value1,
    key2, value2,
    key3, value3


    array: [(key1,key2),(key2,key3)]



    Result:



    (key1,key2),value1,value2
    (key2,key3),value2,value3


    I have tried



    spark.parallize(array).map(lambda x:)









    share|improve this question



























      -1












      -1








      -1








      There is a Spark RDD, called rdd1. It has(key, value) pair and I have a list, whose elements are a tuple(key1,key2).



      I want to get a rdd2, with rows `((key1,key2), (value of key1 in rdd1, value of key2 in rdd1)).



      Can somebody help me?



      rdd1:



      key1, value1,
      key2, value2,
      key3, value3


      array: [(key1,key2),(key2,key3)]



      Result:



      (key1,key2),value1,value2
      (key2,key3),value2,value3


      I have tried



      spark.parallize(array).map(lambda x:)









      share|improve this question
















      There is a Spark RDD, called rdd1. It has(key, value) pair and I have a list, whose elements are a tuple(key1,key2).



      I want to get a rdd2, with rows `((key1,key2), (value of key1 in rdd1, value of key2 in rdd1)).



      Can somebody help me?



      rdd1:



      key1, value1,
      key2, value2,
      key3, value3


      array: [(key1,key2),(key2,key3)]



      Result:



      (key1,key2),value1,value2
      (key2,key3),value2,value3


      I have tried



      spark.parallize(array).map(lambda x:)






      apache-spark join pyspark rdd






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 23 '18 at 11:32









      thebluephantom

      3,37141033




      3,37141033










      asked Nov 22 '18 at 13:35









      user9465775user9465775

      6




      6
























          1 Answer
          1






          active

          oldest

          votes


















          -1














          sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:



          import org.apache.spark.mllib.rdd.RDDFunctions._
          val rdd1 = sc.parallelize(Seq(
          ( "key1", "value1"),
          ( "key2", "value2"),
          ( "key3", "value3"),
          ( "key4", "value4"),
          ( "key5", "value5")
          ))
          val rdd2 = rdd1.sliding(2)
          val rdd3 = rdd2.map(x => (x(0), x(1)))
          val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
          rdd4.collect


          also, the following and this is actually better of course... :



          val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
          rdd5.collect


          returns in both cases:



          res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))


          which I believe meets your needs, but not in pyspark.



          On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.






          share|improve this answer


























          • You will need to convert to pyspark.

            – thebluephantom
            Nov 22 '18 at 22:52












          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53432190%2fspark-rdd-windowing-using-pyspark%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          -1














          sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:



          import org.apache.spark.mllib.rdd.RDDFunctions._
          val rdd1 = sc.parallelize(Seq(
          ( "key1", "value1"),
          ( "key2", "value2"),
          ( "key3", "value3"),
          ( "key4", "value4"),
          ( "key5", "value5")
          ))
          val rdd2 = rdd1.sliding(2)
          val rdd3 = rdd2.map(x => (x(0), x(1)))
          val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
          rdd4.collect


          also, the following and this is actually better of course... :



          val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
          rdd5.collect


          returns in both cases:



          res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))


          which I believe meets your needs, but not in pyspark.



          On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.






          share|improve this answer


























          • You will need to convert to pyspark.

            – thebluephantom
            Nov 22 '18 at 22:52
















          -1














          sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:



          import org.apache.spark.mllib.rdd.RDDFunctions._
          val rdd1 = sc.parallelize(Seq(
          ( "key1", "value1"),
          ( "key2", "value2"),
          ( "key3", "value3"),
          ( "key4", "value4"),
          ( "key5", "value5")
          ))
          val rdd2 = rdd1.sliding(2)
          val rdd3 = rdd2.map(x => (x(0), x(1)))
          val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
          rdd4.collect


          also, the following and this is actually better of course... :



          val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
          rdd5.collect


          returns in both cases:



          res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))


          which I believe meets your needs, but not in pyspark.



          On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.






          share|improve this answer


























          • You will need to convert to pyspark.

            – thebluephantom
            Nov 22 '18 at 22:52














          -1












          -1








          -1







          sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:



          import org.apache.spark.mllib.rdd.RDDFunctions._
          val rdd1 = sc.parallelize(Seq(
          ( "key1", "value1"),
          ( "key2", "value2"),
          ( "key3", "value3"),
          ( "key4", "value4"),
          ( "key5", "value5")
          ))
          val rdd2 = rdd1.sliding(2)
          val rdd3 = rdd2.map(x => (x(0), x(1)))
          val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
          rdd4.collect


          also, the following and this is actually better of course... :



          val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
          rdd5.collect


          returns in both cases:



          res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))


          which I believe meets your needs, but not in pyspark.



          On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.






          share|improve this answer















          sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:



          import org.apache.spark.mllib.rdd.RDDFunctions._
          val rdd1 = sc.parallelize(Seq(
          ( "key1", "value1"),
          ( "key2", "value2"),
          ( "key3", "value3"),
          ( "key4", "value4"),
          ( "key5", "value5")
          ))
          val rdd2 = rdd1.sliding(2)
          val rdd3 = rdd2.map(x => (x(0), x(1)))
          val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
          rdd4.collect


          also, the following and this is actually better of course... :



          val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
          rdd5.collect


          returns in both cases:



          res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))


          which I believe meets your needs, but not in pyspark.



          On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Nov 23 '18 at 19:32

























          answered Nov 22 '18 at 21:55









          thebluephantomthebluephantom

          3,37141033




          3,37141033













          • You will need to convert to pyspark.

            – thebluephantom
            Nov 22 '18 at 22:52



















          • You will need to convert to pyspark.

            – thebluephantom
            Nov 22 '18 at 22:52

















          You will need to convert to pyspark.

          – thebluephantom
          Nov 22 '18 at 22:52





          You will need to convert to pyspark.

          – thebluephantom
          Nov 22 '18 at 22:52




















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53432190%2fspark-rdd-windowing-using-pyspark%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Guess what letter conforming each word

          Port of Spain

          Run scheduled task as local user group (not BUILTIN)