Spark-shell Git Bash
1
I cannot get the spark-shell command to work on Git bash. I am fairly sure my environment is set up correctly as if I run spark-shell in any directory in command prompt, it works just fine for me.
However, when I run spark-shell in bash, it outputs this value instead of running the actual shell:
"C:Program FilesJavajdk1.8.0_191binjava" -cp "C:sparkspark-2.4.0-bin-
hadoop2.7/conf;C:sparkspark-2.4.0-bin-hadoop2.7jars*" "-
Dscala.usejavacp=true"-Xmx1g org.apache.spark.deploy.SparkSubmit --class
org.apache.spark.repl.Main --name "Spark shell" spark-shell
java bash scala apache-spark git-bash
asked Nov 16 '18 at 23:29
FoxloooFoxlooo
145
add a comment |
1
I cannot get the spark-shell command to work on Git bash. I am fairly sure my environment is set up correctly as if I run spark-shell in any directory in command prompt, it works just fine for me.
However, when I run spark-shell in bash, it outputs this value instead of running the actual shell:
"C:Program FilesJavajdk1.8.0_191binjava" -cp "C:sparkspark-2.4.0-bin-
hadoop2.7/conf;C:sparkspark-2.4.0-bin-hadoop2.7jars*" "-
Dscala.usejavacp=true"-Xmx1g org.apache.spark.deploy.SparkSubmit --class
org.apache.spark.repl.Main --name "Spark shell" spark-shell
java bash scala apache-spark git-bash
asked Nov 16 '18 at 23:29
FoxloooFoxlooo
145
Do you mean you are running spark from within a shell script file? If so, can you show us that script? If not, how are you running it?
– dave
Nov 17 '18 at 0:19
No, I'm just calling spark-shell without a script. I just call spark-shell in any directory and it runs in command prompt, however, for git bash, it displays the above output.
– Foxlooo
Nov 17 '18 at 5:31
add a comment |
1
1
1
1
I cannot get the spark-shell command to work on Git bash. I am fairly sure my environment is set up correctly as if I run spark-shell in any directory in command prompt, it works just fine for me.
However, when I run spark-shell in bash, it outputs this value instead of running the actual shell:
"C:Program FilesJavajdk1.8.0_191binjava" -cp "C:sparkspark-2.4.0-bin-
hadoop2.7/conf;C:sparkspark-2.4.0-bin-hadoop2.7jars*" "-
Dscala.usejavacp=true"-Xmx1g org.apache.spark.deploy.SparkSubmit --class
org.apache.spark.repl.Main --name "Spark shell" spark-shell
java bash scala apache-spark git-bash
asked Nov 16 '18 at 23:29
FoxloooFoxlooo
145
I cannot get the spark-shell command to work on Git bash. I am fairly sure my environment is set up correctly as if I run spark-shell in any directory in command prompt, it works just fine for me.
However, when I run spark-shell in bash, it outputs this value instead of running the actual shell:
"C:Program FilesJavajdk1.8.0_191binjava" -cp "C:sparkspark-2.4.0-bin-
hadoop2.7/conf;C:sparkspark-2.4.0-bin-hadoop2.7jars*" "-
Dscala.usejavacp=true"-Xmx1g org.apache.spark.deploy.SparkSubmit --class
org.apache.spark.repl.Main --name "Spark shell" spark-shell
java bash scala apache-spark git-bash
java bash scala apache-spark git-bash
asked Nov 16 '18 at 23:29
FoxloooFoxlooo
145
asked Nov 16 '18 at 23:29
FoxloooFoxlooo
145
asked Nov 16 '18 at 23:29
FoxloooFoxlooo
145
asked Nov 16 '18 at 23:29
FoxloooFoxlooo
145
asked Nov 16 '18 at 23:29
FoxloooFoxlooo
145
145
Do you mean you are running spark from within a shell script file? If so, can you show us that script? If not, how are you running it?
– dave
Nov 17 '18 at 0:19
No, I'm just calling spark-shell without a script. I just call spark-shell in any directory and it runs in command prompt, however, for git bash, it displays the above output.
– Foxlooo
Nov 17 '18 at 5:31
add a comment |
Do you mean you are running spark from within a shell script file? If so, can you show us that script? If not, how are you running it?
– dave
Nov 17 '18 at 0:19
No, I'm just calling spark-shell without a script. I just call spark-shell in any directory and it runs in command prompt, however, for git bash, it displays the above output.
– Foxlooo
Nov 17 '18 at 5:31
Do you mean you are running spark from within a shell script file? If so, can you show us that script? If not, how are you running it?
– dave
Nov 17 '18 at 0:19
Do you mean you are running spark from within a shell script file? If so, can you show us that script? If not, how are you running it?
– dave
Nov 17 '18 at 0:19
No, I'm just calling spark-shell without a script. I just call spark-shell in any directory and it runs in command prompt, however, for git bash, it displays the above output.
– Foxlooo
Nov 17 '18 at 5:31
No, I'm just calling spark-shell without a script. I just call spark-shell in any directory and it runs in command prompt, however, for git bash, it displays the above output.
– Foxlooo
Nov 17 '18 at 5:31
add a comment |
1 Answer
1
active
oldest
votes
0
In a git bash, try and use:
- simple quotes
'(instead of double-quotes")
/instead of:C:/.../...or/c/.../...
That would be more compatible with the shell used by Git bash.
answered Nov 17 '18 at 7:44
VonCVonC
834k29026253170
To clarify, the above is the output that I am returned, I am just callingspark-shell
– Foxlooo
Nov 18 '18 at 21:03
@Foxlooo Any chance to modify thespark-shellscript in order to test the syntax I propose?
– VonC
Nov 18 '18 at 21:05
I'm not sure what script it's running, I'm running it from the default built-in command from the installation of spark and scala. Also, keep in mind it works in command prompt
– Foxlooo
Nov 18 '18 at 21:35
@Foxlooo OK. I am not surprised it does work in a CMD command prompt, and not in a bash (where quotes are interpreted differently)
– VonC
Nov 18 '18 at 21:37
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53346696%2fspark-shell-git-bash%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
In a git bash, try and use:
- simple quotes
'(instead of double-quotes")
/instead of:C:/.../...or/c/.../...
That would be more compatible with the shell used by Git bash.
answered Nov 17 '18 at 7:44
VonCVonC
834k29026253170
To clarify, the above is the output that I am returned, I am just callingspark-shell
– Foxlooo
Nov 18 '18 at 21:03
@Foxlooo Any chance to modify thespark-shellscript in order to test the syntax I propose?
– VonC
Nov 18 '18 at 21:05
I'm not sure what script it's running, I'm running it from the default built-in command from the installation of spark and scala. Also, keep in mind it works in command prompt
– Foxlooo
Nov 18 '18 at 21:35
@Foxlooo OK. I am not surprised it does work in a CMD command prompt, and not in a bash (where quotes are interpreted differently)
– VonC
Nov 18 '18 at 21:37
add a comment |
0
In a git bash, try and use:
- simple quotes
'(instead of double-quotes")
/instead of:C:/.../...or/c/.../...
That would be more compatible with the shell used by Git bash.
answered Nov 17 '18 at 7:44
VonCVonC
834k29026253170
To clarify, the above is the output that I am returned, I am just callingspark-shell
– Foxlooo
Nov 18 '18 at 21:03
@Foxlooo Any chance to modify thespark-shellscript in order to test the syntax I propose?
– VonC
Nov 18 '18 at 21:05
I'm not sure what script it's running, I'm running it from the default built-in command from the installation of spark and scala. Also, keep in mind it works in command prompt
– Foxlooo
Nov 18 '18 at 21:35
@Foxlooo OK. I am not surprised it does work in a CMD command prompt, and not in a bash (where quotes are interpreted differently)
– VonC
Nov 18 '18 at 21:37
add a comment |
0
0
0
In a git bash, try and use:
- simple quotes
'(instead of double-quotes")
/instead of:C:/.../...or/c/.../...
That would be more compatible with the shell used by Git bash.
answered Nov 17 '18 at 7:44
VonCVonC
834k29026253170
In a git bash, try and use:
- simple quotes
'(instead of double-quotes")
/instead of:C:/.../...or/c/.../...
That would be more compatible with the shell used by Git bash.
answered Nov 17 '18 at 7:44
VonCVonC
834k29026253170
answered Nov 17 '18 at 7:44
VonCVonC
834k29026253170
answered Nov 17 '18 at 7:44
VonCVonC
834k29026253170
answered Nov 17 '18 at 7:44
VonCVonC
834k29026253170
834k29026253170
To clarify, the above is the output that I am returned, I am just callingspark-shell
– Foxlooo
Nov 18 '18 at 21:03
@Foxlooo Any chance to modify thespark-shellscript in order to test the syntax I propose?
– VonC
Nov 18 '18 at 21:05
I'm not sure what script it's running, I'm running it from the default built-in command from the installation of spark and scala. Also, keep in mind it works in command prompt
– Foxlooo
Nov 18 '18 at 21:35
@Foxlooo OK. I am not surprised it does work in a CMD command prompt, and not in a bash (where quotes are interpreted differently)
– VonC
Nov 18 '18 at 21:37
add a comment |
To clarify, the above is the output that I am returned, I am just callingspark-shell
– Foxlooo
Nov 18 '18 at 21:03
@Foxlooo Any chance to modify thespark-shellscript in order to test the syntax I propose?
– VonC
Nov 18 '18 at 21:05
I'm not sure what script it's running, I'm running it from the default built-in command from the installation of spark and scala. Also, keep in mind it works in command prompt
– Foxlooo
Nov 18 '18 at 21:35
@Foxlooo OK. I am not surprised it does work in a CMD command prompt, and not in a bash (where quotes are interpreted differently)
– VonC
Nov 18 '18 at 21:37
To clarify, the above is the output that I am returned, I am just calling
spark-shell– Foxlooo
Nov 18 '18 at 21:03
To clarify, the above is the output that I am returned, I am just calling
spark-shell– Foxlooo
Nov 18 '18 at 21:03
@Foxlooo Any chance to modify the
spark-shell script in order to test the syntax I propose?– VonC
Nov 18 '18 at 21:05
@Foxlooo Any chance to modify the
spark-shell script in order to test the syntax I propose?– VonC
Nov 18 '18 at 21:05
I'm not sure what script it's running, I'm running it from the default built-in command from the installation of spark and scala. Also, keep in mind it works in command prompt
– Foxlooo
Nov 18 '18 at 21:35
I'm not sure what script it's running, I'm running it from the default built-in command from the installation of spark and scala. Also, keep in mind it works in command prompt
– Foxlooo
Nov 18 '18 at 21:35
@Foxlooo OK. I am not surprised it does work in a CMD command prompt, and not in a bash (where quotes are interpreted differently)
– VonC
Nov 18 '18 at 21:37
@Foxlooo OK. I am not surprised it does work in a CMD command prompt, and not in a bash (where quotes are interpreted differently)
– VonC
Nov 18 '18 at 21:37
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53346696%2fspark-shell-git-bash%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Do you mean you are running spark from within a shell script file? If so, can you show us that script? If not, how are you running it?
– dave
Nov 17 '18 at 0:19
No, I'm just calling spark-shell without a script. I just call spark-shell in any directory and it runs in command prompt, however, for git bash, it displays the above output.
– Foxlooo
Nov 17 '18 at 5:31