Agfdhyk

I am writing this as I cannot find the answer to this problem. I am using PySpark and running a script, except I cannot see where my other Spark sessions are or close them down. What is the best way to do the following?

I am not initializing any other SparkContext within this spark-submit job so it must be contexts left behind from some previous run? Note that i do not want to allowmultiplecontexts.

1. Check running SparkContexts.


2. Use existing SparkContexts if they exist.


3. Best way to modify this code.

from src import config

import pandas as pd

import plotly.graph_objs as go

from visualize_main import app

from dash.dependencies import Input, Output, State

from pyspark.sql import SparkSession

from datetime import datetime

import dash_core_components as dcc

import dash_html_components as html

from pyspark import SparkContext, SparkConf



spark =SparkSession.builder.master('local').appName("morphy_test111_dgdfgdf").config(conf=conf).getOrCreate()

I submit the job with teh following

spark-submit /app.py

I get this error

WARN SparkContext: Multiple running SparkContexts detected in the same JVM!

org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:

Any help much appreciated as I have not found a proper explanation for this.

a) Check running sparkcontexts b) Use existing sparkcontexts if they exist c) best way to modify this code

In that case remove:

SparkContext.stop("morphy_test11")

conf = SparkConf()

conf.set("spark.driver.allowMultipleContexts", "true")

and leave only

spark = SparkSession.builder.master('local').appName("morphy_test111_dgdfgdf").config(conf=conf).getOrCreate()

If there is an active context it will be reused.