Agfdhyk

It happens that I am manipulating some data using Azure Databricks. Such data is in an Azure Data Lake Storage Gen1. I mounted the data into DBFS, but now, after transforming the data I would like to write it back into my data lake.

To mount the data I used the following:

configs = {"dfs.adls.oauth2.access.token.provider.type": "ClientCredential",

       "dfs.adls.oauth2.client.id": "<your-service-client-id>",

       "dfs.adls.oauth2.credential": "<your-service-credentials>",

       "dfs.adls.oauth2.refresh.url": "https://login.microsoftonline.com/<your-directory-id>/oauth2/token"}



dbutils.fs.mount(source = "adl://<your-data-lake-store-account-name>.azuredatalakestore.net/<your-directory-name>", mount_point = "/mnt/<mount-name>",extra_configs = configs)

I want to write back a .csv file. For this task I am using the following line

dfGPS.write.mode("overwrite").format("com.databricks.spark.csv").option("header", "true").csv("adl://<your-data-lake-store-account-name>.azuredatalakestore.net/<your-directory-name>")

However, I get the following error:

IllegalArgumentException: u'No value for dfs.adls.oauth2.access.token.provider found in conf file.'

Any piece of code that can help me? Or link that walks me through.

Thanks.

If you mount Azure Data Lake Store, you should use the mountpoint to store your data, instead of "adl://...". For details how to mount Azure Data Lake Store
(ADLS ) Gen1 see the Azure Databricks documentation. You can verify if the mountpoint works with:

dbutils.fs.ls("/mnt/<newmountpoint>")

So try after mounting ADLS Gen 1:

dfGPS.write.mode("overwrite").format("com.databricks.spark.csv").option("header", "true").csv("mnt/<mount-name>/<your-directory-name>")

This should work if you added the mountpoint properly and you have also the access rights with the Service Principal on the ADLS.

Spark writes always multiple files in a directory, because each partition is saved individually. See also the following stackoverflow question.