To access a csv file in a S3 storage, I run the following scala code in spark.

sc.hadoopConfiguration.set("fs.s3a.awsAccessKeyId", "*****")
sc.hadoopConfiguration.set("fs.s3a.awsSecretAccessKey", "*****")
val myRDD = sc.textFile("s3a://*****/test_data_1.csv")
myRDD.count

This results in an error message:

java.nio.file.AccessDeniedException: s3a://*****/test_data_1.csv: getFileStatus on 
s3a://*****/test_data_1.csv: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden 
(Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden...)

I can access the respective bucket and file via a client using the same credentials.

Do you know why this is not possible via spark? Any help/ideas would be much appreciated!

Importantly, the underlying S3 storage is a DELL EMC ECS.

Best

targa