I have a dataframe with timestamp in the following format “yyyy-MM-dd HH:mm:ss.SSSSSSS” I want to trim the milliseconds and nanoseconds from the given string and convert that into datetime type.

I tried using the to_timestamp() method to convert from string to timestamp format, I am successful in that but I am getting the milliseconds and Nanoseconds at the end.

I tried following to remove milliseconds but none of them worked.

1. I tried date Trucate method to remove the milliseconds it worked but
   it converts the column to string format.

2. I tried with to_timestamp($”column_name”, “YYYY-mm-dd HH:MM:ss”) but
   I am getting the default format as output. This method did not recoganize my
   custom date time format. Default format I got is –>
   “YYYY-mm-ddTHH:MM:ss.sssss+sss”

   .withColumn(“datetype_timestamp”,
   to_timestamp(col(“RunStartTime”),”YYYY-mm-dd HH:MM:ss”))

Above is my code sample, can someone suggest what I should do here please? Thank you for your time 🙂

Cluster details:
10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)