I have a dataframe with timestamp in the following format “yyyy-MM-dd HH:mm:ss.SSSSSSS” I want to trim the milliseconds and nanoseconds from the given string and convert that into datetime type.
I tried using the to_timestamp() method to convert from string to timestamp format, I am successful in that but I am getting the milliseconds and Nanoseconds at the end.
I tried following to remove milliseconds but none of them worked.
I tried date Trucate method to remove the milliseconds it worked but
it converts the column to string format.
I tried with to_timestamp($”column_name”, “YYYY-mm-dd HH:MM:ss”) but
I am getting the default format as output. This method did not recoganize my
custom date time format. Default format I got is –>
Above is my code sample, can someone suggest what I should do here please? Thank you for your time 🙂
10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)
Leave an answer