PySpark custom shape function

Custom df.shape() function for PySpark dataframe.

import pyspark

def spark_shape(self) :
    return (self.count(), len(self.columns) )

pyspark.sql.dataframe.DataFrame.shape = spark_shape

df.shape()

Just reminder that .count() could be very slow for large tables.

---

Last update 2024-12-06