Dataset.records doesn't create enough partitions for Databricks autoscaling

[tdsmith]

The value of sc.defaultParallelism increases as an autoscaling Databricks Spark cluster grows. At startup, it's only 8. This means that _group_by_size_greedy(summaries, 10 * sc.defaultParallelism) in Dataset.records creates only 80 partitions on a quiet autoscaling cluster. A Databricks cluster set to autoscale up to 30 nodes will eventually have 240 worker processes available, but will only be able to use 80 of them to parse pings.

It would be nice if it were possible to pass in a number of partitions, or if the library could detect that it's on an autoscaling Databricks cluster and do something clever.