如何创建具有指定架构的空DataFrame?

问题描述

我想在Scala中使用指定的架构在DataFrame上创建.我尝试使用JSON读取(我的意思是读取空文件)，但我认为这不是最佳实践.

I want to create on DataFrame with a specified schema in Scala. I have tried to use JSON read (I mean reading empty file) but I don't think that's the best practice.

推荐答案

让我们假设您想要一个具有以下架构的数据框:

Lets assume you want a data frame with the following schema:

root
 |-- k: string (nullable = true)
 |-- v: integer (nullable = false)

您只需为数据框定义架构，然后使用空的RDD[Row]:

You simply define schema for a data frame and use empty RDD[Row]:

import org.apache.spark.sql.types.{
    StructType, StructField, StringType, IntegerType}
import org.apache.spark.sql.Row

val schema = StructType(
    StructField("k", StringType, true) ::
    StructField("v", IntegerType, false) :: Nil)

// Spark < 2.0
// sqlContext.createDataFrame(sc.emptyRDD[Row], schema) 
spark.createDataFrame(sc.emptyRDD[Row], schema)

PySpark的等效功能几乎相同:

PySpark equivalent is almost identical:

from pyspark.sql.types import StructType, StructField, IntegerType, StringType

schema = StructType([
    StructField("k", StringType(), True), StructField("v", IntegerType(), False)
])

# or df = sc.parallelize([]).toDF(schema)

# Spark < 2.0 
# sqlContext.createDataFrame([], schema)
df = spark.createDataFrame([], schema)

使用类型为Tuple的Product的隐式编码器(仅限Scala):

Using implicit encoders (Scala only) with Product types like Tuple:

import spark.implicits._

Seq.empty[(String, Int)].toDF("k", "v")

或案例类:

case class KV(k: String, v: Int)

Seq.empty[KV].toDF

或

spark.emptyDataset[KV].toDF

这篇关于如何创建具有指定架构的空DataFrame?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！