使用pyspark连接PostgreSQL

问题描述

我正在尝试使用 pyspark 连接到数据库，并且正在使用以下代码:

I am trying to connect to a database with pyspark and I am using the following code:

sqlctx = SQLContext(sc)
df = sqlctx.load(
    url = "jdbc:postgresql://[hostname]/[database]",
    dbtable = "(SELECT * FROM talent LIMIT 1000) as blah",
    password = "MichaelJordan",
    user =  "ScottyPippen",
    source = "jdbc",
    driver = "org.postgresql.Driver"
)

我收到以下错误:

知道为什么会这样吗?

编辑:我正在尝试在我的计算机上本地运行代码.

Edit: I am trying to run the code locally in my computer.

推荐答案

从 https 下载 PostgreSQL JDBC 驱动程序://jdbc.postgresql.org/download.html

然后用您的数据库配置值替换.

Then replace the database configuration values by yours.

from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.jars", "/path_to_postgresDriver/postgresql-42.2.5.jar") \
    .getOrCreate()

df = spark.read \
    .format("jdbc") \
    .option("url", "jdbc:postgresql://localhost:5432/databasename") \
    .option("dbtable", "tablename") \
    .option("user", "username") \
    .option("password", "password") \
    .option("driver", "org.postgresql.Driver") \
    .load()

df.printSchema()

更多信息:https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html

这篇关于使用pyspark连接PostgreSQL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！