我要添加graphframes库。通常,此库是通过(例如)添加的:

 pyspark --packages graphframes:graphframes:0.7.0-spark2.4-s_2.11

然后你应该得到这样的东西:
     Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.3.0
      /_/

Using Python version 3.7.0 (default, Sep 25 2018 18:19:16)
SparkSession available as 'spark'.
>>>

我可以导入图形参数:>>>import graphframes
当我执行spark submit test.py时出现问题,其中test.py是:
import numpy as np
from operator import add
from pyspark.sql import *
from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import DoubleType, StringType, IntegerType
from pyspark.sql.functions import udf
from pyspark.sql.types import *
from pyspark.sql import SQLContext
import graphframes
from graphframes import *

if __name__ == "__main__":
    print ("==================================================================")
    print ("|                            main                                |")
    print ("==================================================================")
    # create Spark context with Spark configuration
    conf = SparkConf().setMaster("local[*]").setAppName("test").set('spark.executor.memory', '60G').set('spark.driver.memory', '60G').set('spark.driver.maxResultSize', '10G')

    spark = SparkSession.builder.master("localhost").config(conf=conf).getOrCreate()

    start_time = time.time()
    sc = spark.sparkContext
    sqlContext = SQLContext(sparkContext = sc)

    # Create a Vertex DataFrame with unique ID column "id"
    v = spark.createDataFrame([("a", "Alice", 34),("b", "Bob", 36), ("c", "Charlie", 30),], ["id", "name", "age"])
    v.show()
    # Create an Edge DataFrame with "src" and "dst" columns
    e = spark.createDataFrame([("a", "b", "friend"),("b", "c", "follow"),("c", "b", "follow"),], ["src", "dst", "relationship"])

    # Create a GraphFrame
    g = GraphFrame(v, e)

    # Query: Get in-degree of each vertex.
    g.inDegrees.show()

    # Query: Count the number of "follow" connections in the graph.
    g.edges.filter("relationship = 'follow'").count()

我得到以下异常:
import graphframes
ModuleNotFoundError: No module named 'graphframes'

我认为这个问题与——那些不能使python包可用或不能从Spark客户机/驱动程序加载的包有关。
我认为graphframe应该添加到python路径中。
如何解决这个问题?
如何在windows和Linux中应用1)中提出的解决方案?
我尝试了以下方法:
装上葡萄架
取出罐子里的东西
导航到“graphframe”目录并压缩其中的内容。
将生成的zip复制到我的主目录中:home/tam/
在.bashrc I集合中:
export PYSPARK_PYTHONPYSPARK_PYTHON=/home/tam/.local/easybuild/software/2017/Core/miniconda3/4.3.27/envs/testEnv/bin/python
export PYTHONPATH=${PYTHONPATH}:/home/tam/graphframes.zip

spark submit test.py,我得到以下错误:
 Traceback (most recent call last):


 File "/home/tam/test.py", line 3, in <module>
    import graphframes
  File "/home/tam/graphframes/__init__.py", line 2, in <module>
    from .graphframe import GraphFrame
  File "/home/tam/graphframes/graphframe.py", line 26, in <module>
    from graphframes.lib import Pregel
  File "/home/tam/graphframes/lib/__init__.py", line 3, in <module>
    from .pregel import Pregel
  File "/home/tam/graphframes/lib/pregel.py", line 24, in <module>
    from pyspark.ml.wrapper import JavaWrapper, _jvm
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/spark/2.3.0/python/lib/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module>
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/spark/2.3.0/python/lib/pyspark.zip/pyspark/ml/base.py", line 24, in <module>
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/spark/2.3.0/python/lib/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module>
ModuleNotFoundError: No module named 'numpy'
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

最佳答案

我通过以下方法解决了这个问题:
装上葡萄架
取出罐子里的东西
导航到“graphframe”目录并压缩其中的内容。
将生成的zip复制到我的主目录中:home/tam/
然后在.bashrc配置文件中:
只需将graphframes.zip的路径添加到$PYTHONPATH
然后:
运行以下命令:
spark submit--driver-class path/path to graphframes-0.7.0-spark2.3-s庠2.11.jar--jars“/path to graphframes-0.7.0-spark2.3-s庠2.11.jar”your庠script.py
注意:
--驱动类路径用于提到要添加到spark作业的“驱动程序”的“额外”jar
--驱动程序类路径只将jar推送到驱动程序机器。如果要将jar发送给“执行者”,则需要使用--jar
把上述两个论点加起来是非常重要的。否则会出现以下错误:

py4j.protocol.Py4JJavaError: An error occurred while calling o94.loadClass.
: java.lang.ClassNotFoundException: org.graphframes.GraphFramePythonAPI
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:214)
        at java.lang.Thread.run(Thread.java:745)

有助于

10-08 02:58