如何通过happybase将值放入hbase表?

问题描述

我的开发环境是centos7，hbase 1.2.5，happybase 1.1.0，python 2.7，PyCharm，hadoop 2.7.3，spark 2.1我正在开发一个大数据软件.我需要将值放入HBase表中.这些值来自Spark RDD.以下是代码:

My development environment is centos7, hbase 1.2.5, happybase 1.1.0, python 2.7, PyCharm, hadoop 2.7.3, spark 2.1I am developing a big data software. I need put the values into HBase table. The values are from Spark RDD. The following are the codes:

import happybase
from pyspark import SparkContext, SparkConf

connection = happybase.Connection('localhost')
table = connection.table('tablename')
conf = SparkConf().setAppName("myFirstSparkApp").setMaster("local")
sc = SparkContext(conf=conf)
distFile = sc.textFile("/inputFilePath/")
newLines = distFile.filter(lambda x: 'filter":' in x)
newLines = newLines.map(lambda line:line.split('"'))
# The following line is working. Insert a row into the table.
table.put(b'row-key0', {'billCode:': '222', 'trayCode:': '222', 'pipeline:': '333'})
# But the following line is not working. what is wrong? Why?
newLines.foreach(lambda x: table.put(b'row-key', {'billCode:': x[7], 'trayCode:': x[3], 'pipeline:': x[11]}))

但是最后一行代码不起作用.错误消息是:

But the last line code is not working. The error messages are:

ImportError:没有名为cybin的模块pickle.PicklingError:无法序列化对象:ImportError:没有名为cybin的模块

ImportError: No module named cybinpickle.PicklingError: Could not serialize object: ImportError: No module named cybin

我是spark + happybase + python的新开发人员.怎么解决呢?请您的帮助.谢谢.

I am a new developer of spark+happybase+python. how to resolve it? Kindly need your help, please. Thank you.

推荐答案

这是一个简单的示例.

import happybase
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName("App").setMaster("local")
sc = SparkContext(conf=conf)
rdd = parallelize([("a","1"),("b","2")])
def func(x):
    conn = happybase.Connection('localhost')
    table = conn.table("table_name")
    table.put(x[0],{"cf:c":x[1]})
    conn.close()
rdd.foreach(func)

但并非十全十美，您可以参考 http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd 祝你好运.

But not perfect,You can refer http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrddGood luck.

这篇关于如何通过happybase将值放入hbase表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！