大数据魔法师

大数据魔法师

关注
发信
关注(28)粉丝(399)

Sqoop安装与配置-shell脚本一键安装配置

前言

本文介绍了如何使用Shell脚本一键安装Sqoop。Sqoop是一个用于在Apache Hadoop和结构化数据存储（如关系数据库）之间传输数据的工具。通过使用Sqoop，您可以方便地将数据从关系数据库导入到Hadoop集群中进行分析，并将结果数据导出回关系数据库。

在本文中，我们将提供一个Shell脚本，该脚本将自动下载、安装和配置Sqoop。您只需按照步骤执行脚本，即可完成Sqoop的安装和配置过程。

一、使用shell脚本一键安装

1. 复制脚本

首先，将以下脚本内容复制并保存为/tmp/install_sqoop.sh文件。

#!/bin/bash

installDir="/opt/module/sqoop"
sqoop_version="1.4.6"
mysql_connector_version="5.1.37"

if [ -z "$(command -v wget)" ]; then
  sudo yum install -y wget
  echo "wget安装完成"
fi
if [ -z "$JAVA_HOME" ]; then
  echo "JAVA_HOME未设置，请安装jdk1.8，设置Java环境变量再来执行此脚本"
  exit 1
fi
if [ ! -d "${installDir}" ]; then
  sudo mkdir -p "${installDir}"
  if [ $? -eq 0 ]; then
    echo "安装目录${installDir}已创建"
  else
    echo "请确保您有足够的权限来创建目录，请增加权限后再次执行"
    exit 1
  fi
fi
if [ ! -f /tmp/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz ]; then
  wget https://archive.apache.org/dist/sqoop/$sqoop_version/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz -P /tmp/
  if [ $? -eq 0 ]; then
    echo "sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz下载成功"
  else
    echo "sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz下载失败，请重试或手动下载到/tmp目录下再次执行"
    echo "下载地址：https://archive.apache.org/dist/sqoop/$sqoop_version/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz"
    exit 1
  fi
fi

if [ -d $installDir/sqoop-$sqoop_version ]; then
  echo "$installDir/sqoop-$sqoop_version 已存在，正在删除..."
  sudo rm -rf $installDir/sqoop-$sqoop_version
fi
tar -zxvf /tmp/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz -C $installDir
if [ $? -eq 0 ]; then
  echo "/tmp/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz解压成功"
else
  echo "/tmp/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz解压失败，请查看异常信息后重试"
  exit 1
fi

#设置sqoop环境变量
if [ -z "$SQOOP_HOME" ]; then
  echo >> ~/.bashrc
  echo '#SQOOP_HOME' >> ~/.bashrc
  echo "export SQOOP_HOME=$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha" >> ~/.bashrc
  echo 'export PATH=$PATH:$SQOOP_HOME/bin' >> ~/.bashrc
else
  echo "SQOOP_HOME已有设置：$SQOOP_HOME"
fi

cp $installDir/sqoop-"$sqoop_version".bin__hadoop-2.0.4-alpha/conf/sqoop-env-template.sh $installDir/sqoop-"$sqoop_version".bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh
if [ $? -ne 0 ]; then
  echo "cp $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env-template.sh $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh 失败"
  exit 1
fi
#设置bin/hadoop所在的路径
if [ -n "$HADOOP_HOME" ]; then
  sed -i "s|^#export HADOOP_COMMON_HOME=.*|export HADOOP_COMMON_HOME=$HADOOP_HOME|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"
  if [ $? -ne 0 ]; then
    echo "修改export HADOOP_COMMON_HOME失败"
    exit 1
  fi
fi
#设置hadoop-*-core.jar可用的路径
if [ -n "$HADOOP_HOME" ]; then
  sed -i "s|^#export HADOOP_MAPRED_HOME=.*|export HADOOP_MAPRED_HOME=$HADOOP_HOME|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"
  if [ $? -ne 0 ]; then
    echo "修改export HADOOP_MAPRED_HOME失败"
    exit 1
  fi
fi
#设置bin/hbase所在路径
if [ -n "$HBASE_HOME" ]; then
  sed -i "s|^#export HBASE_HOME=.*|export HBASE_HOME=$HBASE_HOME|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"
  if [ $? -ne 0 ]; then
    echo "修改export HBASE_HOME失败"
    exit 1
  fi
fi
#设置bin/hive可用的路径
if [ -n "$HIVE_HOME" ]; then
  sed -i "s|^#export HIVE_HOME=.*|export HIVE_HOME=$HIVE_HOME|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"
  if [ $? -ne 0 ]; then
    echo "修改export HIVE_HOME失败"
    exit 1
  fi
fi
#设置zookeeper config dir所在的路径
if [ -n "$ZOOKEEPER_HOME" ]; then
  sed -i "s|^#export ZOOCFGDIR=.*|export ZOOCFGDIR=$ZOOKEEPER_HOME/conf|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"
  if [ $? -ne 0 ]; then
    echo "修改export ZOOCFGDIR失败"
    exit 1
  fi
fi

if [ ! -f /tmp/mysql-connector-java-"$mysql_connector_version".jar ]; then
  wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/"$mysql_connector_version"/mysql-connector-java-"$mysql_connector_version".jar -P /tmp
  if [ $? -eq 0 ]; then
    echo "mysql驱动下载成功"
  else
    echo "mysql驱动下载失败，请手动下载到/tmp目录下再次执行次脚本"
    echo "下载地址：https://repo1.maven.org/maven2/mysql/mysql-connector-java/$mysql_connector_version/mysql-connector-java-$mysql_connector_version.jar"
    exit 1
  fi
fi

cp /tmp/mysql-connector-java-"$mysql_connector_version".jar $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/lib
if [ $? -ne 0 ]; then
  echo "cp /tmp/mysql-connector-java-"$mysql_connector_version".jar $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/lib 失败"
  exit 1
fi

cp $HIVE_HOME/lib/hive-common-*.jar $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/lib
if [ $? -ne 0 ]; then
  echo "cp $HIVE_HOME/lib/hive-common-*.jar $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/lib 失败"
  exit 1
fi

echo "sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha 下载、安装、配置成功"

exit 0

2. 增加执行权限

在终端中执行以下命令，为脚本添加执行权限。

chmod a+x /tmp/install_sqoop.sh

3. 执行脚本

执行以下命令，运行脚本开始下载、安装和配置Kafka。

/tmp/install_sqoop.sh

请等待安装完成，如有异常会有提示。

成功如下图所示：
Sqoop安装与配置-shell脚本一键安装配置-LMLPHP

4. 加载用户环境变量

source ~/.bashrc

5. 查看是否安装成功

sqoop version

成功如下图所示：
Sqoop安装与配置-shell脚本一键安装配置-LMLPHP

总结

通过本文提供的Shell脚本，您可以方便地一键安装Sqoop。Sqoop是一个强大的工具，可用于在Hadoop和关系数据库之间进行数据传输。安装Sqoop后，您可以使用其丰富的功能来导入和导出数据，从而更好地利用Hadoop集群进行数据分析和处理。

希望本教程对您有所帮助！如有任何疑问或问题，请随时在评论区留言。感谢阅读！

12-16 11:49
Powered by LMLPHP ©2024 大数据魔法师 0.001536