CDH6.3.2 多 Spark 版本共存

news/2025/1/23 2:07:05/

一 部署Spark客户端

1.1 部署spark3客户端

tar -zxvf spark-3.3.1-bin-3.0.0-cdh6.3.2.tgz -C /opt/cloudera/parcels/CDH/lib
cd /opt/cloudera/parcels/CDH/lib
mv spark-3.3.1-bin-3.0.0-cdh6.3.2/ spark3

将 CDH 集群的 spark-env.sh 复制到 /opt/cloudera/parcels/CDH/lib/spark3/conf 下:

cp /etc/spark/conf/spark-env.sh  /opt/cloudera/parcels/CDH/lib/spark3/conf
chmod +x /opt/cloudera/parcels/CDH/lib/spark3/conf/spark-env.sh#修改 spark-env.sh
vim /opt/cloudera/parcels/CDH/lib/spark3/conf/spark-env.shexport SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark3
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}

将 gateway 节点的 hive-site.xml 复制到 spark3/conf 目录下，不需要做变动:

cp /etc/hive/conf/hive-site.xml /opt/cloudera/parcels/CDH/lib/spark3/conf/

1.2 部署Spark2客户端

tar -zxvf spark-2.4.0-bin-hadoop2.7.tgz -C /opt/cloudera/parcels/CDH/lib
cd /opt/cloudera/parcels/CDH/lib
mv spark-2.4.0-bin-hadoop2.7/ spark2

将 CDH 集群的 spark-env.sh 复制到 /opt/cloudera/parcels/CDH/lib/spark2/conf 下:

cp /etc/spark/conf/spark-env.sh  /opt/cloudera/parcels/CDH/lib/spark2/conf
chmod +x /opt/cloudera/parcels/CDH/lib/spark2/conf/spark-env.sh#修改 spark-env.sh
vim /opt/cloudera/parcels/CDH/lib/spark2/conf/spark-env.shexport SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark2
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}

将 gateway 节点的 hive-site.xml 复制到 spark2/conf 目录下，不需要做变动:

cp /etc/hive/conf/hive-site.xml /opt/cloudera/parcels/CDH/lib/spark2/conf/

二 创建spark-sql

2.1 spark3

vim /opt/cloudera/parcels/CDH/bin/spark3-sql#!/bin/bash 
export HADOOP_CONF_DIR=/etc/hadoop/conf
export YARN_CONF_DIR=/etc/hadoop/conf
SOURCE="${BASH_SOURCE[0]}"  
BIN_DIR="$( dirname "$SOURCE" )"  
while [ -h "$SOURCE" ]  
do  SOURCE="$(readlink "$SOURCE")"  [[ $SOURCE != /* ]] && SOURCE="$BIN_DIR/$SOURCE"  BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"  
done  
BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"  
LIB_DIR=$BIN_DIR/../lib  
export HADOOP_HOME=$LIB_DIR/hadoop  # Autodetect JAVA_HOME if not defined  
. $LIB_DIR/bigtop-utils/bigtop-detect-javahome  exec $LIB_DIR/spark3/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver "$@"

配置 spark-sql 快捷方式

chmod +x /opt/cloudera/parcels/CDH/bin/spark3-sql
alternatives --install /usr/bin/spark-sql spark-sql /opt/cloudera/parcels/CDH/bin/spark3-sql 1

2.2 spark2

vim /opt/cloudera/parcels/CDH/bin/spark2-sql#!/bin/bash 
export HADOOP_CONF_DIR=/etc/hadoop/conf
export YARN_CONF_DIR=/etc/hadoop/conf
SOURCE="${BASH_SOURCE[0]}"  
BIN_DIR="$( dirname "$SOURCE" )"  
while [ -h "$SOURCE" ]  
do  SOURCE="$(readlink "$SOURCE")"  [[ $SOURCE != /* ]] && SOURCE="$BIN_DIR/$SOURCE"  BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"  
done  
BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"  
LIB_DIR=$BIN_DIR/../lib  
export HADOOP_HOME=$LIB_DIR/hadoop  # Autodetect JAVA_HOME if not defined  
. $LIB_DIR/bigtop-utils/bigtop-detect-javahome  exec $LIB_DIR/spark2/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver "$@"

配置 spark-sql 快捷方式

chmod +x /opt/cloudera/parcels/CDH/bin/spark2-sql
alternatives --install /usr/bin/spark-sql spark-sql /opt/cloudera/parcels/CDH/bin/spark2-sql 2

三 配置conf

3.1 spark3

cd /opt/cloudera/parcels/CDH/lib/spark3/conf
## 开启日志
mv log4j2.properties.template log4j2.properties
## spark-defaults.conf 配置
cp /opt/cloudera/parcels/CDH/lib/spark/conf/spark-defaults.conf ./# 修改 spark-defaults.conf
vim /opt/cloudera/parcels/CDH/lib/spark3/conf/spark-defaults.conf
删除 spark.extraListeners、spark.sql.queryExecutionListeners、spark.yarn.jars
添加 spark.yarn.jars=hdfs:///spark/3versionJars/*hadoop fs -mkdir -p /spark/3versionJars
cd /opt/cloudera/parcels/CDH/lib/spark3/jars
hadoop fs -put *.jar /spark/3versionJars

3.2 spark2

cd /opt/cloudera/parcels/CDH/lib/spark2/conf
## 开启日志
mv log4j2.properties.template log4j2.properties
## spark-defaults.conf 配置
cp /opt/cloudera/parcels/CDH/lib/spark/conf/spark-defaults.conf ./# 修改 spark-defaults.conf
vim /opt/cloudera/parcels/CDH/lib/spark3/conf/spark-defaults.conf
删除 spark.extraListeners、spark.sql.queryExecutionListeners、spark.yarn.jars
添加 spark.yarn.jars=hdfs:///spark/2versionJars/*hadoop fs -mkdir -p /spark/2versionJars
cd /opt/cloudera/parcels/CDH/lib/spark2/jars
hadoop fs -put *.jar /spark/2versionJars

四 创建spark-submit

4.1 spark3

vim /opt/cloudera/parcels/CDH/bin/spark3-submit#!/usr/bin/env bash
export HADOOP_CONF_DIR=/etc/hadoop/conf
export YARN_CONF_DIR=/etc/hadoop/conf
SOURCE="${BASH_SOURCE[0]}"
BIN_DIR="$( dirname "$SOURCE" )"
while [ -h "$SOURCE" ]
doSOURCE="$(readlink "$SOURCE")"[[ $SOURCE != /* ]] && SOURCE="$BIN_DIR/$SOURCE"BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
done
BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
LIB_DIR=/opt/cloudera/parcels/CDH/lib
export HADOOP_HOME=$LIB_DIR/hadoop# Autodetect JAVA_HOME if not defined
. $LIB_DIR/bigtop-utils/bigtop-detect-javahome# disable randomized hash for string in Python 3.3+
export PYTHONHASHSEED=0exec $LIB_DIR/spark3/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"

配置 spark3-submit 快捷方式:

chmod +755 /opt/cloudera/parcels/CDH/bin/spark3-submit
alternatives --install /usr/bin/spark-submit spark-submit /opt/cloudera/parcels/CDH/bin/spark3-submit 1

4.2 spark2

vim /opt/cloudera/parcels/CDH/bin/spark2-submit#!/usr/bin/env bash
export HADOOP_CONF_DIR=/etc/hadoop/conf
export YARN_CONF_DIR=/etc/hadoop/conf
SOURCE="${BASH_SOURCE[0]}"
BIN_DIR="$( dirname "$SOURCE" )"
while [ -h "$SOURCE" ]
doSOURCE="$(readlink "$SOURCE")"[[ $SOURCE != /* ]] && SOURCE="$BIN_DIR/$SOURCE"BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
done
BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
LIB_DIR=/opt/cloudera/parcels/CDH/lib
export HADOOP_HOME=$LIB_DIR/hadoop# Autodetect JAVA_HOME if not defined
. $LIB_DIR/bigtop-utils/bigtop-detect-javahome# disable randomized hash for string in Python 3.3+
export PYTHONHASHSEED=0exec $LIB_DIR/spark2/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"

配置 spark2-submit 快捷方式:

chmod +755 /opt/cloudera/parcels/CDH/bin/spark2-submit
alternatives --install /usr/bin/spark-submit spark-submit /opt/cloudera/parcels/CDH/bin/spark2-submit 1

欢迎关注微信公众号：大数据AI


http://www.ppmy.cn/news/1347038.html

相关文章

gtkmm 与 Cambalache 与 Gtk::Builder (新手向)_

gtkmm 与 Cambalache 与 Gtk::Builder (新手向)_

文章目录 前言Cambalache检查Xml.cpp文件如何写才能显示UI首先creat获取ui里的对象显示 前言 新手刚刚使用时的笔记 Cambalache检查Xml 窗口右键inspect UI Definition切换到Xml视图, 可以全选复制粘贴到你的ui文件里, Cambalache 只能保存为.cmb工程文件, 导出也不知道导出…
阅读更多...
STM32单片机的基本原理与应用（七）

STM32单片机的基本原理与应用（七）

超声波测距实验 基本原理 超声波测距实验是STM32单片机通过控制HC-SR04超声波模块，使其发送超声波，遇到物体反射回超声波来实现距离测量，其原理就是在发射超声波到接收超声波会有一段时间，而超声波在空气中传播的速度为声速&…
阅读更多...
【RL】Bellman Equation （贝尔曼等式）

【RL】Bellman Equation （贝尔曼等式）

Lecture2: Bellman Equation State value 考虑grid-world的单步过程： S t → A t R t 1 , S t 1 S_t \xrightarrow[]{A_t} R_{t 1}, S_{t 1} St​At​ ​Rt1​,St1​ t t t, t 1 t 1 t1：时间戳 S t S_t St​：时间 t t t时所处的sta…
阅读更多...
CSS：三列布局

CSS：三列布局

三列布局是指左右两列定宽&#xff0c;中间自适应。最终效果如下&#xff1a; HTML&#xff1a; <div class"container"><div class"left"></div><div class"center"></div><div class"right">…
阅读更多...
Codeforces Round 923 (Div. 3)(A-E)

Codeforces Round 923 (Div. 3)(A-E)

C o d e f o r c e s R o u n d 923 ( D i v . 3 ) ( A − E ) \Huge{Codeforces Round 923 (Div. 3)(A-E)} CodeforcesRound923(Div.3)(A−E) 比赛地址&#xff1a;Codeforces Round 923 (Div. 3) Problems A. Make it White 思路 题目给出一个用’W’和’B’组成的字符串…
阅读更多...
React + SpringBoot + Minio实现文件的预览

React + SpringBoot + Minio实现文件的预览

思路&#xff1a;后端提供接口&#xff0c;从minio获取文件的预览链接&#xff0c;返回给前端&#xff0c;前端使用组件进行渲染展示 这里我从minio获取文件预览地址用到了一个最近刚开源的项目&#xff0c;挺好用的&#xff0c;大伙可以试试&#xff0c;用法也很简单 官网&am…
阅读更多...
输入：123 输出321

输入：123 输出321

给出一个 32 位的有符号整数&#xff0c;你需要将这个整数中每位上的数字进行反转。若翻转后超出取值范围&#xff0c;则输出0即可输入&#xff1a;123 输出321输入&#xff1a;-123 输出-321 #include <stdio.h> int main(int argc, char const *argv[]) { printf("…
阅读更多...
Win32 SDK Gui编程系列之--ListView自绘OwnerDraw(续)

Win32 SDK Gui编程系列之--ListView自绘OwnerDraw(续)

通过所有者绘制的列表视图（2） 所有者绘制列表视图的基础已在前一页中说明。本页将展示如何在所有者绘制列表视图中显示数据库表数据。 1、访问日志 正如在另一个页面中所述，本网站的访问日志目前是通过SQLite3数据库管理的。 以下是上述程序执行的结果。为…
阅读更多...
最新文章

Copyright @ 2022~2023