【spark源码修改】hive3.1.3 spark3.5.4编译，需要修改源码，最终编译成功
- 1. 准备安装包与maven编译环境
- 1.1 安装环境准备
- 1.2 修改pom
- 1.3 打包命令
- 2. 编译与问题解决
- 2.1 开始编译 失败， 缺包pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde
- 2.2 Hive Spark Remote Client 模块编译失败，类找不到，原因是升级后，类不兼容
- 3. 修改源码
- 4. 继续编译
- 4.1 有一个test的类不兼容，直接删掉就行，重新编译
- 4.2 Hive Llap Server 编译失败
- 4.3 编译完成
- 4.4 编译后的安装包
1. 准备安装包与maven编译环境
1.1 安装环境准备
安装包到官网找， hive.apache.org, spark.apache.org
maven环境准备： 初始化只改了settings的ali源
1.2 修改pom
pom版本修改前：
<spark.version>2.3.0</spark.version><scala.binary.version>2.11</scala.binary.version><scala.version>2.11.8</scala.version>
修改后：
<spark.version>3.5.4</spark.version><scala.binary.version>2.13</scala.binary.version><scala.version>2.13.8</scala.version>
1.3 打包命令
mvn clean package -Pdist -DskipTests -Dmaven.javadoc.skip=true
2. 编译与问题解决
2.1 开始编译 失败， 缺包pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde
pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde 找不到
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.2/hadoop-mapreduce-client-core-2.7.2.pom (3.5 kB at 12 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.6.4/hadoop-common-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.6.4/hadoop-common-2.6.4.pom (31 kB at 77 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project-dist/2.6.4/hadoop-project-dist-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project-dist/2.6.4/hadoop-project-dist-2.6.4.pom (18 kB at 48 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project/2.6.4/hadoop-project-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project/2.6.4/hadoop-project-2.6.4.pom (43 kB at 106 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-main/2.6.4/hadoop-main-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-main/2.6.4/hadoop-main-2.6.4.pom (18 kB at 47 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.4/hadoop-annotations-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.4/hadoop-annotations-2.6.4.pom (2.4 kB at 8.0 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.6.4/hadoop-auth-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.6.4/hadoop-auth-2.6.4.pom (7.7 kB at 20 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.2.1/hive-storage-api-2.2.1.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.2.1/hive-storage-api-2.2.1.pom (5.4 kB at 12 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hive 3.1.3:
[INFO]
[INFO] Hive Upgrade Acid .................................. FAILURE [02:30 min]
[INFO] Hive ............................................... SKIPPED
[INFO] Hive Classifications ............................... SKIPPED
[INFO] Hive Shims Common .................................. SKIPPED
[INFO] Hive Shims 0.23 .................................... SKIPPED
[INFO] Hive Shims Scheduler ............................... SKIPPED
[INFO] Hive Shims ......................................... SKIPPED
[INFO] Hive Common ........................................ SKIPPED
[INFO] Hive Service RPC ................................... SKIPPED
[INFO] Hive Serde ......................................... SKIPPED
[INFO] Hive Standalone Metastore .......................... SKIPPED
[INFO] Hive Metastore ..................................... SKIPPED
[INFO] Hive Vector-Code-Gen Utilities ..................... SKIPPED
[INFO] Hive Llap Common ................................... SKIPPED
[INFO] Hive Llap Client ................................... SKIPPED
[INFO] Hive Llap Tez ...................................... SKIPPED
[INFO] Hive Spark Remote Client ........................... SKIPPED
[INFO] Hive Query Language ................................ SKIPPED
[INFO] Hive Llap Server ................................... SKIPPED
[INFO] Hive Service ....................................... SKIPPED
[INFO] Hive Accumulo Handler .............................. SKIPPED
[INFO] Hive JDBC .......................................... SKIPPED
[INFO] Hive Beeline ....................................... SKIPPED
[INFO] Hive CLI ........................................... SKIPPED
[INFO] Hive Contrib ....................................... SKIPPED
[INFO] Hive Druid Handler ................................. SKIPPED
[INFO] Hive HBase Handler ................................. SKIPPED
[INFO] Hive JDBC Handler .................................. SKIPPED
[INFO] Hive HCatalog ...................................... SKIPPED
[INFO] Hive HCatalog Core ................................. SKIPPED
[INFO] Hive HCatalog Pig Adapter .......................... SKIPPED
[INFO] Hive HCatalog Server Extensions .................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client .................. SKIPPED
[INFO] Hive HCatalog Webhcat .............................. SKIPPED
[INFO] Hive HCatalog Streaming ............................ SKIPPED
[INFO] Hive HPL/SQL ....................................... SKIPPED
[INFO] Hive Streaming ..................................... SKIPPED
[INFO] Hive Llap External Client .......................... SKIPPED
[INFO] Hive Shims Aggregator .....................
