spark 写入mysql 中文数据 显示?? 或者 乱码

news/2024/11/28 11:44:02/

目录

前言

Spark报错:

解决办法:

总结一下:

报错:

解决:


前言

spark写入mysql中,查看中文数据 显示?? 或者 乱码

Spark报错:

Sat Nov 23 19:15:59 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:02,903{yy/MM/dd java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 more
2024-11-23 19:16:02,921{yy/MM/dd Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 76 in stage 12.0 failed 1 times, most recent failure: Lost task 76.0 in stage 12.0 (TID 242) (LAPTOP-P70BBG66 executor driver): java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 moreDriver stacktrace:at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2253)at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2202)at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2201)at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2201)at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1078)at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1078)at scala.Option.foreach(Option.scala:274)at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1078)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2440)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2382)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2371)at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2202)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2223)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2242)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2267)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$1(RDD.scala:1020)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:1018)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.saveTable(JdbcUtils.scala:854)at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:63)at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:301)at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:817)at com._02.gy.GZ.zhibiao.Test01$.main(Test01.scala:61)at com._02.gy.GZ.zhibiao.Test01.main(Test01.scala)
Caused by: java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 more
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,282{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,330{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,386{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)Process finished with exit code 1

解决办法:

我试了网上的很多办法:在saprk里面更改编码,直接把这一列更改为UTF-8

.withColumn("ChangeRecordState", decode($"ChangeRecordState", "UTF-8"))

使用新的driver,然后更改maven里面的jdbc链接版本

properties.setProperty("driver", "com.mysql.cj.jdbc.Driver")

等等....都没有解决问题

所以我开始着手自己解决,问题的核心无非就是编码冲突问题,因为已经写入进去了,所以连接配置啥的都没问题,主要解决编码问题

spark中可以打印出中文字段所以spark基本没有什么问题,我还是用了老方法

mysqlURL的配置项上加了?useUnicode=true&characterEncoding=UTF-8

val mysqlUrl = "jdbc:mysql://bigdata1:3306/shtd_industry?useUnicode=true&characterEncoding=UTF-8"

主要解决的是mysql,所以我开始查询mysql的默认编码是什么,我找到了命令

SHOW TABLE STATUS LIKE 'machine_state_time';

找到Collation字段对应的就是默认的编码  --  latin1_swedish_ci

找到了默认编码不是UTF-8,所以这里问题很明显了,就是把 latin1_swedish_ci 变成 UTF-8

所以我又接着查询,把表转换编码的命令

alter table machine_state_time character set utf8;

结果还是不行,当我试着找出问题,才发现我只把表改成了UTF-8,没有把字段改成UTF-8

alter table machine_state_time convert to character set utf8;

这样mysql的问题也解决了,运行一下还是报错

找到问题,scala语言也需要设置编码方式,所以我又加了一条配置项

System.setProperty("file.encoding", "UTF-8")

这样就可以了,运行成功,然后查看一下mysql中的内容

总结一下:

报错:

主要原因是因为编码冲突,scala,sparkmysql-connect连接,mysql三个编码不一致

解决:

设置scala的编码:

System.setProperty("file.encoding", "UTF-8")

设置spark的编码:

saprk本身就是UTF-8 不用设置

设置mysqlURL的编码:

val mysqlUrl = "jdbc:mysql://bigdata1:3306/shtd_industry?useUnicode=true&characterEncoding=UTF-8"

设置mysql本身的编码:

用两条命令,分别是:

//修改目标表所有字段编码格式为utf8
alter table machine_state_time convert to character set utf8;
//修改目标表表编码格式为utf8
alter table machine_state_time character set utf8;

好的,今天的分享就到这里了,希望对你有帮助,有帮助可以点点赞,评论评论


http://www.ppmy.cn/news/1550608.html

相关文章

iOS 17.4 Not Installed

0x00 系统警告 没有安装 17.4 的模拟器,任何操作都无法进行! 点击 OK 去下载,完成之后,依旧是原样! 0x01 解决办法 1、先去官网下载对应的模拟器: https://developer.apple.com/download/all/?q17.4 …

聚观早报 | 知乎Q3营收;华为Mate 70系列发布

聚观早报每日整理最值得关注的行业重点事件,帮助大家及时了解最新行业动态,每日读报,就读聚观365资讯简报。 整理丨Cutie 11月27日消息 知乎Q3营收 华为Mate 70系列发布 iQOO Neo 10拉力橙配色 英伟达展示AI模型Fugatto vivo S20 Pro核…

3D Gaussian Splatting综述 论文笔记

文章目录 参考资料 参考资料 A Survey on 3D Gaussian Splatting精读 3DGaussianSplatting技术的影响会有多大? - Kedreamix的回答 - 知乎 https://www.zhihu.com/question/626506306/answer/3375816824 3DGS综述《A Survey on 3D Gaussian Splatting》 【论文笔记…

记录pbootcms提示:登录失败:表单提交校验失败,请刷新后重试的解决办法

问题描述 pbootcms后台登录的时候提示“登录失败:表单提交校验失败,请刷新后重试!” 解决办法 删除runtime目录,或尝试切换PHP版本,选择7.3或5.6一般就能解决了。

微积分复习笔记 Calculus Volume 1 - 6.8 Exponential Growth and Decay

6.8 Exponential Growth and Decay - Calculus Volume 1 | OpenStax

Spring MVC练习(前后端分离开发实例)

White graces:个人主页 🙉专栏推荐:Java入门知识🙉 🐹今日诗词:二十五弦弹夜月,不胜清怨却飞来🐹 ⛳️点赞 ☀️收藏⭐️关注💬卑微小博主🙏 ⛳️点赞 ☀️收藏⭐️关注&#x1f4…

hadoop-mapreduce词频统计

一、Map Reduce主要阶段 二、词频统计示例 0.MapReduce 词频统计(Word Count)示例图 1. Input 阶段(输入阶段) 输入数据是一段文本,如下: Hadoop is a big data framework. Hadoop can store vast data. Hadoop processes big …

简单的TCPSocket客户端使用案例(已入OPCommon)

做一个实体操作工具类(包含) public delegate void TcpClientReceivedEventHandler(TcpClientStateEventArgs args);public class TCPSocketClient{ //接收委托public event TcpClientReceivedEventHandler TcpReceived;private CancellationTokenSour…