GeoTrellis 尝试(springboot + swagger + spark + GeoTrellis)

news/2024/11/8 16:39:24/

原始使用springboot + spark 整合在一起做了一个通过请求来驱动spark计算的demo工程，最近看了下GeoSpark,Geotrellis,GeoMeca关于GIS方面的大数据相关技术，就写了一个Demo,我这里用的是Spark 2.4.0 版本 scala 2.11 版本 Geotrellis 2.3.1版本 GeoTools 20.0版本 Guava 14.0.1版本
相关pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.i-tudou.bd</groupId><artifactId>spring-spark-demo</artifactId><packaging>jar</packaging><version>1.0-SNAPSHOT</version><name>A Camel Scala Route</name><properties><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding><project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding><java.version>1.8</java.version><scala.version>2.11</scala.version><spark.version>2.4.0</spark.version><guava.version>14.0.1</guava.version><geotrellis.version>2.3.1</geotrellis.version></properties><parent><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-parent</artifactId><version>1.5.3.RELEASE</version><relativePath /> <!-- lookup parent from repository --></parent><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-aop</artifactId></dependency><dependency><groupId>com.github.ben-manes.caffeine</groupId><artifactId>caffeine</artifactId><version>2.8.0</version></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-spark_${scala.version}</artifactId><version>${geotrellis.version}</version></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-raster_${scala.version}</artifactId><version>${geotrellis.version}</version></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-vector_${scala.version}</artifactId><version>${geotrellis.version}</version></dependency><dependency><groupId>org.geotools</groupId><artifactId>gt-coverage</artifactId><version>20.0</version></dependency><dependency><groupId>org.geotools</groupId><artifactId>gt-referencing</artifactId><version>20.0</version></dependency><dependency><groupId>org.geotools</groupId><artifactId>gt-main</artifactId><version>20.0</version></dependency><dependency><groupId>org.geotools</groupId><artifactId>gt-shapefile</artifactId><version>20.0</version></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-geotools_${scala.version}</artifactId><version>${geotrellis.version}</version><exclusions><exclusion><groupId>org.geotools</groupId><artifactId>gt-coverage</artifactId></exclusion><exclusion><groupId>org.geotools</groupId><artifactId>gt-epsg-hsql</artifactId></exclusion><exclusion><groupId>org.geotools</groupId><artifactId>gt-main</artifactId></exclusion><exclusion><groupId>org.geotools</groupId><artifactId>gt-referencing</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-util_${scala.version}</artifactId><version>${geotrellis.version}</version></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-proj4_${scala.version}</artifactId><version>${geotrellis.version}</version></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-spark-pipeline_${scala.version}</artifactId><version>${geotrellis.version}</version></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-shapefile_${scala.version}</artifactId><version>${geotrellis.version}</version><exclusions><exclusion><groupId>org.geotools</groupId><artifactId>gt-shapefile</artifactId></exclusion><exclusion><groupId>javax.media</groupId><artifactId>jai_core</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-spark-etl_${scala.version}</artifactId><version>${geotrellis.version}</version></dependency><!--<dependency><groupId>org.locationtech.geotrellis</groupId><artifactId>geotrellis-spark-etl_2.12</artifactId><version>${geotrellis.version}</version></dependency>--><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId><exclusions><exclusion><groupId>org.springrfamework.boot</groupId><artifactId>spring-boot-starter-tomcat</artifactId></exclusion><exclusion><groupId>com.google.guava</groupId><artifactId>guava</artifactId></exclusion><!--<exclusion><groupId>org.hibernate</groupId><artifactId> hibernate-validator</artifactId></exclusion>--></exclusions></dependency><dependency><groupId>com.google.guava</groupId><artifactId>guava</artifactId><version>${guava.version}</version></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-undertow</artifactId></dependency><dependency><groupId>io.springfox</groupId><artifactId>springfox-swagger2</artifactId><version>2.8.0</version></dependency><dependency><groupId>io.springfox</groupId><artifactId>springfox-swagger-ui</artifactId><version>2.8.0</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_${scala.version}</artifactId><version>${spark.version}</version><exclusions><exclusion><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId></exclusion><exclusion><groupId>log4j</groupId><artifactId>log4j</artifactId></exclusion></exclusions><scope>compile</scope></dependency><dependency><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId></dependency><dependency><groupId>org.codehaus.janino</groupId><artifactId>janino</artifactId><version>3.0.8</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming_${scala.version}</artifactId><version>${spark.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql_${scala.version}</artifactId><version>${spark.version}</version></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-configuration-processor</artifactId><optional>true</optional></dependency></dependencies><repositories><repository><id>maven2-repository.dev.java.net</id><name>Java.net repository</name><url>http://download.java.net/maven/2</url></repository><repository><id>osgeo</id><name>Open Source Geospatial Foundation Repository</name><url>http://download.osgeo.org/webdav/geotools/</url></repository><repository><snapshots><enabled>true</enabled></snapshots><id>boundless</id><name>Boundless Maven Repository</name><url>http://repo.boundlessgeo.com/main</url></repository></repositories><build><plugins><plugin><inherited>true</inherited><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><configuration><source>1.8</source><target>1.8</target></configuration></plugin></plugins></build>
</project>

app 启动类实现

@ComponentScan
@SpringBootApplication
class Config
object springsparkdemoApplication extends App{SpringApplication.run(classOf[Config])
}

springboot 启动时Configuration加载spark实例

@Configuration
class Sparkconfig {private val sparkHome = "."private val appName = "sparkTest"private val master = "local[*]"@Beandef SparkConf: SparkConf  = {val conf = new SparkConf().setAppName(appName).setMaster(master)return conf}@Beandef SparkContext  = new SparkContext(SparkConf)
}

Swagger配置

@Configuration
@EnableSwagger2
@ConditionalOnProperty(name = Array("swagger.enable"), havingValue = "true")
class SwaggerConfiguration {@Bean def createRestApi: Docket = {val pacakage = "com.itudou.bd.Controller"new Docket(DocumentationType.SWAGGER_2).apiInfo(apiInfo).select.apis(RequestHandlerSelectors.basePackage(pacakage)).paths(PathSelectors.any).build}private def apiInfo = new ApiInfoBuilder().title("smartdata-uc").description("").termsOfServiceUrl("").contact("").version("1.0").build
}

Rest 接口调用GeoTrellis

@Api( value = "GeoTrellisController",description = "GeoTrellisController")
@RestController
@RequestMapping (value = Array("GeoTrellisController/data/"))
@CrossOrigin
class GeoTrellisController {@ApiOperation(value = "vecoterToraster")@GetMapping(value = Array("vecoterToraster"))def vecoterToraster ={/* DemoHandle.vactorToRaster("D:\\log\\61011_geo\\610111.shp","D:\\log\\61011_geo\\610111.tif",Sparkconfig.SparkContext);*/}@ApiOperation(value = "etl")@GetMapping(value = Array("etl"))def etl ={var args = Array[String]("--input","D:\\log\\61011_geo\\input.json","--output","D:\\log\\61011_geo\\output.json","--backend-profiles","D:\\log\\61011_geo\\backend-profiles.json");DemoHandle.etl(args);}
}

网上关于GeoTrellis的两个例子

object DemoHandle {val colorMap1 =ColorMap(Map(0 -> RGB(0,0,0),1 -> RGB(255,255,255)))var minX,minY,maxX,maxY = -180.0def getFeatures(path: String,charset: String = "UTF-8"): mutable.ListBuffer[Geometry] ={val features = mutable.ListBuffer[Geometry]()val shpDataStore = new ShapefileDataStore(new File(path).toURI().toURL())shpDataStore.setCharset(Charset.forName(charset))val typeName = shpDataStore.getTypeNames()(0)val featureSource = shpDataStore.getFeatureSource(typeName)val result = featureSource.getFeatures()val itertor = result.features()while(itertor.hasNext()){val sf = itertor.next();//System.out.println(sf.getAttribute(0).toString())features+= WKT.read(sf.getAttribute(0).toString);}itertor.close()shpDataStore.dispose()return features}def vactorToRaster(orgpath: String,armpath:String, sc:SparkContext)={val features = getFeatures(orgpath)minX = features(0).jtsGeom.getEnvelopeInternal.getMinXminY = features(0).jtsGeom.getEnvelopeInternal.getMinYmaxX = features(0).jtsGeom.getEnvelopeInternal.getMaxXmaxY = features(0).jtsGeom.getEnvelopeInternal.getMaxYfor (feature <- features) {if (feature.jtsGeom.getEnvelopeInternal.getMaxX > maxX)maxX = feature.jtsGeom.getEnvelopeInternal.getMaxXif (feature.jtsGeom.getEnvelopeInternal.getMaxY > maxY)maxY = feature.jtsGeom.getEnvelopeInternal.getMaxYif (feature.jtsGeom.getEnvelopeInternal.getMinX < minX)minX = feature.jtsGeom.getEnvelopeInternal.getMinXif (feature.jtsGeom.getEnvelopeInternal.getMinY < minY)minY = feature.jtsGeom.getEnvelopeInternal.getMinY}val geoms:RDD[Geometry] = sc.parallelize(features)val extent:Extent = Extent(minX, minY, maxX, maxY)val tl = TileLayout(100, 72, 256, 256)val layout = LayoutDefinition(extent, tl)val celltype: CellType = IntCellTypeval re = RasterExtent(extent, 1200, 600)val layer: RDD[(SpatialKey, Tile)] with Metadata[LayoutDefinition] =geoms.rasterize(36, celltype, layout)val layerResult = layer.collect();for(sublayer<-layerResult) {sublayer._2.renderPng(colorMap1).write(armpath)}}def etl(args: Array[String])= {implicit val sc = SparkUtils.createSparkContext("ETL", new SparkConf(true).setMaster("local[*]"))try {//Etl.ingest[ProjectedExtent, SpatialKey, Tile](args)} finally {sc.stop}}
}

http://www.ppmy.cn/news/917787.html

相关文章

批处理Bat教程-第八章：番外篇

批处理Bat教程-第八章：番外篇

转载自&#xff1a;http://docs.30c.org/dosbat/chapter08/ 8.1 批量十六进制二进制格式转换 想把数据(流)以十六进制或二进制的形式显示出来&#xff1f;UltraEdit之类的编辑软件一定是首选。即使是要自己亲自转出来&#xff0c;C/C等语言也会方便的多。如果您跟我一样选择使…
阅读更多...
day5-os、sys模块

day5-os、sys模块

一.概述 开发运维相关支撑系统现今已成为Devops下的一大热门领域&#xff0c;Python在这方面也有着自己独到的优势。这类场景以及其他一些场景下&#xff0c;需要调用一些操作系统的接口&#xff0c;这就涉及到今天要讲述的OS模块和sys模块。 二.OS模块的常见用法 os.getcwd()获…
阅读更多...
goudp文件传输服务器,golang实现ftp实时传输文件的案例

goudp文件传输服务器,golang实现ftp实时传输文件的案例

一、项目简介 本项目主要实现的功能是ftp客户端不断地将xml文件和jpg文件实时地上传到服务器，当然也可以是其他格式的文件。每当ftp客户端取到一个文件之后，将文件上传到服务器后，然后将其删除。 项目实现可配置，如果开发者有类似的需求，只需要修改配置文件就可以使用本项…
阅读更多...
itudou(爱土豆) 免费官方版

itudou(爱土豆) 免费官方版

itudou(爱土豆) 免费官方版 软件大小&#xff1a;29.1MB 软件语言&#xff1a;简体中文 软件授权&#xff1a;官方版 软件类别&#xff1a;下载工具 官方网站&#xff1a;www.33LC.com 应用平台&#xff1a;/WinXP/|Win7|/Vista/ itudou(爱土豆)是用以批量上传和下载土豆视频的…
阅读更多...
总结hmdp项目中遇到的坑

总结hmdp项目中遇到的坑

目录 1、RedisTemplate执行lua脚本抛出异常java.lang.UnsupportedOperationException 原因 2、redis出现WRONGTYPE Operation against a key holding the wrong kind of value错误 原因 3、Hutool BeanToMap问题 1、RedisTemplate执行lua脚本抛出异常java.lang.Unsupporte…
阅读更多...
Flask_使用flask_marshmallow序列化数据

Flask_使用flask_marshmallow序列化数据

代码如下&#xff1a; from flask import Flask from flask_marshmallow import Marshmallow from flask_sqlalchemy import SQLAlchemy from marshmallow import fieldsapp Flask(__name__) app.config["SQLALCHEMY_DATABASE_URI"] "mysqlpymysql://root:12…
阅读更多...
我的嵌入式研发求职

我的嵌入式研发求职

1&#xff0c;找一个可持续发展的产品研发方向&#xff0c;能够不断推出新产品&#xff0c;而且产品都有市场。 2&#xff0c;产品的技术是我自己可以独立控制的&#xff0c;不是哪个部分必须由别人来做。而且推出的是一套完整的产品&#xff0c;不是必须依赖于用 户应用平台的…
阅读更多...
为何手机WiFi在国内不能堂而皇之的存在

为何手机WiFi在国内不能堂而皇之的存在

最近关于手机WiFi的话题渐渐多了起来&#xff0c;流入国内的超过100万台水货手机可以用WiFi功能&#xff0c;而堂堂的行货手机却不行。洋品牌们无可奈何&#xff1a;任凭你的机型在国外卖得如何好&#xff0c;要想进入中国&#xff0c;就得把WiFi功能“咔嚓”掉。中国电信也是有…
阅读更多...
最新文章

Copyright @ 2022~2023