搜索管理

准备环境

创建映射

插入原始数据

插入三条原始数据



具体数据:

创建映射：
post：http://localhost:9200/xc_course/doc/_mapping
{"properties": {"description": {"type": "text","analyzer": "ik_max_word","search_analyzer": "ik_smart"},"name": {"type": "text","analyzer": "ik_max_word","search_analyzer": "ik_smart"},"pic":{"type":"text","index":false},"price": {"type": "float"},"studymodel": {"type": "keyword"},"timestamp": {"type": "date","format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"}}}初始化文档：http://localhost:9200/xc_course/doc/1
{
"name": "Bootstrap开发",
"description": "Bootstrap是由Twitter推出的一个前台页面开发框架，是一个非常流行的开发框架，此框架集成了多种页面效果。此开发框架包含了大量的CSS、JS程序代码，可以帮助开发者（尤其是不擅长页面开发的程序人员）轻松的实现一个不受浏览器限制的精美界面效果。",
"studymodel": "201002",
"price":38.6,
"timestamp":"2018-04-25 19:11:35",
"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg"
}
http://localhost:9200/xc_course/doc/2
{
"name": "java编程基础",
"description": "java语言是世界第一编程语言，在软件开发领域使用人数最多。",
"studymodel": "201001",
"price":68.6,
"timestamp":"2018-03-25 19:11:35",
"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg"
}
http://localhost:9200/xc_course/doc/3
{
"name": "spring开发基础",
"description": "spring 在java领域非常流行，java程序员都在用。",
"studymodel": "201001",
"price":88.6,
"timestamp":"2018-02-24 19:11:35",
"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg"
}

简单搜索

DSL搜索

DSL(Domain Specific Language)是es提出的专门面向JSon数据的搜索方式,在搜索时通过json格式完成索引.
DSL比Url搜索方式功能强大,在项目中建议使用DSL方式完成搜索.

查询所有文档

postman:

{"query":{"match_all":{}//搜索方法},"_source":["name","studymodel"]//内容定向}

搜索结果:

{"took": 46,"timed_out": false,"_shards": {"total": 1,"successful": 1,"skipped": 0,"failed": 0},"hits": {"total": 3,"max_score": 1.0,"hits": [{"_index": "xc_andrew","_type": "doc","_id": "1","_score": 1.0,"_source": {"studymodel": "201002","name": "Bootstrap开发"}},{"_index": "xc_andrew","_type": "doc","_id": "2","_score": 1.0,"_source": {"studymodel": "201001","name": "java编程基础"}},{"_index": "xc_andrew","_type": "doc","_id": "3","_score": 1.0,"_source": {"studymodel": "201001","name": "spring开发基础"}}]}
}
}

结果说明
took:本次操作花费的时间,单位毫秒.
timed_out:请求是否超时
_shards:说明本次操作共搜索了那些内容total:集群部署中的分片数量hits:搜索命中的记录
hits.total:符合条件的文档总数
hits.hits:匹配度较高的n个文档
hits.max_score:文档匹配得分,这里为最高分
_score:分割文档都有一个匹配得分,这里按照从高到低
_source:显示文档内容

在java中使用

    @Testpublic void testSearch() throws IOException, ParseException {SearchRequest searchRequest  = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();searchSourceBuilder.query(QueryBuilders.matchAllQuery());searchSourceBuilder.fetchSource(new String[]{"name","studymodel","price","timestamp"},new String[]{} );searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

分页查询

postman分页

{"from":0,"size":1,"query":{"match_all":{}},"_source":["name","studymodel"]}

java分页

//分页测试@Testpublic void testSearchByPage() throws IOException, ParseException {SearchRequest searchRequest  = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();int page = 1;int size = 1;int from = (page - 1) * size;searchSourceBuilder.from(from);searchSourceBuilder.size(size);searchSourceBuilder.query(QueryBuilders.matchAllQuery());searchSourceBuilder.fetchSource(new String[]{"name","studymodel","price","timestamp"},new String[]{} );searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

Term Query

根据任一项查询
postman

{"query":{"term":{"name":"spring"}},"_source":["name","studymodel"]}

java

//termQuery@Testpublic void testTermQuery() throws IOException, ParseException {SearchRequest searchRequest  = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();searchSourceBuilder.query(QueryBuilders.termQuery("name","spring"));searchSourceBuilder.fetchSource(new String[]{"name","studymodel","price","timestamp"},new String[]{} );searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

根据id精确匹配

postman:

{"query":{"ids":{"type":"doc","values":["1","2"]}}}

这里的values注意后面还有一个s
java:

   @Testpublic void testQueryById() throws IOException, ParseException {SearchRequest searchRequest  = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();String[] ids = new String[]{"1","2"};searchSourceBuilder.query(QueryBuilders.termsQuery("_id",ids));searchSourceBuilder.fetchSource(new String[]{"name","studymodel","price","timestamp"},new String[]{} );searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

注意这里的QueryBuilder.termsQuery的方法,而不是termQuery,和根据任意项选择有区别

match Query

搜索的时候分词
postman

{"query":{"match":{"description":{"query":"spring 开发","operator":"or"}}}}

这里的operator表示or的时候是有一个词就可以搜出
表示and的时候表示只都在的时候才可以搜出

java

    //MatchQuery@Testpublic void testQueryMatch() throws IOException, ParseException {SearchRequest searchRequest  = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();searchSourceBuilder.query(QueryBuilders.matchQuery("description","spring开发").operator(Operator.OR));searchSourceBuilder.fetchSource(new String[]{"name","studymodel","price","timestamp"},new String[]{} );searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

postman:

{"query":{"match":{"description":{"query":"spring 开发","minimum_should_match":"80%"}}}}

这里的变量minimum是搜索重复率如果达到总占比的80%,就搜出

multi Query

可以在多个属性中查找,避免如termQuery只能在一个属性中查找的缺点
postman

       {"query":{"multi_match":{"query":"spring css","minimum_should_match":"50%","fields":["name^10","description"]}}
}

java:

    //Multi_Match@Testpublic void testMultiMatch() throws IOException, ParseException {SearchRequest searchRequest  = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();searchSourceBuilder.query(QueryBuilders.multiMatchQuery("spring css","name","description").minimumShouldMatch("50%").field("name",10));searchSourceBuilder.fetchSource(new String[]{"name","studymodel","price","timestamp"},new String[]{} );searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

postman里的^10代表比重提高10倍,因为标题中的比重大于文章中的比重
而在java中使用field.(name,boot)中的boost表示比重.

布尔查询

布尔查询对应Lucene的BooleanQuery查询,实现将多个查询结合起来

如果要包含以上的多个条件一起查询,应该怎么操作?
就要使用boolean查询了
这样子比较复杂,不过有前面的基础,可以很快掌握,理解
postman:

{"_source":["name","description"],"from":0,"size":1,"query":{"bool":{"must":[{"multi_match":{"query":"spring 开发","minimum_should_match":"50%","fields":["name","description"]}},{ "term":{"studymodel":"201001"}}]}}
}

java:

//BoolerMatch@Testpublic void testBoolerMatch() throws IOException, ParseException {SearchRequest searchRequest  = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("spring css", "name", "description").minimumShouldMatch("50%").field("name", 10);TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("studymodel", "201001");
BoolQueryBuilder builder = new BoolQueryBuilder();
builder.must(multiMatchQueryBuilder);
builder.must(termQueryBuilder);searchSourceBuilder.query(builder);searchSourceBuilder.fetchSource(new String[]{"name","studymodel","price","timestamp"},new String[]{} );searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}
}

must：表示必须，多个查询条件必须都满足。（通常使用must）
should：表示或者，多个查询条件只要有一个满足即可。
must_not：表示非。

过滤器

过虑是针对搜索的结果进行过虑，过虑器主要判断的是文档是否匹配，不去计算和判断文档的匹配度得分，所以过虑器性能比查询要高，且方便缓存，推荐尽量使用过虑器去实现查询或者过虑器和查询共同使用。
过虑器在布尔查询中使用，下边是在搜索结果的基础上进行过虑：
postman:

{"query":{"bool":{"must":[{"multi_match":{"query":"spring 开发","minimum_should_match":"50%","fields":["name","description"]}},{ "term":{"studymodel":"201001"}}],"filter":[{"term":{"studymodel":"201001"}},{"range":{"price":{"gte":60,"lte":100}}}]}}}

注意,filter和must同级,在must[],后面加filter

Java:

    //FilterMatch@Testpublic void testFilterMatch() throws IOException, ParseException {SearchRequest searchRequest = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("spring css", "name", "description").minimumShouldMatch("50%").field("name", 10);BoolQueryBuilder builder = new BoolQueryBuilder();builder.must(multiMatchQueryBuilder);builder.filter(QueryBuilders.termQuery("studymodel", "201001"));builder.filter(QueryBuilders.rangeQuery("price").gte(90).lte(100)   );searchSourceBuilder.query(builder);searchSourceBuilder.fetchSource(new String[]{"name", "studymodel", "price", "timestamp"},new String[]{});searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(price);System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

排序

可以在字段上添加一个或多个排序，支持在keyword、date、float等类型上添加，text类型的字段上不允许添加排序。
发送POSThttp://localhost:9200/xc_course/doc/_search
过虑0–10元价格范围的文档，并且对结果进行排序，先按studymodel降序，再按价格升序

postman:

{"query":{"bool":{"filter":[{"range":{"price":{"gte":0,"lte":100}}}]}},"sort":[{"studymodel":"DESC","price":"ASC"}]

java:

//OrderMatch@Testpublic void testOrderMatch() throws IOException, ParseException {SearchRequest searchRequest = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();BoolQueryBuilder builder = new BoolQueryBuilder();builder.filter(QueryBuilders.rangeQuery("price").gte(0).lte(100)   );searchSourceBuilder.sort(new FieldSortBuilder("studymodel").order(SortOrder.DESC));searchSourceBuilder.sort(new FieldSortBuilder("price").order(SortOrder.ASC));searchSourceBuilder.query(builder);searchSourceBuilder.fetchSource(new String[]{"name", "studymodel", "price", "timestamp"},new String[]{});searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(price);System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

高亮显示

高亮显示可以将搜索结果一个或多个字突出显示，以便向用户展示匹配关键字的位置。在搜索语句中添加highlight即可实现，如下：
Post：http://127.0.0.1:9200/xc_course/doc/_search

postman:

{"query":{"bool":{"must":[{"multi_match":{"query":"spring 开发","minimum_should_match":"50%","fields":["name","description"]}},{ "term":{"studymodel":"201001"}}],"filter":[{"range":{"price":{"gte":0,"lte":100}}}]}}, "sort":[{"studymodel":"DESC","price":"ASC"}],"highlight":{"pre_tags":["<tag>"],"post_tags":["</tag>"],"fields":{"name":{},"description":{}}}
}

java:

    //HighLight@Testpublic void testHighLight() throws IOException, ParseException {SearchRequest searchRequest = new SearchRequest("xc_andrew");searchRequest.types("doc");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("spring 开发", "name", "description").minimumShouldMatch("50%").field("name", 10);BoolQueryBuilder builder = new BoolQueryBuilder();builder.must(multiMatchQueryBuilder );builder.filter(QueryBuilders.rangeQuery("price").gte(0).lte(100)   );searchSourceBuilder.query(builder);searchSourceBuilder.fetchSource(new String[]{"name", "studymodel", "price", "timestamp"},new String[]{});HighlightBuilder highlightBuilder = new HighlightBuilder();highlightBuilder.preTags("<tag>");highlightBuilder.postTags("</tag>");highlightBuilder.fields().add(new HighlightBuilder.Field("name"));searchSourceBuilder.highlighter(highlightBuilder);searchRequest.source(searchSourceBuilder);SearchResponse search = restHighLevelClient.search(searchRequest);SearchHits hits = search.getHits();long totalHits = hits.totalHits;SearchHit[] searchHits = hits.getHits();SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for (SearchHit hit : searchHits) {String id = hit.getId();Map<String, Object> sourceAsMap = hit.getSourceAsMap();String name = (String) sourceAsMap.get("name");Map<String, HighlightField> highlightFields = hit.getHighlightFields();if (highlightFields!=null){HighlightField nameField = highlightFields.get("name");if (nameField!=null){StringBuffer sb = new StringBuffer();Text[] fragments = nameField.getFragments();for (Text fragment : fragments) {sb.append(fragment.string());}name = sb.toString();}}String description = (String) sourceAsMap.get("description");// String studymodel = (String) sourceAsMap.get("studymodel");Double price = (Double) sourceAsMap.get("price");Date timestamp = simpleDateFormat.parse((String) sourceAsMap.get("timestamp"));System.out.println(price);System.out.println(name);// System.out.println(studymodel);System.out.println(timestamp);System.out.println(description);}}

这个高亮设置,一定是要有检索内容才可以添加的,应为高亮添加的就是你的检索内容的词汇

集群管理

集群结构

es通常以集群的方式部署,工作,这样不仅可以提高处理PB及数据的能力,还可以很好的增加系统的容错性和可用性.
ES集群结构图:

每一个es服务器可以看作是一个节点,
而节点有主节点和副节点直接点负责集群管理,
索引可以分布在不同的片上,片也分主片和副片,当主片失效时,可以由副片分担,这样实现了高可用性,和高容错性,使得ES可以成为PB级数据的处理器,一个搜索请求过来时,就会分片取查询,最后将结果返回非用户.
一个及集群中会有多个主节点,主节点管理集群,比如增加节点,移除节点,主节点挂掉之后,ES会重新选择一个节点.
节点转发,每个节点都知道其他节点的信息,我们可以对任意节点发起请求,接受请求的节点会转发给其他节点查询数据.

搭建集群

复制之前的集群节点,然后进入config中修改elasticsearch.yml
将其中的name修改,本身的端口加一,
然后点击bin中的elasticsearch.bat,即可.

我这里部署了三个,分成两篇,由于不同节点之间可以转发,所以在任何一个节点上,都可以看到全部的内容.

集群的健康

通过访问Get /_cluster/health来查看Elasticsearch的集群健康情况
用三种颜色表示健康的状态:green,yellow,red

green 所有主分片和副分片都正常运行.

yellow:所有的主分片正常运行,但是有些副分片不正常

这里是我关掉node_3后的情况.

red:存在主分片不正常运行