PlainElastic.Net

PlainElastic.Net

PlainElastic.Net

The really plain Elastic Search .Net client.

Plain Idea

Usually connectivity clients built using BLACK BOX principle: there is a client interface and some unknown magic behind it.
(call of the client method internally generate some commands and queries to external system, get responses, somehow process them and then retrieve result to user)
As the result user hardly can debug connectivity issues or extend client functional with missed features.

The main Idea of PlainElastic.Net is to be a GLASS BOX. e.g. provide a full control over connectivity process to user.

Installation

NuGet support

You can find PlainElastic.Net in NuGet Gallery or just install it using VS NuGet Packages Manager. 
Or just type Install-Package PlainElastic.Net in Package Manager Console.

Building from Source

The easiest way to build PlainElastic.Net from source is to clone the git repository on GitHub and build the PlainElastic.Net solution.

git clone git://github.com/Yegoroff/PlainElastic.Net.git

The solution file PlainElastic.Net.sln is located in the root of the repo.

How Its works

1) The only thing you need to connect to ES is a HTTP connection.

  var connection  = new ElasticConnection();

2) Than you can declare sting with ES command

  string command = "http://localhost:9200/twitter/user/test";

3) And JSON string with data

  string jsonData = "{ \"name\": \"Some Name\" }";

4) And pass them using connection to ES.

  string response = connection.Put(command, jsonData);

5) Get JSON string response and analyze it.

  if(response.Contains("\"ok\":true")) {
   ... // do something useful
  }

So, how PlainElastic can help you here?

  // 1. It provides ES HTTP connection
  var connection  = new ElasticConnection("localhost", 9200);

  // 2. And sophisticated ES command builders:
  string command = Commands.Index(index: "twitter", type: "user", id: test)

  // 3. And gives you the ability to serialize your objects to JSON:
  var serializer = new JsonNetSerializer();
  var tweet = new Tweet { Name = "Some Name" };
  string jsonData = serializer.ToJson(tweet);

  // 4. Then you can use appropriate HTTP verb to execute ES command:
  string response = connection.Put(command, jsonData);

  // 5. And then you can deserialize operation response to typed object to easily analyze it:
  IndexResult indexResult = serializer.ToIndexResult(result);
  if(indexResult.ok) {
     ... // do something useful.
  }

  // 6. And even more: Typed mapping and condition-less query builders.

Concepts

No addition abstraction upon native Elastic Search query and mapping syntax.

This eliminates requirements to read both ES and driver‘s manuals, and also it allows you not to guess how driver will generate actual ES query when you construct it using driver‘s Query DSL.
So if you want to apply some ES query - all you need is to read ES Query DSL documentation

All you need is strings.

Let‘s take some ES query sample in a format that you will see in ES documentation:

$ curl -XGET http://localhost:9200/twitter/tweet/_search -d ‘{
     "query" : {
         "term" : { "User": "somebody" }
     }
}‘

In PlainElastic.Net this could be done using:

var connection  = new ElasticConnection("localhost", 9200);
string command = new SearchCommand("twitter", "tweet"); // This will generate: twitter/tweet/_search
string query = new QueryBuilder<Tweet>()        // This will generate:
          .Query(q => q                         // { "query": { "term": { "User": "somebody" } } }
            .Term(t => t
              .Field(tweet=> tweet.User).Value("somebody")
            )
          ).Build();
string result = connection.Get( command, query);

// Than we can convert search results to typed results
var serializer = new JsonNetSerializer();
var foundTweets = serializer.ToSearchResults<Tweet>(result);
foreach (Tweet tweet in  foundTweets.Documents)
{
  ...
}

As you can see all parameters passed to and returned from Get HTTP verb execution are just strings.
This gives us complete control over generated commands and queries. You can copy/paste and debug them in any ES tool that allows to execute JSON queries (e.g. CURL or ElasticHead ).

Command building

PlainElastic.Net commands represent URL part of ElasticSearch requests.
All commands have corresponding links to ES documentation in their XML comments, so you can use these links to access detailed command description.

Most of the commands have Index ,Type and Id constructor parameters, (these parameters forms address part) all other options could be set using fluent builder interface.

string indexCommand = new IndexCommand(index: "twitter", type: "tweet", id: "10")
               .Routing("route_value")
               .Refresh();

There is also a Commands class that represents a command registry and allows you to easily build commands, without necessity to remember command class name.

string searchCommand = Commands.Index(index: "twitter", type: "tweet", id: "10")
               .Routing("route_value")
               .Refresh();

Indexing

ES documentation: http://www.elasticsearch.org/guide/reference/api/index_.html

The easiest way to index document is to serialize your document object to JSON and pass it to PUT index command:

var connection  = new ElasticConnection("localhost", 9200);
var serializer = new JsonNetSerializer();

var tweet = new Tweet { User = "testUser" };
string tweetJson = serializer.ToJson(tweet);

string result = connection.Put(new IndexCommand("twitter", "tweet", id: "10"), tweetJson);

// Convert result to typed index result object.
var indexResult = serializer.ToIndexResult(result);

Note: You can specify additional indexing parameters such as Parent or Refresh in IndexCommand builder.

string indexCommand = new IndexCommand("twitter", "tweet", id: "10").Parent("5").Refresh();

Bulk Operations

ES documentation: http://www.elasticsearch.org/guide/reference/api/bulk.html

There are two options to build Bulk operations JSONs. First is to build all Bulk operations at once:

IEnumerable<Tweet> tweets = new List<Tweet>();

string bulkCommand = new BulkCommand(index: "twitter", type: "tweet");

string bulkJson =
    new BulkBuilder(serializer)
       .BuildCollection(tweets,
            (builder, tweet) => builder.Index(data: tweet,  id: tweet.Id)
                       // You can apply any custom logic here
                       // to generate Indexes, Creates or Deletes.
);

string result = connection.Post(bulkCommand, bulkJson);

//Parse bulk result;
BulkResult bulkResult = serializer.ToBulkResult(result);
...

Second allows you to build Bulk operations in batches of desired size.
This will prevent from constructing huge in-memory strings, and allows to process input collection on-the-fly, without enumerating them to the end.

IEnumerable<Tweet> tweets = new List<Tweet>();

string bulkCommand = new BulkCommand(index: "twitter", type: "tweet");

IEnumerable<string> bulkJsons =
    new BulkBuilder(serializer)
        .PipelineCollection(tweets,
            (builder, tweet) => builder.Index(data: tweet,  id: myObject.Id))
        .JoinInBatches(batchSize: 10); // returns deferred IEnumerable of JSONs
                            // with at most 10 bulk operations in each element,
                            // this will allow to process input elements on-the-fly
                            // and not to generate all bulk JSON at once

foreach(string bulk in bulkJsons )
{
  // Send bulk batch.
  string result = connection.Post(bulkCommand, bulk);

  // Parse bulk batch result.
  BulkResult bulkResult = serializer.ToBulkResult(result);
  ...
}

Note: You can build not only Index Bulk operations but also Create and Delete.

IEnumerable<string> bulkJsons =
  new BulkBuilder(serializer)
     .PipelineCollection(tweets,
            (builder, tweet) => {
              switch (tweet.State) {
                case State.Added:
                  builder.Create(data: tweet,  id: myObject.Id))
                case State.Updated:
                  builder.Index(data: tweet,  id: myObject.Id))
                case State.Deleted:
                  builder.Delete(id: myObject.Id))
              }
            });

Queries

ES documentation: http://www.elasticsearch.org/guide/reference/query-dsl/

The main idea of QueryBuilder is to repeat JSON syntaxes of ES queries.
Besides this it provides intellisense with fluent builder interface 
and property references:

for single property .Field(tweet => tweet.Name) 
for collection type property .FieldOfCollection(collection: user => user.Tweets, field: tweet => tweet.Name)

So let’s see how it works.

We have http://localhost:9200/twitter index with type user. Below we add sample "user" document to it:

PUT http://localhost:9200/twitter/user/1
{
    "Id": 1,
    "Active": true,
    "Name": "John Smith",
    "Alias": "Johnnie"
}

Now let‘s create some synthetic JSON query to get this document:

POST http://localhost:9200/twitter/user/_search
{
    "query": {
        "bool": {
            "must": [
                {
                   "query_string": {
                      "fields": ["Name","Alias"], "query" : "John"
                    }
                },
                {
                   "prefix" : {
                      "Alias": { "prefix": "john" }
                   }
                }
            ]
        }
    },
    "filter": {
        "term": { "Active": "true" }
    }
}

Assuming that we have defined class User:

class User
{
    public int Id { get; set; }
    public bool Active { get; set; }
    public string Name { get; set; }
    public string Alias { get; set; }
}

This query could be constructed using:

string query = new QueryBuilder<User>()
    .Query(q => q
        .Bool(b => b
           .Must(m => m
               .QueryString(qs => qs
                   .Fields(user => user.Name, user => user.Alias).Query("John")
               )
               .Prefix(p => p
                    .Field(user => user.Alias).Prefix("john")
               )
           )
        )
    )
    .Filter(f => f
        .Term(t => t
            .Field(user=> user.Active).Value("true")
        )
    )
    .BuildBeautified();

And then to execute this query we can use the following code:

var connection = new ElasticConnection("localhost", 9200);
var serializer = new JsonNetSerializer();

string result = connection.Post(Commands.Search("twitter", "user"), query);
User foundUser = serializer.ToSearchResult<User>(result).Documents.First();

See Query Builder Gist for complete sample.

Condition-less Queries:

Its usual case when you have a bunch of UI filters to define full-text query, price range filter, category filter etc.
None of these filters are mandatory, so when you construct final query you should use only defined filters. This brings ugly conditional logic to your query-building code.

So how PlainElastic.Net addresses this?

The idea behind is really simple:
If provided condition value is null or empty - the corresponding query or filter will not be generated.

Expression

string query = new QueryBuilder<User>()
    .Query(q => q
        .QueryString(qs => qs
           .Fields(user => user.Name, user => user.Alias).Query("")
        )
    )
    .Filter(f => f
        .Term(t => t
            .Field(user=> user.Active).Value(null)
        )
    )
    .Build();

will generate "{}" string that will return all documents from the index.

The real life usage sample: 
Let‘s say we have criterion object that represents UI filters:

class Criterion
{
    public string FullText { get; set; }
    public double? MinPrice { get; set; }
    public double? MaxPrice { get; set; }
    public bool? Active { get; set; }
}

So our query builder could look like this:

public string BuildQuery(Criterion criterion)
{
    string query = new QueryBuilder<Item>()
        .Query(q => q
            .QueryString(qs => qs
                .Fields(item => item.Name, item => item.Description)
                .Query(criterion.FullText)
            )
        )
        .Filter(f => f
            .And(a => a
                .Range(r => r
                    .Field(item => item.Price)
                    // AsString extension allows to convert nullable values to string or null
                    .From(criterion.MinPrice.AsString())
                    .To(criterion.MaxPrice.AsString())
                )
                .Term(t => t
                    .Field(user => user.Active).Value(criterion.Active.AsString())
                )
            )
        ).BuildBeautified();
}

And that‘s all - no ugly ifs or switches.
You just write query builder using most complex scenario, and then it will build only defined criterions.

If we call this function with BuildQuery( new Criterion { FullText = "text" }) then it will generate:

{
    "query": {
        "query_string": {
            "fields": ["Name", "Description"],
            "query": "text"
        }
    }
}

so it omits all not defined filters.

See Condion-less Query Builder Gist for complete sample.

Facets

ES documentation: http://www.elasticsearch.org/guide/reference/api/search/facets/index.html

For now only Terms facet, Terms Stats facet, Statistical facet, Range facet and Filter Facet supported.

You can construct facet queries using the following syntax:

public string BuildFacetQuery(Criterion criterion)
{
  return new QueryBuilder<Item>()
        .Query(q => q
            .QueryString(qs => qs
                .Fields(item => item.Name, item => item.Description)
                .Query(criterion.FullText)
            )
        )

        // Facets Part
        .Facets(facets => facets
            .Terms(t => t
                .FacetName("ItemsPerCategoryCount")
                .Field(item => item.Category)
                .Size(100)
                )
        )
        .BuildBeautified();
}

To read facets result you need to deserialize it to SearchResults and access its .facet property:

  // Build faceted query with FullText criterion defined.
  string query = BuildFacetQuery(new Criterion { FullText = "text" });
  string result = connection.Post(Commands.Search("store", "item"), query);

  // Parse facets query result
  var searchResults = serializer.ToSearchResult<Item>(result);
  var itemsPerCategoryTerms = searchResults.facets.Facet<TermsFacetResult>("ItemsPerCategoryCount").terms;

  foreach (var facetTerm in itemsPerCategoryTerms)
  {
      Console.WriteLine("Category: {0}  Items Count: {1}".F(facetTerm.term, facetTerm.count));
  }

See Facet Query Builder Gist for complete sample.

Highlighting

ES documentation: http://www.elasticsearch.org/guide/reference/api/search/highlighting/

You can construct highlighted queries using the following syntax:

string query = new QueryBuilder<Note>()
    .Query(q => q
        .QueryString(qs => qs
            .Fields(c => c.Caption)
            .Query("Note")
        )
     )
     .Highlight(h => h
        .PreTags("<b>")
        .PostTags("</b>")
        .Fields(
             f => f.FieldName(n => n.Caption).Order(HighlightOrder.score),
             f => f.FieldName("_all")
        )
     )
    .BuildBeautified();

To get highlighted fragments you need to deserialize results to SearchResult<T> and access highlight property of each hit:

// Execute query and deserialize results.
string results = connection.Post(Commands.Search("notes", "note"), query);
var noteResults = serializer.ToSearchResult<Note>(results);

// Array of higlighted fragments for Caption field for the first hit.
var hit = noteResults.hits.hits[0];
string[] fragments = hit.highlight["Caption"];

See Highlighting Gist for complete sample.

Scrolling

ES documentation: http://www.elasticsearch.org/guide/reference/api/search/scroll/

You can construct scrolling search request by specifing scroll keep alive time in SearchCommand:

string scrollingSearchCommand = new SearchCommand(index:"notes", type:"note")
                                      .Scroll("5m")
                                      .SearchType(SearchType.scan);

To scroll found documents you need to deserialize results to SearchResult<T> and get the _scroll_id field. Then you should execute SearchScrollCommand with acquired scroll_id

// Execute query and deserialize results.
string results = connection.Post(scrollingSearchCommand, queryJson);
var noteResults = serializer.ToSearchResult<Note>(results);

// Get the initial scroll ID
string scrollId = scrollResults._scroll_id;

// Execute SearchScroll request to scroll found documents.
results = connection.Get(Commands.SearchScroll(scrollId).Scroll("5m"));

See Scrolling Gist for complete sample.

Mapping

ES documentation: http://www.elasticsearch.org/guide/reference/mapping/

Mapping of core and object types could be performed in the following manner:

private static string BuildCompanyMapping()
    {
        return new MapBuilder<Company>()
            .RootObject(typeName: "company",
                        map: r => r
                .All(a => a.Enabled(false))
                .Dynamic(false)
                .Properties(pr => pr
                    .String(company => company.Name, f => f.Analyzer(DefaultAnalyzers.standard).Boost(2))
                    .String(company => company.Description, f => f.Analyzer(DefaultAnalyzers.standard))
                    .String(company => company.Fax, f => f.Analyzer(DefaultAnalyzers.keyword))

                    .Object(company => company.Address, address => address
                        .Properties(ap => ap
                            .String(addr => addr.City)
                            .String(addr => addr.State)
                            .String(addr => addr.Country)
                        )
                    )

                    .NestedObject(company => company.Contacts, o => o
                        .Properties(p => p
                            .String(contact => contact.Name)
                            .String(contact => contact.Department)
                            .String(contact => contact.Email)

                            // It‘s unnecessary to specify opt.Type(NumberMappingType.Integer)
                            // cause it will be inferred from property type.
                            // Showed here only for educational purpose.
                            .Number(contact => contact.Age, opt => opt.Type(NumberMappingType.Integer))

                            .Object(ct => ct.Address, oa => oa
                                .Properties( pp => pp
                                    .String(a => a.City)
                                    .String(a => a.State)
                                    .String(a => a.Country)
                                )
                            )
                        )
                    )
                )
          )
          .BuildBeautified();

To apply mapping you need to use PutMappingCommand:

var connection = new ElasticConnection("localhost", 9200);
string jsonMapping = BuildCompanyMapping();

connection.Put(new PutMappingCommand("store", "company"), jsonMapping);

See Mapping Builder Gist for complete sample.

Index Settings

ES documentation: http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html

You can build index settings by using IndexSettinsBuilder:

private static string BuildIndexSettings()
{
    return new IndexSettingsBuilder()
        .Analysis(als => als
            .Analyzer(a => a
                .Custom("lowerkey", custom => custom
                    .Tokenizer(DefaultTokenizers.keyword)
                    .Filter(DefaultTokenFilters.lowercase)
                )
                .Custom("fulltext", custom => custom
                    .CharFilter(DefaultCharFilters.html_strip)
                    .Tokenizer(DefaultTokenizers.standard)
                    .Filter(DefaultTokenFilters.word_delimiter,
                            DefaultTokenFilters.lowercase,
                            DefaultTokenFilters.stop,
                            DefaultTokenFilters.standard)
                )
            )
        )
        .BuildBeautified();
}

You can put index settings to index by UpdateSettingsCommand or by passing settings to index creation command:

var connection = new ElasticConnection("localhost", 9200);

var settings = BuildIndexSettings();

if (IsIndexExists("store", connection))
{
    // We can‘t update settings on active index.
    // So we need to close it, then update settings and then open index back.
    connection.Post(new CloseCommand("store"));

    connection.Put(new UpdateSettingsCommand("store"), settings);

    connection.Post(new OpenCommand("store"));
}
else
{
    // Create Index with settings.
    connection.Put(Commands.Index("store").Refresh(), settings);
}

See Index Settings Gist for complete sample.

Special thanks to devoyster (Andriy Kozachuk) for providing Index Settings support.

Samples

If something is missed

In case you need ElasticSearch feature that not yet covered by PlainElastic.Net, just remember that everything passed to ES connection is a string, so you can add missed functionality using .Custom(string) function, that exists in every builder.

return new QueryBuilder<Item>()
    .Query(q => q
        .Term(t => t
              .Field(user => user.Active)
              .Value(true.ToString())

              // Custom string representing boost part.
              .Custom("\"boost\": 3")
          )
    )
    .BuildBeautified();

or even more - just pass you string with JSON to ES connection.

Also don‘t forget to add an issue to PlainElastic.Net github repository PlainElastic Issues so I can add this functionality to the future builds.

License

PlainElastic.Net is free software distributed under the terms of MIT License (see LICENSE.txt) these terms don’t apply to other 3rd party tools, utilities or code which may be used to develop this application.

时间： 2024-08-23 13:11:47

PlainElastic.Net的相关文章

学习MVC之租房网站（九）-房源显示和搜索

在上一篇<学习MVC之租房网站(八)- 前台注册和登录>完成了前台用户的注册.登录.重置密码等功能,然后要实现与业务相关的功能,包括房源的显示.检索等. 一 房源显示 房源显示内容较多,涉及到的有House.Attachment.HousePic,处理的信息包括房屋类型.朝向.楼层.装修状态.家具等. 这里显示的房源是通过后台的房源管理维护的,后台添加房源时会上传图片.使用UEditor编辑文本,前台显示房源时也要把图片和富文本显示出来.在前台使用后台上传的图片是个问题:UEditor产生的富

ElasticSearch(站内搜索）

简介 Elasticsearch是一个实时的分布式搜索和分析引擎.它可以帮助你用前所未有的速度去处理大规模数据.它可以用于全文搜索,结构化搜索以及分析,当然你也可以将这三者进行组合.Elasticsearch是一个建立在全文搜索引擎 Apache Lucene 基础上的搜索引擎,可以说Lucene是当今最先进,最高效的全功能开源搜索引擎框架.但是Lucene只是一个框架,要充分利用它的功能,需要使用JAVA,并且在程序中集成Lucene.需要很多的学习了解,才能明白它是如何运行的,Lucene确

DotNet 资源大全中文版（Awesome最新版）

Awesome系列的.Net资源整理.awesome-dotnet是由quozd发起和维护.内容包括:编译器.压缩.应用框架.应用模板.加密.数据库.反编译.IDE.日志.风格指南等. API 框架 NancyFx:轻量.用于构建 HTTP 基础服务的非正式(low-ceremony)框架,基于.Net 及 Mono 平台. 官网 ASP.NET WebAPI:快捷创建 HTTP 服务的框架,可以广泛用于多种不同的客户端,包括浏览器和移动设备. 官网 ServiceStack:架构缜密.速度飞快

ASP.NET SignalR 与 LayIM2.0 配合轻松实现Web聊天室（七） 之 历史记录查询（时间，关键字，图片，文件），关键字高亮显示。

前言 上一篇讲解了如何自定义右键菜单,都是前端的内容,本篇内容就一个:查询.聊天历史纪录查询,在之前介绍查找好友的那篇博客里已经提到过 Elasticsearch,今天它又要上场了.对于Elasticsearch不感冒的同学呢,本篇可以不用看啦. from baidu: ElasticSearch是一个基于Lucene的搜索服务器.它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口.Elasticsearch是用Java开发的,并作为Apache许可条款下的开放源码发布,

Func&lt;T,T&gt;应用之Elasticsearch查询语句构造器的开发

前言 之前项目中做Elasticsearch相关开发的时候,虽然借助了第三方的组件PlainElastic.Net,但是由于当时不熟悉用法,而选择了自己拼接查询语句.例如: string queryGroup = "{\"query\": {\"match\": { \"roomid\": \"FRIEND_12686_10035\" }}}"; //关键字查询 string queryKeyWord =

ASP.NET SignalR 与 LayIM2.0 配合轻松实现Web聊天室（四） 之 用户搜索（Elasticsearch），加好友流程（1）。

前面几篇基本已经实现了大部分即时通讯功能:聊天,群聊,发送文件,图片,消息.不过这些业务都是比较粗犷的.下面我们就把业务细化,之前用的是死数据,那我们就从加好友开始吧.加好友,首先你得知道你要加谁.Layim界面右下角有个+号,点击它之后就会弹出查找好友的界面,不过那个界面需要自定义.由于前端不是我的强项,勉强凑了个页面.不过不要在意这些细节.这些都不重要,今天主要介绍一下ElasticSearch搜索解决方案.它是一个基于Lucene的搜索服务器.它提供了一个分布式多用户能力的全文搜索引擎,基

DotNet 资源

DotNet 资源 目录 API 应用框架(Application Frameworks) 应用模板(Application Templates) 人工智能(Artificial Intelligence) 程序集处理(Assembly Manipulation) 资源(Assets) 认证和授权(Authentication and Authorization) 自动构建(Build Automation) 缓存(Caching) CLI CLR CMS 代码分析和度量(Code Analys

.Net开源框架列表

API 框架 NancyFx:轻量.用于构建 HTTP 基础服务的非正式(low-ceremony)框架,基于.Net 及 Mono 平台.官网 ASP.NET WebAPI:快捷创建 HTTP 服务的框架,可以广泛用于多种不同的客户端,包括浏览器和移动设备.官网 ServiceStack:架构缜密.速度飞快.令人愉悦的 web 服务.官网 Nelibur:Nelibur 是一个使用纯 WCF 构建的基于消息的 web 服务框架.Nelibur 可以便捷地创建高性能.基于消息的 web 服务,使

.Net 开源项目资源大全

Awesome DotNet,这又是一个 Awesome XXX 系列的资源整理,由 quozd 发起和维护.内容包括:编译器.压缩.应用框架.应用模板.加密.数据库.反编译.IDE.日志.风格指南等. 伯乐在线已在 GitHub 上发起「DotNet 资源大全中文版」的整理.欢迎扩散.欢迎加入. https://github.com/jobbole/awesome-dotnet-cn (注:下面用 [$] 标注的表示收费工具,但部分收费工具针对开源软件的开发/部署/托管是免费的) API 框架