【scikit-learn】网格搜索来进行高效的参数调优



内容概要?

  • 如何使用K折交叉验证来搜索最优调节参数
  • 如何让搜索参数的流程更加高效
  • 如何一次性的搜索多个调节参数
  • 在进行真正的预测之前，如何对调节参数进行处理
  • 如何削减该过程的计算代价

1. K折交叉验证回顾?

交叉验证的过程

  • 选择K的值（一般是10），将数据集分成K等份
  • 使用其中的K-1份数据作为训练数据，另外一份数据作为测试数据，进行模型的训练
  • 使用一种度量测度来衡量模型的预测性能

交叉验证的优点

  • 交叉验证通过降低模型在一次数据分割中性能表现上的方差来保证模型性能的稳定性
  • 交叉验证可以用于选择调节参数、比较模型性能差别、选择特征

交叉验证的缺点

  • 交叉验证带来一定的计算代价，尤其是当数据集很大的时候，导致计算过程会变得很慢

2. 使用GridSearchCV进行高效调参?

GridSearchCV根据你给定的模型自动进行交叉验证，通过调节每一个参数来跟踪评分结果，实际上，该过程代替了进行参数搜索时的for循环过程。

In [1]:

from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.grid_search import GridSearchCV

In [2]:

# read in the iris data
iris = load_iris()

# create X (features) and y (response)
X = iris.data
y = iris.target

In [3]:

# define the parameter values that should be searched
k_range = range(1, 31)
print k_range

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]

In [4]:

# create a parameter grid: map the parameter names to the values that should be searched
# 下面是构建parameter grid，其结构是key为参数名称，value是待搜索的数值列表的一个字典结构
param_grid = dict(n_neighbors=k_range)
print param_grid

{‘n_neighbors‘: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]}

In [5]:

knn = KNeighborsClassifier(n_neighbors=5)
# instantiate the grid
# 这里GridSearchCV的参数形式和cross_val_score的形式差不多，其中param_grid是parameter grid所对应的参数
# GridSearchCV中的n_jobs设置为-1时，可以实现并行计算（如果你的电脑支持的情况下）
grid = GridSearchCV(knn, param_grid, cv=10, scoring=‘accuracy‘)

我们可以知道，这里的grid search针对每个参数进行了10次交叉验证，并且一共对30个参数进行相同过程的交叉验证

In [6]:

grid.fit(X, y)

Out[6]:

GridSearchCV(cv=10, error_score=‘raise‘,
       estimator=KNeighborsClassifier(algorithm=‘auto‘, leaf_size=30, metric=‘minkowski‘,
           metric_params=None, n_neighbors=5, p=2, weights=‘uniform‘),
       fit_params={}, iid=True, loss_func=None, n_jobs=1,
       param_grid={‘n_neighbors‘: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]},
       pre_dispatch=‘2*n_jobs‘, refit=True, score_func=None,
       scoring=‘accuracy‘, verbose=0)

In [7]:

# view the complete results (list of named tuples)
grid.grid_scores_

Out[7]:

[mean: 0.96000, std: 0.05333, params: {‘n_neighbors‘: 1},
 mean: 0.95333, std: 0.05207, params: {‘n_neighbors‘: 2},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 3},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 4},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 5},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 6},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 7},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 8},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 9},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 10},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 11},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 12},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 13},
 mean: 0.97333, std: 0.04422, params: {‘n_neighbors‘: 14},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 15},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 16},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 17},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 18},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 19},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 20},
 mean: 0.96667, std: 0.03333, params: {‘n_neighbors‘: 21},
 mean: 0.96667, std: 0.03333, params: {‘n_neighbors‘: 22},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 23},
 mean: 0.96000, std: 0.04422, params: {‘n_neighbors‘: 24},
 mean: 0.96667, std: 0.03333, params: {‘n_neighbors‘: 25},
 mean: 0.96000, std: 0.04422, params: {‘n_neighbors‘: 26},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 27},
 mean: 0.95333, std: 0.04269, params: {‘n_neighbors‘: 28},
 mean: 0.95333, std: 0.04269, params: {‘n_neighbors‘: 29},
 mean: 0.95333, std: 0.04269, params: {‘n_neighbors‘: 30}]

In [8]:

# examine the first tuple
print grid.grid_scores_[0].parameters
print grid.grid_scores_[0].cv_validation_scores
print grid.grid_scores_[0].mean_validation_score

{‘n_neighbors‘: 1}
[ 1.          0.93333333  1.          0.93333333  0.86666667  1.
  0.86666667  1.          1.          1.        ]
0.96

In [9]:

# create a list of the mean scores only
grid_mean_scores = [result.mean_validation_score for result in grid.grid_scores_]
print grid_mean_scores

[0.95999999999999996, 0.95333333333333337, 0.96666666666666667, 0.96666666666666667, 0.96666666666666667, 0.96666666666666667, 0.96666666666666667, 0.96666666666666667, 0.97333333333333338, 0.96666666666666667, 0.96666666666666667, 0.97333333333333338, 0.97999999999999998, 0.97333333333333338, 0.97333333333333338, 0.97333333333333338, 0.97333333333333338, 0.97999999999999998, 0.97333333333333338, 0.97999999999999998, 0.96666666666666667, 0.96666666666666667, 0.97333333333333338, 0.95999999999999996, 0.96666666666666667, 0.95999999999999996, 0.96666666666666667, 0.95333333333333337, 0.95333333333333337, 0.95333333333333337]

In [10]:

# plot the results
plt.plot(k_range, grid_mean_scores)
plt.xlabel(‘Value of K for KNN‘)
plt.ylabel(‘Cross-Validated Accuracy‘)

Out[10]:

<matplotlib.text.Text at 0x6e34090>

In [11]:

# examine the best model
print grid.best_score_
print grid.best_params_
print grid.best_estimator_

0.98
{‘n_neighbors‘: 13}
KNeighborsClassifier(algorithm=‘auto‘, leaf_size=30, metric=‘minkowski‘,
           metric_params=None, n_neighbors=13, p=2, weights=‘uniform‘)

3. 同时对多个参数进行搜索?

这里我们使用knn的两个参数，分别是n_neighbors和weights，其中weights参数默认是uniform，该参数将所有数据看成等同的，而另一值是distance，它将近邻的数据赋予更高的权重，而较远的数据赋予较低权重。

In [12]:

# define the parameter values that should be searched
k_range = range(1, 31)
weight_options = [‘uniform‘, ‘distance‘]

In [13]:

# create a parameter grid: map the parameter names to the values that should be searched
param_grid = dict(n_neighbors=k_range, weights=weight_options)
print param_grid

{‘n_neighbors‘: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], ‘weights‘: [‘uniform‘, ‘distance‘]}

In [14]:

# instantiate and fit the grid
grid = GridSearchCV(knn, param_grid, cv=10, scoring=‘accuracy‘)
grid.fit(X, y)

Out[14]:

GridSearchCV(cv=10, error_score=‘raise‘,
       estimator=KNeighborsClassifier(algorithm=‘auto‘, leaf_size=30, metric=‘minkowski‘,
           metric_params=None, n_neighbors=5, p=2, weights=‘uniform‘),
       fit_params={}, iid=True, loss_func=None, n_jobs=1,
       param_grid={‘n_neighbors‘: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], ‘weights‘: [‘uniform‘, ‘distance‘]},
       pre_dispatch=‘2*n_jobs‘, refit=True, score_func=None,
       scoring=‘accuracy‘, verbose=0)

In [15]:

# view the complete results
grid.grid_scores_

Out[15]:

[mean: 0.96000, std: 0.05333, params: {‘n_neighbors‘: 1, ‘weights‘: ‘uniform‘},
 mean: 0.96000, std: 0.05333, params: {‘n_neighbors‘: 1, ‘weights‘: ‘distance‘},
 mean: 0.95333, std: 0.05207, params: {‘n_neighbors‘: 2, ‘weights‘: ‘uniform‘},
 mean: 0.96000, std: 0.05333, params: {‘n_neighbors‘: 2, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 3, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 3, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 4, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 4, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 5, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 5, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 6, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 6, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 7, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 7, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 8, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 8, ‘weights‘: ‘distance‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 9, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 9, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 10, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 10, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 11, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 11, ‘weights‘: ‘distance‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 12, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.04422, params: {‘n_neighbors‘: 12, ‘weights‘: ‘distance‘},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 13, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 13, ‘weights‘: ‘distance‘},
 mean: 0.97333, std: 0.04422, params: {‘n_neighbors‘: 14, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 14, ‘weights‘: ‘distance‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 15, ‘weights‘: ‘uniform‘},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 15, ‘weights‘: ‘distance‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 16, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 16, ‘weights‘: ‘distance‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 17, ‘weights‘: ‘uniform‘},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 17, ‘weights‘: ‘distance‘},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 18, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 18, ‘weights‘: ‘distance‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 19, ‘weights‘: ‘uniform‘},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 19, ‘weights‘: ‘distance‘},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 20, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 20, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.03333, params: {‘n_neighbors‘: 21, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 21, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.03333, params: {‘n_neighbors‘: 22, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 22, ‘weights‘: ‘distance‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 23, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 23, ‘weights‘: ‘distance‘},
 mean: 0.96000, std: 0.04422, params: {‘n_neighbors‘: 24, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 24, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.03333, params: {‘n_neighbors‘: 25, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 25, ‘weights‘: ‘distance‘},
 mean: 0.96000, std: 0.04422, params: {‘n_neighbors‘: 26, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 26, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 27, ‘weights‘: ‘uniform‘},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 27, ‘weights‘: ‘distance‘},
 mean: 0.95333, std: 0.04269, params: {‘n_neighbors‘: 28, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 28, ‘weights‘: ‘distance‘},
 mean: 0.95333, std: 0.04269, params: {‘n_neighbors‘: 29, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 29, ‘weights‘: ‘distance‘},
 mean: 0.95333, std: 0.04269, params: {‘n_neighbors‘: 30, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.03333, params: {‘n_neighbors‘: 30, ‘weights‘: ‘distance‘}]

In [16]:

# examine the best model
print grid.best_score_
print grid.best_params_

0.98
{‘n_neighbors‘: 13, ‘weights‘: ‘uniform‘}

4. 使用最佳参数做出预测?

In [17]:

# train your model using all data and the best known parameters
knn = KNeighborsClassifier(n_neighbors=13, weights=‘uniform‘)
knn.fit(X, y)

# make a prediction on out-of-sample data
knn.predict([3, 5, 4, 2])

Out[17]:

array([1])

这里使用之前得到的最佳参数对模型进行重新训练，在训练时，就可以将所有的数据都作为训练数据全部投入到模型中去，这样就不会浪费个别数据了。

In [18]:

# shortcut: GridSearchCV automatically refits the best model using all of the data
grid.predict([3, 5, 4, 2])

Out[18]:

array([1])

5. 使用RandomizeSearchCV来降低计算代价?

  • RandomizeSearchCV用于解决多个参数的搜索过程中计算代价过高的问题
  • RandomizeSearchCV搜索参数中的一个子集，这样你可以控制计算代价

In [19]:

from sklearn.grid_search import RandomizedSearchCV

In [20]:

# specify "parameter distributions" rather than a "parameter grid"
param_dist = dict(n_neighbors=k_range, weights=weight_options)

In [21]:

# n_iter controls the number of searches
rand = RandomizedSearchCV(knn, param_dist, cv=10, scoring=‘accuracy‘, n_iter=10, random_state=5)
rand.fit(X, y)
rand.grid_scores_

Out[21]:

[mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 18, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 8, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 24, ‘weights‘: ‘distance‘},
 mean: 0.98000, std: 0.03055, params: {‘n_neighbors‘: 20, ‘weights‘: ‘uniform‘},
 mean: 0.95333, std: 0.04269, params: {‘n_neighbors‘: 28, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 9, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 5, ‘weights‘: ‘distance‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 5, ‘weights‘: ‘uniform‘},
 mean: 0.97333, std: 0.03266, params: {‘n_neighbors‘: 19, ‘weights‘: ‘uniform‘},
 mean: 0.96667, std: 0.04472, params: {‘n_neighbors‘: 20, ‘weights‘: ‘distance‘}]

In [22]:

# examine the best model
print rand.best_score_
print rand.best_params_

0.98
{‘n_neighbors‘: 20, ‘weights‘: ‘uniform‘}

In [23]:

# run RandomizedSearchCV 20 times (with n_iter=10) and record the best score
best_scores = []
for _ in range(20):
    rand = RandomizedSearchCV(knn, param_dist, cv=10, scoring=‘accuracy‘, n_iter=10)
    rand.fit(X, y)
    best_scores.append(round(rand.best_score_, 3))
print best_scores

[0.98, 0.98, 0.973, 0.98, 0.98, 0.98, 0.98, 0.98, 0.98, 0.98, 0.98, 0.973, 0.98, 0.98, 0.98, 0.973, 0.98, 0.98, 0.973, 0.973]

当你的调节参数是连续的，比如回归问题的正则化参数，有必要指定一个连续分布而不是可能值的列表，这样RandomizeSearchCV就可以执行更好的grid search。

参考资料?

时间： 2024-07-30 13:50:01

【scikit-learn】网格搜索来进行高效的参数调优的相关文章

调参必备---GridSearch网格搜索

什么是Grid Search 网格搜索? Grid Search:一种调参手段:穷举搜索:在所有候选的参数选择中,通过循环遍历,尝试每一种可能性,表现最好的参数就是最终的结果.其原理就像是在数组里找最大值.(为什么叫网格搜索?以有两个参数的模型为例,参数a有3种可能,参数b有4种可能,把所有可能性列出来,可以表示成一个3*4的表格,其中每个cell就是一个网格,循环过程就像是在每个网格里遍历.搜索,所以叫grid search) Simple Grid Search:简单的网格搜索 以2个参数的

Query意图分析：记一次完整的机器学习过程（scikit learn library学习笔记）

所谓学习问题,是指观察由n个样本组成的集合,并根据这些数据来预测未知数据的性质. 学习任务(一个二分类问题): 区分一个普通的互联网检索Query是否具有某个垂直领域的意图.假设现在有一个O2O领域的垂直搜索引擎,专门为用户提供团购.优惠券的检索:同时存在一个通用的搜索引擎,比如百度,通用搜索引擎希望能够识别出一个Query是否具有O2O检索意图,如果有则调用O2O垂直搜索引擎,获取结果作为通用搜索引擎的结果补充. 我们的目的是学习出一个分类器(classifier),分类器可以理解为一个函数,

机器学习 libsvm交叉验证与网格搜索（参数选择）

首先说交叉验证. 交叉验证(Cross validation)是一种评估统计分析.机器学习算法对独立于训练数据的数据集的泛化能力(generalize), 能够避免过拟合问题. 交叉验证一般要尽量满足: 1)训练集的比例要足够多,一般大于一半 2)训练集和测试集要均匀抽样 交叉验证主要分成以下几类: 1)Double cross-validation Double cross-validation也称2-fold cross-validation(2-CV),作法是将数据集分成两个相等大小的子集

Python之扩展包安装（scikit learn）

scikit learn 是Python下开源的机器学习包.(安装环境:win7.0 32bit和Python2.7) Python安装第三方扩展包较为方便的方法:easy_install + packages name 在官网 https://pypi.python.org/pypi/setuptools/#windows-simplified 下载名字为 的文件. 在命令行窗口运行 ,安装后,可在python2.7文件夹下生成Scripts文件夹.把路径D:\Python27\Scripts

libsvm交叉验证与网格搜索（参数选择）

首先说交叉验证.交叉验证(Cross validation)是一种评估统计分析.机器学习算法对独立于训练数据的数据集的泛化能力(generalize), 能够避免过拟合问题.交叉验证一般要尽量满足:1)训练集的比例要足够多,一般大于一半2)训练集和测试集要均匀抽样 交叉验证主要分成以下几类: 1)Double cross-validationDouble cross-validation也称2-fold cross-validation(2-CV),作法是将数据集分成两个相等大小的子集,进行两回

scikit learn 模块 调参 pipeline+girdsearch 数据举例：文档分类

scikit learn 模块 调参 pipeline+girdsearch 数据举例:文档分类数据集 fetch_20newsgroups #-*- coding: UTF-8 -*- import numpy as np from sklearn.pipeline import Pipeline from sklearn.linear_model import SGDClassifier from sklearn.grid_search import GridSearchCV from sk

SQL Server 性能调优4 之书写高效的查询

限制查询的行和列来提高性能 这条规则非常简单,这里就不细说了. 使用搜索可参数化判断(sargable conditions)来提高性能 Sargable 由 Search ARGument Able 简写而来,字面意思是搜索可参数化?还是比较晦涩哎... 总之使用Sargable判断可以帮助查询优化器更有效地利用索引,并提高采用 index seek 的可能性,我们先把所有的操作符分一下组. Sargable操作符 = > >= < <= BETWEEN LIKE (通配符必须出

Elasticsearch搜索调优

最近把搜索后端从AWS cloudsearch迁到了AWS ES和自建ES集群.测试发现search latency高于之前的benchmark,可见模拟数据远不如真实数据来的实在.这次在产线的backup ES上直接进行测试和优化,通过本文记录search调优的主要过程. 问题1:发现AWS ES shard级别的search latency是非常小的,符合期望,但是最终的查询耗时却非常大(ES response的took), 整体的耗时比预期要高出200ms~300ms. troublesh

Scikit Learn安装教程

Windows下安装scikit-learn 准备工作 Python (>= 2.6 or >= 3.3), Numpy (>= 1.6.1) Scipy (>= 0.9), Matplotlib(可选). NumPy NumPy系统是Python的一种开源的数值计算扩展.这种工具可用来存储和处理大型矩阵,比Python自身的嵌套列表(nested list structure)结构要高效的多(该结构也可以用来表示矩阵(matrix)). Scipy SciPy是一款方便.易于使用