Spill data to tempdb

查看Execution Plan时,在Sort Operator上,发现一个Warning:Operator used tempdb to spill data during execution with spill level 1

以XML格式查看执行计划,发现一个SpillToTempDb的节点:

<Warnings>
   <SpillToTempDb SpillLevel="1" />
</Warnings>

Sort Warnings are raised by SQL Server when a sort operation cannot be done in memory and must spill to tempdb.

一,Grant Memory

在 SQL Server 编译查询语句时, Optimizer会 Estimate 查询语句完成Sort或Hash操作所需要的内存(Grant Memory),在语句真正执行时,SQL Server 必须申请到这些内存,否则查询语句不会执行。在查询语句执行的过程中,Grant Memory不会被其他Thread占用,也不会减少,这样就能使每一个查询语句都有一个稳定的内存环境,保证查询语句能够顺利执行。由于Grant Memory是Optimizer在 compile-time预估的值,在查询语句执行时,sort 或 hash操作真正需要的内存(run-time memory),可能大于Grant Memory。对于Index Sort 操作,SQL Server从Buffer Pools申请内测,增加Grant Memory,以避免Spill Data to tempdb,如果是非Index Sort操作,那么SQL Server 为了完成Sort 或Hash 操作,就必须将Sort中间表或Hast Table 的一部分数据溢出到tempdb,在低速的disk中做排序或hash 操作(build hash table 或 probe hash match),降低了查询性能。同时,SQL Server 执行器会抛出Sort Warning 或 Hash Warning,这些信息是在run-time时抛出的,不能通过compile-time 查询计划(sys.dm_exec_query_plan 和 sys.dm_exec_text_query_plan)获取到。

引用《Never Ignore a Sort Warning in SQL Server》:

SQL Server does ask for more memory for an in-progress sort to avoid spilling to TempDB. But (there is always a ‘but’), it is not for regular sorts. It only works for index sorts. The interesting thing here is that it can “steal” from the buffer pool as much memory as it needs, so as to avoid a spill to disk.

1,Query Memory Spills

引用:Query Memory Spills

When you sometimes look at Execution Plans, you can see that the SELECT operator has sometimes a so-called Memory Grant assigned. This Memory Grant is specified in kilobytes and is needed for the query execution, when some operators (like Sort/Hash operators) in the Execution Plans need memory for execution – the so called Query Memory.

This query memory must be granted by SQL Server before the query is actually executed. The Query Optimizer uses the underlying Statistics to determine how much Query Memory must be acquired for a given query. The problem is now, when the Statistics are out-of-date, and SQL Server underestimates the processed rows. In this case, SQL Server will also request to less Query Memory for the given query. But when the query actually executes, the query can’t resize its granted Query Memory, and can’t just request more. The query must operate within the granted Query Memory. In this case, SQL Server has to spill the Sort/Hash-Operation into TempDb, which means that our very fast in-memory operation becomes a very slow physical On-Disk operation. SQL Server Profiler will report those Query Memory Spills through the events Sort Warnings and Hash Warning.

2,Hash Join

Before a hash join begins execution, SQL Server tries to estimate how much memory it will need to build its hash table. It uses the cardinality estimate for the size of the build input along with the expected average row size to estimate the memory requirement. To minimize the memory required by the hash join, the optimizer chooses the smaller of the two tables as the build table. SQL Server then tries to reserve sufficient memory to ensure that the hash join can successfully store the entire build table in memory. If SQL Server grants the hash join less memory than it requests or if the estimate is too low, the hash join might run out of memory during the build phase. If the hash join runs out of memory, it begins spilling a small percentage of the total hash table to disk (to a workfile in tempdb). The hash join keeps track of which buckets of the hash table are still in memory and which ones have been spilled to disk. As it reads each new row from the build table, it checks to see whether it hashes to an in-memory or an on-disk bucket. If it hashes to an in-memory bucket, it proceeds as usual. If it hashes to an on-disk bucket, it writes the row to disk. This process of running out of memory and spilling buckets to disk can repeat multiple times until the build phase is complete.

二,在SQL Server 2012中,通过Extended Events来Track memory spills.

--step1,create event session
create event session TrackSortWarning
on server
add event sqlserver.sort_warning
(
action (sqlserver.sql_text)
where(sqlserver.database_id=15)
)
add target package0.event_file
(
set filename=N‘D:\TrackSortWarning.xel‘,max_file_size=50
)
WITH
(
max_memory=4096 KB,
event_retention_mode=allow_single_event_loss,
max_dispatch_latency=30 seconds,
startup_state=on

);

--step2, start the event session

--step3,query 

;with cte as
(
select object_name,cast(event_data as xml) as event_data
from sys.fn_xe_file_target_read_file(‘D:\TrackSortWarning*.xel‘,null,null,null)
)
select event_data.value(‘(event/@timestamp)[1]‘,‘datetime‘) as [timestamp],
    event_data.value(‘(event/data[@name="sort_warning_type"]/text)[1]‘,‘varchar(20)‘) as sort_warning_type,
    event_data.value(‘(event/action[@name="sql_text"]/value)[1]‘,‘varchar(max)‘) as sql_text
from cte;

三,解决memory spills问题

1,Parameter Sniffing

在查询计划执行时,编译器探测变量的当前值,生成一个对变量的当前值“足够好”的执行计划。

select ...
from dbo.dt_test
where co1>@var_id
option(RECOMPILE);

When sort warnings are caused by parameter sniffing, the solution is to use the RECOMPILE query hint in the stored procedure definition.This option forces the optimizer to regenerate the execution plan for the statement containing the hint, rather than reusing an existing execution plan. The optimizer still sniffs the parameter value, but this process happens whenever a stored procedure is executed, not only for the first invocation. When an execution plan is generated, the optimizer checks the value of the parameter, chooses an optimal execution plan and grants sufficient memory to the appropriate operators.

2,更新Outdated stats

如果没有开启自动更新stats info,那么SQL Server Optimizer根据过期的统计信息生成的执行计划,可能会导致Estimated 数据行数小于actual 数据行数,致使Grant memory 偏小,使用 UPDATE STATISTICS (Transact-SQL) 更新统计信息。

3,建立Index,或使用Searchable Arguments(SARG)重写查询语句

如果查询语句不符合SARG的要求,SQL Server Optimizer无法使用index 或 统计信息,导致Optimizer预估的 “Estimated number of rows" 出现偏差。这种情况,必须重写查询语句,使用SARG。必要时,通过创建index 来动态增加Grant Memory,避免Spill data to tempdb。

推荐阅读《Never Ignore a Sort Warning in SQL Server

参考文档:

Never Ignore a Sort Warning in SQL Server

Correct SQL Server TempDB Spills in Query Plans Caused by Outdated Statistics

Query Memory Spills

Identifying and Solving Sort Warnings Problems in SQL Server

Operator used tempdb to spill data during execution with spill level

时间: 2024-08-01 15:50:43

Spill data to tempdb的相关文章

Operator used tempdb to spill data during execution with spill level 1

在执行一个查询语句时,发现 TOP(10) 和 TOP(100)所用时间差距很大.在对其调优时,发现 Sort Operator 消耗的时间高达95%,并抛出Warning: 1,为什么会出现warning? To quote: Actual number of rows is greater then estimated one. SQL Server grants memory before execution, looking on estimated values. At run tim

SQL Server中关于跟踪(Trace)那点事

前言 一提到跟踪俩字,很多人想到警匪片中的场景,同样在我们的SQL Server数据库中“跟踪”也是无处不在的,如果我们利用好了跟踪技巧,就可以针对某些特定的场景做定向分析,找出充足的证据来破案. 简单的举几个应用场景: 在线生产库为何突然宕机?数百张数据表为何不翼而飞?刚打好补丁的系统为何屡遭黑手?新添加的信息表为何频频丢失?某张表字段的突然更改,究竟为何人所为?这些个匿名的访问背后,究竟是人是鬼?突然增加的增量数据,究竟是对是错?数百兆的日志爆炸式的增长背后又隐藏着什么?这一且的背后,是应用

SQL TRACE过程中的事件号详细解释

我们定位数据库性能问题时经常会用到Trace跟踪,下面列举了一下Trace跟踪事件号的含义,方便查看 下表列出了可以在跟踪中添加或删除的事件. 事件号 事件名称 说明 0-9 保留 保留 10 RPC:Completed 在完成了远程过程调用 (RPC) 时发生. 11 RPC:Starting 在启动了 RPC 时发生. 12 SQL:BatchCompleted 在完成了 Transact-SQL 批处理时发生. 13 SQL:BatchStarting 在启动了 Transact-SQL

SQL Server中关于跟踪(Trace)那点事(转载)

前言 一提到跟踪俩字,很多人想到警匪片中的场景,同样在我们的SQL Server数据库中“跟踪”也是无处不在的,如果我们利用好了跟踪技巧,就可以针对某些特定的场景做定向分析,找出充足的证据来破案. 简单的举几个应用场景: 在线生产库为何突然宕机?数百张数据表为何不翼而飞?刚打好补丁的系统为何屡遭黑手?新添加的信息表为何频频丢失?某张表字段的突然更改,究竟为何人所为?这些个匿名的访问背后,究竟是人是鬼?突然增加的增量数据,究竟是对是错?数百兆的日志爆炸式的增长背后又隐藏着什么?这一且的背后,是应用

批量导出存储在msdb库的SSIS包

use msdb   go IF OBJECT_ID('msdb.dbo.usp_ExportSSISPkgs') IS NOT NULL     DROP PROCEDURE dbo.usp_ExportSSISPkgs;    go CREATE PROCEDURE dbo.usp_ExportSSISPkgs   @exportPath NVARCHAR(2000)='D:\temp\'    AS    BEGIN DECLARE @pkgData XML, @pkgName NVARC

Flink - Juggling with Bits and Bytes

http://www.36dsj.com/archives/33650 http://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html http://www.bigsynapse.com/addressing-big-data-performance ,addressing-big-data-performance   第一篇描述,当前JVM存在的问题, 1. Java对象开销 Java对象的存储密度相对偏低,对

SQL Server 移动系统数据库位置(非master)

以移动tempdb为例子: --------------------------------------------------------------------------------------------------------------------------- 第一步: 用alter database modify file 来指定新的物理文件位置 例子 alter database tempdb         modify file (name ='tempdev',filen

SQL Server 2012笔记分享-51:理解系统数据库恢复

下图是一个很重要的表格,详细描述了系统数据库的备份需求,支持的恢复模式和还原的选项. master数据库:需要备份,需要在单用户模式下恢复 model数据库:需要备份,恢复数据库的方式为T3608 trace flag msdb数据库:需要备份,恢复模式默认为简单,恢复数据库的方式和恢复普通数据库一样,没有特殊要求 tempdb数据库:不需要备份,因为在实例启动的时候tempdb会重建:     详细的步骤可以参考 http://msdn.microsoft.com/zh-cn/library/

《MSSQL2008技术内幕:T-SQL语言基础》读书笔记(下)

索引: 一.SQL Server的体系结构 二.查询 三.表表达式 四.集合运算 五.透视.逆透视及分组 六.数据修改 七.事务和并发 八.可编程对象 五.透视.逆透视及分组 5.1 透视 所谓透视(Pivoting)就是把数据从行的状态旋转为列的状态的处理.其处理步骤为: 相信很多人在笔试或面试的时候被问到如何通过SQL实现行转列或列转行的问题,可能很多人当时懵逼了,没关系,下面我们通过例子来理解. (1)准备数据 --1.0准备数据 USE tempdb; IF OBJECT_ID('dbo