删除大表的数据:普通删除语句和游标处理的比较

一、前言

WHERE条件用不到索引的情况下如何删除大表记录?文章中只是列举出其中一种方式而已,但仍然存在很多不严谨的地方.只是

轻描淡写随着数据表越来越大,历史数据的处理将越来越困难.且过滤字段没有索引(如时间)字段,而创建索引是需要临时表空间排序的.有越来越大,创建的维护工作也随之越来越难.

除了本文介绍的方法以外,还可以通过以下几种方法来维护历史数据:

1. 创建历史表,将历史数据定期移至历史表,让源表保持"瘦小身材"(源表会有碎片,需要定期对表和索引进行重建)

2. ORACLE系统包DBMS_REDEFINITION实现表的在线重定义

3. 把源表做分区处理,历史数据的维护可以转化成表/索引分区的维护.

本文将对以下几种方法做比较:

1. 普通删除语句(delete from tabname where condition1...condition2...)

2. 通过存储过程(每次提取一条)

3. 通过存储过程(批量提取)

以一张近500M的表作为实验对象,该表没有任何索引.总条数400多万:

SQL> select bytes/1024/1024 from user_segments where segment_name=‘ROBO‘;

BYTES/1024/1024

---------------

472         ---

SQL> select count(*) from robo;

COUNT(*)

----------

4154240  ----约400万条数据

SQL> select count(*) from robo where owner=‘TEST‘;

COUNT(*)

----------

4992  ---近5000条数据

SQL> select count(*) from robo where owner=‘SYS‘;

COUNT(*)

----------

1909760  ---近200万条数据

二、实验过程

1)假设待删除数据量小:

1. 普通删除的方法:

SQL> delete from robo where owner=‘TEST‘;

delete from robo

where

owner=‘TEST‘

性能数据:

-------------

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          1          1          0           0

Execute      1      0.65       0.64      59842      60092       5280        4992

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total        2      0.65       0.64      59843      60093       5280        4992

Misses in library cache during parse: 1

Optimizer mode: ALL_ROWS

Parsing user id: 52

Rows     Row Source Operation

-------  ---------------------------------------------------

0  DELETE  ROBO (cr=60092 pr=59842 pw=0 time=642682 us)

4992   TABLE ACCESS FULL ROBO (cr=60092 pr=59840 pw=0 time=485683 us)

只要全表扫描一次,删除效率很快

2. 游标处理(每次提取一条)

declare

v_rowid varchar2(25);

v_sqltext varchar2(200);

cursor c1 is select rowid from robo where owner=‘TEST‘;

begin

v_sqltext := ‘delete from robo where rowid=:1‘;

open c1;

loop

fetch c1 into v_rowid;

execute immediate v_sqltext using v_rowid;

exit when c1%notfound;

end loop;

dbms_output.put_line(c1%rowcount||‘ row deleted.‘);

close c1;

end;

/

4992 row deleted.

PL/SQL procedure successfully completed.

性能数据:

-------------

delete from robo

where

rowid=:1

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute   4993      0.33       0.31          1       4993       5280        4992

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total     4994      0.33       0.31          1       4993       5280        4992

Misses in library cache during parse: 0

Optimizer mode: ALL_ROWS

Parsing user id: 52     (recursive depth: 1)

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

db file sequential read                         1        0.00          0.00

********************************************************************************

declare

v_rowid varchar2(25);

v_sqltext varchar2(200);

cursor c1 is select rowid from robo where owner=‘TEST‘;

begin

v_sqltext := ‘delete from robo where rowid=:1‘;

open c1;

loop

fetch c1 into v_rowid;

execute immediate v_sqltext using v_rowid;

exit when c1%notfound;

end loop;

dbms_output.put_line(c1%rowcount||‘ row deleted.‘);

close c1;

end;

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute      1      0.26       0.22          0          0          0           1

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total        2      0.26       0.22          0          0          0           1

Misses in library cache during parse: 0

Optimizer mode: ALL_ROWS

Parsing user id: 52

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

SQL*Net message to client                       1        0.00          0.00

********************************************************************************

SELECT ROWID

FROM

ROBO WHERE OWNER=‘TEST‘

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute      1      0.00       0.00          0          0          0           0

Fetch     4993      0.70       0.73      60078      69059          0        4992

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total     4995      0.70       0.73      60078      69059          0        4992

Misses in library cache during parse: 0

Optimizer mode: ALL_ROWS

Parsing user id: 52     (recursive depth: 1)

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

SQL*Net message to client                       1        0.00          0.00

SQL*Net message from client                     1        3.13          3.13

db file sequential read                         1        0.00          0.00

db file scattered read                       3784        0.00          0.27

性能数据:

-----------------

delete from robo

where

rowid=:1

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute   4993      0.33       0.31          1       4993       5280        4992

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total     4994      0.33       0.31          1       4993       5280        4992

Misses in library cache during parse: 0

Optimizer mode: ALL_ROWS

Parsing user id: 52     (recursive depth: 1)

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

db file sequential read                         1        0.00          0.00

********************************************************************************

全表扫描次数为一次,但游标提取和删除次数等于结果集数目+1次。执行效率比普通删除来得低

2)假设待删除数据量小:

1. 普通删除方法

SQL> delete from robo where owner=‘SYS‘;

1909760 rows deleted.

Elapsed: 00:01:20.23

性能数据:

-------------

delete from robo

where

owner=‘SYS‘

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00         31         87          0           0

Execute      1     17.52      78.34      27888      60322    2123625     1909760

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total        2     17.53      78.34      27919      60409    2123625     1909760

Misses in library cache during parse: 1

Optimizer mode: ALL_ROWS

Parsing user id: 52

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

db file sequential read                      2998        0.00          0.02

db file scattered read                       2318        0.00          0.07

log file switch completion                     17        0.97         14.43

log file switch (checkpoint incomplete)        84        0.97         45.38

log buffer space                                4        0.19          0.29

SQL*Net message to client                       1        0.00          0.00

********************************************************************************

主要耗时是等待在线日志的切换,过程当中产生大量在线日志,执行效率大。

2. 游标处理(每交提取一条)

SQL> declare

v_rowid varchar2(25);

v_sqltext varchar2(200);

cursor c1 is select rowid from robo where owner=‘SYS‘;

begin

v_sqltext := ‘delete from robo where rowid=:1‘;

open c1;

loop

fetch c1 into v_rowid;

execute immediate v_sqltext using  2    3    4    5    6    7    8    9   10   v_rowid;

exit when c1%notfound;

end loop;

dbms_output.put_line(c1%rowcount||‘ row deleted.‘);

close c1;

end;

/

11   12   13   14   15   16

PL/SQL procedure successfully completed.

Elapsed: 00:04:12.89

性能数据:

-------------

delete from robo

where

rowid=:1

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute 1909761    112.14     162.39        344    1909874    2124331     1909760

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total   1909762    112.14     162.39        344    1909874    2124331     1909760

Misses in library cache during parse: 0

Optimizer mode: ALL_ROWS

Parsing user id: 52     (recursive depth: 1)

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

db file sequential read                       342        0.00          0.00

log file switch completion                     19        0.97         12.66

log file switch (checkpoint incomplete)        76        0.97         39.72

********************************************************************************

declare

v_rowid varchar2(25);

v_sqltext varchar2(200);

cursor c1 is select rowid from robo where owner=‘SYS‘;

begin

v_sqltext := ‘delete from robo where rowid=:1‘;

open c1;

loop

fetch c1 into v_rowid;

execute immediate v_sqltext using v_rowid;

exit when c1%notfound;

end loop;

dbms_output.put_line(c1%rowcount||‘ row deleted.‘);

close c1;

end;

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute      1     77.46      75.43          0          0          0           1

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total        2     77.46      75.43          0          0          0           1

Misses in library cache during parse: 1

Optimizer mode: ALL_ROWS

Parsing user id: 52

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

SQL*Net message to client                       1        0.00          0.00

********************************************************************************

SELECT ROWID

FROM

ROBO WHERE OWNER=‘SYS‘

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          1          1          0           0

Execute      1      0.00       0.00          0          0          0           0

Fetch   1909761     11.68      11.27      59936    2035069          0     1909760

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total   1909763     11.69      11.27      59937    2035070          0     1909760

Misses in library cache during parse: 1

Optimizer mode: ALL_ROWS

Parsing user id: 52     (recursive depth: 1)

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

SQL*Net message to client                       1        0.00          0.00

SQL*Net message from client                     1       18.98         18.98

db file sequential read                         7        0.00          0.00

db file scattered read                       3798        0.00          0.21

游标执行一次,将所有符合条件的ROWID一次性查询出来,但提取次数和删除次数与结果集相当。比起直接删除效率来得慢.

3. 游标处理(批量提取)

declare

type var_tab is table of varchar2(25) index by pls_integer;

v_rowid var_tab;

v_sqltext varchar2(200);

cursor c1 is select rowid from robo where owner=‘SYS‘;

begin

v_sqltext := ‘delete from robo where rowid=:1‘;

open c1;

loop

fetch c1 bulk collect  into v_rowid limit 50000;

for i in 1..v_rowid.count loop

execute immediate v_sqltext using v_rowid(i);

end loop;

exit when c1%notfound;

end loop;

dbms_output.put_line(c1%rowcount||‘ row deleted.‘);

close c1;

end;

/

性能数据:

------------

delete from robo

where

rowid=:1

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute 1909760    113.98     154.81      36737    2053459    2121671     1909760

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total   1909761    113.98     154.81      36737    2053459    2121671     1909760

Misses in library cache during parse: 0

Optimizer mode: ALL_ROWS

Parsing user id: 52     (recursive depth: 1)

Rows     Row Source Operation

-------  ---------------------------------------------------

1909760  DELETE  ROBO (cr=2053679 pr=36737 pw=0 time=79817096 us)

1909760   TABLE ACCESS BY USER ROWID ROBO (cr=2053394 pr=36266 pw=0 time=12527588 us)

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

db file sequential read                     36735        0.00          0.22

log file switch completion                     21        0.97         12.30

log file switch (checkpoint incomplete)        72        0.97         30.14

********************************************************************************

declare

type var_tab is table of varchar2(25) index by pls_integer;

v_rowid var_tab;

v_sqltext varchar2(200);

cursor c1 is select rowid from robo where owner=‘SYS‘;

begin

v_sqltext := ‘delete from robo where rowid=:1‘;

open c1;

loop

fetch c1 bulk collect  into v_rowid limit 50000;

for i in 1..v_rowid.count loop

execute immediate v_sqltext using v_rowid(i);

end loop;

exit when c1%notfound;

end loop;

dbms_output.put_line(c1%rowcount||‘ row deleted.‘);

close c1;

end;

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute      1     49.21      47.42          0          0          0           1

Fetch        0      0.00       0.00          0          0          0           0

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total        2     49.21      47.42          0          0          0           1

Misses in library cache during parse: 0

Optimizer mode: ALL_ROWS

Parsing user id: 52

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

SQL*Net message to client                       2        0.00          0.00

SQL*Net message from client                     2        0.00          0.00

********************************************************************************

SELECT ROWID

FROM

ROBO WHERE OWNER=‘SYS‘

call     count       cpu    elapsed       disk      query    current        rows

------- ------  -------- ---------- ---------- ---------- ----------  ----------

Parse        1      0.00       0.00          0          0          0           0

Execute      1      0.00       0.00          0          0          0           0

Fetch       39      2.48       2.42     126813     221222          1     1909760

------- ------  -------- ---------- ---------- ---------- ----------  ----------

total       41      2.48       2.42     126813     221222          1     1909760

Misses in library cache during parse: 0

Optimizer mode: ALL_ROWS

Parsing user id: 52     (recursive depth: 1)

Rows     Row Source Operation

-------  ---------------------------------------------------

1909760  TABLE ACCESS FULL ROBO (cr=221222 pr=126813 pw=0 time=1910539 us)

Elapsed times include waiting on following events:

Event waited on                             Times   Max. Wait  Total Waited

----------------------------------------   Waited  ----------  ------------

db file sequential read                       306        0.00          0.00

db file scattered read                       7945        0.00          0.47

latch: object queue header operation            2        0.00          0.00

db file parallel read                           1        0.00          0.00

全表扫描一次,但由于是批量提取数据(每次定为5万条)提高,减少提取次数,比起上一种方法节省了很大一部分时间.删除的次数依然与结果集相当.

三、总结:

普通删除和游标处理两种方法,删除操作的执行次数都相同。

1. 普通删除的方法:

较游标处理的方法相比较,虽然时间快,但记录数不好控制,从头删到尾,加上在线日志的归档,和不能批量提交给回滚段的压力非常大。

2. 游标处理的方法:

结果集的查询次数为一次,但提取次数跟方法相关.

批量提取的带来很大优势,主要体现在节省两个方面的时间: 1) 游标提取次数的减少,结果集提取时间缩短;2)游标执行时间的缩短

游标处理的最大优势是,虽然花费更多的时间,但能通过游标自由控制结果集,如批量删除和提交,减少回滚段的压力。

---------------------------------------------------

道行尚浅,欢迎拍砖。

转载请注明出处.

时间: 2024-11-03 05:33:05

删除大表的数据:普通删除语句和游标处理的比较的相关文章

MySQL 存储过程删除大表

1.权限问题 alter routine 编辑或删除存储过程 create routine 建立存储过程 execute 创建存储过程 2.存储过程相关的一些命令 show procedure status\G 查看数据库中有哪些存储过程 show procedure status where db = 'db_name'\G 查看指定数据库(db_name)中有哪些存储过程 select name from mysql.proc where db = 'db_name'; 查看指定数据库(db

cmd 命令行模式操作数据库 添加查询 修改 删除 ( 表 字段 数据)

一 查看数据库.表.数据字段.数据 1 首先配置环境变量 进入mysql  或者通过一键集成工具 打开mysql命令行  或者mysql 可视化工具 打开命令行 进入这样的界面   不同的机器操作不同,这里就不一一描述了 2 查看当前所有的数据库 show  databases: 3 选择(进入) 数据库 use   数据库名: 4  查看当前数据库所有的表 show tables: 5 查看 某个表的字段结构 desc  表明: 6 查询表数据 select * from  表名: 二  新建

WHERE条件用不到索引的情况下如何删除大表记录?

如果开发人员在做表设计阶段没有考虑到大表历史数据的维护性,随着数据量越来越大,表就越来越难管理和维护.连创建索引的失败都可能会失败(因为创建索引要利用临时表空间的的排序,当临时表空间不够大创建索引的动作就会报错).如果过滤条件用不到索引,每成功完成一次delete操作就需要全表扫描一次...那么几十GB的大表,情何以堪...? 那么有什么方法可以解决这种问题吗?当然改造生产表是可以的(利用DBMS_REDEFINITION在线重定义的方法可以很好的控制阻塞时间)再者可以利用游标获取结果集每行的R

sql:表中数据全部删除之后,重新插入时,从1开始增加

数据库中设置了自增列,有时候需要清楚数据库从新录入数据.最常见的做法就是使用sql语句"delete 表明名"或是直接选中数据,然后删除数据.但是再次插入数据的时候,你就会发现自增列会从上次删除数据的最大值的下一个值开始,而不是从1开始的.有没有一种感觉,感觉挺苦恼的. 现在有一个最简单暴力的方法,让你删除数据后,再次插入记录的时候,自增列从1开始. 直接从表中,将需要自增的字段进行修改,把自增这个选项进行取消,然后保存,保存之后再次对这个字段进行自增定义,这样子字段就会重新从1开始自

Oracle 删除大表中部分数据

需求: 项目中有一张表大概有7000多万条数据,造成表空间已满,需要清理部分数据,打算清理3000万. 2B 做法: delete from table_name where ID > '40000000'; 备注:select count(1) from table_name where ID > 'his_batch_4000000';  的结果大概有3000万条数据. 影响: 删了N个小时也没执行完,最终强制停止,造成表被锁.(没有管理员权限,需要联系DBA 才能解锁) 改进: decl

删除大表数据

declare cursor mycursor is SELECT ROWID FROM tpr_zjjx1 WHERE jxrq=to_date('2013-06-30','yyyy-mm-dd') order by rowid; type rowid_table_type is table of rowid index by pls_integer; v_rowid rowid_table_type; BEGIN open mycursor; loop fetch mycursor bulk

Oracle Delete inner的方式,级联删除子表的数据方式。

例子1: delete from table1 a where exists (select 1 from table2 b where a.id=b.id) 例子2: rebatepolicy表是主表,rebatepolicyitems是从表,从表有主表的主键,现在对于主表一些条件的数据的对应子表要求删除. 如下方式: delete from rebatepolicyitems rs where exists ( select 1 from rebatepolicy r where rs.re

【MySQL】使用硬链接的方式删除大表

Intro MySQL中删除比较大的表时,如果直接用drop table的方式进行删除,有可能会对整个实例产生影响甚至使得实例夯住.因此可以通过硬链接的方式对表进行删除,使得对生产环境的影响降到最低. drop table 的过程 持有 buffer pool mutex: 持有 buffer pool 中的 flush list mutex: 开始扫描 LRU list: 如果 dirty page 属于 drop table,那么就直接从 LRU list 中移除: 如果删除的 page 个

MySQL多表关联数据同时删除

MySQL多表关联时的多表删除: DELETE t1, t2FROM    t1LEFT JOIN t2 ON t1.id = t2.idWHERE    t1.id = 25 原文地址:https://www.cnblogs.com/leeego-123/p/10821106.html