如何更快地批量数据处理？

如何更快地批量数据处理？

作者 xzh2000

 

在批量处理数据时，比如从上亿条记录的表中删除上1000万条数据，不仅要考虑对undo的影响，还要考虑对redo，对磁盘i/o，锁、阻塞等的影响，在某种意思上讲，同样的任务，执行的sql次越少，总的消耗的时间就越少。。。

 

 

create table tmp_object1 as select * from dba_objects;
create index idx_object1_name on tmp_object1(object_name);
create table tmp_object2 as select * from dba_objects;

alter session set events '10046 trace name context forever, level 12';

declare
cursor cur_obj1 is select object_name from tmp_object1;

type v_array is table of tmp_object1.object_name%type;
v_obj1 v_array;
begin
open cur_obj1;
loop
fetch cur_obj1 bulk collect into v_obj1 limit 10000;

forall i in 1..v_obj1.count
delete from tmp_object1 where object_name = v_obj1(i);
commit;

exit when cur_obj1%notfound;
end loop;
close cur_obj1;
end;
/
alter session set events '10046 trace name context off';

OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- -------
Parse 16 0.00 0.00 0 2 0 0
Execute 26 9.23 9.03 254 52778 121592 51712
Fetch 39 0.07 0.07 1 1363 0 51735
------- ------ -------- ---------- ---------- ---------- ---------- -------
total 81 9.31 9.11 255 54143 121592 103447

Misses in library cache during parse: 5
Misses in library cache during execute: 4

Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ----------
db file sequential read 254 0.00 0.00

alter session set events '10046 trace name context forever, level 12';

declare
v_cnt number;
begin
select count(*) into v_cnt
from tmp_object2;

v_cnt := floor(v_cnt/10000) + 1;
for i in 1..v_cnt loop
delete from tmp_object2 where rownum <= 10000;
commit;
end loop;
end;
/
alter session set events '10046 trace name context off';

OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- -------
Parse 14 0.00 0.00 0 2 0 0
Execute 24 1.15 1.70 0 2785 57665 51712
Fetch 31 0.04 0.04 712 865 0 22
------- ------ -------- ---------- ---------- ---------- ---------- -------
total 69 1.20 1.75 712 3652 57665 51734

Misses in library cache during parse: 3
Misses in library cache during execute: 1

Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ----------
db file sequential read 17 0.00 0.00
db file scattered read 67 0.00 0.01
log file switch completion 3 0.02 0.04
log buffer space 1 0.11 0.11

说明：在上面的前半部分示例代码中，若把object_name = v_obj(i)改成rowid = v_obj(i)，删除数据时用rowid，而不是通过索引扫描，性能当然会略有提升，但总体来讲，比后半部分的代码，性能上还是大大不如。。。