用Hibernate插入大量数据
突然有个想法,把ip数据导入到数据库里,于是找了纯真qq的ip库,一共是331482条记录
开始的时候按常规的一条条导进去
经过观察发现save一条commit一次很慢,于是改为
速度虽然快了很多,但是依然是内存溢出
后来发现hibernate没有释放缓存,于是改成这样
这次终于可以了,331482条记录,从读取txt文件,并处理ip信息,再导入mysql的数据库,总共花了219秒
开始的时候按常规的一条条导进去
1
public int save(Object transientInstance) {
2
log.debug("saving "+Object.class.toString()+" instance");
3
Transaction tx = null;
4
Session session = getSession();
5
try {
6
tx = session.beginTransaction();
7
session.save(transientInstance);
8
tx.commit();
9
log.debug("save successful");
10
return 0;
11
} catch (Exception re) {
12
log.error("save failed", re);
13
if (tx != null && tx.isActive())
14
tx.rollback();
15
return -1;
16
}
17
}
导了一个晚上还是导完,原以为再等一段时间就可以了,后来居然已内存溢出告终,真是气死。![](/Images/OutliningIndicators/ExpandedBlockStart.gif)
2
![](/Images/OutliningIndicators/InBlock.gif)
3
![](/Images/OutliningIndicators/InBlock.gif)
4
![](/Images/OutliningIndicators/InBlock.gif)
5
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
6
![](/Images/OutliningIndicators/InBlock.gif)
7
![](/Images/OutliningIndicators/InBlock.gif)
8
![](/Images/OutliningIndicators/InBlock.gif)
9
![](/Images/OutliningIndicators/InBlock.gif)
10
![](/Images/OutliningIndicators/InBlock.gif)
11
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
12
![](/Images/OutliningIndicators/InBlock.gif)
13
![](/Images/OutliningIndicators/InBlock.gif)
14
![](/Images/OutliningIndicators/InBlock.gif)
15
![](/Images/OutliningIndicators/InBlock.gif)
16
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
17
![](/Images/OutliningIndicators/ExpandedBlockEnd.gif)
经过观察发现save一条commit一次很慢,于是改为
1
public int save(List transientInstances, int commitNum) {
2
log.debug("saving larger transientInstances");
3
Transaction tx = null;
4
Session session = getSession();
5
try {
6
tx = session.beginTransaction();
7
int i = 0;
8
for (Object transientInstance : transientInstances) {
9
session.save(transientInstance);
10
i++;
11
if (i == commitNum) {
12
tx.commit();
13
log.debug("save " + commitNum + "data successful");
14
i = 0;
15
16
tx = session.beginTransaction();
17
}
18
}
19
if (i != 0) {
20
tx.commit();
21![](/Images/OutliningIndicators/InBlock.gif)
22
}
23
// session.
24
log.debug("save larger successful");
25
return 0;
26
} catch (Exception re) {
27
log.error("save failed", re);
28
if (tx != null && tx.isActive())
29
tx.rollback();
30
return -1;
31
}
32
}
![](/Images/OutliningIndicators/ExpandedBlockStart.gif)
2
![](/Images/OutliningIndicators/InBlock.gif)
3
![](/Images/OutliningIndicators/InBlock.gif)
4
![](/Images/OutliningIndicators/InBlock.gif)
5
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
6
![](/Images/OutliningIndicators/InBlock.gif)
7
![](/Images/OutliningIndicators/InBlock.gif)
8
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
9
![](/Images/OutliningIndicators/InBlock.gif)
10
![](/Images/OutliningIndicators/InBlock.gif)
11
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
12
![](/Images/OutliningIndicators/InBlock.gif)
13
![](/Images/OutliningIndicators/InBlock.gif)
14
![](/Images/OutliningIndicators/InBlock.gif)
15
![](/Images/OutliningIndicators/InBlock.gif)
16
![](/Images/OutliningIndicators/InBlock.gif)
17
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
18
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
19
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
20
![](/Images/OutliningIndicators/InBlock.gif)
21
![](/Images/OutliningIndicators/InBlock.gif)
22
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
23
![](/Images/OutliningIndicators/InBlock.gif)
24
![](/Images/OutliningIndicators/InBlock.gif)
25
![](/Images/OutliningIndicators/InBlock.gif)
26
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
27
![](/Images/OutliningIndicators/InBlock.gif)
28
![](/Images/OutliningIndicators/InBlock.gif)
29
![](/Images/OutliningIndicators/InBlock.gif)
30
![](/Images/OutliningIndicators/InBlock.gif)
31
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
32
![](/Images/OutliningIndicators/ExpandedBlockEnd.gif)
速度虽然快了很多,但是依然是内存溢出
后来发现hibernate没有释放缓存,于是改成这样
1
public int save(List transientInstances, int commitNum) {
2
log.debug("saving larger transientInstances");
3
Transaction tx = null;
4
Session session = getSession();
5
try {
6
tx = session.beginTransaction();
7
int i = 0;
8
for (Object transientInstance : transientInstances) {
9
session.save(transientInstance);
10
i++;
11
if (i == commitNum) {
12
tx.commit();
13
log.debug("save " + commitNum + "data successful");
14
i = 0;
15
session.flush(); session.clear();
16
tx = session.beginTransaction();
17
}
18
}
19
if (i != 0) {
20
tx.commit();
21
session.flush(); session.clear();
22
}
23
// session.
24
log.debug("save larger successful");
25
return 0;
26
} catch (Exception re) {
27
log.error("save failed", re);
28
if (tx != null && tx.isActive())
29
tx.rollback();
30
return -1;
31
}
32
}
![](/Images/OutliningIndicators/ExpandedBlockStart.gif)
2
![](/Images/OutliningIndicators/InBlock.gif)
3
![](/Images/OutliningIndicators/InBlock.gif)
4
![](/Images/OutliningIndicators/InBlock.gif)
5
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
6
![](/Images/OutliningIndicators/InBlock.gif)
7
![](/Images/OutliningIndicators/InBlock.gif)
8
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
9
![](/Images/OutliningIndicators/InBlock.gif)
10
![](/Images/OutliningIndicators/InBlock.gif)
11
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
12
![](/Images/OutliningIndicators/InBlock.gif)
13
![](/Images/OutliningIndicators/InBlock.gif)
14
![](/Images/OutliningIndicators/InBlock.gif)
15
![](/Images/OutliningIndicators/InBlock.gif)
16
![](/Images/OutliningIndicators/InBlock.gif)
17
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
18
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
19
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
20
![](/Images/OutliningIndicators/InBlock.gif)
21
![](/Images/OutliningIndicators/InBlock.gif)
22
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
23
![](/Images/OutliningIndicators/InBlock.gif)
24
![](/Images/OutliningIndicators/InBlock.gif)
25
![](/Images/OutliningIndicators/InBlock.gif)
26
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
27
![](/Images/OutliningIndicators/InBlock.gif)
28
![](/Images/OutliningIndicators/InBlock.gif)
29
![](/Images/OutliningIndicators/InBlock.gif)
30
![](/Images/OutliningIndicators/InBlock.gif)
31
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
32
![](/Images/OutliningIndicators/ExpandedBlockEnd.gif)
这次终于可以了,331482条记录,从读取txt文件,并处理ip信息,再导入mysql的数据库,总共花了219秒