daixinet

  博客园  :: 首页  :: 新随笔  :: 联系 :: 订阅 订阅  :: 管理
2秒统计一亿数据

题目如下：

1、生成两个文件：
1）文件名Person.txt
字段：
personid，long，主键
name，string，内容随机
记录条数 1千万条
格式：txt，csv或者二进制，都可以。

2）文件名Order.txt
字段：
orderid，long，主键
personid，long，外键，内容必须在Person.txt里面，取值随机
amount，double，金额，数值随机
记录条数 1亿条，personid的模拟尽量均匀
格式：txt，csv或者二进制，都可以。

2、写程序以刚才生成的文件为依据，读取其内容，统计出等同与以下语句的一组数据，并写入文件 sqlresult.txt
1）给定一个person name ，SELECT o.amount FROM person p,order o WHERE o.personid=p. personid AND p.name=? ORDER BY o.amount limit 1000
2）SELECT p.name, sum(o.amount) sum, count(*) from person p,order o where o.personid=p. personid group by o.personid order by sum limit 1000 【选做】

Java要求：-Xmx1024m，即JVM最大内存超过1G
.NET要求在32位机器上跑。
开发要求：可以使用任何开源组件，包括单机数据库，但不允许使用SQL语句。

 

1、程序运行时间：
    1）本机时间测试：
    按名称查询：50ms左右
    汇总查询：2000ms左右
    2）局域网测试
    按名称查询：13573ms
    汇总查询：287162ms（5分钟左右，建议把文件Copy到本机）
   
2、文件构成：
    1）存储方式
    二进制，按位存储
    2）文件构成（共5个文件）
    person.nk：                 Person信息（190M），           ID：long（8），Name：String（12）
    person.nk.name：       Person索引文件（190M），    Name、PersonID
    order.nk：                    订单信息（2.23G），                ID：long（8），PersonID：long（8），Amount：double（8）
    order.nk.personid：     订单索引文件（1.48G），        personID，OrderID
    order.nk.amount：       订单索引文件（1.48G），        PersonID，Amount
3、设计思路：
    1）按照名称查询订单信息说明：
    因为person有1千万条，Order有1亿条，平均每人有10个订单；
    首先根据名称查询PersonID：person.nk.name，根据名称排序，二分法查找；
    然后再根据PersonID查询金额：order.nk.personid，根据PersonID排序，二分法查找订单；
    2）查询汇总信息：
    循环订单，统计金额和数量。
    直接查询order.nk文件，因为文件比较大，会花费大量的时间，所以创建PersonID和金额的索引，按PersonID排序，就可以统计出需要的结果。

 

源码：

test.java

 1 import java.util.ArrayList;
 2 import java.util.List;
 3 
 4 public class test {
 5 
 6     public static void main(String[] args) {
 7 
 8         //测试名称（也可通过 PersonOperate.printPersonData(); 打印所有人员）
 9         // Id:233745 Name:尉全平
10         // Id:233746 Name:汪哲宁
11         // Id:233747 Name:宦英凡
12         // Id:233748 Name:余彪朗
13         // Id:233749 Name:贾栋信
14         // Id:233750 Name:秋泽波
15         // Id:233751 Name:司栋固
16         // Id:233752 Name:黎苑芬
17         // Id:233753 Name:吉士天
18         // Id:233754 Name:轩兰兰
19         // Id:233755 Name:西冰凤
20         // Id:233756 Name:冀克新
21         // Id:233757 Name:梁韵婵
22         // Id:233758 Name:吉兰琳
23         // Id:233759 Name:齐艳纨
24         // Id:233760 Name:马枫聪
25         // Id:233761 Name:阙菊婉
26 
27 
28         //如有重名，默认取第一个人员，请通过 PersonOperate.getPersonsByName("东楠固") 方法确认
29         PersonOperate.printOrderByPersonName("支浩毅", 1000, true);
30 
31         OrderOperate.printTop1000SumData(1000, true);
32 
33 
34 
35     }
36 
37 }
View Code

PersonOperate.java

  1 import java.io.File;
  2 import java.io.IOException;
  3 import java.io.RandomAccessFile;
  4 import java.nio.MappedByteBuffer;
  5 import java.nio.channels.FileChannel;
  6 import java.util.ArrayList;
  7 import java.util.Collections;
  8 import java.util.Comparator;
  9 import java.util.List;
 10 
 11 public class PersonOperate {
 12 
 13     /**
 14      * 单条插入
 15      *
 16      * @param person
 17      */
 18     public static void insert(Person person) {
 19 
 20         File file = new File(CommonFunc.psnPath);
 21         long fileLen = file.length();
 22 
 23         RandomAccessFile raf = null;
 24         try {
 25             raf = new RandomAccessFile(file, "rw");
 26 
 27             long id = person.getPersonId();
 28             byte[] idByte = CommonFunc.getBytes(id);
 29             String name = person.getName();
 30             byte[] nameByte = CommonFunc.getBytes(name, CommonFunc.nameLen);
 31 
 32             byte[] idNameByte = new byte[CommonFunc.psnIdLen + CommonFunc.nameLen];
 33             System.arraycopy(idByte, 0, idNameByte, 0, CommonFunc.psnIdLen);
 34             System.arraycopy(nameByte, 0, idNameByte, CommonFunc.psnIdLen, CommonFunc.nameLen);
 35 
 36             raf.seek(fileLen);
 37             raf.write(idNameByte);
 38 
 39         } catch (Exception ex) {
 40             System.out.println(ex.getMessage());
 41         } finally {
 42             if (null != raf) {
 43                 try {
 44                     raf.close();
 45                 } catch (IOException e) {
 46                     System.out.println(e.getMessage());
 47                 }
 48             }
 49         }
 50     }
 51 
 52     /**
 53      * 批量插入
 54      *
 55      * @param personList
 56      */
 57     public static void batchInsert(Person[] personList) {
 58 
 59         File file = new File(CommonFunc.psnPath);
 60         long fileLen = file.length();
 61 
 62         RandomAccessFile raf = null;
 63         try {
 64             byte[] idNameBytes = new byte[(CommonFunc.psnIdLen + CommonFunc.nameLen) * personList.length];
 65             for (int i = 0; i < personList.length; i++) {
 66                 Person person = personList[i];
 67 
 68                 long id = person.getPersonId();
 69                 byte[] idByte = CommonFunc.getBytes(id);
 70                 String name = person.getName();
 71                 byte[] nameByte = CommonFunc.getBytes(name, CommonFunc.nameLen);
 72 
 73                 System.arraycopy(idByte, 0, idNameBytes, i * (CommonFunc.psnIdLen + CommonFunc.nameLen), CommonFunc.psnIdLen);
 74                 System.arraycopy(nameByte, 0, idNameBytes, i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen, CommonFunc.nameLen);
 75             }
 76 
 77             raf = new RandomAccessFile(file, "rw");
 78             raf.seek(fileLen);
 79             raf.write(idNameBytes);
 80         } catch (Exception ex) {
 81             System.out.println(ex.getMessage());
 82         } finally {
 83             if (null != raf) {
 84                 try {
 85                     raf.close();
 86                 } catch (IOException e) {
 87                     System.out.println(e.getMessage());
 88                 }
 89             }
 90         }
 91     }
 92 
 93     /**
 94      * 根据ID查对象
 95      *
 96      * @param personId
 97      * @return
 98      */
 99     public static Person getPersonById(long personId) {
100         Person rtn = new Person();
101 
102         File file = new File(CommonFunc.psnPath);
103         //获取位置
104         long pos = (personId - 1) * (CommonFunc.psnIdLen + CommonFunc.nameLen);
105 
106         RandomAccessFile raf = null;
107         try {
108             raf = new RandomAccessFile(file, "r");
109 
110             raf.seek(pos);
111             rtn.setPersonId(raf.readLong());
112             pos += CommonFunc.psnIdLen;
113 
114             raf.seek(pos);
115             byte[] nameByte = new byte[CommonFunc.nameLen];
116             raf.read(nameByte);
117             rtn.setName(CommonFunc.getString(nameByte));
118         } catch (Exception ex) {
119             System.out.println(ex.getMessage());
120         } finally {
121             if (null != raf) {
122                 try {
123                     raf.close();
124                 } catch (IOException e) {
125                     System.out.println(e.getMessage());
126                 }
127             }
128         }
129 
130         return rtn;
131     }
132 
133     /**
134      * 根据名字查询Person对象
135      *
136      * @param name
137      * @return
138      */
139     public static List<Person> getPersonsByName(String name) {
140         List<Person> rtn = new ArrayList<>();
141 
142         //获取要查询的ID列表
143         File idxFile = new File(CommonFunc.psnIdxPath);
144         long fileLen = idxFile.length();
145         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
146 
147         FileChannel idxFileChannel = null;
148         try {
149             idxFileChannel = new RandomAccessFile(idxFile, "r").getChannel();
150             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, idxFile.length());
151 
152             byte[] nameBytes = CommonFunc.getBytes(name, 12);
153             long firstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameBytes[0], nameBytes[1], nameBytes[2]});
154 
155             //二分法查找，确定位置
156             long findRow = -1;
157             long minRow = 1;
158             long maxRow = rowNum;
159             while (maxRow >= minRow) {
160                 long midRow = (minRow + maxRow) / 2;
161                 idxMappedByteBuffer.position((int) (midRow - 1) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
162                 byte[] nameByte = new byte[CommonFunc.nameLen];
163                 idxMappedByteBuffer.get(nameByte);
164                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
165 
166                 if (tempFirstNameNum > firstNameNum) {
167                     if (maxRow == midRow) {
168                         break;
169                     }
170                     maxRow = midRow;
171                 } else if (tempFirstNameNum < firstNameNum) {
172                     if (minRow == midRow) {
173                         minRow++;
174                         continue;
175                     }
176                     minRow = midRow;
177                 } else {
178                     findRow = midRow - 1;
179                     break;
180                 }
181             }
182 
183             if (findRow == -1) {
184                 return rtn;
185             }
186 
187             //前后搜索，确定该数据的最大行和最小行
188             long minRowIndex = 0;
189             while ((findRow - minRowIndex) >= 0) {
190                 idxMappedByteBuffer.position((int) (findRow - minRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
191                 byte[] nameByte = new byte[CommonFunc.nameLen];
192                 idxMappedByteBuffer.get(nameByte);
193                 String tempName = CommonFunc.getString(nameByte);
194                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
195 
196                 if (tempFirstNameNum != firstNameNum) {
197                     break;
198                 }
199 
200                 if (tempName.equals(name) == true) {
201                     idxMappedByteBuffer.position((int) (findRow - minRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
202                     long personId = idxMappedByteBuffer.getLong();
203 
204                     Person person = new Person();
205                     person.setPersonId(personId);
206                     person.setName(tempName);
207                     rtn.add(person);
208                 }
209                 minRowIndex++;
210             }
211 
212             long maxRowIndex = 1;
213             while ((findRow + maxRowIndex) <= rowNum) {
214                 idxMappedByteBuffer.position((int) (findRow + maxRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
215                 byte[] nameByte = new byte[CommonFunc.nameLen];
216                 idxMappedByteBuffer.get(nameByte);
217                 String tempName = CommonFunc.getString(nameByte);
218                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
219 
220                 if (tempFirstNameNum != firstNameNum) {
221                     break;
222                 }
223 
224                 if (tempName.equals(name) == true) {
225                     idxMappedByteBuffer.position((int) (findRow + maxRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
226                     long personId = idxMappedByteBuffer.getLong();
227 
228                     Person person = new Person();
229                     person.setPersonId(personId);
230                     person.setName(tempName);
231                     rtn.add(person);
232                 }
233 
234                 maxRowIndex++;
235             }
236         } catch (Exception ex) {
237             System.out.println(ex.getMessage());
238         } finally {
239             if (null != idxFileChannel) {
240                 try {
241                     idxFileChannel.close();
242                 } catch (IOException e) {
243                     System.out.println(e.getMessage());
244                 }
245             }
246         }
247 
248         return rtn;
249     }
250 
251     public static List<Person> getPersonsByName_bak(String name) {
252         List<Person> rtn = new ArrayList<>();
253 
254         File file = new File(CommonFunc.psnPath);
255         long fileLen = file.length();
256         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
257 
258         RandomAccessFile raf = null;
259         try {
260             raf = new RandomAccessFile(file, "r");
261             for (int i = 0; i < rowNum; i++) {
262                 Person person = new Person();
263                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen));
264                 person.setPersonId(raf.readLong());
265 
266                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen);
267                 byte[] nameByte = new byte[CommonFunc.nameLen];
268                 raf.read(nameByte);
269                 String pName = CommonFunc.getString(nameByte);
270                 person.setName(pName);
271 
272                 if (name.equals(pName) == true) {
273                     rtn.add(person);
274                 }
275             }
276         } catch (Exception ex) {
277             System.out.println(ex.getMessage());
278         } finally {
279             if (null != raf) {
280                 try {
281                     raf.close();
282                 } catch (IOException e) {
283                     System.out.println(e.getMessage());
284                 }
285             }
286         }
287 
288         return rtn;
289     }
290 
291     public static List<Person> getPersonsByName_bak2(String name) {
292         List<Person> rtn = new ArrayList<>();
293 
294         //获取要查询的ID列表
295         File idxFile = new File(CommonFunc.psnIdxPath);
296         long fileLen = idxFile.length();
297         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
298 
299         RandomAccessFile idxRaf = null;
300         try {
301             idxRaf = new RandomAccessFile(idxFile, "r");
302 
303             byte[] nameBytes = CommonFunc.getBytes(name, 12);
304             long firstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameBytes[0], nameBytes[1], nameBytes[2]});
305 
306             //二分法查找，确定位置
307             long findRow = -1;
308             long minRow = 1;
309             long maxRow = rowNum;
310             while (maxRow >= minRow) {
311                 long midRow = (minRow + maxRow) / 2;
312                 idxRaf.seek((midRow - 1) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
313                 byte[] nameByte = new byte[CommonFunc.nameLen];
314                 idxRaf.read(nameByte);
315                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
316 
317                 if (tempFirstNameNum > firstNameNum) {
318                     if (maxRow == midRow) {
319                         break;
320                     }
321                     maxRow = midRow;
322                 } else if (tempFirstNameNum < firstNameNum) {
323                     if (minRow == midRow) {
324                         minRow++;
325                         continue;
326                     }
327                     minRow = midRow;
328                 } else {
329                     findRow = midRow - 1;
330                     break;
331                 }
332             }
333 
334             if (findRow == -1) {
335                 return rtn;
336             }
337 
338             //前后搜索，确定该数据的最大行和最小行
339             long minRowIndex = 0;
340             while ((findRow - minRowIndex) >= 0) {
341                 idxRaf.seek((findRow - minRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
342                 byte[] nameByte = new byte[CommonFunc.nameLen];
343                 idxRaf.read(nameByte);
344                 String tempName = CommonFunc.getString(nameByte);
345                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
346 
347                 if (tempFirstNameNum != firstNameNum) {
348                     break;
349                 }
350 
351                 if (tempName.equals(name) == true) {
352                     idxRaf.seek((findRow - minRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
353                     byte[] idByte = new byte[CommonFunc.psnIdLen];
354                     idxRaf.read(idByte);
355                     long personId = CommonFunc.getLong(idByte);
356 
357                     Person person = getPersonById(personId);
358                     rtn.add(person);
359                 }
360                 minRowIndex++;
361             }
362 
363             long maxRowIndex = 1;
364             while ((findRow + maxRowIndex) <= rowNum) {
365                 idxRaf.seek((findRow + maxRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
366                 byte[] nameByte = new byte[CommonFunc.nameLen];
367                 idxRaf.read(nameByte);
368                 String tempName = CommonFunc.getString(nameByte);
369                 long tempFirstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
370 
371                 if (tempFirstNameNum != firstNameNum) {
372                     break;
373                 }
374 
375                 if (tempName.equals(name) == true) {
376                     idxRaf.seek((findRow + maxRowIndex) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
377                     byte[] idByte = new byte[CommonFunc.psnIdLen];
378                     idxRaf.read(idByte);
379                     long personId = CommonFunc.getLong(idByte);
380 
381                     Person person = getPersonById(personId);
382                     rtn.add(person);
383                 }
384 
385                 maxRowIndex++;
386             }
387         } catch (Exception ex) {
388             System.out.println(ex.getMessage());
389         } finally {
390             if (null != idxRaf) {
391                 try {
392                     idxRaf.close();
393                 } catch (IOException e) {
394                     System.out.println(e.getMessage());
395                 }
396             }
397         }
398 
399         return rtn;
400     }
401 
402     /**
403      * 创建名称索引
404      * <p/>
405      * 每个字符占3个字节，取Int值，占4个字节
406      * 3个字符，进行相加，然后进行排序
407      * <p/>
408      * 1个字节的最大值为 256*256*256 = 16777216
409      * 3个字节的最大值为 16777216 * 3 = 50331648
410      * <p/>
411      * 如：孙(15052185)天(15049897)天(15049897):45151979
412      */
413     public static void createfirstNameIndex() {
414 
415         long totalStartTime = System.currentTimeMillis();
416         //1、创建临时文件
417         System.out.println("开始创建临时文件");
418         long startTime = System.currentTimeMillis();
419 
420         createTempFile();
421 
422         long endTime = System.currentTimeMillis();
423         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
424         System.out.println("临时文件创建完成");
425 
426         //2、整理临时文件
427         System.out.println("整理临时文件");
428         startTime = System.currentTimeMillis();
429 
430         fillTempFile();
431 
432         endTime = System.currentTimeMillis();
433         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
434         System.out.println("整理临时文件完成");
435 
436         //3、创建索引文件
437         System.out.println("开始创建索引文件");
438         startTime = System.currentTimeMillis();
439 
440         createIdxFile();
441 
442         endTime = System.currentTimeMillis();
443         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
444         System.out.println("索引文件创建完成");
445 
446         long totalEndTime = System.currentTimeMillis();
447         System.out.println("程序运行总时间： " + (totalEndTime - totalStartTime) + "ms");
448     }
449 
450     private static void createTempFile() {
451         File tempFile = new File(CommonFunc.tempPath);
452         File file = new File(CommonFunc.psnPath);
453         long fileLen = file.length();
454         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
455 
456         FileChannel tempFileChannel = null;
457         RandomAccessFile raf = null;
458         try {
459             if (tempFile.exists() == true) {
460                 tempFile.delete();
461             }
462             tempFile.createNewFile();
463 
464             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
465             raf = new RandomAccessFile(file, "r");
466             for (int i = 0; i < rowNum; i++) {
467                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen);
468                 byte[] nameByte = new byte[CommonFunc.nameLen];
469                 raf.read(nameByte);
470                 long firstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
471 
472                 MappedByteBuffer mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_ONLY, (firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen), CommonFunc.tempLen);
473                 byte[] longByte = new byte[CommonFunc.tempLen];
474                 mappedByteBuffer.get(longByte);
475                 long tempVal = CommonFunc.getLong(longByte);
476 
477                 mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, (firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen), CommonFunc.tempLen + CommonFunc.tempLen);
478                 tempVal++;
479                 longByte = CommonFunc.getBytes(tempVal);
480                 byte[] zeroByte = new byte[CommonFunc.tempLen];
481                 byte[] idNameByte = new byte[CommonFunc.tempLen + CommonFunc.tempLen];
482                 System.arraycopy(longByte, 0, idNameByte, 0, CommonFunc.tempLen);
483                 System.arraycopy(zeroByte, 0, idNameByte, CommonFunc.tempLen, CommonFunc.tempLen);
484                 mappedByteBuffer.put(idNameByte);
485 
486                 if (i % 100000 == 0) {
487                     System.out.println("创建临时文件 " + i);
488                 }
489             }
490         } catch (Exception ex) {
491             System.out.println(ex.getMessage());
492         } finally {
493             if (null != tempFileChannel) {
494                 try {
495                     tempFileChannel.close();
496                 } catch (IOException e) {
497                     System.out.println(e.getMessage());
498                 }
499             }
500             if (null != raf) {
501                 try {
502                     raf.close();
503                 } catch (IOException e) {
504                     System.out.println(e.getMessage());
505                 }
506             }
507         }
508     }
509 
510     private static void fillTempFile() {
511         File tempFile = new File(CommonFunc.tempPath);
512         long fileLen = tempFile.length();
513         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
514 
515         FileChannel tempFileChannel = null;
516         try {
517             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
518             long preNum = 0;
519             for (int i = 0; i < rowNum; i++) {
520                 MappedByteBuffer mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_ONLY, i * (CommonFunc.tempLen + CommonFunc.tempLen), CommonFunc.tempLen);
521                 byte[] longByte = new byte[8];
522                 mappedByteBuffer.get(longByte);
523                 long tempVal = CommonFunc.getLong(longByte);
524 
525                 if (tempVal <= 0) {
526                     continue;
527                 }
528 
529                 mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, i * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen, CommonFunc.tempLen);
530                 longByte = CommonFunc.getBytes(preNum);
531                 mappedByteBuffer.put(longByte);
532                 preNum += tempVal;
533 
534                 if (i % 100000 == 0) {
535                     System.out.println("填充临时文件 " + i);
536                 }
537             }
538         } catch (Exception ex) {
539             System.out.println(ex.getMessage());
540         } finally {
541             if (null != tempFileChannel) {
542                 try {
543                     tempFileChannel.close();
544                 } catch (IOException e) {
545                     System.out.println(e.getMessage());
546                 }
547             }
548         }
549     }
550 
551     private static void createIdxFile() {
552         File file = new File(CommonFunc.psnPath);
553         File tempFile = new File(CommonFunc.tempPath);
554         File idxFile = new File(CommonFunc.psnIdxPath);
555         long fileLen = file.length();
556         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
557 
558         RandomAccessFile raf = null;
559         RandomAccessFile tempRaf = null;
560         RandomAccessFile idxRaf = null;
561         try {
562             if (idxFile.exists() == true) {
563                 idxFile.delete();
564             }
565             idxFile.createNewFile();
566 
567             raf = new RandomAccessFile(file, "r");
568             tempRaf = new RandomAccessFile(tempFile, "rw");
569             idxRaf = new RandomAccessFile(idxFile, "rw");
570 
571             for (int i = 0; i < rowNum; i++) {
572                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen));
573                 byte[] idByte = new byte[CommonFunc.psnIdLen];
574                 raf.read(idByte);
575                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen);
576                 byte[] nameByte = new byte[CommonFunc.nameLen];
577                 raf.read(nameByte);
578                 long firstNameNum = CommonFunc.getLong(new byte[]{0, 0, 0, 0, 0, nameByte[0], nameByte[1], nameByte[2]});
579 
580                 tempRaf.seek((firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
581                 byte[] longByte = new byte[CommonFunc.tempLen];
582                 tempRaf.read(longByte);
583                 long tempVal = CommonFunc.getLong(longByte);
584                 tempRaf.seek((firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
585                 longByte = new byte[CommonFunc.tempLen];
586                 tempRaf.read(longByte);
587                 long tempPreNum = CommonFunc.getLong(longByte);
588                 tempRaf.seek((firstNameNum - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
589                 tempVal--;
590                 longByte = CommonFunc.getBytes(tempVal);
591                 tempRaf.write(longByte);
592 
593                 //写入
594                 idxRaf.seek((tempPreNum + tempVal) * (CommonFunc.psnIdLen + CommonFunc.nameLen));
595                 idxRaf.write(nameByte);
596                 idxRaf.seek((tempPreNum + tempVal) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
597                 idxRaf.write(idByte);
598 
599                 if (i % 100000 == 0) {
600                     System.out.println("创建索引文件 " + i);
601                 }
602             }
603         } catch (Exception ex) {
604             System.out.println(ex.getMessage());
605         } finally {
606             if (null != raf) {
607                 try {
608                     raf.close();
609                 } catch (IOException e) {
610                     System.out.println(e.getMessage());
611                 }
612             }
613             if (null != tempRaf) {
614                 try {
615                     tempRaf.close();
616                 } catch (IOException e) {
617                     System.out.println(e.getMessage());
618                 }
619             }
620             if (null != idxRaf) {
621                 try {
622                     idxRaf.close();
623                 } catch (IOException e) {
624                     System.out.println(e.getMessage());
625                 }
626             }
627         }
628     }
629 
630     //以下为测试用方法
631 
632     //标准数据：1千万条
633     public static void createData() {
634 
635         File file = new File(CommonFunc.psnPath);
636         if (file.exists() == true) {
637             file.delete();
638         }
639         try {
640             file.createNewFile();
641         } catch (IOException e) {
642             System.out.println(e.getMessage());
643         }
644 
645         long totalStartTime = System.currentTimeMillis();
646         for (int j = 0; j < 10; j++) {
647 
648             long startTime = System.currentTimeMillis();
649             Person[] personList = new Person[1000000];
650             for (int i = (j * 1000000); i < (j * 1000000 + 1000000); i++) {
651 
652                 Person person = new Person();
653                 personList[i - (j * 1000000)] = person;
654 
655                 person.setPersonId(i + 1);
656                 person.setName(CommonFunc.getPersonName());
657             }
658 
659             PersonOperate.batchInsert(personList);
660             long endTime = System.currentTimeMillis();
661             System.out.println("程序第 " + j + " 次运行时间： " + (endTime - startTime) + "ms");
662         }
663         long totalEndTime = System.currentTimeMillis();
664         System.out.println("程序运行总时间： " + (totalEndTime - totalStartTime) + "ms");
665     }
666 
667     //测试数据 3条
668     public static void createTestData() {
669 
670         File file = new File(CommonFunc.psnPath);
671         if (file.exists() == true) {
672             file.delete();
673         }
674         try {
675             file.createNewFile();
676         } catch (IOException e) {
677             System.out.println(e.getMessage());
678         }
679 
680         //批量提交:1KW数据
681         long totalStartTime = System.currentTimeMillis();
682         for (int j = 0; j < 1; j++) {
683 
684             long startTime = System.currentTimeMillis();
685             Person[] personList = new Person[3];
686             for (int i = (j * 3); i < (j * 3 + 3); i++) {
687 
688                 Person person = new Person();
689                 personList[i - (j * 3)] = person;
690 
691                 person.setPersonId(i + 1);
692                 person.setName(CommonFunc.getPersonName());
693             }
694 
695             PersonOperate.batchInsert(personList);
696             long endTime = System.currentTimeMillis();
697             System.out.println("程序第 " + j + " 次运行时间： " + (endTime - startTime) + "ms");
698         }
699         long totalEndTime = System.currentTimeMillis();
700         System.out.println("程序运行总时间： " + (totalEndTime - totalStartTime) + "ms");
701     }
702 
703     public static void printPersonData() {
704 
705         List<Person> rtn = new ArrayList<>();
706 
707         File file = new File(CommonFunc.psnPath);
708         long fileLen = file.length();
709         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
710 
711         RandomAccessFile raf = null;
712         try {
713             raf = new RandomAccessFile(file, "r");
714             for (int i = 0; i < rowNum; i++) {
715                 Person person = new Person();
716                 rtn.add(person);
717 
718                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen));
719                 person.setPersonId(raf.readLong());
720 
721                 raf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen);
722                 byte[] nameByte = new byte[CommonFunc.nameLen];
723                 raf.read(nameByte);
724                 person.setName(CommonFunc.getString(nameByte));
725 
726                 System.out.println("Id:" + person.getPersonId() + " Name:" + person.getName());
727             }
728         } catch (Exception ex) {
729             System.out.println(ex.getMessage());
730         } finally {
731             if (null != raf) {
732                 try {
733                     raf.close();
734                 } catch (IOException e) {
735                     System.out.println(e.getMessage());
736                 }
737             }
738         }
739     }
740 
741     public static void printTempData() {
742 
743         File tempFile = new File(CommonFunc.tempPath);
744         long fileLen = tempFile.length();
745         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
746 
747         RandomAccessFile tempRaf = null;
748         try {
749             tempRaf = new RandomAccessFile(tempFile, "rw");
750             for (int i = 0; i < rowNum; i++) {
751                 tempRaf.seek(i * (CommonFunc.tempLen + CommonFunc.tempLen));
752                 byte[] longByte = new byte[8];
753                 tempRaf.read(longByte);
754                 long tempVal = CommonFunc.getLong(longByte);
755 
756                 if (tempVal <= 0) {
757                     continue;
758                 }
759 
760                 tempRaf.seek(i * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
761                 longByte = new byte[8];
762                 tempRaf.read(longByte);
763                 long tempPreNum = CommonFunc.getLong(longByte);
764 
765                 System.out.println("PersonNameNum : " + (i + 1) + " ; Num : " + tempVal + " ; PreNum : " + tempPreNum);
766             }
767         } catch (Exception ex) {
768             System.out.println(ex.getMessage());
769         } finally {
770             if (null != tempRaf) {
771                 try {
772                     tempRaf.close();
773                 } catch (IOException e) {
774                     System.out.println(e.getMessage());
775                 }
776             }
777         }
778     }
779 
780     public static void printIdxData() {
781 
782         File tempFile = new File(CommonFunc.psnIdxPath);
783         long fileLen = tempFile.length();
784         long rowNum = fileLen / (CommonFunc.psnIdLen + CommonFunc.nameLen);
785 
786         RandomAccessFile tempRaf = null;
787         try {
788             tempRaf = new RandomAccessFile(tempFile, "rw");
789             for (int i = 0; i < rowNum; i++) {
790 
791                 tempRaf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen));
792                 byte[] nameByte = new byte[CommonFunc.nameLen];
793                 tempRaf.read(nameByte);
794                 String name = CommonFunc.getString(nameByte);
795 
796                 tempRaf.seek(i * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.nameLen);
797                 byte[] idByte = new byte[CommonFunc.psnIdLen];
798                 tempRaf.read(idByte);
799                 long id = CommonFunc.getLong(idByte);
800 
801                 System.out.println("idx : " + i + " ; name : " + name + " ; id : " + id);
802             }
803         } catch (Exception ex) {
804             System.out.println(ex.getMessage());
805         } finally {
806             if (null != tempRaf) {
807                 try {
808                     tempRaf.close();
809                 } catch (IOException e) {
810                     System.out.println(e.getMessage());
811                 }
812             }
813         }
814     }
815 
816     public static void printOrderByPersonName(String name) {
817 
818         printOrderByPersonName(name, 0, false);
819     }
820 
821     public static void printOrderByPersonName(String name, int totalNum, final boolean desc) {
822 
823         long startTime = System.currentTimeMillis();
824         List<Person> personList = PersonOperate.getPersonsByName(name);
825         if (personList.size() <= 0) {
826             System.out.println("该名称没有对应的人员");
827             return;
828         }
829         //默认取第一个人
830         Person person = personList.get(0);
831         List<Order> orderList = OrderOperate.getOrderByPersonId(person.getPersonId());
832 
833         //排序
834         Collections.sort(orderList, new Comparator<Order>() {
835             @Override
836             public int compare(Order o1, Order o2) {
837                 int rtn = 0;
838                 if (o1.getAmount() < o2.getAmount()) {
839                     rtn = desc == false ? -1 : 1;
840                 } else if (o1.getAmount() > o2.getAmount()) {
841                     rtn = desc == false ? 1 : -1;
842                 }
843                 return rtn;
844             }
845         });
846 
847         List<Order> rtn;
848         if (totalNum > 0 && orderList.size() > totalNum) {
849             rtn = orderList.subList(0, totalNum);
850         } else {
851             rtn = orderList;
852         }
853 
854         if (rtn.size() <= 0) {
855             System.out.println("该名称对应的订单为0");
856         }
857 
858         long endTime = System.currentTimeMillis();
859 
860         for (Order order : rtn) {
861             System.out.println("订单ID：" + order.getOrderId() + " 价格：" + order.getAmount());
862         }
863         System.out.println("共查出 " + rtn.size() + " 条");
864         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
865     }
866 }
View Code

OrderOperate.java

   1 import java.io.File;
   2 import java.io.IOException;
   3 import java.io.RandomAccessFile;
   4 import java.nio.MappedByteBuffer;
   5 import java.nio.channels.FileChannel;
   6 import java.util.ArrayList;
   7 import java.util.List;
   8 
   9 public class OrderOperate {
  10 
  11     /**
  12      * 插入订单
  13      *
  14      * @param order
  15      */
  16     public static void insert(Order order) {
  17 
  18         File file = new File(CommonFunc.orderPath);
  19         long fileLen = file.length();
  20 
  21         RandomAccessFile raf = null;
  22         try {
  23             raf = new RandomAccessFile(file, "rw");
  24 
  25             long id = order.getOrderId();
  26             byte[] idByte = CommonFunc.getBytes(id);
  27             long personId = order.getPersonId();
  28             byte[] personIdByte = CommonFunc.getBytes(personId);
  29             double amount = order.getAmount();
  30             byte[] amountByte = CommonFunc.getBytes(amount);
  31 
  32             byte[] idNameByte = new byte[CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen];
  33             System.arraycopy(idByte, 0, idNameByte, 0, CommonFunc.orderIdLen);
  34             System.arraycopy(personIdByte, 0, idNameByte, CommonFunc.orderIdLen, CommonFunc.personIdLen);
  35             System.arraycopy(amountByte, 0, idNameByte, CommonFunc.orderIdLen + CommonFunc.personIdLen, CommonFunc.amountLen);
  36 
  37             raf.seek(fileLen);
  38             raf.write(idNameByte);
  39         } catch (Exception ex) {
  40             System.out.println(ex.getMessage());
  41         } finally {
  42             if (null != raf) {
  43                 try {
  44                     raf.close();
  45                 } catch (IOException e) {
  46                     System.out.println(e.getMessage());
  47                 }
  48             }
  49         }
  50     }
  51 
  52     /**
  53      * 批量插入订单
  54      *
  55      * @param orders
  56      */
  57     public static void batchInsert(Order[] orders) {
  58 
  59         File file = new File(CommonFunc.orderPath);
  60         long fileLen = file.length();
  61 
  62         RandomAccessFile raf = null;
  63         try {
  64             raf = new RandomAccessFile(file, "rw");
  65 
  66             byte[] idNameBytes = new byte[(CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) * orders.length];
  67             for (int i = 0; i < orders.length; i++) {
  68                 Order order = orders[i];
  69 
  70                 long id = order.getOrderId();
  71                 byte[] idByte = CommonFunc.getBytes(id);
  72                 long personId = order.getPersonId();
  73                 byte[] personIdByte = CommonFunc.getBytes(personId);
  74                 double amount = order.getAmount();
  75                 byte[] amountByte = CommonFunc.getBytes(amount);
  76 
  77                 System.arraycopy(idByte, 0, idNameBytes, i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen), CommonFunc.orderIdLen);
  78                 System.arraycopy(personIdByte, 0, idNameBytes, i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen, CommonFunc.personIdLen);
  79                 System.arraycopy(amountByte, 0, idNameBytes, i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + (CommonFunc.orderIdLen + CommonFunc.personIdLen), CommonFunc.amountLen);
  80             }
  81             raf.seek(fileLen);
  82             raf.write(idNameBytes);
  83 
  84         } catch (Exception ex) {
  85             System.out.println(ex.getMessage());
  86         } finally {
  87             if (null != raf) {
  88                 try {
  89                     raf.close();
  90                 } catch (IOException e) {
  91                     System.out.println(e.getMessage());
  92                 }
  93             }
  94         }
  95     }
  96 
  97     /**
  98      * 根据ID查询Order（二分法查询）
  99      *
 100      * @param orderId
 101      * @return
 102      */
 103     public static Order getOrderById(long orderId) {
 104         Order rtn = new Order();
 105 
 106         File file = new File(CommonFunc.orderPath);
 107         //获取位置
 108         long pos = (orderId - 1) * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
 109 
 110         RandomAccessFile raf = null;
 111         try {
 112             raf = new RandomAccessFile(file, "r");
 113 
 114             raf.seek(pos);
 115             rtn.setOrderId(raf.readLong());
 116             pos += CommonFunc.orderIdLen;
 117 
 118             raf.seek(pos);
 119             rtn.setPersonId(raf.readLong());
 120             pos += CommonFunc.personIdLen;
 121 
 122             raf.seek(pos);
 123             rtn.setAmount(raf.readDouble());
 124         } catch (Exception ex) {
 125             System.out.println(ex.getMessage());
 126         } finally {
 127             if (null != raf) {
 128                 try {
 129                     raf.close();
 130                 } catch (IOException e) {
 131                     System.out.println(e.getMessage());
 132                 }
 133             }
 134         }
 135 
 136         return rtn;
 137     }
 138 
 139     /**
 140      * 根据PersonID查询所有订单
 141      *
 142      * @param personId
 143      * @return
 144      */
 145     public static List<Order> getOrderByPersonId(long personId) {
 146         List<Order> rtn = new ArrayList<>();
 147 
 148         //获取要查询的ID列表
 149         File idxFile = new File(CommonFunc.orderIdxPath);
 150         long fileLen = idxFile.length();
 151         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
 152 
 153         FileChannel idxFileChannel = null;
 154         try {
 155             idxFileChannel = new RandomAccessFile(idxFile, "r").getChannel();
 156             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, idxFile.length());
 157 
 158             //二分法查找，确定位置
 159             long findRow = -1;
 160             long minRow = 1;
 161             long maxRow = rowNum;
 162             while (maxRow > minRow) {
 163                 long midRow = (minRow + maxRow) / 2;
 164                 idxMappedByteBuffer.position((int) (midRow - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
 165                 long tempPersonId = idxMappedByteBuffer.getLong();
 166                 if (tempPersonId > personId) {
 167                     if (maxRow == midRow) {
 168                         break;
 169                     }
 170                     maxRow = midRow;
 171                 } else if (tempPersonId < personId) {
 172                     if (minRow == midRow) {
 173                         minRow++;
 174                         continue;
 175                     }
 176                     minRow = midRow;
 177                 } else {
 178                     findRow = midRow - 1;
 179                     break;
 180                 }
 181             }
 182 
 183             if (findRow == -1) {
 184                 return rtn;
 185             }
 186 
 187             //前后搜索，确定该数据的最大行和最小行
 188             long minRowIndex = 0;
 189             while ((findRow - minRowIndex) >= 0) {
 190                 idxMappedByteBuffer.position((int) (findRow - minRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen));
 191                 long tempPersonId = idxMappedByteBuffer.getLong();
 192                 if (tempPersonId != personId) {
 193                     break;
 194                 }
 195 
 196                 idxMappedByteBuffer.position((int) (findRow - minRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
 197                 long orderId = idxMappedByteBuffer.getLong();
 198                 Order order = getOrderById(orderId);
 199                 rtn.add(order);
 200                 minRowIndex++;
 201             }
 202 
 203             long maxRowIndex = 1;
 204             while ((findRow + maxRowIndex) <= rowNum) {
 205                 idxMappedByteBuffer.position((int) (findRow + maxRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen));
 206                 long tempPersonId = idxMappedByteBuffer.getLong();
 207                 if (tempPersonId != personId) {
 208                     break;
 209                 }
 210 
 211                 idxMappedByteBuffer.position((int) (findRow + maxRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
 212                 long orderId = idxMappedByteBuffer.getLong();
 213                 Order order = getOrderById(orderId);
 214                 rtn.add(order);
 215                 maxRowIndex++;
 216             }
 217         } catch (Exception ex) {
 218             System.out.println(ex.getMessage());
 219         } finally {
 220             if (null != idxFileChannel) {
 221                 try {
 222                     idxFileChannel.close();
 223                 } catch (IOException e) {
 224                     System.out.println(e.getMessage());
 225                 }
 226             }
 227         }
 228 
 229         return rtn;
 230     }
 231 
 232     public static List<Order> getOrderByPersonId_bak(long personId) {
 233         List<Order> rtn = new ArrayList<>();
 234 
 235         //获取要查询的ID列表
 236         File idxFile = new File(CommonFunc.orderIdxPath);
 237         long fileLen = idxFile.length();
 238         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
 239 
 240         RandomAccessFile idxRaf = null;
 241         try {
 242             idxRaf = new RandomAccessFile(idxFile, "r");
 243 
 244             //二分法查找，确定位置
 245             long findRow = -1;
 246             long minRow = 1;
 247             long maxRow = rowNum;
 248             while (maxRow > minRow) {
 249                 long midRow = (minRow + maxRow) / 2;
 250                 idxRaf.seek((midRow - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
 251                 byte[] longByte = new byte[8];
 252                 idxRaf.read(longByte);
 253                 long tempPersonId = CommonFunc.getLong(longByte);
 254                 if (tempPersonId > personId) {
 255                     if (maxRow == midRow) {
 256                         break;
 257                     }
 258                     maxRow = midRow;
 259                 } else if (tempPersonId < personId) {
 260                     if (minRow == midRow) {
 261                         minRow++;
 262                         continue;
 263                     }
 264                     minRow = midRow;
 265                 } else {
 266                     findRow = midRow - 1;
 267                     break;
 268                 }
 269             }
 270 
 271             if (findRow == -1) {
 272                 return rtn;
 273             }
 274 
 275             //前后搜索，确定该数据的最大行和最小行
 276             long minRowIndex = 0;
 277             while ((findRow - minRowIndex) >= 0) {
 278                 idxRaf.seek((findRow - minRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen));
 279                 byte[] longByte = new byte[8];
 280                 idxRaf.read(longByte);
 281                 long tempPersonId = CommonFunc.getLong(longByte);
 282 
 283                 if (tempPersonId != personId) {
 284                     break;
 285                 }
 286 
 287                 idxRaf.seek((findRow - minRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
 288                 longByte = new byte[8];
 289                 idxRaf.read(longByte);
 290                 long orderId = CommonFunc.getLong(longByte);
 291 
 292                 Order order = getOrderById(orderId);
 293                 rtn.add(order);
 294 
 295                 minRowIndex++;
 296             }
 297 
 298             long maxRowIndex = 1;
 299             while ((findRow + maxRowIndex) <= rowNum) {
 300                 idxRaf.seek((findRow + maxRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen));
 301                 byte[] longByte = new byte[8];
 302                 idxRaf.read(longByte);
 303                 long tempPersonId = CommonFunc.getLong(longByte);
 304 
 305                 if (tempPersonId != personId) {
 306                     break;
 307                 }
 308 
 309                 idxRaf.seek((findRow + maxRowIndex) * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
 310                 longByte = new byte[8];
 311                 idxRaf.read(longByte);
 312                 long orderId = CommonFunc.getLong(longByte);
 313 
 314                 Order order = getOrderById(orderId);
 315                 rtn.add(order);
 316 
 317                 maxRowIndex++;
 318             }
 319         } catch (Exception ex) {
 320             System.out.println(ex.getMessage());
 321         } finally {
 322             if (null != idxRaf) {
 323                 try {
 324                     idxRaf.close();
 325                 } catch (IOException e) {
 326                     System.out.println(e.getMessage());
 327                 }
 328             }
 329         }
 330 
 331         return rtn;
 332     }
 333 
 334     /**
 335      * 创建PersonID索引（时间太长，待进一步优化）
 336      */
 337     public static void createPersonIdIndex() {
 338 
 339         long totalStartTime = System.currentTimeMillis();
 340 
 341         // 1）做一个PersonId，数量的文件；
 342         System.out.println("开始创建临时文件");
 343         long startTime = System.currentTimeMillis();
 344 
 345         createTempFile();
 346 
 347         long endTime = System.currentTimeMillis();
 348         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
 349         System.out.println("临时文件创建完成");
 350 
 351         // 2）做一个空间；
 352         System.out.println("整理临时文件");
 353         startTime = System.currentTimeMillis();
 354 
 355         fillTempFile();
 356 
 357         endTime = System.currentTimeMillis();
 358         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
 359         System.out.println("整理临时文件完成");
 360 
 361         // 3）填充空间；
 362         System.out.println("开始创建索引文件");
 363         startTime = System.currentTimeMillis();
 364 
 365         createIdxFile();
 366 
 367         endTime = System.currentTimeMillis();
 368         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
 369         System.out.println("索引文件创建完成");
 370 
 371         long totalEndTime = System.currentTimeMillis();
 372         System.out.println("程序运行总时间： " + (totalEndTime - totalStartTime) + "ms");
 373 
 374     }
 375 
 376     public static void createPersonIdAmontIndex() {
 377 
 378         long totalStartTime = System.currentTimeMillis();
 379 
 380         // 1）做一个PersonId，数量的文件；
 381         System.out.println("开始创建临时文件");
 382         long startTime = System.currentTimeMillis();
 383 
 384         createTempFile();
 385 
 386         long endTime = System.currentTimeMillis();
 387         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
 388         System.out.println("临时文件创建完成");
 389 
 390         // 2）做一个空间；
 391         System.out.println("整理临时文件");
 392         startTime = System.currentTimeMillis();
 393 
 394         fillTempFile();
 395 
 396         endTime = System.currentTimeMillis();
 397         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
 398         System.out.println("整理临时文件完成");
 399 
 400         // 3）填充空间；
 401         System.out.println("开始创建索引文件");
 402         startTime = System.currentTimeMillis();
 403 
 404         createIdxAmountFile();
 405 
 406         endTime = System.currentTimeMillis();
 407         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
 408         System.out.println("索引文件创建完成");
 409 
 410         long totalEndTime = System.currentTimeMillis();
 411         System.out.println("程序运行总时间： " + (totalEndTime - totalStartTime) + "ms");
 412 
 413     }
 414 
 415     private static void createTempFile() {
 416         File tempFile = new File(CommonFunc.tempPath);
 417         File file = new File(CommonFunc.orderPath);
 418         long fileLen = file.length();
 419         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
 420 
 421         FileChannel tempFileChannel = null;
 422         RandomAccessFile raf = null;
 423         try {
 424             if (tempFile.exists() == true) {
 425                 tempFile.delete();
 426             }
 427             tempFile.createNewFile();
 428 
 429             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
 430             MappedByteBuffer mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, tempFile.length());
 431             raf = new RandomAccessFile(file, "r");
 432             for (long i = 0; i < rowNum; i++) {
 433                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen);
 434                 byte[] longByte = new byte[8];
 435                 raf.read(longByte);
 436                 long personId = CommonFunc.getLong(longByte);
 437 
 438                 int currentLen = (int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen));
 439                 if (mappedByteBuffer.capacity() < currentLen + (CommonFunc.tempLen + CommonFunc.tempLen)) {
 440                     mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, currentLen + (CommonFunc.tempLen + CommonFunc.tempLen));
 441                 }
 442 
 443                 mappedByteBuffer.position(currentLen);
 444                 long tempVal = mappedByteBuffer.getLong();
 445 
 446                 tempVal++;
 447                 mappedByteBuffer.position(currentLen);
 448                 mappedByteBuffer.putLong(tempVal);
 449 
 450                 if (i % 1000000 == 0) {
 451                     System.out.println("创建临时文件 " + i);
 452                 }
 453             }
 454         } catch (Exception ex) {
 455             System.out.println(ex.getMessage());
 456         } finally {
 457             if (null != tempFileChannel) {
 458                 try {
 459                     tempFileChannel.close();
 460                 } catch (IOException e) {
 461                     System.out.println(e.getMessage());
 462                 }
 463             }
 464             if (null != raf) {
 465                 try {
 466                     raf.close();
 467                 } catch (IOException e) {
 468                     System.out.println(e.getMessage());
 469                 }
 470             }
 471         }
 472     }
 473 
 474     private static void fillTempFile() {
 475         File tempFile = new File(CommonFunc.tempPath);
 476         long fileLen = tempFile.length();
 477         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
 478 
 479         FileChannel tempFileChannel = null;
 480         try {
 481             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
 482             MappedByteBuffer mappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, fileLen);
 483             long preNum = 0;
 484             for (int i = 0; i < rowNum; i++) {
 485                 mappedByteBuffer.position(i * (CommonFunc.tempLen + CommonFunc.tempLen));
 486                 long tempVal = mappedByteBuffer.getLong();
 487 
 488                 if (tempVal <= 0) {
 489                     continue;
 490                 }
 491 
 492                 mappedByteBuffer.position(i * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
 493                 mappedByteBuffer.putLong(preNum);
 494                 preNum += tempVal;
 495 
 496                 if (i % 1000000 == 0) {
 497                     System.out.println("填充临时文件 " + i);
 498                 }
 499             }
 500         } catch (Exception ex) {
 501             System.out.println(ex.getMessage());
 502         } finally {
 503             if (null != tempFileChannel) {
 504                 try {
 505                     tempFileChannel.close();
 506                 } catch (IOException e) {
 507                     System.out.println(e.getMessage());
 508                 }
 509             }
 510         }
 511     }
 512 
 513     private static void createIdxFile() {
 514         File file = new File(CommonFunc.orderPath);
 515         File tempFile = new File(CommonFunc.tempPath);
 516         File idxFile = new File(CommonFunc.orderIdxPath);
 517         long fileLen = file.length();
 518         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
 519 
 520         RandomAccessFile raf = null;
 521         FileChannel tempFileChannel = null;
 522         FileChannel idxFileChannel = null;
 523         try {
 524             if (idxFile.exists() == true) {
 525                 idxFile.delete();
 526             }
 527             idxFile.createNewFile();
 528 
 529             raf = new RandomAccessFile(file, "r");
 530             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
 531             idxFileChannel = new RandomAccessFile(idxFile, "rw").getChannel();
 532             MappedByteBuffer tempMappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, tempFile.length());
 533             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, rowNum * (CommonFunc.personIdLen + CommonFunc.orderIdLen));
 534             for (long i = 0; i < rowNum; i++) {
 535                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen));
 536                 byte[] idByte = new byte[CommonFunc.orderIdLen];
 537                 raf.read(idByte);
 538                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen);
 539                 byte[] personIdByte = new byte[CommonFunc.personIdLen];
 540                 raf.read(personIdByte);
 541                 long personId = CommonFunc.getLong(personIdByte);
 542 
 543                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)));
 544                 long tempVal = tempMappedByteBuffer.getLong();
 545                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)) + CommonFunc.tempLen);
 546                 long tempPreNum = tempMappedByteBuffer.getLong();
 547                 tempVal--;
 548                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)));
 549                 tempMappedByteBuffer.putLong(tempVal);
 550                 //写入
 551                 idxMappedByteBuffer.position((int) ((tempPreNum + tempVal) * (CommonFunc.orderIdLen + CommonFunc.personIdLen)));
 552                 idxMappedByteBuffer.put(personIdByte);
 553                 idxMappedByteBuffer.position((int) ((tempPreNum + tempVal) * (CommonFunc.orderIdLen + CommonFunc.personIdLen)) + CommonFunc.personIdLen);
 554                 idxMappedByteBuffer.put(idByte);
 555 
 556                 if (i % 1000000 == 0) {
 557                     System.out.println("创建索引文件 " + i);
 558                 }
 559             }
 560         } catch (Exception ex) {
 561             System.out.println(ex.getMessage());
 562         } finally {
 563             if (null != raf) {
 564                 try {
 565                     raf.close();
 566                 } catch (IOException e) {
 567                     System.out.println(e.getMessage());
 568                 }
 569             }
 570             if (null != tempFileChannel) {
 571                 try {
 572                     tempFileChannel.close();
 573                 } catch (IOException e) {
 574                     System.out.println(e.getMessage());
 575                 }
 576             }
 577             if (null != idxFileChannel) {
 578                 try {
 579                     idxFileChannel.close();
 580                 } catch (IOException e) {
 581                     System.out.println(e.getMessage());
 582                 }
 583             }
 584         }
 585     }
 586 
 587     private static void createIdxAmountFile() {
 588         File file = new File(CommonFunc.orderPath);
 589         File tempFile = new File(CommonFunc.tempPath);
 590         File idxFile = new File(CommonFunc.orderIdxAmountPath);
 591         long fileLen = file.length();
 592         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
 593 
 594         RandomAccessFile raf = null;
 595         FileChannel tempFileChannel = null;
 596         FileChannel idxFileChannel = null;
 597         try {
 598             if (idxFile.exists() == true) {
 599                 idxFile.delete();
 600             }
 601             idxFile.createNewFile();
 602 
 603             raf = new RandomAccessFile(file, "r");
 604             tempFileChannel = new RandomAccessFile(tempFile, "rw").getChannel();
 605             idxFileChannel = new RandomAccessFile(idxFile, "rw").getChannel();
 606             MappedByteBuffer tempMappedByteBuffer = tempFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, tempFile.length());
 607             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_WRITE, 0, rowNum * (CommonFunc.personIdLen + CommonFunc.orderIdLen));
 608             for (long i = 0; i < rowNum; i++) {
 609                 //raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen));
 610                 //byte[] idByte = new byte[CommonFunc.orderIdLen];
 611                 //raf.read(idByte);
 612                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen);
 613                 byte[] personIdByte = new byte[CommonFunc.personIdLen];
 614                 raf.read(personIdByte);
 615                 long personId = CommonFunc.getLong(personIdByte);
 616                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen + CommonFunc.personIdLen);
 617                 double amount = raf.readDouble();
 618 
 619                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)));
 620                 long tempVal = tempMappedByteBuffer.getLong();
 621                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)) + CommonFunc.tempLen);
 622                 long tempPreNum = tempMappedByteBuffer.getLong();
 623                 tempVal--;
 624                 tempMappedByteBuffer.position((int) ((personId - 1) * (CommonFunc.tempLen + CommonFunc.tempLen)));
 625                 tempMappedByteBuffer.putLong(tempVal);
 626                 //写入
 627                 idxMappedByteBuffer.position((int) ((tempPreNum + tempVal) * (CommonFunc.orderIdLen + CommonFunc.personIdLen)));
 628                 idxMappedByteBuffer.put(personIdByte);
 629                 idxMappedByteBuffer.position((int) ((tempPreNum + tempVal) * (CommonFunc.orderIdLen + CommonFunc.personIdLen)) + CommonFunc.personIdLen);
 630                 idxMappedByteBuffer.putDouble(amount);
 631 
 632                 if (i % 1000000 == 0) {
 633                     System.out.println("创建索引文件 " + i);
 634                 }
 635             }
 636         } catch (Exception ex) {
 637             System.out.println(ex.getMessage());
 638         } finally {
 639             if (null != raf) {
 640                 try {
 641                     raf.close();
 642                 } catch (IOException e) {
 643                     System.out.println(e.getMessage());
 644                 }
 645             }
 646             if (null != tempFileChannel) {
 647                 try {
 648                     tempFileChannel.close();
 649                 } catch (IOException e) {
 650                     System.out.println(e.getMessage());
 651                 }
 652             }
 653             if (null != idxFileChannel) {
 654                 try {
 655                     idxFileChannel.close();
 656                 } catch (IOException e) {
 657                     System.out.println(e.getMessage());
 658                 }
 659             }
 660         }
 661     }
 662 
 663     //以下为测试用方法
 664 
 665     //标准数据:一亿条
 666     public static void createData() {
 667 
 668         File file = new File(CommonFunc.orderPath);
 669         if (file.exists() == true) {
 670             file.delete();
 671         }
 672         try {
 673             file.createNewFile();
 674         } catch (IOException e) {
 675             System.out.println(e.getMessage());
 676         }
 677 
 678         long totalStartTime = System.currentTimeMillis();
 679         for (int j = 0; j < 100; j++) {
 680 
 681             long startTime = System.currentTimeMillis();
 682             Order[] orderList = new Order[1000000];
 683             for (int i = (j * 1000000); i < (j * 1000000 + 1000000); i++) {
 684 
 685                 Order order = new Order();
 686                 orderList[i - (j * 1000000)] = order;
 687 
 688                 order.setOrderId(i + 1);
 689                 order.setPersonId(CommonFunc.getRandomPersonId(10000000));
 690                 order.setAmount(CommonFunc.getRandomDouble(10000));
 691             }
 692 
 693             OrderOperate.batchInsert(orderList);
 694             long endTime = System.currentTimeMillis();
 695             System.out.println("程序第 " + j + " 次运行时间： " + (endTime - startTime) + "ms");
 696         }
 697         long totalEndTime = System.currentTimeMillis();
 698         System.out.println("程序运行总时间： " + (totalEndTime - totalStartTime) + "ms");
 699 
 700     }
 701 
 702     //测试数据：10条
 703     public static void createTestData() {
 704 
 705         File file = new File(CommonFunc.orderPath);
 706         if (file.exists() == true) {
 707             file.delete();
 708         }
 709         try {
 710             file.createNewFile();
 711         } catch (IOException e) {
 712             System.out.println(e.getMessage());
 713         }
 714 
 715         long totalStartTime = System.currentTimeMillis();
 716         for (int j = 0; j < 1; j++) {
 717 
 718             long startTime = System.currentTimeMillis();
 719             Order[] orderList = new Order[10];
 720             for (int i = (j * 10); i < (j * 10 + 10); i++) {
 721 
 722                 Order order = new Order();
 723                 orderList[i - (j * 10)] = order;
 724 
 725                 order.setOrderId(i + 1);
 726                 order.setPersonId(CommonFunc.getRandomPersonId(3));
 727                 order.setAmount(CommonFunc.getRandomDouble(10000));
 728             }
 729 
 730             OrderOperate.batchInsert(orderList);
 731             long endTime = System.currentTimeMillis();
 732             System.out.println("程序第 " + j + " 次运行时间： " + (endTime - startTime) + "ms");
 733         }
 734         long totalEndTime = System.currentTimeMillis();
 735         System.out.println("程序运行总时间： " + (totalEndTime - totalStartTime) + "ms");
 736 
 737     }
 738 
 739     public static void printOrderData() {
 740 
 741         List<Order> rtn = new ArrayList<>();
 742 
 743         File file = new File(CommonFunc.orderPath);
 744         long fileLen = file.length();
 745         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen);
 746 
 747         RandomAccessFile raf = null;
 748         try {
 749             raf = new RandomAccessFile(file, "r");
 750             for (long i = 0; i < rowNum; i++) {
 751                 Order order = new Order();
 752                 rtn.add(order);
 753 
 754                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen));
 755                 order.setOrderId(raf.readLong());
 756 
 757                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + CommonFunc.orderIdLen);
 758                 byte[] personIdByte = new byte[CommonFunc.personIdLen];
 759                 raf.read(personIdByte);
 760                 order.setPersonId(CommonFunc.getLong(personIdByte));
 761 
 762                 raf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen + CommonFunc.amountLen) + (CommonFunc.orderIdLen + CommonFunc.personIdLen));
 763                 byte[] amountLenByte = new byte[CommonFunc.amountLen];
 764                 raf.read(amountLenByte);
 765                 order.setAmount(CommonFunc.getDouble(amountLenByte));
 766 
 767                 System.out.println("ID:" + order.getOrderId() + " PersonId:" + order.getPersonId() + " Amount:" + order.getAmount());
 768             }
 769         } catch (Exception ex) {
 770             System.out.println(ex.getMessage());
 771         } finally {
 772             if (null != raf) {
 773                 try {
 774                     raf.close();
 775                 } catch (IOException e) {
 776                     System.out.println(e.getMessage());
 777                 }
 778             }
 779         }
 780     }
 781 
 782     public static void printTempData() {
 783 
 784         File tempFile = new File(CommonFunc.tempPath);
 785         long fileLen = tempFile.length();
 786         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
 787 
 788         RandomAccessFile tempRaf = null;
 789         try {
 790             tempRaf = new RandomAccessFile(tempFile, "rw");
 791             for (int i = 0; i < rowNum; i++) {
 792                 tempRaf.seek(i * (CommonFunc.tempLen + CommonFunc.tempLen));
 793                 byte[] longByte = new byte[8];
 794                 tempRaf.read(longByte);
 795                 long tempVal = CommonFunc.getLong(longByte);
 796 
 797                 if (tempVal <= 0) {
 798                     continue;
 799                 }
 800 
 801                 tempRaf.seek(i * (CommonFunc.tempLen + CommonFunc.tempLen) + CommonFunc.tempLen);
 802                 longByte = new byte[8];
 803                 tempRaf.read(longByte);
 804                 long tempPreNum = CommonFunc.getLong(longByte);
 805 
 806                 System.out.println("PersonId : " + (i + 1) + " ; Num : " + tempVal + " ; PreNum : " + tempPreNum);
 807             }
 808         } catch (Exception ex) {
 809             System.out.println(ex.getMessage());
 810         } finally {
 811             if (null != tempRaf) {
 812                 try {
 813                     tempRaf.close();
 814                 } catch (IOException e) {
 815                     System.out.println(e.getMessage());
 816                 }
 817             }
 818         }
 819     }
 820 
 821     public static void printIdxData() {
 822 
 823         File tempFile = new File(CommonFunc.orderIdxPath);
 824         long fileLen = tempFile.length();
 825         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen);
 826 
 827         RandomAccessFile tempRaf = null;
 828         try {
 829             tempRaf = new RandomAccessFile(tempFile, "rw");
 830             for (long i = 0; i < rowNum; i++) {
 831 
 832                 tempRaf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen));
 833                 byte[] longByte = new byte[CommonFunc.personIdLen];
 834                 tempRaf.read(longByte);
 835                 long personid = CommonFunc.getLong(longByte);
 836 
 837                 tempRaf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen) + CommonFunc.personIdLen);
 838                 longByte = new byte[CommonFunc.orderIdLen];
 839                 tempRaf.read(longByte);
 840                 long orderid = CommonFunc.getLong(longByte);
 841 
 842                 System.out.println("idx : " + i + " ; personid : " + personid + " ; orderid : " + orderid);
 843             }
 844         } catch (Exception ex) {
 845             System.out.println(ex.getMessage());
 846         } finally {
 847             if (null != tempRaf) {
 848                 try {
 849                     tempRaf.close();
 850                 } catch (IOException e) {
 851                     System.out.println(e.getMessage());
 852                 }
 853             }
 854         }
 855     }
 856 
 857     public static void printIdxAmountData() {
 858 
 859         File tempFile = new File(CommonFunc.orderIdxAmountPath);
 860         long fileLen = tempFile.length();
 861         long rowNum = fileLen / (CommonFunc.orderIdLen + CommonFunc.personIdLen);
 862 
 863         RandomAccessFile tempRaf = null;
 864         try {
 865             tempRaf = new RandomAccessFile(tempFile, "rw");
 866             for (long i = 0; i < rowNum; i++) {
 867 
 868                 tempRaf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen));
 869                 byte[] longByte = new byte[CommonFunc.personIdLen];
 870                 tempRaf.read(longByte);
 871                 long personid = CommonFunc.getLong(longByte);
 872 
 873                 tempRaf.seek(i * (CommonFunc.orderIdLen + CommonFunc.personIdLen) + CommonFunc.personIdLen);
 874                 double amount = tempRaf.readDouble();
 875 
 876                 System.out.println("idx : " + i + " ; personid : " + personid + " ; amount : " + amount);
 877             }
 878         } catch (Exception ex) {
 879             System.out.println(ex.getMessage());
 880         } finally {
 881             if (null != tempRaf) {
 882                 try {
 883                     tempRaf.close();
 884                 } catch (IOException e) {
 885                     System.out.println(e.getMessage());
 886                 }
 887             }
 888         }
 889     }
 890 
 891     public static void printTop1000SumData(int totalNum, boolean desc) {
 892 
 893         long startTime = System.currentTimeMillis();
 894 
 895         List<OrderQuery> rtn = getOrderQueryList(totalNum, desc);
 896 
 897         long endTime = System.currentTimeMillis();
 898 
 899         for (OrderQuery orderQuery : rtn) {
 900             System.out.println("姓名：" + orderQuery.getPersonName() + " 数量：" + orderQuery.getOrderCount() + " 总价：" + orderQuery.getOrderSum());
 901         }
 902         System.out.println("共查出 " + rtn.size() + " 条");
 903         System.out.println("程序运行时间： " + (endTime - startTime) + "ms");
 904     }
 905 
 906     private static List<OrderQuery> getOrderQueryList(int totalNum, boolean desc) {
 907 
 908         List<OrderQuery> rtn = new ArrayList<>();
 909 
 910         File idxFile = new File(CommonFunc.orderIdxAmountPath);
 911         long fileLen = idxFile.length();
 912         long rowNum = fileLen / (CommonFunc.tempLen + CommonFunc.tempLen);
 913         File psnFile = new File(CommonFunc.psnPath);
 914 
 915         FileChannel idxFileChannel = null;
 916         FileChannel psnFileChannel = null;
 917         try {
 918             idxFileChannel = new RandomAccessFile(idxFile, "r").getChannel();
 919             MappedByteBuffer idxMappedByteBuffer = idxFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, idxFile.length());
 920 
 921             psnFileChannel = new RandomAccessFile(psnFile, "r").getChannel();
 922             MappedByteBuffer psnMappedByteBuffer = psnFileChannel.map(FileChannel.MapMode.READ_ONLY, 0, psnFile.length());
 923 
 924             OrderQuery orderQuery = null;
 925             long prePersonId = -1;
 926             for (int i = 0; i < rowNum; i++) {
 927                 long personId = idxMappedByteBuffer.getLong();
 928                 double amount = idxMappedByteBuffer.getDouble();
 929 
 930                 if (prePersonId == personId) {
 931                     orderQuery.setOrderCount(orderQuery.getOrderCount() + 1);
 932                     orderQuery.setOrderSum(orderQuery.getOrderSum() + amount);
 933                 } else {
 934                     //处理完的数据，添加入列表
 935                     if (orderQuery != null) {
 936                         insertOrderQueryList(rtn, orderQuery, totalNum, desc);
 937                     }
 938 
 939                     //获取Person姓名
 940                     int psnNamePos = (int) (personId - 1) * (CommonFunc.psnIdLen + CommonFunc.nameLen) + CommonFunc.psnIdLen;
 941                     psnMappedByteBuffer.position(psnNamePos);
 942                     byte[] psnNameByte = new byte[CommonFunc.nameLen];
 943                     psnMappedByteBuffer.get(psnNameByte);
 944                     String personName = CommonFunc.getString(psnNameByte);
 945                     //设置对象
 946                     orderQuery = new OrderQuery();
 947                     orderQuery.setPersonName(personName);
 948                     orderQuery.setOrderCount(1);
 949                     orderQuery.setOrderSum(amount);
 950 
 951                     prePersonId = personId;
 952                 }
 953 
 954                 if (i % 1000000 == 0) {
 955                     System.out.println("处理条数 " + i);
 956                 }
 957             }
 958 
 959             //添加最后一条数据
 960             if (orderQuery != null) {
 961                 insertOrderQueryList(rtn, orderQuery, totalNum, desc);
 962             }
 963 
 964         } catch (Exception ex) {
 965             System.out.println(ex.getMessage());
 966         } finally {
 967             if (null != idxFileChannel) {
 968                 try {
 969                     idxFileChannel.close();
 970                 } catch (IOException e) {
 971                     System.out.println(e.getMessage());
 972                 }
 973             }
 974             if (null != psnFileChannel) {
 975                 try {
 976                     psnFileChannel.close();
 977                 } catch (IOException e) {
 978                     System.out.println(e.getMessage());
 979                 }
 980             }
 981         }
 982 
 983         return rtn;
 984     }
 985 
 986     private static void insertOrderQueryList(List<OrderQuery> list, OrderQuery orderQuery, int listMaxNum, boolean desc) {
 987 
 988         int insertIdx = desc ? getInsertDescIndex(list, orderQuery) : getInsertIndex(list, orderQuery);
 989         if (insertIdx < listMaxNum || listMaxNum <= 0) {
 990             list.add(insertIdx, orderQuery);
 991         }
 992 
 993         if (list.size() > listMaxNum && listMaxNum > 0) {
 994             for (int i = list.size() - 1; i >= listMaxNum; i--) {
 995                 list.remove(i);
 996 
 997             }
 998         }
 999     }
1000 
1001     private static int getInsertIndex(List<OrderQuery> list, OrderQuery orderQuery) {
1002         int rtn = -1;
1003 
1004         //处理边缘点
1005         if (list.size() <= 0 || list.get(0).getOrderSum() > orderQuery.getOrderSum()) {
1006             return 0;
1007         }
1008         if (list.get(list.size() - 1).getOrderSum() <= orderQuery.getOrderSum()) {
1009             return list.size();
1010         }
1011 
1012         //常规处理
1013         int minIdx = 0;
1014         int maxIdx = list.size() - 1;
1015         while (maxIdx > minIdx) {
1016             //中间数据处理
1017             if (minIdx + 1 == maxIdx) {
1018                 rtn = maxIdx;
1019                 break;
1020             }
1021 
1022             int midIdx = (minIdx + maxIdx) / 2;
1023             double currentNum = list.get(midIdx).getOrderSum();
1024             if (currentNum > orderQuery.getOrderSum()) {
1025                 maxIdx = midIdx;
1026             } else if (currentNum < orderQuery.getOrderSum()) {
1027                 minIdx = midIdx;
1028             } else {
1029                 rtn = midIdx + 1;
1030                 break;
1031             }
1032         }
1033 
1034         return rtn;
1035     }
1036 
1037     private static int getInsertDescIndex(List<OrderQuery> list, OrderQuery orderQuery) {
1038         int rtn = -1;
1039 
1040         //处理边缘点
1041         if (list.size() <= 0 || list.get(0).getOrderSum() < orderQuery.getOrderSum()) {
1042             return 0;
1043         }
1044         if (list.get(list.size() - 1).getOrderSum() >= orderQuery.getOrderSum()) {
1045             return list.size();
1046         }
1047 
1048         //常规处理
1049         int minIdx = 0;
1050         int maxIdx = list.size() - 1;
1051         while (maxIdx > minIdx) {
1052             //中间数据处理
1053             if (minIdx + 1 == maxIdx) {
1054                 rtn = maxIdx;
1055                 break;
1056             }
1057 
1058             int midIdx = (minIdx + maxIdx) / 2;
1059             double currentNum = list.get(midIdx).getOrderSum();
1060             if (currentNum > orderQuery.getOrderSum()) {
1061                 minIdx = midIdx;
1062             } else if (currentNum < orderQuery.getOrderSum()) {
1063                 maxIdx = midIdx;
1064             } else {
1065                 rtn = midIdx + 1;
1066                 break;
1067             }
1068         }
1069 
1070         return rtn;
1071     }
1072 
1073 }
View Code

CommonFunc.java

  1 import java.nio.charset.Charset;
  2 import java.util.Random;
  3 
  4 public class CommonFunc {
  5 
  6     //private static String dbPath = "\\\\192.168.1.163\\Theme\\zuoye\\order.nk";
  7 
  8 //    public static String psnPath = "D:\\test\\person.nk";
  9 //    public static String psnIdxPath = "D:\\test\\person.nk.name";
 10 //    public static String orderPath = "D:\\test\\order.nk";
 11 //    public static String orderIdxPath = "D:\\test\\order.nk.personid";
 12 //    public static String orderIdxAmountPath = "D:\\test\\order.nk.amount";
 13 //    public static String tempPath = "D:\\test\\temp.nk";
 14 
 15     public static String psnPath = "\\\\192.168.1.163\\Theme\\zuoye\\person.nk";
 16     public static String psnIdxPath = "\\\\192.168.1.163\\Theme\\zuoye\\person.nk.name";
 17     public static String orderPath = "\\\\192.168.1.163\\Theme\\zuoye\\order.nk";
 18     public static String orderIdxPath = "\\\\192.168.1.163\\Theme\\zuoye\\order.nk.personid";
 19     public static String orderIdxAmountPath = "\\\\192.168.1.163\\Theme\\zuoye\\order.nk.amount";
 20     public static String tempPath = "\\\\192.168.1.163\\Theme\\zuoye\\temp.nk";
 21 
 22     public static int psnIdLen = 8;
 23     public static int nameLen = 12;
 24     public static int orderIdLen = 8;
 25     public static int personIdLen = 8;
 26     public static int amountLen = 8;
 27     public static int tempLen = 8;
 28 
 29     private static String firstName = "赵钱孙李周吴郑王冯陈褚卫蒋沈韩杨朱秦尤许何吕施张孔曹严华金魏陶姜戚谢邹喻柏水窦章云苏潘葛奚范彭郎鲁韦昌马苗凤花方俞任袁柳酆鲍史唐费廉岑薛雷贺倪汤滕殷罗毕郝邬安常乐于时傅皮卞齐康伍余元卜顾孟平黄和穆萧尹姚邵湛汪祁毛禹狄米贝明臧计伏成戴谈宋茅庞熊纪舒屈项祝董梁杜阮蓝闵席季麻强贾路娄危江童颜郭梅盛林***钟徐邱骆高夏蔡田樊胡凌霍虞万支柯咎管卢莫经房裘缪干解应宗宣丁贲邓郁单杭洪包诸左石崔吉钮龚程嵇邢滑裴陆荣翁荀羊於惠甄魏加封芮羿储靳汲邴糜松井段富巫乌焦巴弓牧隗山谷车侯宓蓬全郗班仰秋仲伊宫宁仇栾暴甘钭厉戎祖武符刘姜詹束龙叶幸司韶郜黎蓟薄印宿白怀蒲台从鄂索咸籍赖卓蔺屠蒙池乔阴郁胥能苍双闻莘党翟谭贡劳逄姬申扶堵冉宰郦雍却璩桑桂濮牛寿通边扈燕冀郏浦尚农温别庄晏柴瞿阎充慕连茹习宦艾鱼容向古易慎戈廖庚终暨居衡步都耿满弘匡国文寇广禄阙东殴殳沃利蔚越夔隆师巩厍聂晁勾敖融冷訾辛阚那简饶空曾毋沙乜养鞠须丰巢关蒯相查后江红游竺权逯盖益桓公万俟司马上官欧阳夏侯诸葛闻人东方赫连皇甫尉迟公羊澹台公冶宗政濮阳淳于仲孙太叔申屠公孙乐正轩辕令狐钟离闾丘长孙慕容鲜于宇文司徒司空亓官司寇仉督子车颛孙端木巫马公西漆雕乐正壤驷公良拓拔夹谷宰父谷粱晋楚阎法汝鄢涂钦段干百里东郭南门呼延归海羊舌微生岳帅缑亢况后有琴梁丘左丘东门西门商牟佘佴伯赏南宫墨哈谯笪年爱阳佟第五言福百家姓续";
 30     private static String girl = "秀娟英华慧巧美娜静淑惠珠翠雅芝玉萍红娥玲芬芳燕彩春菊兰凤洁梅琳素云莲真环雪荣爱妹霞香月莺媛艳瑞凡佳嘉琼勤珍贞莉桂娣叶璧璐娅琦晶妍茜秋珊莎锦黛青倩婷姣婉娴瑾颖露瑶怡婵雁蓓纨仪荷丹蓉眉君琴蕊薇菁梦岚苑婕馨瑗琰韵融园艺咏卿聪澜纯毓悦昭冰爽琬茗羽希宁欣飘育滢馥筠柔竹霭凝晓欢霄枫芸菲寒伊亚宜可姬舒影荔枝思丽 ";
 31     private static String boy = "伟刚勇毅俊峰强军平保东文辉力明永健世广志义兴良海山仁波宁贵福生龙元全国胜学祥才发武新利清飞彬富顺信子杰涛昌成康星光天达安岩中茂进林有坚和彪博诚先敬震振壮会思群豪心邦承乐绍功松善厚庆磊民友裕河哲江超浩亮政谦亨奇固之轮翰朗伯宏言若鸣朋斌梁栋维启克伦翔旭鹏泽晨辰士以建家致树炎德行时泰盛雄琛钧冠策腾楠榕风航弘";
 32 
 33     /**
 34      * 随机获取中文名字
 35      *
 36      * @return
 37      */
 38     public static String getPersonName() {
 39 
 40         String nameStr = getRandomNum(2) == 1 ? boy : girl;
 41 
 42         int firstRandom = getRandomNum(firstName.length() - 1);
 43         String first = firstName.substring(firstRandom, firstRandom + 1);
 44         int secondRandom = getRandomNum(nameStr.length() - 1);
 45         String second = nameStr.substring(secondRandom, secondRandom + 1);
 46         int thridRandom = getRandomNum(nameStr.length() - 1);
 47         String third = nameStr.substring(thridRandom, thridRandom + 1);
 48 
 49         return first.concat(second).concat(third);
 50     }
 51 
 52     /**
 53      * 随机获取PersonID
 54      *
 55      * @return
 56      */
 57     public static long getRandomPersonId(int max) {
 58         return getRandomNum(max) + 1;
 59     }
 60 
 61     /**
 62      * 获取随机数，不包含传入的最大数
 63      *
 64      * @param maxNum
 65      * @return
 66      */
 67     public static int getRandomNum(int maxNum) {
 68         return (int) (Math.random() * maxNum);
 69     }
 70 
 71     /**
 72      * 获取Double类型的随机数
 73      *
 74      * @param maxNum
 75      * @return
 76      */
 77     public static double getRandomDouble(int maxNum) {
 78         Random randomno = new Random();
 79         return randomno.nextDouble() * maxNum;
 80     }
 81 
 82     public static byte[] getBytes(long data) {
 83         byte[] bytes = new byte[8];
 84         bytes[7] = (byte) (data & 0xff);
 85         bytes[6] = (byte) ((data >> 8) & 0xff);
 86         bytes[5] = (byte) ((data >> 16) & 0xff);
 87         bytes[4] = (byte) ((data >> 24) & 0xff);
 88         bytes[3] = (byte) ((data >> 32) & 0xff);
 89         bytes[2] = (byte) ((data >> 40) & 0xff);
 90         bytes[1] = (byte) ((data >> 48) & 0xff);
 91         bytes[0] = (byte) ((data >> 56) & 0xff);
 92         return bytes;
 93     }
 94 
 95     public static byte[] getBytes(int data) {
 96         byte[] bytes = new byte[4];
 97         bytes[3] = (byte) (data & 0xff);
 98         bytes[2] = (byte) ((data & 0xff00) >> 8);
 99         bytes[1] = (byte) ((data & 0xff0000) >> 16);
100         bytes[0] = (byte) ((data & 0xff000000) >> 24);
101         return bytes;
102     }
103 
104     public static byte[] getBytes(double data) {
105         long intBits = Double.doubleToLongBits(data);
106         return getBytes(intBits);
107     }
108 
109     public static byte[] getBytes(String data, int len) {
110         return getBytes(data, len, "UTF-8");
111     }
112 
113     public static byte[] getBytes(String data, int len, String charsetName) {
114         byte[] rtn = new byte[len];
115 
116         Charset charset = Charset.forName(charsetName);
117         byte[] strByte = data.getBytes(charset);
118         //取小的数据
119         int copyLen = strByte.length < len ? strByte.length : len;
120         System.arraycopy(strByte, 0, rtn, 0, copyLen);
121 
122         return rtn;
123     }
124 
125     public static long getLong(byte[] bytes) {
126         return (0xffL & (long) bytes[7]) |
127                 (0xff00L & ((long) bytes[6] << 8)) |
128                 (0xff0000L & ((long) bytes[5] << 16)) |
129                 (0xff000000L & ((long) bytes[4] << 24)) |
130                 (0xff00000000L & ((long) bytes[3] << 32)) |
131                 (0xff0000000000L & ((long) bytes[2] << 40)) |
132                 (0xff000000000000L & ((long) bytes[1] << 48)) |
133                 (0xff00000000000000L & ((long) bytes[0] << 56));
134     }
135 
136     public static int getInt(byte[] bytes) {
137         return (0xff & bytes[3]) |
138                 (0xff00 & (bytes[2] << 8)) |
139                 (0xff0000 & (bytes[1] << 16)) |
140                 (0xff000000 & (bytes[0] << 24));
141     }
142 
143     public static double getDouble(byte[] bytes) {
144         long l = getLong(bytes);
145         return Double.longBitsToDouble(l);
146     }
147 
148     public static String getString(byte[] bytes) {
149         return getString(bytes, "UTF-8");
150     }
151 
152     public static String getString(byte[] bytes, String charsetName) {
153         Charset charset = Charset.forName(charsetName);
154         return new String(bytes, charset).trim();
155     }
156 }
View Code

OrderQuery.java

 1 public class OrderQuery {
 2 
 3     private String personName;
 4     private int OrderCount;
 5     private double OrderSum;
 6 
 7     public String getPersonName() {
 8         return personName;
 9     }
10 
11     public void setPersonName(String personName) {
12         this.personName = personName;
13     }
14 
15     public int getOrderCount() {
16         return OrderCount;
17     }
18 
19     public void setOrderCount(int orderCount) {
20         OrderCount = orderCount;
21     }
22 
23     public double getOrderSum() {
24         return OrderSum;
25     }
26 
27     public void setOrderSum(double orderSum) {
28         OrderSum = orderSum;
29     }
30 }
View Code

Person.java

 1 public class Person {
 2 
 3     private long personId;
 4     private String name;
 5 
 6     public long getPersonId() {
 7         return personId;
 8     }
 9 
10     public void setPersonId(long personId) {
11         this.personId = personId;
12     }
13 
14     public String getName() {
15         return name;
16     }
17 
18     public void setName(String name) {
19         this.name = name;
20     }
21 }
View Code

Order.java

 1 public class Order {
 2 
 3     private long orderId;
 4     private long personId;
 5     private double amount;
 6 
 7     public long getOrderId() {
 8         return orderId;
 9     }
10 
11     public void setOrderId(long orderId) {
12         this.orderId = orderId;
13     }
14 
15     public long getPersonId() {
16         return personId;
17     }
18 
19     public void setPersonId(long personId) {
20         this.personId = personId;
21     }
22 
23     public double getAmount() {
24         return amount;
25     }
26 
27     public void setAmount(double amount) {
28         this.amount = amount;
29     }
30 }
View Code

 

posted on 2015-10-09 22:48  daixinet.com  阅读(955)  评论(0)  编辑  收藏  举报