《编程之美》读书笔记07: 2.5 寻找最大的K个数
问题:
从n个数中找出最大的k个数。
两种思路:
1 保存目前找到的最大k个数,每访问一个数,就与这k个数中的最小值比较,决定是否更新这k个数。储存k个数的数据结构可采用:败者树、二叉查找树、最小堆。
C++ STL提供了multiset和priority_queue容器,另外还提供了make_heap,push_heap,pop_heap方便手动构建堆结构。(测试发现,手工建堆的效率最高,当n和k增大到一定值时,采用红黑树的multiset的效率极差。手动建堆的效率相比priority_queue有略微提高。)
2 修改排序方法,去除不必要的过程。
选择排序: 只要选k次。
冒泡排序: 只要冒泡k次即可。
堆排序: 构建好最大堆后,取 k次最大值
快速排序: 分区时,根据数P将数组分为两部分,设大于P的数个数为a,小于P的数的个数为b。如果,a>=k,则从这a个数取最大的k个数,若a<k,则从b个数取最大的k-a-1个。
归并排序: 当待合并的两个数组,两数组长度和大等于k时,合并时只取前k个。或者:以(k+1)/2个数为一组,将数组分成几个组,对每组进行排序(可以采用任何一种高效的排序方法)后,两两合并时只取前k个。
计数排序: 如果都是整数,先扫描一遍找出最大值max,最小值min,再扫一遍,将每个值减去min,对这个值计数,最后从max-min开始统计,找出最大的k个数。另外,也可采用桶排序。
桶排序: 可以不对桶内的数据进行排序。
基数排序: 可以采用最高关键字比较方法,并免去相关的排序。
STL中的nth_element就是基于对intorsort的修改(introtsort是对快速排序的改进,当递归深度达到一定值时,可切换到堆排序),而partial_sort和partial_sort_copy是基于堆排序的修改,因而在k很小时,其效率可能会高于nth_element。遗憾的是:STL没有提供完全基于堆排序的nth_element。
从下面的测试结果,可以看出:在M不是很大,M/N很小时,partial_sort和partial_sort_copy尽管多了“对堆结构进行排序”这个不必要的操作,其效率仍然高于nth_element,但相差不多。而在其它情况下nth_element的效率则比其它的几种方法要高很多。
如果源数据都是整数,多数情况下(即使允许修改源数据),桶排序方法(结合计数方法)的效率比nth_element高。桶排序只需256K的内存,效率很高。在M和N至少有一个大于当前内存大小的情况下,桶排序是最佳选择,其性能远高于其它方法。
如果源数据是浮点数,根据浮点数在内存中的表示,可以对桶排序方法进行适当修改,使之对浮点数也适用。
测试程序相关说明:
① 测试程序要求不得改变源数据,某些方法要多一个复制源数据操作,可以从partial_sort_copy和partial_sort效率的差异,看出这个复制操作的影响。
② 桶排序方法对应nth_count;
③ 对堆结构的调整,采用三种途径(分别对应三个程序):利用push_heap和pop_heap、只用pop_heap、手写代码调整。(
④ multiset和heapsort方法,在相同N和M情况下,所用时间起伏很大,即所用时间对原始数据依赖性很高。
![](//www.cppblog.com/Images/OutliningIndicators/ContractedBlock.gif)
程序代码
//2.5_nth.cpp by flyinghearts#qq.com
2
#include <iostream>
3
#include <vector>
4
#include <queue>
5
#include <set>
6
#include <utility>
7
#include <algorithm>
8
#include <ctime>
9
#include <cassert>
10
#include <cstdlib>
11
using namespace std;
12![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
13
const int check_result=0; //是否检查结果的正确性
14![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
15
template<typename T> void nth_mset(const T*, size_t, T*, size_t);
16
template<typename T> void nth_queue(const T*, size_t, T*, size_t);
17
template<typename T> void nth_stl(const T*, size_t, T*, size_t);
18
template<typename T> void nth_partial(const T*, size_t, T*, size_t);
19
template<typename T> void nth_partial_copy(const T*, size_t, T*, size_t);
20
template<typename T> void nth_heap_1(const T*, size_t, T*, size_t);
21
template<typename T> void nth_heap_2(const T*, size_t, T*, size_t);
22
template<typename T> void nth_heap_3(const T*, size_t, T*, size_t);
23
template<typename T> void nth_heapsort(const T*, size_t, T*, size_t);
24![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
25
//使用桶排序法,仅适用于32位整数且 整数的最大重复次数小于2^32
26
void nth_count(const int *src, size_t src_size, int *dst, size_t dst_size);
27![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
28
//从N个数(取值范围(-2^BITS,2^BITS) )取M个最大的数
29
template<typename T, typename R, size_t size>
30
void test(R (&ff)[size], size_t count=1, size_t N=1e8,
31
size_t M=1e4, size_t BITS=31);
32![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
33![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
class Rand_num {
34
int mask;
35
public:
36![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
Rand_num(int i=31){ if(i>31 || i<1) i=31; mask=((1<<i)-1)&(1<<31);}
37![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
int operator()(){ return (((rand()&3)<<30)+(rand()<<15)+rand())&mask;}
38
};
39![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
40![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
struct Func {
41
const char *name;
42
void (*func)(const int *, size_t, int *, size_t);
43
};
44![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
45
int main()
46![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
47![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
Func ff[]={
48![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"nth_elmemnt",nth_stl},
49![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"nth_count",nth_count},
50![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"priority_queue",nth_queue},
51![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"partial_sort",nth_partial},
52![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"partial_sort_copy",nth_partial_copy},
53![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"heap(pop/push) ",nth_heap_1},
54![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"heap(pop/copy)",nth_heap_2},
55![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"heap(custom_pop)",nth_heap_3},
56![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"multiset",nth_mset},
57![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
{"heap_sort",nth_heapsort},
58
};
59
test<int>(ff,1,1e6,8e5);
60
test<int>(ff,1,1e8,1e4,10);
61
test<int>(ff,1,1e8,1e4);
62
test<int>(ff,1,1e8,1e5,10);
63
test<int>(ff,1,1e8,1e5);
64
test<int>(ff,1,1e8,1e6,10);
65
test<int>(ff,1,1e8,1e6);
66
test<int>(ff,1,1e8,5e6,10);
67
test<int>(ff,1,1e8,5e6);
68
}
69![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
70
template<typename T>
71
void nth_heapsort(const T *src, size_t src_size, T *dst, size_t dst_size)
72![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
73
assert( dst_size >0 && dst_size <= src_size);
74
vector<T> cache(src,src+src_size);
75
T *first=&cache[0];
76
T *last=first+src_size;
77
T *pp=last;
78
make_heap(first, last);
79
for (size_t i=0; i<dst_size; ++i) pop_heap(first, pp--);
80
copy(last-dst_size,last, dst);
81
}
82![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
83![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
84
template<typename T>
85
void nth_mset(const T *src, size_t src_size, T *dst, size_t dst_size)
86![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
87
assert( dst_size >0 && dst_size <= src_size);
88
assert(src);
89
assert(dst);
90
multiset<T> tmp(src, src+dst_size);
91
const T *p=src + dst_size -1;
92
const T *end=src + src_size;
93
T min=*(tmp.begin());
94![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
while ( ++p < end){
95![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if (*p > min){
96
tmp.insert(*p);
97
tmp.erase(tmp.begin());
98
min=*(tmp.begin());
99
}
100
}
101
copy(tmp.begin(),tmp.end(),dst);
102
}
103![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
104
template<typename T>
105
void nth_queue(const T *src, size_t src_size, T *dst, size_t dst_size)
106![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
107
assert( dst_size >0 && dst_size <= src_size);
108
assert(src);
109
assert(dst);
110
typedef greater<T> CMP;
111
CMP cmp;
112
priority_queue<T, vector<T>, CMP > tmp(src, src+dst_size);
113
const T *p=src + dst_size -1;
114
const T *end=src + src_size;
115
T toppest=tmp.top();
116![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
while ( ++p < end){
117![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if ( cmp(*p,toppest) ){
118
tmp.pop();
119
tmp.push(*p);
120
toppest=tmp.top();
121
}
122
}
123
copy( &(tmp.top()), &(tmp.top()) + dst_size,dst);
124
}
125![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
126![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
127
template<typename T>
128
void nth_stl(const T *src, size_t src_size, T *dst, size_t dst_size)
129![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
130
assert( dst_size >0 && dst_size <= src_size);
131
assert(src);
132
assert(dst);
133
vector<T> tmp(src, src+src_size);
134
greater<T> cmp;
135
nth_element(tmp.begin(),tmp.begin()+dst_size, tmp.end(), cmp);
136
copy(tmp.begin(),tmp.begin()+dst_size,dst);
137
}
138![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
139
template<typename T>
140
void nth_partial(const T *src, size_t src_size, T *dst, size_t dst_size)
141![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
142
assert( dst_size >0 && dst_size <= src_size);
143
assert(src);
144
assert(dst);
145
vector<T> tmp(src, src+src_size);
146
greater<T> cmp;
147
partial_sort(tmp.begin(),tmp.begin()+dst_size, tmp.end(), cmp);
148
copy(tmp.begin(),tmp.begin()+dst_size,dst);
149
}
150![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
151
template<typename T>
152
void nth_partial_copy(const T *src, size_t src_size, T *dst, size_t dst_size)
153![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
154
assert( dst_size >0 && dst_size <= src_size);
155
assert(src);
156
assert(dst);
157
greater<T> cmp;
158
partial_sort_copy(src,src+src_size,dst,dst+dst_size,cmp);
159
}
160![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
161![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
162
template<typename T>
163
void nth_heap_1(const T *src, size_t src_size, T *dst, size_t dst_size)
164![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
165
assert( dst_size >0 && dst_size <= src_size);
166
greater<T> cmp;
167
copy(src, src+dst_size, dst);
168
const T *p=src + dst_size -1;
169
const T *end=src + src_size;
170
T * const first=dst;
171
T * const last=dst+dst_size;
172
make_heap( first, last, cmp);
173
T toppest=*first;
174![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
while ( ++p < end){
175![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if ( cmp(*p, toppest) ){
176
pop_heap( first, last, cmp);
177
*(last-1)=*p;
178
push_heap( first,last, cmp);
179
toppest=*first;
180
}
181
}
182
}
183![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
184![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
185
template<typename T>
186
void nth_heap_2(const T *src, size_t src_size, T *dst, size_t dst_size)
187![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
188
greater<T> cmp;
189
assert( dst_size >0 && dst_size <= src_size);
190
vector<int> tmp(src, src+dst_size+1);
191
const T *p=src + dst_size -1;
192
const T *end=src + src_size;
193
T * const first=&tmp[0];
194
T * const last=first+dst_size;
195
make_heap( first, last, cmp);
196
T toppest=*first;
197![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
while ( ++p < end){
198![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if ( cmp(*p, toppest) ){
199
*last=*p;
200
pop_heap( first, last+1, cmp);
201
toppest=*first;
202
}
203
}
204
copy(first, last,dst);
205
}
206![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
207![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
208
template<typename T>
209
void nth_heap_3(const T *src, size_t src_size, T *dst, size_t dst_size)
210![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
211
assert( dst_size >0 && dst_size <= src_size);
212
copy(src, src+dst_size, dst);
213
const T *p=src + dst_size -1;
214
const T *end=src + src_size;
215
size_t i,left;
216
make_heap( &dst[0],&dst[0]+dst_size, greater<T>() );
217
T min=dst[0];
218![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
while ( ++p < end){
219![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if (*p > min){
220
dst[0]=*p;
221
i=0;
222![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
while( (left=(i<<1)+1)<dst_size-1){
223
if (dst[left] > dst[left+1]) ++left;
224
if (dst[left] >= *p) break;
225
dst[i]=dst[left];
226
i=left;
227
}
228![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if (left==dst_size-1 && dst[left]<*p){
229
dst[i]=dst[left];
230
i=left;
231
}
232
dst[i]=*p;
233
min=dst[0];
234
}
235
}
236
}
237![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
238![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
239
void nth_count(const int *src, size_t src_size, int *dst, size_t dst_size)
240![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
241
assert( dst_size >0 && dst_size <= src_size);
242
assert(sizeof(int)==4);
243
const unsigned TOTAL=0x10000; //桶总数
244
const unsigned MIDDLE= TOTAL >> 1; //0到TOTAL的中间位置
245![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
unsigned count[TOTAL]={0};
246
unsigned * const mid=count+MIDDLE;
247
size_t sum=0;
248
const int *p=src, *end=src+src_size;
249
for (; p<end; ++p) ++mid[ *p>>16];
250
unsigned pos=TOTAL;
251
while (sum < dst_size) sum += count[--pos];
252
size_t new_dst_size = count[pos]+ dst_size - sum;
253
int *q=dst;
254
int high16=pos-MIDDLE;
255![](//www.cppblog.com/Images/OutliningIndicators/InBlock.gif)
256![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if (new_dst_size == 0) {
257
for (p=src,q=dst; p<end; ++p)
258
if ( (*p>>16)>= high16) *q++=*p;
259
return ;
260
}
261![](//www.cppblog.com/Images/OutliningIndicators/InBlock.gif)
262
fill(count, count+TOTAL, 0);
263![](//www.cppblog.com/Images/OutliningIndicators/InBlock.gif)
264![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
for (p=src,q=dst; p<end; ++p){
265
if ((*p>>16)>high16) *q++=*p;
266
else if ((*p>>16)==high16) ++count[*p&0xFFFF];
267
}
268![](//www.cppblog.com/Images/OutliningIndicators/InBlock.gif)
269
int low16=TOTAL, high16_v=high16<<16, value=0, number=0;
270
sum=0;
271![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
while (1){
272
number=count[--low16];
273![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if (number) {
274
sum += number;
275
value=high16_v+ low16;
276![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if (sum<new_dst_size){
277
for (; number>0; --number) *q++=value;
278![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
}else{
279
for (number -= sum - new_dst_size; number>0; --number) *q++=value;
280
return;
281
}
282
}
283
}
284
}
285![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
286![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
287
template<typename T, typename R, size_t size>
288
void test(R (&ff)[size], unsigned count, unsigned N, unsigned M, unsigned BITS)
289![](//www.cppblog.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
{
290
assert( N>=M && M>0);
291
assert( size>0);
292
assert(ff);
293
size_t j=0, i=0;
294
vector<T> arr(N),result(M),tmp(M);
295
greater<T> cmp;
296
cout<<"\nN,M: "<<N<<" "<<M<<" "<< 1.0*M/N<< endl;
297
clock_t tb;
298
srand(time(0));
299
Rand_num rand_num(BITS);
300![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
for( j=0; j <count; ++j ){
301
tb=clock();
302
generate(arr.begin(),arr.end(), rand_num);
303
tb=clock()-tb;
304
cout<<"Randomizing: "<<tb<<" ms\n";
305![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if(check_result){
306
tb=clock();
307
partial_sort_copy(arr.begin(), arr.end(), tmp.begin(), tmp.end(), cmp);
308
tb=clock()-tb;
309
cout<<"partial_sort_copy: "<<tb<<" ms\n";
310
}
311![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
for(i=0; i<size; ++i){
312
tb=clock();
313
ff[i].func(&arr[0],N,&result[0],M);
314
tb=clock()-tb;
315
cout<< ff[i].name<<" "<< tb<<" ms\n";
316![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if(check_result){
317
sort(result.begin(),result.end(), cmp);
318![](//www.cppblog.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
if( !equal(result.begin(), result.end(), tmp.begin())) {
319
cout<< ff[i].name<< " failed to pass!\nresult:\n";
320
}
321
}
322
}
323
cout<<"\n";
324
}
325
}
326![](//www.cppblog.com/Images/OutliningIndicators/None.gif)
327![](//www.cppblog.com/Images/OutliningIndicators/None.gif)