Redis数据结构之HperLogLog

一、HyperLogLog

HyperLogLog是用来做基数统计的。

其可以非常省内存的去统计各种计数，比如注册ip数、每日访问IP数、页面实时UV（PV肯定字符串就搞定了）、在线用户数等在对准确性不是很重要的应用场景。

 

HyperLogLog的优点是：

在输入元素的数量或者体积非常非常大时，计算基数所需的空间总是固定的、并且是很小的，

HyperLogLog的缺点:

它是估计基数的算法，所以会有一定误差0.81%。

每个HyperLogLog键只需要花费12KB内存，就可以计算接近264个不同元素的基数。这和计算基数时，元素越多耗费内存就越多的集合形成鲜明对比。 

但是，因为 HyperLogLog 只会根据输入元素来计算基数，而不会储存输入元素本身，所以 HyperLogLog 不能像集合那样，返回输入的各个元素即无法知道统计的详细内容。

 

二、基数和估算值

1、基数

基数是集合中不同元素的数量。

比如数据集 {1, 3, 5, 7, 5, 7, 8}， 那么这个数据集的基数集为 {1, 3, 5 ,7, 8}, 基数(不重复元素)为5。 

基数估计就是在误差可接受的范围内，快速计算基数。

 

2、估算值

算法给出的基数并不是精确的，可能会比实际稍微多一些或者稍微少一些，但会控制在合理的范围之内。

 

三、HperLogLog基本命令

redis HyperLogLog 的基本命令：

1 PFADD key element [element ...] 

添加指定元素到 HyperLogLog 中。

2 PFCOUNT key [key ...] 

返回给定 HyperLogLog 的基数估算值。

3 PFMERGE destkey sourcekey [sourcekey ...] 

将多个 HyperLogLog 合并为一个 HyperLogLog

 

PFADD

将任意数量的元素添加到指定的 HyperLogLog 里面。在执行这个命令之后，HyperLogLog内部的结构会被更新，并有所反馈，

如果执行完之后HyperLogLog内部的基数估算发生了变化，那么就会返回1，否则（认为已经存在）就返回0。

这个命令还有一个比较神器的就是可以只有键，没有值，这样的意思就是只是创建空的键，不放值。

如果这个键存在，不做任何事情，返回0；不存在的话就创建，并返回1。

这个命令的时间复杂度为O(1)，所以就放心用吧~

 

PFCOUNT

当命令作用于单个键的时候，返回这个键的基数估算值。如果键不存在，则返回0。

当 PFCOUNT 命令作用于多个键时， 返回所有给定 HyperLogLog 的并集的近似基数， 这个近似基数是通过将所有给定 HyperLogLog 合并至一个临时 HyperLogLog 来计算得出的。

这个命令在作用于单个值的时候，时间复杂度为O(1)，并且具有非常低的平均常数时间；在作用于N个值的时候，时间复杂度为O(N)，这个命令的常数复杂度会比较低些。

命令返回的可见集合（observed set）基数并不是精确值， 而是一个带有 0.81% 标准错误（standard error）的近似值。

举个例子， 为了记录一天会执行多少次各不相同的搜索查询， 一个程序可以在每次执行搜索查询时调用一次 PFADD ， 并通过调用 PFCOUNT 命令来获取这个记录的近似结果。

 

PFMERGE

合并（merge）多个HyperLogLog为一个HyperLogLog。 合并后的 HyperLogLog 的基数接近于所有输入 HyperLogLog 的可见集合（observed set）的并集。

合并得出的 HyperLogLog 会被储存在 destkey 键里面， 如果该键并不存在， 那么命令在执行之前， 会先为该键创建一个空的 HyperLogLog 。

这个命令的第一个参数为目标键，剩下的参数为要合并的HyperLogLog。命令执行时，如果目标键不存在，则创建后再执行合并。

这个命令的时间复杂度为O(N)，其中N为要合并的HyperLogLog的个数。不过这个命令的常数时间复杂度比较高。

 

redis> PFADD  ip:20170626  "192.168.0.10"  "192.168.0.20"  "192.168.0.30"

(integer) 1

redis> PFADD  ip:20170626 "192.168.0.20"  "192.168.0.40"  "192.168.0.50"  # 存在就只加新的

(integer) 1

redis> PFCOUNT ip:20170626  # 元素估计数量没有变化

(integer) 5

redis> PFADD  ip:20170626 "192.168.0.20"  # 存在就不会增加

(integer) 0

edis> PFMERGE ip:20170626   ip:20170627   ip:20170628

OK

redis> PFCOUNT  ip:201706

(integer) 5

 

四、hperloglog 描述

由于hperloglog，这种数据结构在实际应用场景中并不多。因此，这里就不再详细讨论了。

我们看下hperloglog.c文件，对HperLogLog的描述

/* The Redis HyperLogLog implementation is based on the following ideas:

 *

 * * The use of a 64 bit hash function as proposed in [1], in order to don't

 *   limited to cardinalities up to 10^9, at the cost of just 1 additional

 *   bit per register.

 * * The use of 16384 6-bit registers for a great level of accuracy, using

 *   a total of 12k per key.

 * * The use of the Redis string data type. No new type is introduced.

 * * No attempt is made to compress the data structure as in [1]. Also the

 *   algorithm used is the original HyperLogLog Algorithm as in [2], with

 *   the only difference that a 64 bit hash function is used, so no correction

 *   is performed for values near 2^32 as in [1].

 *

 * [1] Heule, Nunkesser, Hall: HyperLogLog in Practice: Algorithmic

 *     Engineering of a State of The Art Cardinality Estimation Algorithm.

 *

 * [2] P. Flajolet, éric Fusy, O. Gandouet, and F. Meunier. Hyperloglog: The

 *     analysis of a near-optimal cardinality estimation algorithm.

 *

 * Redis uses two representations:

 *

 * 1) A "dense" representation where every entry is represented by

 *    a 6-bit integer.

 * 2) A "sparse" representation using run length compression suitable

 *    for representing HyperLogLogs with many registers set to 0 in

 *    a memory efficient way.

 *

 *

 * HLL header

 * ===

 *

 * Both the dense and sparse representation have a 16 byte header as follows:

 *

 * +------+---+-----+----------+

 * | HYLL | E | N/U | Cardin.  |

 * +------+---+-----+----------+

 *

 * The first 4 bytes are a magic string set to the bytes "HYLL".

 * "E" is one byte encoding, currently set to HLL_DENSE or

 * HLL_SPARSE. N/U are three not used bytes.

 *

 * The "Cardin." field is a 64 bit integer stored in little endian format

 * with the latest cardinality computed that can be reused if the data

 * structure was not modified since the last computation (this is useful

 * because there are high probabilities that HLLADD operations don't

 * modify the actual data structure and hence the approximated cardinality).

 *

 * When the most significant bit in the most significant byte of the cached

 * cardinality is set, it means that the data structure was modified and

 * we can't reuse the cached value that must be recomputed.

 *

 * Dense representation

 * ===

 *

 * The dense representation used by Redis is the following:

 *

 * +--------+--------+--------+------//      //--+

 * |11000000|22221111|33333322|55444444 ....     |

 * +--------+--------+--------+------//      //--+

 *

 * The 6 bits counters are encoded one after the other starting from the

 * LSB to the MSB, and using the next bytes as needed.

 *

 * Sparse representation

 * ===

 *

 * The sparse representation encodes registers using a run length

 * encoding composed of three opcodes, two using one byte, and one using

 * of two bytes. The opcodes are called ZERO, XZERO and VAL.

 *

 * ZERO opcode is represented as 00xxxxxx. The 6-bit integer represented

 * by the six bits 'xxxxxx', plus 1, means that there are N registers set

 * to 0. This opcode can represent from 1 to 64 contiguous registers set

 * to the value of 0.

 *

 * XZERO opcode is represented by two bytes 01xxxxxx yyyyyyyy. The 14-bit

 * integer represented by the bits 'xxxxxx' as most significant bits and

 * 'yyyyyyyy' as least significant bits, plus 1, means that there are N

 * registers set to 0. This opcode can represent from 0 to 16384 contiguous

 * registers set to the value of 0.

 *

 * VAL opcode is represented as 1vvvvvxx. It contains a 5-bit integer

 * representing the value of a register, and a 2-bit integer representing

 * the number of contiguous registers set to that value 'vvvvv'.

 * To obtain the value and run length, the integers vvvvv and xx must be

 * incremented by one. This opcode can represent values from 1 to 32,

 * repeated from 1 to 4 times.

 *

 * The sparse representation can't represent registers with a value greater

 * than 32, however it is very unlikely that we find such a register in an

 * HLL with a cardinality where the sparse representation is still more

 * memory efficient than the dense representation. When this happens the

 * HLL is converted to the dense representation.

 *

 * The sparse representation is purely positional. For example a sparse

 * representation of an empty HLL is just: XZERO:16384.

 *

 * An HLL having only 3 non-zero registers at position 1000, 1020, 1021

 * respectively set to 2, 3, 3, is represented by the following three

 * opcodes:

 *

 * XZERO:1000 (Registers 0-999 are set to 0)

 * VAL:2,1    (1 register set to value 2, that is register 1000)

 * ZERO:19    (Registers 1001-1019 set to 0)

 * VAL:3,2    (2 registers set to value 3, that is registers 1020,1021)

 * XZERO:15362 (Registers 1022-16383 set to 0)

 *

 * In the example the sparse representation used just 7 bytes instead

 * of 12k in order to represent the HLL registers. In general for low

 * cardinality there is a big win in terms of space efficiency, traded

 * with CPU time since the sparse representation is slower to access:

 *

 * The following table shows average cardinality vs bytes used, 100

 * samples per cardinality (when the set was not representable because

 * of registers with too big value, the dense representation size was used

 * as a sample).

 *

 * 100 267

 * 200 485

 * 300 678

 * 400 859

 * 500 1033

 * 600 1205

 * 700 1375

 * 800 1544

 * 900 1713

 * 1000 1882

 * 2000 3480

 * 3000 4879

 * 4000 6089

 * 5000 7138

 * 6000 8042

 * 7000 8823

 * 8000 9500

 * 9000 10088

 * 10000 10591

 *

 * The dense representation uses 12288 bytes, so there is a big win up to

 * a cardinality of ~2000-3000. For bigger cardinalities the constant times

 * involved in updating the sparse representation is not justified by the

 * memory savings. The exact maximum length of the sparse representation

 * when this implementation switches to the dense representation is

 * configured via the define server.hll_sparse_max_bytes.

 */