Redis持久化方案

Redis可以实现数据的持久化存储，即将数据保存到磁盘上。 
Redis的持久化存储提供两种方式：RDB与AOF。RDB是默认配置。AOF需要手动开启。 

 

默认redis是会以快照的形式将数据持久化到磁盘（一个二进制文件，dump.rdb，这个文件名字可以指定），在配置文件中的格式是：save N M表示在N秒之内，redis至少发生M次修改则redis抓快照到磁盘。当然我们也可以手动执行save或者bgsave（异步）做快照。当redis需要做持久化时，redis会fork一个子进程；子进程将数据写到磁盘上一个临时RDB文件中；当子进程完成写临时文件后，将原来的RDB替换掉，这样的好处就是可以copy-on-write.

# Save the DB on disk:
#
#   save <seconds> <changes>
#
#   Will save the DB if both the given number of seconds and the given
#   number of write operations against the DB occurred.
#
#   In the example below the behaviour will be to save:
#   after 900 sec (15 min) if at least 1 key changed
#   after 300 sec (5 min) if at least 10 keys changed
#   after 60 sec if at least 10000 keys changed
#
#   Note: you can disable saving completely by commenting out all "save" lines.
#
#   It is also possible to remove all the previously configured save
#   points by adding a save directive with a single empty string argument
#   like in the following example:
#
#   save ""

save 900 1
save 300 10
save 60 10000

 

还有一种持久化方法是Append-only：filesnapshotting方法在redis异常宕机时，最近的数据会丢失（丢失数据的多少视你save策略的配置），所以这是它最大的缺点，当数据量越大，丢失的数据就越多（可以看下文官方说明）。如果要开启AOF模式，修改Redis的配置文件redis.conf。

appendonly yes  
appendfilename "appendonly.aof" 

# appendfsync always
appendfsync everysec  
# appendfsync no

no-appendfsync-on-rewrite no    

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb  

aof-load-truncated yes  

 

官方说明：

By default Redis asynchronously dumps the dataset on disk. This mode is good enough in many applications, but an issue with the Redis process or a power outage may result into a few minutes of writes lost (depending on the configured save points).

The Append Only File is an alternative persistence mode that provides much better durability. For instance using the default data fsync policy (see later in the config file) Redis can lose just one second of writes in a dramatic event like a server power outage, or a single write if something wrong with the Redis process itself happens, but the operating system is still running correctly.

AOF and RDB persistence can be enabled at the same time without problems. If the AOF is enabled on startup Redis will load the AOF, that is the file with the better durability guarantees.

Redis默认采用异步的方式将数据存放到磁盘上，这个模式对大部份应用来说是足够好的，但是在Redis进程或电源发生故障的情况下，可能会造成小部份的数据丢失，这取决于配置的保存时间点。 
Appendonly是一种能够提供非常好的持久化的模式，例如使用默认的Fsync方案，Redis能在发生服务器电源故障或操作系统仍然正常运行但Redis进程莫名挂掉的情况下，只丢失1秒的数据。 
AOF与RDB模式可以同时启用，这并不冲突。如果AOF是可用的，那Redis启动时将自动加载AOF，这个文件能够提供更好的持久性保障。

 

The fsync() call tells the Operating System to actually write data on disk instead of waiting for more data in the output buffer. Some OS will really flush data on disk, some other OS will just try to do it ASAP.

Redis supports three different modes:

no: don’t fsync, just let the OS flush the data when it wants. Faster. 
always: fsync after every write to the append only log. Slow, Safest. 
everysec: fsync only one time every second. Compromise.

The default is “everysec”, as that’s usually the right compromise between speed and data safety. It’s up to you to understand if you can relax this to “no” that will let the operating system flush the output buffer when it wants, for better performances (but if you can live with the idea of some data loss consider the default persistence mode that’s snapshotting), or on the contrary, use “always” that’s very slow but a bit safer than everysec.

If unsure, use “everysec”.

fsync()调用告诉操作系统将数据真实的写入磁盘而不是放到缓冲区中，一些操作系统会真实的执行请求，还有一些操作系统只会尽力的尝试。

Redis支持3种不同的模式： 
no：不即时同步，由操作系统控制何时刷写到磁盘上，这种模式速度最快； 
always：每次只写日志，速度较慢，但最安全； 
everysec：每秒钟同步一次，折中的方案。

默认的模式是“everysec”，它通常是在速度和数据安全之间折中的方法。如果你可以控制操作系统在Redis需要的时候去刷写缓冲区那可以使用“no”模式，能够提供更好的性能（但如果你能接受一些数据丢失，可以考虑默认的持久化模式–快照方式），相反，使用“always”模式很慢，但是它比“everysec”模式要安全一点。

如果不确定，就使用“everysec”。

 

When the AOF fsync policy is set to always or everysec, and a background saving process (a background save or AOF log background rewriting) is performing a lot of I/O against the disk, in some Linux configurations Redis may block too long on the fsync() call. Note that there is no fix for this currently, as even performing fsync in a different thread will block our synchronous write(2) call.

In order to mitigate this problem it’s possible to use the following option that will prevent fsync() from being called in the main process while a BGSAVE or BGREWRITEAOF is in progress.

This means that while another child is saving, the durability of Redis is the same as “appendfsync none”. In practical terms, this means that it is possible to lose up to 30 seconds of log in the worst scenario (with the default Linux settings).

If you have latency problems turn this to “yes”. Otherwise leave it as “no” that is the safest pick from the point of view of durability.

当使用AOF的fsync方案设置为“always”或“everysec”时，后台的存储进程会执行大量的磁盘I/O操作，在一些Linux架构中，Redis在fsync()调用时可能会阻塞很久。这个问题当前并没有修复，即使是在一个不同的线程执行fsync也将会阻塞我们的同步写调用。

为了缓解这个问题，可以使用以下选项，它将会在有一个BGSAVE或BGREWRITEAOF正在运行时，阻止主进程调用fsync()。

这意味着有另一个子进程在存储时，Redis的持久性等同于“appendfsync none”。在实践中，意味着在最坏的情况下它可能丢失多达30秒的日志（默认的Linux设置）。

如果你有潜在的问题需要更改它为“yes”。否则从持久性的观点来看“no”是最安全的选择。

 

Automatic rewrite of the append only file. 
Redis is able to automatically rewrite the log file implicitly calling BGREWRITEAOF when the AOF log size grows by the specified percentage.

This is how it works: Redis remembers the size of the AOF file after the latest rewrite (if no rewrite has happened since the restart, the size of the AOF at startup is used).

This base size is compared to the current size. If the current size is bigger than the specified percentage, the rewrite is triggered. Also you need to specify a minimal size for the AOF file to be rewritten, this is useful to avoid rewriting the AOF file even if the percentage increase is reached but it is still pretty small.

Specify a percentage of zero in order to disable the automatic AOF rewrite feature. 

自动重写append only文件。 
当AOF日志的大小根据指定的百分比增长时，Redis会暗中调用BGREWRITEAOF去自动重写日志文件。

工作原理：Redis记忆AOF文件最后一次重写的大小（如果重启后没有重写发生，AOF的大小在启动时会被使用）。

基本大小对比当前大小。如果当前大小比指定的百分比大，触发重写。并且你要为AOF文件指定一个最小的尺寸去重写，这对于避免重写AOF文件是有用的，即使达到了百分比增长率但它仍然是非常小的。

指定百分比为0以便禁用自动AOF重写。

 

An AOF file may be found to be truncated at the end during the Redis startup process, when the AOF data gets loaded back into memory. 
This may happen when the system where Redis is running crashes, especially when an ext4 filesystem is mounted without the data=ordered option (however this can’t happen when Redis itself crashes or aborts but the operating system still works correctly).

Redis can either exit with an error when this happens, or load as much data as possible (the default now) and start if the AOF file is found to be truncated at the end. The following option controls this behavior.

If aof-load-truncated is set to yes, a truncated AOF file is loaded and the Redis server starts emitting a log to inform the user of the event. Otherwise if the option is set to no, the server aborts with an error and refuses to start. When the option is set to no, the user requires to fix the AOF file using the “redis-check-aof” utility before to restart the server.

Note that if the AOF file will be found to be corrupted in the middle the server will still exit with an error. This option only applies when Redis will try to read more data from the AOF file but not enough bytes will be found.

可能发现一个AOF文件在Redis启动过程中被截断，当AOF数据被加载回内存时。 
这可能发生在系统中Redis运行崩溃，尤其是挂载一个EXT4文件系统而没有使用“data=ordered”选项时（但是这不会发生在Redis自身崩溃或中断而操作系统仍然正常运行的时候）。

当有一个错误发生时Redis要么退出，要么加载尽可能多的数据（当前默认）启动，如果AOF文件最后发现被截断。以下选项控制这个行为。

如果aof-load-truncated被设置为“yes”，一个被截断的AOF文件将会被加载，Redis服务启动发出一个日志告知用户这个事件。如果这个选项设置为“no”，服务器中止一个错误并拒绝启动。当选项被设置为“no”，用户需要在重启服务之前使用“redis-check-aof”功能确定AOF文件。

需要注意的是如果AOF文件被发现是损坏的，那服务器仍然会以一个错误退出。这个选项仅适用于Redis希望读取更多的数据但是没有发现足够的字节时。

 

 

http://zheng-ji.info/blog/2016/03/10/gai-xuan-ze-na-chong-redischi-jiu-hua-pei-zhi/

https://my.oschina.net/davehe/blog/174662