gearman mysql持久化

gearman 创建Mysql持久化队列的方式如下：

 

  启动gearman，命令如下：

gearmand命令：

-b, –backlog=BACKLOG 连接请求队列的最大值
-d, –daemon Daemon 守护进程化
-f, –file-descriptors=FDS 可打开的文件描述符数量
-h, –help
-l, –log-file=FILE Log 日志文件
-L, –listen=ADDRESS 开启监听的地址
-p, –port=PORT 开启监听的端口
-P, –pid-file=FILE File pid file
-r,–protocol=PROTOCOL 使用的协议
-q, –queue-type=QUEUE 持久化队列类型
-t, –threads=THREADS I/O线程数量
-u, –user=USER 进程的有效用户名
libdrizzle Options:
--libdrizzle-host=HOST Host of server.
--libdrizzle-port=PORT Port of server.
--libdrizzle-uds=UDS Unix domain socket for server.
--libdrizzle-user=USER User name for authentication.
--libdrizzle-password=PASSWORD Password for authentication.
--libdrizzle-db=DB Database to use.
--libdrizzle-table=TABLE Table to use.
--libdrizzle-mysql Use MySQL protocol.
libmemcached Options:
--libmemcached-servers=SERVER_LIST List of Memcached servers to use.
libsqlite3 Options:
--libsqlite3-db=DB Database file to use.
--libsqlite3-table=TABLE Table to use.
libpq Options:
--libpq-conninfo=STRING PostgreSQL connection information string.
--libpq-table=TABLE Table to use.
http Options:
--http-port=PORT Port to listen on.

 

/usr/local/gearman/sbin/gearmand -p 4730 -L 0.0.0.0 --log-file=/tmp/gearmand-4730.log --pid-file=/tmp/gearmand-4730.pid -q MySQL --mysql-host=localhost --mysql-user=root --mysql-db=gearman --verbose DEBUG-d

我自己看gearmand 参数与上面不同：

[root@localhost sbin]# ./gearmand  -h
Allowed options:

Allowed options:

General options:
  -b [ --backlog ] arg (=32)                         Number of backlog 
                                                     connections for listen.
  -d [ --daemon ]                                    Daemon, detach and run in 
                                                     the background.
  --exceptions                                       Enable protocol exceptions
                                                     by default.
  -f [ --file-descriptors ] arg                      Number of file descriptors
                                                     to allow for the process 
                                                     (total connections will be
                                                     slightly less). Default is
                                                     max allowed for user.
  -h [ --help ]                                      Print this help menu.
  -j [ --job-retries ] arg (=0)                      Number of attempts to run 
                                                     the job before the job 
                                                     server removes it. This is
                                                     helpful to ensure a bad 
                                                     job does not crash all 
                                                     available workers. Default
                                                     is no limit.
  --job-handle-prefix arg                            Prefix used to generate a 
                                                     job handle string. If not 
                                                     provided, the default 
                                                     "H:<host_name>" is used.
  --hashtable-buckets arg (=991)                     Number of buckets in the 
                                                     internal job hash tables. 
                                                     The default of 991 works 
                                                     well for about three 
                                                     million jobs in queue. If 
                                                     the number of jobs in the 
                                                     queue at any time will 
                                                     exceed three million, use 
                                                     proportionally larger 
                                                     values (991 * # of jobs / 
                                                     3M). For example, to 
                                                     accomodate 2^32 jobs, use 
                                                     1733003. This will consume
                                                     ~26MB of extra memory. 
                                                     Gearmand cannot support 
                                                     more than 2^32 jobs in 
                                                     queue at this time.
  --keepalive                                        Enable keepalive on 
                                                     sockets.
  --keepalive-idle arg (=-1)                         If keepalive is enabled, 
                                                     set the value for 
                                                     TCP_KEEPIDLE for systems 
                                                     that support it. A value 
                                                     of -1 means that either 
                                                     the system does not 
                                                     support it or an error 
                                                     occurred when trying to 
                                                     retrieve the default 
                                                     value.
  --keepalive-interval arg (=-1)                     If keepalive is enabled, 
                                                     set the value for 
                                                     TCP_KEEPINTVL for systems 
                                                     that support it. A value 
                                                     of -1 means that either 
                                                     the system does not 
                                                     support it or an error 
                                                     occurred when trying to 
                                                     retrieve the default 
                                                     value.
  --keepalive-count arg (=-1)                        If keepalive is enabled, 
                                                     set the value for 
                                                     TCP_KEEPCNT for systems 
                                                     that support it. A value 
                                                     of -1 means that either 
                                                     the system does not 
                                                     support it or an error 
                                                     occurred when trying to 
                                                     retrieve the default 
                                                     value.
  -l [ --log-file ] arg (=/opt/var/log/gearmand.log) Log file to write errors 
                                                     and information to. If the
                                                     log-file parameter is 
                                                     specified as 'stderr', 
                                                     then output will go to 
                                                     stderr. If 'none', then no
                                                     logfile will be generated.
  -L [ --listen ] arg                                Address the server should 
                                                     listen on. Default is 
                                                     INADDR_ANY.
  -P [ --pid-file ] arg (=/opt/var/gearmand.pid)     File to write process ID 
                                                     out to.
  -r [ --protocol ] arg                              Load protocol module.
  -R [ --round-robin ]                               Assign work in round-robin
                                                     order per worker 
                                                     connection. The default is
                                                     to assign work in the 
                                                     order of functions added 
                                                     by the worker.
  -q [ --queue-type ] arg (=builtin)                 Persistent queue type to 
                                                     use.
  --config-file arg (=/opt/etc/gearmand.conf)        Can be specified with 
                                                     '@name', too
  --syslog                                           Use syslog.
  --coredump                                         Whether to create a core 
                                                     dump for uncaught signals.
  -t [ --threads ] arg (=4)                          Number of I/O threads to 
                                                     use, 0 means that gearmand
                                                     will try to guess the 
                                                     maximum number it can use.
                                                     Default=4.
  -u [ --user ] arg                                  Switch to given user after
                                                     startup.
  --verbose arg (=ERROR)                             Set verbose level (FATAL, 
                                                     ALERT, CRITICAL, ERROR, 
                                                     WARNING, NOTICE, INFO, 
                                                     DEBUG).
  -V [ --version ]                                   Display the version of 
                                                     gearmand and exit.
  -w [ --worker-wakeup ] arg (=0)                    Number of workers to 
                                                     wakeup for each job 
                                                     received. The default is 
                                                     to wakeup all available 
                                                     workers.

HTTP:
  --http-port arg (=8080) Port to listen on.

Gear:
  -p [ --port ] arg (=4730) Port the server should listen on.
  --ssl                     Enable ssl connections.
  --ssl-ca-file arg         CA file.
  --ssl-certificate arg     SSL certificate.
  --ssl-key arg             SSL key for certificate.

builtin:

MySQL:
  --mysql-host arg (=localhost)      MySQL host.
  --mysql-port arg (=3306)           Port of server. (by default 3306)
  --mysql-user arg                   MySQL user.
  --mysql-password arg               MySQL user password.
  --mysql-db arg                     MySQL database.
  --mysql-table arg (=gearman_queue) MySQL table name.

/sbin/gearmand -p 4730 -L 0.0.0.0 -p 4730  -q MySQL --mysql-host=localhost  --mysql-port 3306 
--mysql-user=root --mysql-password pwd --mysql-db=gearman --verbose DEBUG -d

为了安全起见，可以把-L指定为127.0.0.0 

执行上面的错误：./gearmand: Error while initializing the queue : MySQL

具体原因查看日志：gearmand.log。显示：

DEBUG 2014-12-05 06:49:31.300168 [ main ] THREADS: 4 -> libgearman-server/gearmand.cc:263
INFO 2014-12-05 06:49:31.300259 [ main ] Initializing MySQL module
ERROR 2014-12-05 06:49:31.341972 [ main ] Failed to connect to database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) -> libgearman-server/plugins/queue/mysql/queue.cc:228
ERROR 2014-12-05 06:49:31.342008 [ main ] Failed to initialize mysql: initialize(QUEUE_ERROR) -> libgearman-server/queue.cc:246
DEBUG 2014-12-05 06:49:31.342081 [ main ] Shutting down all threads -> libgearman-server/gearmand.cc:276
DEBUG 2014-12-05 06:49:31.342093 [ main ] MySQL queue flush -> libgearman-server/plugins/queue/mysql/queue.cc:370
DEBUG 2014-12-05 06:49:31.342103 [ main ] removing queue: FUNCTION -> libgearman-server/gearmand.cc:178
INFO 2014-12-05 06:49:31.342112 [ main ] Shutdown complete
[root@localhost log]# pwd

我去/var/lib/mysql/这个目录没有mysql.sock这个文件。

 

由于mysql 默认的mysql.sock 是在/var/lib/mysql/mysql.sock，创建符号连接：、（参考文章：

 Can't connect to local MySQL server through socket '/tmp/mysql.sock' (11)

解决不能通过mysql .sock连接MySQL问题 这个问题主要提示是，不能通过 '/tmp/mysql .sock'连到服务器，而php标准配置正是用过'/tmp/mysql .sock'，但是一些mysql 安装方法 将 mysql .sock放在/var/lib/mysql .sock或者其他的什么地方，你可以通过修改/etc/my.cnf文件来修正它，打开文件，可以看到如下的东东： 
　　[mysql d] 
　　socket=/var/lib/mysql .sock 
　　改一下就好了，但也会引起其他的问题，如mysql 程序连不上了，再加一点： 
　　[mysql ] 
　　socket=/tmp/mysql .sock 
　　或者还可以通过修改php.ini中的配置来使php用其他的mysql .sock来连，这个大家自己去找找
　　
　　或者用这样的方法:
　　ln -s /var/lib/mysql /mysql .sock /tmp/mysql .sock 

成功了,就是这样ln -s /var/lib/mysql /mysql .sock /tmp/mysql .sock 

）

locate mysql.sock所在目录。然后创建连接。

ln -s   /data/xx/data/mysql.sock  /var/lib/mysql/mysql.sock

在 /var/lib/mysql/创建连接后，再运行，果然成功。成功自动创建gearman_queue表。表结构：

create table `gearman_queue` (
`unique_key` varchar(64) NOT NULL,
`function_name` varchar(255) NOT NULL,
`priority` int(11) NOT NULL,
`data` LONGBLOB NOT NULL,
`when_to_run` INT, PRIMARY KEY  (`unique_key`)
)

 

 

 具体的参数，根据自己的服务器状况去修改就行了。

3. 再次登入mysql命令行，执行：

use gearman;show tables;

 就可以看到下面多了个“gearman_queue”的表。

这样，gearman就变成了持久化的方式。

=============================================================================

gearman用mysql持久化之后，其实会带来一些问题。

1. 每个任务都会写入数据库，这样会带来磁盘IO的损耗，并且gearman的性能瓶颈又多了一个可能，就是由数据库引起的性能问题。

2. mysql有个“wait_timeout”的参数，在mysql命令行中运行

show variables like "%timeout%";

 可以看到wait_timeout的值，默认是28800。也就是说，如果一个mysql的连接，超过28800s没有任何响应，就会断开。

3. gearman持久化的方式，如果超过了mysql的wait_timeouts的时间没有任何响应，和数据库的连接就会被mysql断开，而且gearman目前是没有mysql重新连接的，结果就是，会导致如下错误，必须重启gearman才能重新正常工作。

gearman报错 

ERROR 2014-04-01 02:10:02.897899 [ proc ] mysql_stmt_execute failed: -> libgearman-server/plugins/queue/mysql/queue.cc:357
ERROR 2014-04-01 02:10:02.897910 [ proc ] gearman_server_job_add gearman_server_run_command(QUEUE_ERROR) -> libgearman-server/server.cc:301

所以，gearman持久化方式带来的缺点显而易见，该方式下，要避免gearman对mysql连接超时断开，可以将mysql的wait_timeout参数改大。

或者，直接放弃用mysql的持久化方式。

 持久化队列将后台作业存放在一个外部持久的队列中。持久化队列只对后台jobs有效，因为前台jobs依附于客户端。如果job服务器挡掉了，客户端会检测到，将会从其他地方重新启动这个前台job或者返回错误。而后台jobs没有依附于客户端，如果要想让它运行则需要提交。

 

The persistent queue works by calling a module callback function right before putting a new job in the internal queue for pending jobs to be run. This allows the module to store the job about to be run in some persistent way so that it can later be replayed during a restart. Once it is stored through the module, the job is put onto the active runnable queue, waking up available workers if needed. Once the job has been successfully completed by a worker, another module callback function is called to notify the module the job is done and can be removed. If a job server crashes or is restarted between these two calls for a job, the jobs are reloaded during the next job server start. When the job server starts up, it will call a replay callback function in the module to provide a list of all jobs that were not complete. This is used to populate the internal memory queue of jobs to be run. Once this replay is complete, the job server finishes its initialization and the jobs are now runnable once workers connect (the queue should be in the same state as when it crashed). These jobs are removed from the persistent queue when completed as normal. NOTE: Deleting jobs from the persistent queue storage will not remove them from the in-memory queue while the server is running.

 

创建一个后台job

 ./gearman -f sayHello -b  xx00

 

-f函数名 -b 后台  xx00为数据，

执行后。

数据表多了一条记录：

 

执行队列中的job：

 ./gearman -f sayHello  -w

 

 

 

gearmanMonitor可以可视化显示：

 

 

 

 

http://www.ttlsa.com/gearman/gearman-mysql/

http://huoding.com/2012/10/30/196

更多：

http://www.linuxeye.com/database/mysql-replication-to-redis-by-gearman.html