【mysql】不可不知的Metadata Lock

一、问题发生

说一个现象，当收到服务器报警之后，数据库服务器CPU使用超过90%，通过 show processlist 一看，满屏都是 Waiting for table metadata lock 状态的连接。第一反应就是kill掉这些连接，奈何连接实在太多，实在kill不过来，于是重启服务，重启果真能解决90%的问题，但如果不找到问题原因，问题也肯定会再次出现。

类似于下图（没找到Waiting for table metadata lock的示例，将就看下）

 Id	User	Host	db	Command	Time	State	Info	Memory_used	Memory_used_by_query	Logical_read	Physical_sync_read	Physical_async_read	Temp_user_table_size	Temp_sort_table_size	Temp_sort_file_size
1277733	data01	172.17.22.204:26173	birds	Query	50	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2275632	12912	1012951	1	0	0	0	0
1277751	data01	172.17.22.204:26745	birds	Query	49	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2245288	12912	936956	4	0	0	0	0
1277802	data01	172.17.22.204:29587	birds	Query	58	Creating sort index	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2584048	12912	761473	2	0	0	0	0
1277825	data01	172.17.22.204:30637	birds	Query	57	Creating sort index	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	1414176	12912	751966	0	0	0	0	0
1277827	data01	172.17.22.204:30639	birds	Query	56	Creating sort index	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2497392	12912	757188	0	0	0	0	0
1277828	data01	172.17.22.204:30641	birds	Query	57	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	1134712	12912	739028	0	0	0	0	0
1277829	data01	172.17.22.204:30723	birds	Query	56	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	1134712	12912	725881	0	0	0	0	0
1277830	data01	172.17.22.204:30721	birds	Query	55	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2238288	12912	732720	0	0	0	0	0
1277831	data01	172.17.22.204:30765	birds	Query	56	Creating sort index	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2497520	12912	757099	0	0	0	0	0
1277832	data01	172.17.22.204:30801	birds	Query	55	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	693261	0	0	0	0	0
1277879	data01	172.17.22.204:32905	birds	Query	34	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '300780'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536	8544	462365	0	0	0	0	0
1277880	data01	172.17.22.204:32907	birds	Query	33	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	425506	3	0	0	0	0
1277881	data01	172.17.22.204:32993	birds	Query	29	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '185711'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1138896	8544	483586	1	0	0	0	0
1277914	data01	172.17.22.204:34571	birds	Query	25	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2217736	12912	315580	0	0	0	0	0
1277915	data01	172.17.22.204:34573	birds	Query	0	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '557415'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1138640	8544	283988	0	0	0	0	0
1277916	data01	172.17.22.204:34677	birds	Query	25	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	309707	0	0	0	0	0
1277918	data01	172.17.22.204:34747	birds	Query	25	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	299223	0	0	0	0	0
1277919	data01	172.17.22.204:34805	birds	Sleep	0	NULL	1123104	8208	283855	3	0	0	0	0
1277920	data01	172.17.22.204:34829	birds	Query	6	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '512151'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536	8544	305759	5	0	0	0	0
1277922	data01	172.17.22.204:34909	birds	Query	20	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	253827	0	0	0	0	0
1277925	data01	172.17.22.204:35133	birds	Query	21	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2218056	12912	276768	0	0	0	0	0
1277929	data01	172.17.22.204:35319	birds	Query	20	Sending data	SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139  ORDER BY a.tid DESC  LIMIT 0,500	2217928	12912	251582	0	0	0	0	0
1277971	data01	172.17.22.204:36879	birds	Query	3	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '489209'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536	8544	45250	0	0	0	0	0
1277972	data01	172.17.22.204:36881	birds	Query	3	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '187583'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1150288	8544	40179	0	0	0	0	0
1277981	data01	172.17.22.204:37213	birds	Query	0	Sending data	SELECT tid,subject FROM pre_forum_thread WHERE tid < '194152'  AND fid = '302'  AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536	8544	7289	0	0	0	0	0
1277982	data01	172.17.22.204:37231	birds	Query	0	Writing to net	SELECT * FROM pre_forum_post WHERE tid=585078 AND position>=14361 AND position<14371 ORDER BY position	1164064	10280	374	0 0	0	0	0
1277984	data01	172.17.22.204:37285	NULL	Query	0	init	show full processlist	73800	8472	0	0	0	0	0	0

原因是什么：

MySQL在进行一些alter table等DDL操作时，如果该表上有未提交的事务则会出现 Waiting for table metadata lock ，而一旦出现metadata lock，该表上的后续操作都会被阻塞，由于当时直接把ALTER给kill了，所以基本没有现场

二、关于Table Metadata Lock

1、为什么会有 Metadata  Lock

源起于一个bug，MySQL官方文档链接：http://bugs.mysql.com/bug.php?id=989

Description:
If user1 has an active transaction on a table and then user2 drops this table, then user1 does COMMIT, then in the binlog we have something like:
DROP TABLE t;
BEGIN;
INSERT INTO t ... ;
COMMIT;
which is wrong.

 

这个bug大致意思是说：当一个会话在主库执行DML操作还没提交时，另一个会话对同一个对象执行了DDL操作如drop table，而由于MySQL的binlog是基于事务提交的先后顺序进行记录的，因此在从库上应用时，就出现了先drop table，然后再向table中insert的情况，导致从库应用出错。

因此，MySQL在5.5.3版本后引入了Metadata lock，只有在事务结束后才会释放Metadata lock，因此在事务提交或回滚前，是无法进行DDL操作的。

2、什么是Table Metadata Lock

首先，看看官方的说法，

MySQL官方文档位置：http://dev.mysql.com/doc/refman/5.6/en/metadata-locking.html

To ensure transaction serializability, the server must not permit one session to perform a data definition language (DDL) statement on a table that is used in an uncompleted explicitly or implicitly started transaction in another session.

The server achieves this by acquiring metadata locks on tables used within a transaction and deferring release of those locks until the transaction ends.

A metadata lock on a table prevents changes to the table's structure.

This locking approach has the implication that a table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends.

从上面的描述可以看到，

  • MDL出现的初衷就是为了保护一个处于事务中的表的结构不被修改。

  • 这里提到的事务包括两类，显式事务和AC-NL-RO（auto-commit non-locking read-only）事务。显式事务包括两类：1. 关闭AutoCommit下的操作，2. 以begin或start transaction开始的操作。AC-NL-RO可理解为AutoCommit开启下的select操作。

  • MDL是事务级别的，只有在事务结束后才会释放。在此之前，其实也有类似的保护机制，只不过是语句级别的。

需要注意的是，MDL不仅仅适用于表，同样也适用于其它对象，如下表所示，其中，"等待状态"对应的是"show processlist"中的State。

 

为了提高数据库的并发度，MDL被细分为了11种类型。

  • MDL_INTENTION_EXCLUSIVE

  • MDL_SHARED

  • MDL_SHARED_HIGH_PRIO

  • MDL_SHARED_READ

  • MDL_SHARED_WRITE

  • MDL_SHARED_WRITE_LOW_PRIO

  • MDL_SHARED_UPGRADABLE

  • MDL_SHARED_READ_ONLY

  • MDL_SHARED_NO_WRITE

  • MDL_SHARED_NO_READ_WRITE

  • MDL_EXCLUSIVE

常用的有MDL_SHARED_READ,MDL_SHARE D_WRITE及MDL_EXCLUSIVE，其分别用于SELECT操作，DML操作及DDL操作。其它类型的对应操作可参考源码sql/mdl.h。

对于MDL_EXCLUSIVE，官方的解释是， 

An exclusive metadata lock.

A connection holding this lock can modify both table's metadata and data.

No other type of metadata lock can be granted while this lock is held.

To be used for CREATE/DROP/RENAME TABLE statements and for execution of certain phases of other DDL statements.

简而言之，MDL_EXCLUSIVE是独占锁，在其持有期间是不允许其它类型的MDL被授予，自然也包括SELECT和DML操作。

这也就是为什么DDL操作被阻塞时，后续其它操作也会被阻塞。

关于MDL的补充

1. MDL的最大等待时间由lock_wait_timeout参数决定，其默认值为31536000（365天）。在使用工具进行DDL操作时，这个值就不太合理。事实上，pt-online-schema-change和gh-ost对其就进行了相应的调整，其中，前者60s，后者3s。

2. 如果一个SQL语法上有效，但执行时报错，如，列名不存在，其同样会获取MDL锁，直到事务结束才释放。

三、Metadata Lock解决了什么问题

MySQL 从5.5.3版本，对Metadata lock进行了调整，主要是MDL锁持有的周期从语句变成了事务， 其原因主要是解决两个问题：

  • 问题1: 破坏事务隔离级别 在repeatable read的隔离级别下，多次的select语句执行过程中，会因为其它session的DDL语句，而导致select语句执行的结果不相同，破坏了RR的隔离级别。

  • 问题2: 破坏binlog的顺序 在对表的DML过程中，会因为其它session的DDL语句，导致binlog里的event顺序在备库执行的结果和主库不一致。

从MySQL 5.5.3开始，MDL锁的持有周期变成了事务，解决了上面提到的两个问题，但在autocommit=off的情况下，也大大增加了阻塞的可能性。DBA对于阻塞的case，处理起来又比较麻烦，原因就是MDL锁的阻塞情况没有暴露明确的信息。

从MySQL 5.7.6开始，可以通过performance schema来查询MDL锁的持有情况。

RR事务隔离级别下不可重复读的问题，演示环境，MySQL 5.5.0。

session1> begin;
Query OK, 0 rows affected (0.00 sec)
session1> select * from t1;
+------+------+
| id  | name |
+------+------+
|    1 | a    |
|    2 | b    |
+------+------+
2 rows in set (0.00 sec)
session2> alter table t1 add c1 int;
Query OK, 2 rows affected (0.02 sec)
Records: 2  Duplicates: 0  Warnings: 0
session1> select * from t1;
Empty set (0.00 sec)
session1> commit;
Query OK, 0 rows affected (0.00 sec)
session1> select * from t1;
+------+------+------+
| id  | name | c1  |
+------+------+------+
|    1 | a    | NULL |
|    2 | b    | NULL |
+------+------+------+
2 rows in set (0.00 sec)

可以看到，虽然是RR隔离级别，但在开启事务的情况下，第二次查询却没有结果

主从复制问题

包括主从数据不一致，主从复制中断等。 如下面的主从数据不一致。

session1> create table t1(id int,name varchar(10)) engine=innodb;
Query OK, 0 rows affected (0.00 sec)
session1> begin;
Query OK, 0 rows affected (0.00 sec)
session1> insert into t1 values(1,'a');
Query OK, 1 row affected (0.00 sec)
session2> truncate table t1;
Query OK, 0 rows affected (0.46 sec)
session1> commit;
Query OK, 0 rows affected (0.35 sec)
session1> select * from t1;
Empty set (0.00 sec)

再来看看从库的结果

session1> select * from t1;
+------+------+------+
| id   | name | c1   |
+------+------+------+
|    1 | a    | NULL |
+------+------+------+
1 row in set (0.00 sec)

看看binlog的内容，可以看到，truncate操作记录在前，insert操作记录在后。

# at 7140
#180714 19:32:14 server id 1  end_log_pos 7261    Query    thread_id=31    exec_time=0    error_code=0
SET TIMESTAMP=1531567934/*!*/;
create table t1(id int,name varchar(10)) engine=innodb
/*!*/;
# at 7261
#180714 19:32:30 server id 1  end_log_pos 7333    Query    thread_id=32    exec_time=0    error_code=0
SET TIMESTAMP=1531567950/*!*/;
BEGIN
/*!*/;
# at 7333
#180714 19:32:30 server id 1  end_log_pos 7417    Query    thread_id=32    exec_time=0    error_code=0
SET TIMESTAMP=1531567950/*!*/;
truncate table t1
/*!*/;
# at 7417
#180714 19:32:30 server id 1  end_log_pos 7444    Xid = 422
COMMIT/*!*/;
# at 7444
#180714 19:32:34 server id 1  end_log_pos 7516    Query    thread_id=31    exec_time=0    error_code=0
SET TIMESTAMP=1531567954/*!*/;
BEGIN
/*!*/;
# at 7516
#180714 19:32:24 server id 1  end_log_pos 7611    Query    thread_id=31    exec_time=0    error_code=0
SET TIMESTAMP=1531567944/*!*/;
insert into t1 values(1,'a')
/*!*/;
# at 7611
#180714 19:32:34 server id 1  end_log_pos 7638    Xid = 421
COMMIT/*!*/;

如果会话2执行的是drop table操作，还会导致主从中断。

有意思的是，如果会话2执行的是alter table操作，其依旧会被阻塞，阻塞时间受innodb_lock_wait_timeout参数限制。

mysql> show processlist;
+----+------+-----------+----------+---------+------+-------------------+---------------------------+
| Id | User | Host      | db       | Command | Time | State             | Info                      |
+----+------+-----------+----------+---------+------+-------------------+---------------------------+
| 54 | root | localhost | NULL     | Query   |    0 | NULL              | show processlist          |
| 58 | root | localhost | slowtech | Sleep   | 1062 |                   | NULL                      |
| 60 | root | localhost | slowtech | Query   |   11 | copy to tmp table | alter table t1 add c1 int |
+----+------+-----------+----------+---------+------+-------------------+---------------------------+
3 rows in set (0.00 sec)

可以看到没有MDL锁的情况下，会出现以上两种问题，当有MDL锁保护之后，就正常了

InnoDB层已经有了IS、IX这样的意向锁，有同学觉得可以用来实现上述例子的并发控制。但由于MySQL是Server-Engine架构，所以MDL锁是在Server中实现。另外，MDL锁还能实现其他粒度级别的锁，比如全局锁、库级别的锁、表空间级别的锁，这是InnoDB存储引擎层不能直接实现的锁。

但与InnoDB锁的实现一样，MDL锁也是类似对一颗树的各个对象从上至下进行加锁（对树进行加锁具体见：《MySQL技术内幕：InnoDB存储引擎》）。但是MDL锁对象的层次更多，简单来看有如下的层次：

上图中显示了最常见的4种MDL锁的对象，并且注明了常见的SQL语句会触发的锁。与InnoDB层类似的是，某些类型的MDL锁会从上往下一层层进行加锁。比如LOCK TABLE … WRITE这样的SQL语句，其首先会对GLOBAL级别加INTENTION_EXCLUSIVE锁，再对SCHEMA级别加INTENTION_EXCLUSIVE锁，最后对TABLE级别加SHARED_NO_READ_WRITE锁。这里最令人意外的是还有COMMIT对象层次的锁，其实这主要用于XA事务中。比如分布式事务已经PREPARE成功，但是在XA COMMIT之前有其他会话执行了FLUSH TABLES WITH READ LOCK这样的操作，那么分布式事务的提交就需要等待。

除了上图标注的对象，其实还有TABLESPACE、FUNCTION、PROCEDURE、EVENT等其他对象类型，其实都是为了进行并发控制。只是这些在MySQL数据库中都不常用，故不再赘述（当然也是为了偷懒）。

GLOBAL=0,TABLESPACE,SCHEMA,TABLE,FUNCTION,PROCEDURE,TRIGGER,EVENT,COMMIT,USER_LEVEL_LOCK,LOCKING_SERVICE,NAMESPACE_END

目前MDL有如下锁模式，锁之间的兼容性可见源码mdl.cc：

锁模式 对应SQL
MDL_INTENTION_EXCLUSIVE GLOBAL对象、SCHEMA对象操作会加此锁
MDL_SHARED FLUSH TABLES with READ LOCK
MDL_SHARED_HIGH_PRIO 仅对MyISAM存储引擎有效
MDL_SHARED_READ SELECT查询
MDL_SHARED_WRITE DML语句
MDL_SHARED_WRITE_LOW_PRIO 仅对MyISAM存储引擎有效
MDL_SHARED_UPGRADABLE ALTER TABLE
MDL_SHARED_READ_ONLY LOCK xxx READ
MDL_SHARED_NO_WRITE FLUSH TABLES xxx,yyy,zzz READ
MDL_SHARED_NO_READ_WRITE FLUSH TABLE xxx WRITE
MDL_EXCLUSIVE ALTER TABLE xxx PARTITION BY …

MDL锁的性能与并发改进

讲到这同学们会发现MDL锁的开销并不比InnoDB层的行锁要小，而且这可能是一个更为密集的并发瓶颈。MySQL 5.6和5.5版本通常通过调整如下两个参数来进行并发调优:

  • metadata_locks_cache_size: MDL锁的缓存大小

  • metadata_locks_hash_instances：通过分片来提高并发度，与InnoDB AHI类似

MySQL 5.7 MDL锁的最大改进之处在于将MDL锁的机制通过lock free算法来实现，从而提高了在多核并发下数据库的整体性能提升。

MDL锁的诊断

MySQL 5.7版本之前并没有提供一个方便的途径来查看MDL锁，github上有一名为mysql-plugin-mdl-info的项目，通过插件的方式来查看，非常有想法的实现，大赞。好在官方也意识到了这个问题，于是在MySQL 5.7中的performance_schea库下新增了一张表metadata_locks，用其来查看MDL锁那是相当的方便:

不过默认PS并没有打开此功能，需要手工将wait/lock/metadata/sql/mdl监控给打开

SELECT * FROM performance_schema.setup_instruments;  
UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME ='global_instrumentation';
UPDATE performance_schema.setup_instruments SET ENABLED = 'YES' WHERE NAME ='wait/lock/metadata/sql/mdl';
select * from performance_schema.metadata_locks\G

　　

参考文章

http://mysql.taobao.org/monthly/2015/10/02/

http://www.yunweipai.com/archives/19818.html

https://www.jianshu.com/p/683bd7fcb7a0

https://www.cnblogs.com/zengkefu/p/5690385.html

https://www.cnblogs.com/cchust/p/3826398.html

https://www.cnblogs.com/xpchild/p/3789068.html

http://www.ywnds.com/?p=7209&viewuser=42

https://mp.weixin.qq.com/s?__biz=MzU3ODU4MTgyOQ==&mid=2247483729&idx=1&sn=03711c826fcef68ea530a9cfa4decb25&chksm=fd72657bca05ec6d356040139c0f649800b5d5c454a4d675241fdd6a8809bb1657778a4aa032&scene=21#wechat_redirect

https://mp.weixin.qq.com/s?__biz=MzU3ODU4MTgyOQ==&mid=2247483735&idx=1&sn=a3de9f2632581b4643781b44cdbfe819&chksm=fd72657dca05ec6b1a82338e8416b58b58278daee749620393ec268a98833e6d2b0b25fe20ff&mpshare=1&scene=1&srcid=0913ujxgOYxYZzV6fuLfAMhn#rd

http://www.ywnds.com/?p=7209&viewuser=42

https://www.cnblogs.com/zengkefu/p/5690385.html

https://m.aliyun.com/yunqi/articles/51031

https://www.cnblogs.com/xpchild/p/3790139.html

posted @ 2018-09-20 14:18  踏雪无痕SS  阅读(7818)  评论(0)  编辑  收藏  举报