浅析libcurl多线程安全问题
浅析libcurl多线程安全问题
背景：使用多线程libcurl发送请求，在未设置超时或长超时的情况下程序运行良好。但只要设置了较短超时（小于180s），程序就会出现随机的coredump。并且栈里面找不到任何有用的信息。
问题：1.为什么未设置超时，或者长超时时间（比如601s）的情况下多线程libcurl不会core？
问题：2.进程coredump并不是必现，是否在libcurl内多线程同时修改了全局变量导致？
先来看下官方libcurl的说明：
libcurl is free, thread-safe, IPv6 compatible, feature rich, well supported, fast, thoroughly documented and is already used by many known, big and successful companies and numerous applications.
可以看到官方自称licurl是线程安全的，是否真的如此？再来看看代码中用到的超时选项的说明：
CURLOPT_TIMEOUT
Pass a long as parameter containing the maximum time in seconds that you allow the libcurl transfer operation to take. Normally, name lookups can take a considerable time and limiting operations to less than a few minutes risk aborting perfectly normal operations. This option will cause curl to use the SIGALRM to enable time-outing system calls.
In unix-like systems, this might cause signals to be used unless CURLOPT_NOSIGNAL is set.
Default timeout is 0 (zero) which means it never times out.
选项提到了超时机制是使用SIGALRM信号量来实现的，并且在unix-like操作系统中又提到了另外一个选项CURLOPT_NOSIGNAL：
CURLOPT_NOSIGNAL
Pass a long. If it is 1, libcurl will not use any functions that install signal handlers or any functions that cause signals to be sent to the process. This option is mainly here to allow multi-threaded unix applications to still set/use all timeout options etc, without risking getting signals. The default value for this parameter is 0. (Added in 7.10)
If this option is set and libcurl has been built with the standard name resolver, timeouts will not occur while the name resolve takes place. Consider building libcurl with c-ares support to enable asynchronous DNS lookups, which enables nice timeouts for name resolves without signals.
Setting CURLOPT_NOSIGNAL to 1 makes libcurl NOT ask the system to ignore SIGPIPE signals, which otherwise are sent by the system when trying to send data to a socket which is closed in the other end. libcurl makes an effort to never cause such SIGPIPEs to trigger, but some operating systems have no way to avoid them and even on those that have there are some corner cases when they may still happen, contrary to our desire. In addition, usingCURLAUTH_NTLM_WB authentication could cause a SIGCHLD signal to be raised.
该选项说明提到，为了在多线程中允许程序去设置timeout选项，但不是使用signals，需要设置CURLOPT_NOSIGNAL为1 。
于是在代码中加上了这句，测试再没有发现有coredump的情况。
1 easy_setopt(curl, CURLOPT_NOSIGNAL, (long)1);
问题：3.timeout机制实现机制是什么，为什么设置了选项CURLOPT_NOSIGNAL线程就安全了？
为了解答上面的问题，需要查看libcurl的相关源代码，以下是DNS解析的函数：
1 int Curl_resolv_timeout(struct connectdata *conn, 2 const char *hostname, 3 int port, 4 struct Curl_dns_entry **entry, 5 long timeoutms) 6 { 7 #ifdef USE_ALARM_TIMEOUT 8 #ifdef HAVE_SIGACTION 9 struct sigaction keep_sigact; /* store the old struct here */ 10 volatile bool keep_copysig = FALSE; /* wether old sigact has been saved */ 11 struct sigaction sigact; 12 #else 13 #ifdef HAVE_SIGNAL 14 void (*keep_sigact)(int); /* store the old handler here */ 15 #endif /* HAVE_SIGNAL */ 16 #endif /* HAVE_SIGACTION */ 17 volatile long timeout; 18 volatile unsigned int prev_alarm = 0; 19 struct SessionHandle *data = conn->data; 20 #endif /* USE_ALARM_TIMEOUT */ 21 int rc; 22 23 *entry = NULL; 24 25 if(timeoutms < 0) 26 /* got an already expired timeout */ 27 return CURLRESOLV_TIMEDOUT; 28 29 #ifdef USE_ALARM_TIMEOUT 30 if(data->set.no_signal) 31 /* Ignore the timeout when signals are disabled */ 32 timeout = 0; 33 else 34 timeout = timeoutms; 35 36 if(!timeout) 37 /* USE_ALARM_TIMEOUT defined, but no timeout actually requested */ 38 return Curl_resolv(conn, hostname, port, entry); 39 40 if(timeout < 1000) 41 /* The alarm() function only provides integer second resolution, so if 42 we want to wait less than one second we must bail out already now. */ 43 return CURLRESOLV_TIMEDOUT; 44 45 /************************************************************* 46 * Set signal handler to catch SIGALRM 47 * Store the old value to be able to set it back later! 48 *************************************************************/ 49 #ifdef HAVE_SIGACTION 50 sigaction(SIGALRM, NULL, &sigact); 51 keep_sigact = sigact; 52 keep_copysig = TRUE; /* yes, we have a copy */ 53 sigact.sa_handler = alarmfunc; 54 #ifdef SA_RESTART 55 /* HPUX doesn't have SA_RESTART but defaults to that behaviour! */ 56 sigact.sa_flags &= ~SA_RESTART; 57 #endif 58 /* now set the new struct */ 59 sigaction(SIGALRM, &sigact, NULL); 60 #else /* HAVE_SIGACTION */ 61 /* no sigaction(), revert to the much lamer signal() */ 62 #ifdef HAVE_SIGNAL 63 keep_sigact = signal(SIGALRM, alarmfunc); 64 #endif 65 #endif /* HAVE_SIGACTION */ 66 67 /* alarm() makes a signal get sent when the timeout fires off, and that 68 will abort system calls */ 69 prev_alarm = alarm(curlx_sltoui(timeout/1000L)); 70 71 /* This allows us to time-out from the name resolver, as the timeout 72 will generate a signal and we will siglongjmp() from that here. 73 This technique has problems (see alarmfunc). 74 This should be the last thing we do before calling Curl_resolv(), 75 as otherwise we'd have to worry about variables that get modified 76 before we invoke Curl_resolv() (and thus use "volatile"). */ 77 if(sigsetjmp(curl_jmpenv, 1)) { 78 /* this is coming from a siglongjmp() after an alarm signal */ 79 failf(data, "name lookup timed out"); 80 rc = CURLRESOLV_ERROR; 81 goto clean_up; 82 } 83 84 #else 85 #ifndef CURLRES_ASYNCH 86 if(timeoutms) 87 infof(conn->data, "timeout on name lookup is not supported\n"); 88 #else 89 (void)timeoutms; /* timeoutms not used with an async resolver */ 90 #endif 91 #endif /* USE_ALARM_TIMEOUT */ 92 93 /* Perform the actual name resolution. This might be interrupted by an 94 * alarm if it takes too long. 95 */ 96 rc = Curl_resolv(conn, hostname, port, entry); 97 98 #ifdef USE_ALARM_TIMEOUT 99 clean_up: 100 101 if(!prev_alarm) 102 /* deactivate a possibly active alarm before uninstalling the handler */ 103 alarm(0); 104 105 #ifdef HAVE_SIGACTION 106 if(keep_copysig) { 107 /* we got a struct as it looked before, now put that one back nice 108 and clean */ 109 sigaction(SIGALRM, &keep_sigact, NULL); /* put it back */ 110 } 111 #else 112 #ifdef HAVE_SIGNAL 113 /* restore the previous SIGALRM handler */ 114 signal(SIGALRM, keep_sigact); 115 #endif 116 #endif /* HAVE_SIGACTION */ 117 118 /* switch back the alarm() to either zero or to what it was before minus 119 the time we spent until now! */ 120 if(prev_alarm) { 121 /* there was an alarm() set before us, now put it back */ 122 unsigned long elapsed_ms = Curl_tvdiff(Curl_tvnow(), conn->created); 123 124 /* the alarm period is counted in even number of seconds */ 125 unsigned long alarm_set = prev_alarm - elapsed_ms/1000; 126 127 if(!alarm_set || 128 ((alarm_set >= 0x80000000) && (prev_alarm < 0x80000000)) ) { 129 /* if the alarm time-left reached zero or turned "negative" (counted 130 with unsigned values), we should fire off a SIGALRM here, but we 131 won't, and zero would be to switch it off so we never set it to 132 less than 1! */ 133 alarm(1); 134 rc = CURLRESOLV_TIMEDOUT; 135 failf(data, "Previous alarm fired off!"); 136 } 137 else 138 alarm((unsigned int)alarm_set); 139 } 140 #endif /* USE_ALARM_TIMEOUT */ 141 142 return rc; 143 }
由此可见，DNS解析阶段timeout的实现机制是通过SIGALRM+sigsetjmp/siglongjmp来实现的。
解析前，通过alarm设定超时时间，并设置跳转的标记：
1 /* alarm() makes a signal get sent when the timeout fires off, and that 2 will abort system calls */ 3 prev_alarm = alarm(curlx_sltoui(timeout/1000L)); 4 5 /* This allows us to time-out from the name resolver, as the timeout 6 will generate a signal and we will siglongjmp() from that here. 7 This technique has problems (see alarmfunc). 8 This should be the last thing we do before calling Curl_resolv(), 9 as otherwise we'd have to worry about variables that get modified 10 before we invoke Curl_resolv() (and thus use "volatile"). */ 11 if(sigsetjmp(curl_jmpenv, 1)) { 12 /* this is coming from a siglongjmp() after an alarm signal */ 13 failf(data, "name lookup timed out"); 14 rc = CURLRESOLV_ERROR; 15 goto clean_up; 16 }
在等到超时后，进入alarmfunc函数实现跳转：
1 #ifdef USE_ALARM_TIMEOUT 2 /* 3 * This signal handler jumps back into the main libcurl code and continues 4 * execution. This effectively causes the remainder of the application to run 5 * within a signal handler which is nonportable and could lead to problems. 6 */ 7 static 8 RETSIGTYPE alarmfunc(int sig) 9 { 10 /* this is for "-ansi -Wall -pedantic" to stop complaining! (rabe) */ 11 (void)sig; 12 siglongjmp(curl_jmpenv, 1); 13 return; 14 } 15 #endif /* USE_ALARM_TIMEOUT */
而CURLOPT_NOSIGNAL选项的作用是什么呢？
1 case CURLOPT_NOSIGNAL: 2 /* 3 * The application asks not to set any signal() or alarm() handlers, 4 * even when using a timeout. 5 */ 6 data->set.no_signal = (0 != va_arg(param, long))?TRUE:FALSE; 7 break;
再回过头看看DNS解析的那段代码，你会发现在超时设定前有如下代码：
1 #ifdef USE_ALARM_TIMEOUT 2 if(data->set.no_signal) 3 /* Ignore the timeout when signals are disabled */ 4 timeout = 0; 5 else 6 timeout = timeoutms; 7 8 if(!timeout) 9 /* USE_ALARM_TIMEOUT defined, but no timeout actually requested */ 10 return Curl_resolv(conn, hostname, port, entry);
没错！设置了CURLOPT_NOSIGNAL选项，会把超时时间设置为0，也就是DNS解析不设置超时时间！以此来绕过SIGALRM+sigsetjmp/siglongjmp的超时机制。以此引来新的问题，DNS解析没有超时限制，不过这个官方有推荐的解决方法了。
了解了这些选项的原理之后，回到问题3，为什么使用了CURLOPT_NOSIGNAL选项后就保证了线程安全？继续看sigsetjmp/siglongjmp实现就会发现：
1 #ifdef HAVE_SIGSETJMP 2 /* Beware this is a global and unique instance. This is used to store the 3 return address that we can jump back to from inside a signal handler. This 4 is not thread-safe stuff. */ 5 sigjmp_buf curl_jmpenv; 6 #endif
sigsetjmp/siglongjmp使用的curl_jmpenv是个全局唯一的变量！多个线程都会去修改该变量，破坏了栈的内容并导致coredump。看来这还是libcurl的实现问题，如果每个线程都有一个sigjmp_buf变量，是否就可以解决上面的问题呢？
看到这里，问题2也有了答案：当多个线程同时修改sigjmp_buf会出现问题，但线程间是串行的sigsetjmp/siglongjmp并不会出现问题，这有一定的随机性。
问题1，未设置超时不会有问题这很好理解，但是为什么设置长超时也不会出现问题？
原因就是在libcurl超时前，apache服务器端先超时返回了。apache超时时间一般是180s。只要libcurl超时大于180s，libcurl客户端永远都不会触发超时。而是直接返回504的错误。
是否设置了CURLOPT_NOSIGNAL就可以保证线程安全了呢？官方文档还提到了另外两个函数：
CURL *curl_easy_init( );
This function must be the first function to call, and it returns a CURL easy handle that you must use as input to other easy-functions. curl_easy_init initializes curl and this call MUST have a corresponding call to curl_easy_cleanup(3) when the operation is complete.
If you did not already call curl_global_init(3), curl_easy_init(3) does it automatically. This may be lethal in multi-threaded cases, since curl_global_init(3) is not thread-safe, and it may result in resource problems because there is no corresponding cleanup.
You are strongly advised to not allow this automatic behaviour, by calling curl_global_init(3) yourself properly. See the description in libcurl(3) of global environment requirements for details of how to use this function.
其中curl_easy_init函数体内会调用curl_global_init，而后者是非线程安全的。
在curl_easy_init函数体内，有且仅调用一次curl_global_init：
1 /* 2 * curl_easy_init() is the external interface to alloc, setup and init an 3 * easy handle that is returned. If anything goes wrong, NULL is returned. 4 */ 5 CURL *curl_easy_init(void) 6 { 7 CURLcode res; 8 struct SessionHandle *data; 9 10 /* Make sure we inited the global SSL stuff */ 11 if(!initialized) { 12 res = curl_global_init(CURL_GLOBAL_DEFAULT); 13 if(res) { 14 /* something in the global init failed, return nothing */ 15 DEBUGF(fprintf(stderr, "Error: curl_global_init failed\n")); 16 return NULL; 17 } 18 } 19 20 /* We use curl_open() with undefined URL so far */ 21 res = Curl_open(&data); 22 if(res != CURLE_OK) { 23 DEBUGF(fprintf(stderr, "Error: Curl_open failed\n")); 24 return NULL; 25 } 26 27 return data; 28 }
但是在curl_global_init函数体内，是非线程安全的。initialized++并非原子操作，有可能出现多个线程重复执行curl_global_init。
1 CURLcode curl_global_init(long flags) 2 { 3 if(initialized++) 4 return CURLE_OK; 5 6 /* Setup the default memory functions here (again) */ 7 Curl_cmalloc = (curl_malloc_callback)malloc; 8 Curl_cfree = (curl_free_callback)free; 9 Curl_crealloc = (curl_realloc_callback)realloc; 10 Curl_cstrdup = (curl_strdup_callback)system_strdup; 11 Curl_ccalloc = (curl_calloc_callback)calloc; 12 #if defined(WIN32) && defined(UNICODE) 13 Curl_cwcsdup = (curl_wcsdup_callback)_wcsdup; 14 #endif 15 16 if(flags & CURL_GLOBAL_SSL) 17 if(!Curl_ssl_init()) { 18 DEBUGF(fprintf(stderr, "Error: Curl_ssl_init failed\n")); 19 return CURLE_FAILED_INIT; 20 } 21 22 if(flags & CURL_GLOBAL_WIN32) 23 if(win32_init() != CURLE_OK) { 24 DEBUGF(fprintf(stderr, "Error: win32_init failed\n")); 25 return CURLE_FAILED_INIT; 26 } 27 28 #ifdef __AMIGA__ 29 if(!Curl_amiga_init()) { 30 DEBUGF(fprintf(stderr, "Error: Curl_amiga_init failed\n")); 31 return CURLE_FAILED_INIT; 32 } 33 #endif 34 35 #ifdef NETWARE 36 if(netware_init()) { 37 DEBUGF(fprintf(stderr, "Warning: LONG namespace not available\n")); 38 } 39 #endif 40 41 #ifdef USE_LIBIDN 42 idna_init(); 43 #endif 44 45 if(Curl_resolver_global_init() != CURLE_OK) { 46 DEBUGF(fprintf(stderr, "Error: resolver_global_init failed\n")); 47 return CURLE_FAILED_INIT; 48 } 49 50 #if defined(USE_LIBSSH2) && defined(HAVE_LIBSSH2_INIT) 51 if(libssh2_init(0)) { 52 DEBUGF(fprintf(stderr, "Error: libssh2_init failed\n")); 53 return CURLE_FAILED_INIT; 54 } 55 #endif 56 57 if(flags & CURL_GLOBAL_ACK_EINTR) 58 Curl_ack_eintr = 1; 59 60 init_flags = flags; 61 62 return CURLE_OK; 63 }
所以curl_global_init需要在单线程中执行，例如在程序的开头。
最后，贴一个官方给出的多线程例子，稍作修改（docs/examples/multithreads.c）：
1 /*************************************************************************** 2 * _ _ ____ _ 3 * Project ___| | | | _ \| | 4 * / __| | | | |_) | | 5 * | (__| |_| | _ <| |___ 6 * \___|\___/|_| \_\_____| 7 * 8 * Copyright (C) 1998 - 2011, Daniel Stenberg, <daniel@haxx.se>, et al. 9 * 10 * This software is licensed as described in the file COPYING, which 11 * you should have received as part of this distribution. The terms 12 * are also available at http://curl.haxx.se/docs/copyright.html. 13 * 14 * You may opt to use, copy, modify, merge, publish, distribute and/or sell 15 * copies of the Software, and permit persons to whom the Software is 16 * furnished to do so, under the terms of the COPYING file. 17 * 18 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY 19 * KIND, either express or implied. 20 * 21 ***************************************************************************/ 22 /* A multi-threaded example that uses pthreads extensively to fetch 23 * X remote files at once */ 24 25 #include <stdio.h> 26 #include <pthread.h> 27 #include <curl/curl.h> 28 29 #define NUMT 4 30 31 /* 32 List of URLs to fetch. 33 34 If you intend to use a SSL-based protocol here you MUST setup the OpenSSL 35 callback functions as described here: 36 37 http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION 38 39 */ 40 const char * const urls[NUMT]= { 41 "http://curl.haxx.se/", 42 "ftp://cool.haxx.se/", 43 "http://www.contactor.se/", 44 "www.haxx.se" 45 }; 46 47 static void *pull_one_url(void *url) 48 { 49 CURL *curl; 50 51 curl = curl_easy_init(); 52 curl_easy_setopt(curl, CURLOPT_URL, url); 53 curl_easy_setopt(curl, CURLOPT_TIMEOUT, 30L); /*timeout 30s,add by edgeyang*/ 54 curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L); /*no signal,add by edgeyang*/ 55 curl_easy_perform(curl); /* ignores error */ 56 curl_easy_cleanup(curl); 57 58 return NULL; 59 } 60 61 62 /* 63 int pthread_create(pthread_t *new_thread_ID, 64 const pthread_attr_t *attr, 65 void * (*start_func)(void *), void *arg); 66 */ 67 68 int main(int argc, char **argv) 69 { 70 pthread_t tid[NUMT]; 71 int i; 72 int error; 73 74 /* Must initialize libcurl before any threads are started */ 75 curl_global_init(CURL_GLOBAL_ALL); 76 77 for(i=0; i< NUMT; i++) { 78 error = pthread_create(&tid[i], 79 NULL, /* default attributes please */ 80 pull_one_url, 81 (void *)urls[i]); 82 if(0 != error) 83 fprintf(stderr, "Couldn't run thread number %d, errno %d\n", i, error); 84 else 85 fprintf(stderr, "Thread %d, gets %s\n", i, urls[i]); 86 } 87 88 /* now wait for all threads to terminate */ 89 for(i=0; i< NUMT; i++) { 90 error = pthread_join(tid[i], NULL); 91 fprintf(stderr, "Thread %d terminated\n", i); 92 } 93 94 curl_global_cleanup(); /*add by edgeyang*/ 95 return 0; 96 }
