Lean web services using C, FastCGI and MySQL

http://yaikhom.com/2014/09/13/lean-web-services-using-c-fast-cgi-and-mysql.html


September 13, 2014

I like theC programming language.What I like most is that with C you have to be very explicit aboutwhat you would like the computer to do. Consequently, there are fewersurprises compared to other programming languages that I have used. Ialso like the fact that I can recall most of the syntax and semanticsof the language from the top-of-my-head. Finally, if something goeswrong, I like the certainty of being able to debug and pinpoint thecause of failure by simply going through the source code. So, it got methinking:what if I wanted to write lean web services using C? Theseare my notes for future reference.

I have used simpler modern frameworks that are based on Java, PHP andJSP. They are easy to develop with initially but debugging and findingout performance bottlenecks or memory leaks is usually quite ahassle because there are so many components interacting atrun-time at different levels-of-abstraction. I also find that as thecomplexity of the framework grows, the log details become more crypticand less helpful. Furthermore, all of these frameworks can be veryresource intensive. If, say, I wished to replicate several servernodes for scalability on Amazon EC2, thecosts of maintaining several resource hungry applications can beextremely prohibitive. I would have to choose nodes that aresufficiently geared to handle the resource demands, which are moreexpensive to run. So, the question is, is there a way I couldreplicate the same functionalities and scalability using a much leanerframework, a framework that is less resource intensive. This got meinterested to learn more about the older technologies, which existedwhen our servers were not very powerful.

Before I proceed, I wish to apologise for the lack of quantitativeperformance metrics to justify my claims. Perhaps, as I learn more, Ishould be able to gather evidence to support or refute myhypothesis. For now, these are merely hunches based on personalexperience.

With FastCGI, I am hoping that the web servicescould be simple light-weight processes, and hence scaling these withcheaper server nodes would bring the cost significantly down. The cost ofinitially developing the application would inevitably be higher(if this was not the case, why would anyone invent simpler frameworksin the first place); however, once deployed, the long-term pay-off ofrunning and maintaining a lean cost-effective application is a seriousproposition. After all, the costs of running the servers are recurringand surely must affect the profit margin.

Based on my experience developing and usingRESTful web services,I have learned that the key is in defining a clear and effectiveapplication programming interface (API), with which the client-sideJavaScript code can interact efficiently. Once the APIs are welldefined, I find that writing the web services is not thatchallenging. I am hoping that the same would be the case when using Cand FastCGI, albeit with a bit more programming, which I do notmind. With more computational power becoming available at the hands ofthe customers, it would be prudent to leverage this for a much leanerserver framework that are tuned perfectly to collecting and serving dataas fast and cheaply as possible. I hope that this turns out to be aninteresting journey for me.

I studied several blogs and articles to learn FastCGI and how it can bemade to work with Apache, MySQL and C. I have listed these in theReferences section. All I did here is consolidate the main lessonsbased on my hands-on experience. While the examples I havepresented are quite simple, I wanted to get something up-and-runningas soon as I could. I hope that you'll find this note useful.

Create the workspace

In this note, I am using Mac OSX 10.9.4 for development. We shall doall of our work inside the~/fastcgi_learn directory.

<span style="font-size:18px;"><span style="font-size:24px;">$ mkdir -p ~/fastcgi_learn
$ mkdir -p ~/fastcgi_learn/lib
$ mkdir -p ~/fastcgi_learn/mods
$ mkdir -p ~/fastcgi_learn/progs</span></span>

Installing FastCGI library

<span style="font-size:18px;"><span style="font-size:24px;">$ cd ~/fastcgi_learn/lib
$ curl -o fcgi.tar.gz http://www.fastcgi.com/dist/fcgi.tar.gz
$ tar xvozf fcgi.tar.gz
$ cd fcgi-2.4.1-SNAP-0311112127/
$ ./configure
$ make
$ sudo make install</span></span>

Installing Apache mods for FastCGI

<span style="font-size:18px;"><span style="font-size:24px;">$ cd ~/fastcgi_learn/mods
$ curl -o mod_fastcgi-current.tar.gz \
  http://www.fastcgi.com/dist/mod_fastcgi-current.tar.gz
$ tar xvozf mod_fastcgi-current.tar.gz
$ cd mod_fastcgi-2.4.6/
$ sudo apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c \
  fcgi_pm.c fcgi_protocol.c fcgi_util.c</span></span>

If you get the "apxs:Error: Command failed with rc=65536" error during FastCGI mod compilation, the fix is as follows:

<span style="font-size:18px;"><span style="font-size:24px;">$ sudo ln -s \
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/ \
  /Applications/Xcode.app/Contents/Developer/Toolchains/OSX10.9.xctoolchain</span></span>

Set up FastCGI mod

<span style="font-size:18px;"><span style="font-size:24px;">$ sudo emacs -nw /private/etc/apache2/httpd.conf</span></span>

Replace
<span style="font-size:18px;"><span style="font-size:24px;">LoadModule mod_fastcgi_module libexec/apache2/mod_fastcgi.so</span></span>

with
<span style="font-size:18px;"><span style="font-size:24px;">LoadModule fastcgi_module libexec/apache2/mod_fastcgi.so</span></span>

At the end of file, add:

<span style="font-size:18px;"><span style="font-size:24px;"><IfModule mod_fastcgi.c>
        FastCgiIpcDir /tmp/fcgi_ipc/
        AddHandler fastcgi-script .fcgi
</IfModule></span></span>

Create the web server communication channel

<span style="font-size:18px;"><span style="font-size:24px;">$ sudo mkdir /tmp/fcgi_ipc
$ sudo chmod 777 /tmp/fcgi_ipc
$ sudo apachectl restart
$ sudo apachectl -t</span></span>

Compile an example

<span style="font-size:18px;"><span style="font-size:24px;">$ cd ~/fastcgi_learn/progs
$ emacs -nw counter.c</span></span>

Type in the following example code:

<span style="font-size:18px;"><span style="font-size:24px;">#include <fcgi_stdio.h>
#include <stdlib.h>
int main(void)
{
    int count = 0;
    while(FCGI_Accept() >= 0)
        printf("Content-type: text/html\r\n\r\n"
               "Request number %d on host <i>%s</i>\n",
               ++count, getenv("SERVER_NAME"));
    return 0;
}</span></span>

Compile the program

<span style="font-size:18px;"><span style="font-size:24px;">$ gcc -o counter.fcgi counter.c -lfcgi</span></span>

Deploy application

<span style="font-size:18px;"><span style="font-size:24px;">$ sudo cp counter.fcgi /Library/WebServer/CGI-Executables/counter.fcgi
$ sudo apachectl restart</span></span>

Check if it is working http://127.0.0.1/cgi-bin/counter.fcgi. Refreshing the browser should increment the counter.

Example MySQL application

Create the database

<span style="font-size:18px;"><span style="font-size:24px;">create database fastcgi_test;
use fastcgi_test;
create table fruits (
    id int not null auto_increment,
    name varchar(64) not null,
    primary key (id)
);
grant all on fastcgi_test.* to 'testuser'@'localhost' identified by 'testuser';
flush privileges;
insert into fruits (name) values ("Apple"), ("Orange"), ("Banana");</span></span>

The application source

To keep the source code small, we are forgoing error checks.

<span style="font-size:18px;"><span style="font-size:24px;">#include <fcgi_stdio.h>
#include <mysql.h>
#include <stdlib.h>

int main(void)
{
    MYSQL *con = mysql_init(NULL);
    mysql_real_connect(con, "localhost", "testuser", "testuser",
                       "fastcgi_test", 0, NULL, 0);

    while(FCGI_Accept() >= 0) {
            printf("Content-type: application/json\r\n\r\n");
            mysql_query(con, "SELECT * FROM fruits");
            MYSQL_RES *result = mysql_store_result(con);
            int num_fields = mysql_num_fields(result);
            MYSQL_ROW row;
            printf("{names:[");
            int count = 0;
            while ((row = mysql_fetch_row(result))) {
                    printf("%s{", count++ ? "," : "");
                    for (int i = 0; i < num_fields;){
                            printf("\"%d\":\"%s\"", i, row[i] ? row[i] : "");
                            if (++i < num_fields)
                                    printf(",");
                    }
                    printf("}");
            }
            printf("]}");
            mysql_free_result(result);
    }

    mysql_close(con);
    mysql_library_end();
    return 0;
}</span></span>

Compilation and deployment

<span style="font-size:18px;"><span style="font-size:24px;">$ gcc -o example example.c -lfcgi -I /usr/local/mysql/include/ \
  -L /usr/local/mysql/lib/ -lmysqlclient -lm -lz
$ sudo cp example /Library/WebServer/CGI-Executables/example.fcgi
$ sudo apachectl restart</span></span>

If you visithttp://127.0.0.1/cgi-bin/example.fcgi,you should see:

<span style="font-size:18px;"><span style="font-size:24px;">{names:[{"0":"1","1":"Apple"},{"0":"2","1":"Orange"},{"0":"3","1":"Banana"}]}</span></span>

Dynamic linking fix

If you get Library not loaded: libmysqlclient.16.dylib error whenrunningexample, the fix is as follows. The following will give thecurrent incorrect library path:

<span style="font-size:18px;"><span style="font-size:24px;">$ otool -DX /usr/local/mysql-5.5.24-osx10.6-x86_64/lib/libmysqlclient.dylib
libmysqlclient.18.dylib</span></span>

You run the following to fix the path:

<span style="font-size:18px;"><span style="font-size:24px;">$ sudo install_name_tool -id /usr/local/mysql/lib/libmysqlclient.18.dylib \
  /usr/local/mysql/lib/libmysqlclient.dylib</span></span>

You should now see the corrected path:

<span style="font-size:18px;"><span style="font-size:24px;">$ otool -DX /usr/local/mysql-5.5.24-osx10.6-x86_64/lib/libmysqlclient.dylib
/usr/local/mysql/lib/libmysqlclient.18.dylib</span></span>

UPDATE

On 3 January 2015, I noticed that after upgrading to OSX 10.10 (Yosemite),the FastCGI modules fromhttp://www.fastcgi.com/ no longercompiles. This is to do with Apache upgrade from version 2.2 to 2.4,which changed the APIs. I tried to check if there is a newer moduleavailable; unfortunately, the FastCGI web-site was down. As analternative, you could use Apache's own FastCGI implementation namedmod_fcgid. If you haveHomebrew installed, you can do the following toinstall Apache'smod_fcgid FastCGI module:

<span style="font-size:18px;"><span style="font-size:24px;">$ brew install homebrew/apache/mod_fcgid</span></span>

Now edit the Apache config as follows:

<span style="font-size:18px;"><span style="font-size:24px;">$ sudo emacs -nw /private/etc/apache2/httpd.conf</span></span>

Add following, using path as advised at the end of the abovebrew command.

<span style="font-size:18px;"><span style="font-size:24px;">LoadModule fcgid_module /usr/local/Cellar/mod_fcgid/2.3.9/libexec/mod_fcgid.so</span></span>

At the end of file, add:

<span style="font-size:18px;"><span style="font-size:24px;"><IfModule fcgid_module>
        AddHandler fcgid-script .fcgi
</IfModule></span></span>

References





posted @ 2016-04-08 23:19  张同光  阅读(139)  评论(0)  编辑  收藏  举报