[转] Performance vs. load vs. stress testing _Grig Gheorghiu (翻译水平有限,如有错误请帮忙刊正)

作者：Grig Gheorghiu
作者联系方式：http://agiletesting.blogspot.com/

I recently got some comments/questions related to my previous blog entry on performance vs. load vs. stress testing. Many people are still confused as to exactly what the difference is between performance and load testing. I've been thinking more about it and I'd like to propose the following question as a litmus test[试金石] to distinguish between these two types of testing: are you actively profiling your application code and/or monitoring the server(s) running your application? If the answer is yes, then you're engaged in performance testing. If the answer is no, then what you're doing is load testing.

[译：我上一篇发表在自己博客上的文章关于性能、负载、压力的测试文章收到了一些问题和评论，很多人对如何正确的区分性能测试与负载测试仍然很迷惑。为此我思考了一段时间，并且愿意将如下的问题做为试金石来区分出这两类测试：你是积极的剖析你的程序代码或者监测正运行你程序服务器的服务器吗？如果答案是肯定的，则你正从事性能测试。如果答案是否定的，则你做的是负载测试]

Another way to look at it is to see whether you're doing more of a white-box type testing as opposed to black-box testing. In the white-box approach, testers, developers, system administrators and DBAs work together in order to instrument the application code and the database queries (via specialized profilers for example), and the hardware/operating system of the server(s) running the application and the database (via monitoring tools such as vmstat, iostat, top or Windows PerfMon). All these activities belong to performance testing.

[译：从另一方面考虑它的方法是是否你相对于黑盒测试做了更多的白盒测试的工作。在白盒测试里，测试人员，开发人员，系统管理员，数据库管理员在一起工作为了实现程序和数据库查询（例如通过特殊的模具）,而且应用程序和数据运行在服务器(组)上的硬件/操作系统上(通过监视工具如vmstat, iostat, top or Windows PerfMon等).所以这些动作归属为性能测试。]

In practice though the 2 terms are often used interchangeably, and I am as guilty as anyone else of doing this, since I called one of my recent blog entries "HTTP performance testing with httperf, autobench and openload" instead of calling it more precisely "HTTP load testing". I didn't have access to the application code or the servers hosting the applications I tested, so I wasn't really doing performance testing, only load testing.

[译：在实践中这两种术语经常被交替使用，我也愧疚的这样做过，因为我在最近的一篇"HTTP performance testing with httperf, autobench and openload" 的文章题目应该更合适的替换为"HTTP load testing"，我没有接触被测试程序的源码或者服务器主机，所以我没有真正的做性能测试而仅是负载测试]

The black box approach is to run client load tools against the application in order to measure its responsiveness. Such tools range from lightweight, command-line driven tools such as httperf, openload, siege, Apache Flood, to more heavy duty tools such as OpenSTA, The Grinder, JMeter. This type of testing doesn't look at the internal behavīor of the application, nor does it monitor the hardware/OS resources on the server(s) hosting the application. If this sounds like the type of testing you're doing, then I call it load testing.

[译：在黑盒测试过程中是运行客户负载工具来测试程序的反应。 此类工具从轻量级，命令行驱动工具如httperf, openload, siege, Apache Flood 到更重量级工具如OpenSTA, The Grinder, JMeter. 此类测试并不关注程序的内在运行， 也不监视运行在服务器（群）上的硬件/操作系统的资源。如果听起来是此类的你正在进行的工作，则我称它为负载测试。]

I think part of the confusion is that no matter how you look at these two types of testing, they have one common element: the load testing part. Even when you're profiling the application and monitoring the servers (hence doing performance testing), you still need to run load tools against the application, so from that perspective you're doing load testing.

[译：我想有部分的疑惑是无论你如何看待这两类测试，它们有一些通用的元素：负载测试部分。甚至当你正模压程序并监视服务器（因此进行性能测试），你仍然需要去运行负载工具测试运行程序，所以从这个观点上你正在进行负载测试。]

As far as I'm concerned, these definitions don't have much value in and of themselves. What matters most is to have a well-established procedure for tuning the application and the servers so that you can meet your users' or your business customers' requirements. This procedure will use elements of all the types of testing mentioned here and in my previous entry: load, performance and stress testing.

[译：到目前我关心的，这些定义本身不具有价值．问题的关键是让一个良好建立的过程用于调试应用程序和服务器以使自身可以满足用你用户或者商业客户的需求．这个过程将会使用所有在这提到过的测试类型和我上一篇文章：负载，性能和压力测试]

Here's one example of such a procedure. Let's say you're developing a Web application with a database back-end that needs to support 100 concurrent users, with a response time of less than 3 seconds. How would you go about testing your application in order to make sure these requirements are met?

[译：举一个此类程序的例子。 假设你正在开发一个web应用程序和一台终端服务器且需要支持100名并发用户，且响应时间小于3妙。 你如何进行测试使它达符合这些要求呢？]

1. Start with 1 Web/Application server connected to 1 Database server. If you can, put both servers behind a firewall, and if you're thinking about doing load balancing down the road, put the Web server behind the load balancer. This way you'll have one each of different devices that you'll use in a real production environment.

[译：1 开始使用web或者程序服务器连接到一台数据库服务器。如果你可以的话,将两台服务置于防火墙后,要是你考虑以此方法做负载均衡将WEB服务器置于负载器后面。这种方法将你将会使你的每一个不同的设备都在一个真实的产品环境中。]

2. Run a load test against the Web server, starting with 10 concurrent users, each user sending a total of 1000 requests to the server. Step up the number of users in increments of 10, until you reach 100 users.


[译：2 对一台WEB服务器运行一个负载测试,以10个并发用户开始,每个用户向服务器发出1000个请求,设置用户增量为10,直到达到100个用户为止.]

3. While you're blasting the Web server, profile your application and your database to see if there are any hot spots[是否译成拐点比较好] in your code/SQL queries/stored procedures that you need to optimize. I realize I'm glossing over important details here, but this step is obviously highly dependent on your particular application.

[译：3 当你正在冲击WEB 服务器时,监视描绘你的应用程序和你的数据库,观察在你的代码或存储过程是否有拐点需要你优化. 我知道我在这掩饰了很多细节,但此一步很大程序程度上取决于特定的应用程序. ]

4. Also monitor both servers (Web/App and Database) via command line utilities mentioned before (top, vmstat, iostat, netstat, Windows PerfMon). These utilities will let you know what's going on with the servers in terms of hardware resources. Also monitor the firewall and the load balancer (many times you can do this via SNMP) -- but these devices are not likely to be a bottleneck at this level, since they usualy can deal with thousands of connections before they hit a limit, assuming they're hardware-based and not software-based.

[译：4 同样通过前面提过的命令行工具来监视应用程序和数据. 这些工具会使你了解就硬件资源上的一些情况. 还要监视防火墙和负载器(多数情况下你可以使用SNMP)--但这些设备在这个层面上不会成为一个瓶颈,因为它们常常可以处理数千个连接在达到限制前,假设它们是硬件增强而不是软件增强的.(有问题)]

5.This is one of the most important steps in the whole procedure. It's not easy to make sense of the output of these monitoring tools, you need somebody who has a lot of experience in system/network architecture and administration. On Sun/Solaris platforms, there is a tool called the SE Performance Toolkit that tries to alleviate this task via built-in heuristics that kick in when certain thresholds are reached and tell you exactly what resource is being taxed.

[译：5  在整个过程中这是一个最重要的步骤,理解这些监视工具的输出并不容易,需要某人具有很深的系统/网络 结构和管理经验. 在SUN/Solais 平台上有一款名为SE Performace Tookit的工具它可以操作这类工作通过嵌入探索当某些启动项已经达到了并通知你资源真正的负载.]

6. Let's say your Web server's reply rate[回应速度] starts to level off around 50 users. Now you have a repeatable condition that you know causes problems. All the profiling and monitoring you've done in step 3, should have already given you a good idea about hot spots in your applicationm about SQL queries that are not optimized properly, about resource status at the hardware/OS level.

[译：6  如果说你的WEB 服务器的响应率以维持在50人左右开启.现在你有了一个可以复用的引起问题的条件.所有在第三步进行的剖析和监视,应当已经给你提供了一个很好的想法关于在你的应用程序和SQL查询的拐点当其没有适当优化及硬件/操作系统级别上的资源状态.]


At this point, the developers need to take back the profiling measurements and tune the code and the database queries. The system administrators can also increase server performance simply by throwing more hardware at the servers -- especially more RAM at the Web/App server in my experience, the more so if it's Java-based.

[译：7 在这点上,开发人员需要回溯到管理]

5. Let's say the application/database code, as well as the hardware/OS environment have been tuned to the best of everybody's abilities. You re-run the load test from step 2 and now you're at 75 concurrent users before performance starts to degrade.

At this point, there's not much you can do with the existing setup. It's time to think about scaling the system horizontally, by adding other Web servers in a load-balanced Web server farm[web 服务群组], or adding other database servers. Or maybe do content caching, for example with Apache mod_cache. Or maybe adding an external caching server[高速缓存服务器] such as Squid.

One very important product of this whole procedure is that you now have a baseline number for your application for this given "staging" hardware environment. You can use the staging setup for nightly peformance testing runs that will tell you whether changes in your application/database code caused an increase or a decrease in performance.[真正的接触到了开发层面的成本意义]

6. Repeat above steps in a "real" production environment before you actually launch your application.

All this discussion assumed you want to get performance/benchmarking numbers for your application. If you want to actually discover bugs and to see if your application fails and recovers gracefully, you need to do stress testing. Blast your Web server with double the number of users for example. Unplug network cables randomly (or shut down/restart switch ports via SNMP). Take out a disk from a RAID array. That kind of thing.

The conclusion? At the end of the day, it doesn't really matter what you call your testing, as long as you help your team deliver what it promised in terms of application functionality and performance. Performance testing in particular is more art than science, and many times the only way to make progress in optimizing and tuning the application and its environment is by trial-and-error[反复试验] and perseverance. Having lots of excellent open source tools also helps a lot.