From a student who went to Java One and looked into Java code testing... ------------------------------------------------------- http://www.suntest.com/100percent/download.html Now http://www.sun.com/suntest If you are writing Java programs that you hope will ... you will want to do a JavaPureCheck. It reads your programming looking for hard-coded file names, mixing event models (), using classes out of the Sun libraries (change from release to release and are dangerous), ingesting classes into a system package. There is a frequently used little Java idom that hides exceptions: try { ... } catch {} Instead, let error bubble up to the top where they can get reported to the users and ultimately fixed. This may be ok for throw-away code. Can do concurrent test engineering. Can review test interface, can review for testability, can add some methods, for example, to objects that are mostly write-only interfaces to allow you to get at the data for testing. Do decent engineering. Think about your interfaces up front and stabilize them. Build your tests up front. You will change your interfaces, and you will have to change your tests. You don't know everything when you start. ... Don't write one giant test method that tests all of the fields of an object, but instead write multiple small objects so your changes are more contained. Strong type checking is handy when you are changing type definitions, because it will catch errors. Write durable tests. Make your tests durability across revisions. Does everyone use SetName on your AWT components -- if not your components will not be internationalizable. It will also make your tests more durable. Record your interface using an English language interface and play your test back using the French version. That is an example of a durable test across internationalization. Parameterize your tests. Instead of hard coding, ... For example, if your program gets data from a file, send in the information to the test as a parameter -- send this "Fred" and see if "Fred" is displayed, sent this "John" and see of "John" is displayed. The 100% Pure Java program is intended to help people write Java programs that run everywhere. It can and does check for compliance with the ideal, to see if you have "intentionally" wired your program to a particular platform. 100% Pure (JavaPureCheck) advises you of such things. Testing is representative of the user. If you didn't write once and test everywhere, you would be able to "run" everywhere. "Running" is, in effect, testing. If you don't test everywhere, you aren't supporting your customers who are running from everywhere. They have a lot of ideas, but they don't know if they can make money off of it. They are trying to make some money. They are spending more money than they are making, but they are striving to make money. Threads: You can't test threads. You have to think them out. How can you test for race conditions or deadlocks? You can probably test for deadlocks, but it isn't easy. A good reference is "Concurrent Programming in Java" from the Addison Wesley series. Don't tell anyone, but they are working on something. They think they can do enough analysis of your program to be able to tell you that there is a possible race condition or a possible deadlock issue. Are you familiar with the waterfall model? Good luck. That model is outmoded. Consider the spiral model with continued feedback with the customer and there is continued refinement. "Spiral QA" incorporates QA into that spiral development model. Plan for test revisiions, incorporate QA and test people into your cycle, incorporate test development people into your team and into your spiral cycle. This allows you to get to the customer quicker because you have been doing QA all along. By incorporating quality assurance all along, you will catch problems earlier and will minimize the number of big changes that are caused by not finding out about problems until the very end that have a significant structural impact. It also give QA an opportunity to educate the designers about what are or are not good programming practices. Re: Beta testing: I'm ambivalent. My professional pride makes my skin crawl, however. Don't we care enough about the user to test it more thoroughly before it goes to market. However, until the customer really tries it, you will never know if you really understood the customers' requirements. It works better for Microsoft than it does for Joe and Betty in the garage, because they are innovators. This extended Beta is good for innovators, but at the same time, extended beta works against innovators by not allowing them to get their innovative product to market sooner. Regressions tests are very important to avoid re-incorporating the same errors again. Four kinds of program testers: Program checkers (like type-checking compilers, lint, JavaPureCheck, etc.) are the very best kind of testing because it catches things earlier, is comparatively painless, and is relatively certain because you get 100% coverage. Testing is essentially statistical sampling, because you can't test everything. Test Generators -- things that help to spit out a test. They perform the test, get the results back, and perform some checking between what was received back and what was predicted to be return. When you are testing, don't get stuck on the predicate "equal" (==). Often, other predicates are more meaningful or less expensive than testing for equal. For example, when testing floating point results, you want to test for a range and not for equals. Test Execution Frameworks -- Often just gather results, but some will actually associate tests with particular sections of source code that have changed. Test Design Tools -- will analyze the code and help you to create tests. Test Measurement Tools -- such as coverage tests determine how good a job of testing you have done. Test Automation: Tools are automation. It is good and bad. It is good because it legislates what is done and results in better repeatability and more consistent results. You get better leverage because there is one button that says "Do this test", or "Do all theses tests", as opposed to having someone manually trying do the same thing all the time, or in trying to remember what you did to repeat a bug. Test automation makes it easier to . Disadvantages because it takes longer to get the results because you have to configure the tools. Sometimes you are further away from the bug because you have more between you and the bug. The initial cost to build the framework. There is the cost of the test system and the learning curve for using it. Test tools are automation, but not oracle. If you are not investing very much in testing, then you are better off with manual testing because you won't get over the hump to automate testing. If you can afford to spend more up front, then you will get a better return on your investment after the investment. Explicit Plug for Sun's Test Products: JavaStar for Java is a lot like Xrunner, except that it is in Java. Since it is in Java, you can modify the source to include loops or other automation. You have access to all of the fields and all of the methods, unlike other systems where the test system doesn't know about the implementation object. JavaSpec is for unit or API testing if you have a library, etc. It allows you to build method by method with a kind of precondition/postcondition kind of specification -- e.g., setup code, assertions, etc. JavaScope is our coverage analysis. It works by rewriting your source to insert coverage metering code that report back to the JavaScope coverage database. The other way to get this kind of information is to instrument the virtual machine. The JavaScope has some disadvantages in that you have to perturb your code and that timing information isn't available. Advantage is that it will work on any virtual machine. It generates branch coverage counts, but not timing. Quantify for Java runs on the Microsoft VM. JavaLoad is in beta now, is written in Java, and will be announced at JavaONE. JavaSpec How do you test your Test Tools? We use our test tools to test our test tools, and they achieve about 60-70% statement coverage, but he doesn't know for certain.