the Fiddler's picture

Add unit tests to the OpenTK.Math

Project:The Open Toolkit library
Version:0.9.x-dev
Component:Code
Category:task
Priority:normal
Assigned:Unassigned
Status:open
Description

Right now, there is no rigorous way to test OpenTK.Math. Bugs may difficult to catch and may go unnoticed for a long time.

Is there anyone familiar with NUnit, who can describe how it can be integrated into a project and how to write unit tests for it? Can it be integrated into the build system so the tests run automatically? What kind of functionality can it test?


Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
objarni's picture

#1

Dogfight uses NUnit to test some of it's code:

svn checkout http://dogfight2008.googlecode.com/svn/trunk/ dogfight2008-read-only

In that project, I embedded NUnit. Because of that, all I need is a computer with Visual Studio to build Dogfight2008. NUnit is about 2mb and can be stripped down (remove .net1.1 binaries, docs, etc..) if that is too heavy.

Is there anyone familiar with NUnit, who can describe how it can be integrated into a project and how to write unit tests for it? Can it be integrated into the build system so the tests run automatically? What kind of functionality can it test?
What it can test: anything that can be expressed as a boolean expression, generally speaking.

A class used to test another class is called a Fixture in NUnit, eg. class SomeClass would have a class called SomeClassFixture containing the tests for SomeClass. Here is an example from Dogfight, a class to test my math utilities class called Calc:
CalcFixture.cs [unit test]
Calc.cs [unit]

The way I use NUnit:

1. For each project (.csproj), I have a "sibling project" called ProjectName.Tests.dll
2. ProjectName.Tests.dll references ProjectName.dll and nunit.framework.dll and contains all tests
3. ProjectName.Tests.dll has a post-build event "nunit-console.exe $(TargetPath)" which runs all tests after the project has been successfully built
4. If any tests fail, nunit-console returns a non-zero value reported as an error in the Error window, then I can check the Output window to see exactly what tests failed
5. If all tests pass, nunit-console returns zero so no error is reported in the Error window

I guess all this should be translateable to Ant?

the Fiddler's picture

#2

Thanks for the info, I'll have to see what Prebuild supports in the way of post-build events.

kanato's picture

#3

So I've always been curious about Nunit but I've never really looked into it. What does it give us that just using System.Diagnostics.Debug.Assert doesn't?

Inertia's picture

#4

Cookies. And an additional dependency.

I'd rather write a larger test app that benchmarks the functions and only does a few checks to confirm the function's correctness first. What's the point of writing unit tests for most of the Vector*** structs functions? Most are so simple that they cannot possibly be faulty.

the Fiddler's picture

#5

Most are so simple that they cannot possibly be faulty.
Famous last words :)

Myself, I'm more worried about matrices and quaternions. I've been having some problems with the latter and I cannot pinpoint whther it's my fault or I'm hitting a bug. I recall you were having similar problems with an .X loader.

I admit I have no experience wrt to unit-testing, which is why I was looking at the solution used by the Mono guys (nunit). As far as I can see, nunit defines a testing interface and provides tools that interact with that. I have a feeling that a home-made solution would fare worse than nunit from a maintenance standpoint.

objarni's picture

#6

NUnit provides more readable unit test code than using only Debug.Assert(), but of course in essence (from a mathematical standpoint) there is no difference in functionality.

Adding NUnit dependency to the OpenTK development branch does not imply adding a single byte to neither OpenTK.dll nor OpenTK.Utilities.dll (since test projects are separate from production projects, see my first reply above). It is simply like Prebuild - a tool used to develop OpenTK.

And if you are dareful enough to include NUnit binaries in the branch, it does not even imply installing or adding any hassle to OpenTK developers. The only thing added is 2 mb extra download first time you checkout opentk-dev (SVN is smart enough to not perform this download more than once to your computer).

So unless you are on a 300 baud modem I wouldn't worry about the dependency on the open source (and rather popular) NUnit unit testing tool.

What has to be done for this to work:

1) learn the basics of NUnit (it is minimalistic, I can provide support since I've used the tool for two years..)
2) modify build script to include post-compile-and-link step to run nunit-console on each test project
3) build some test fixtures for Vector, Matrix, Quaternion

objarni's picture

#7

If you want to try VisualStudio+NUnit with the post-build-event that runs nunit-console.exe you can download my "template solution" here:

http://dl.getdropbox.com/u/385840/SolutionTemplate-NUnit-stripped.zip

The .zip file contains a "TemplateSolution" (it is not really a Visual Studio template -- it is just a .zip of my empty solution, which I take a copy of when starting new projects) with NUnit 2.5 stripped down quite a lot (1,6 mb unpacked), all set to go.

For example, try changing Expect(true) to Expect(false) in the Fixture file, push F5 and see the error/output windows of Visual Studio.

Inertia's picture

#8

The point is that we're adding a dependency to OpenTK which is only serving to verify that the Math functions were not changed. It cannot be used to verify GL, AL, CL, fonts etc.

My suggestion was trying to kill 2 flies with 1 hit: verify correctness and doing a benchmark. It would not have to output any graphics (Console), does not have to upload to an online database for comparisons or stuff like that. However it could take advantage of a GL context to calculate matrices (GL.Rotate etc.) which can be used to verify against.

objarni's picture

#9

Using GL.Rotate and friends is one way of checking the correctness of Matrix4 algorithms.

One idea is writing an example "Maths SelfTests" which runs unit tests - either via Debug.Assert, NUnit or some other technique.

Unit test code should be written for algorithms that are OpenTKs responsibility, eg. the maths routines. Writing unit tests for the pinvoke API - GL and AL - is pointless. [Though help-algorithms (if any) in the GL/AL code of OpenTK could have unit tests.]

Base rule is test your own code - rely on third party code (GL and AL in OpenTK's case).

NUnit does not upload anything to any database. It only runs unit test code. But if you are sensitive about this dependency (which really is a build dependency, not a library dependency - without even needing to install anything - I just don't think it is a heavy burden to bear for opentk developers - but that is my opinion), I absolutely think having some unit tests (for both correctness and performance) is way better than having none.

Here are some "rules of thumb" regarding unit testing, just wanted to share some of what I've learned:

The reason to write Unit Tests is to increase quality of developed algorithms.

The reason to run Unit Tests often is to decrease the likelyhood of introducing bugs in existing code, at a lot lower cost (in programmer time) than performing manual testing after every change to the code - letting the computer do what it is supremely good at compared to humans - repetitive work at an insanely fast rate, with close to zero calculation errors.

Think of running post-build unit tests as an extension to the compiler - adding algebraic or semantic rules to be fullfilled by the code, not just syntax/type correctness. Like doing a dynamic analysis of the code, not just a static one. Of course it is limited to "pointwise tests" (we cannot test anything completely - for example testing 'substring' algorithm for correctness) but testing a function at two input points is a lot better than testing it at zero input points.

djk's picture

#10

I am going to add my 2 cents, we have just under 5000 visualStudio unit tests for the 15 assemblies in our system. Our policy is to write unit tests for all non-UI classes, and have found a handful of UI related classes that could be tested. In our case, I ran the complete suite of tests on a routine basis. We are currently setting up a build server that will automatically run the unit tests every night, failures will be emailed to the developers.

On a side note, recently a change was made to the persistence mechanism that was thought to be relatively minor and well contained. I was off working another project for a few weeks and came back to everyone scurrying around trying to locate and fix all the breakage. No one had run the unit test, which when I ran them it quickly showed the extent and nature of the problem, which turned out to be significant. Hence the new build server mentioned above.

Now, onto NUint and OpenTK. All the test code can be located in a separate assembly which can be off on another branch of the svn tree and only those who want to use the tests would have to download them.

I strongly encourage this practice and am willing to donate some time to write unit tests if someone else can organize the unit test project and parse out specific classes to work on.

djk