Unit Testing Framework?

dave_cowden

Alright, I'm a TDD guy, and now that i did my first dixie cup/learning FS file, i'm already bothered by not writing any tests.

Are there any unit testing or test harnesses out there for writing tests for features? I've not really seen any discussion along those lines. It bothers me to have more than 100 lines of production code without any test code....

kevin_o_toole_1

Let me clarify what I meant in my last paragraph:

For a unit test that doesn't require a gui, you don't need to define one. Additionally, you may put many tests into a single block so that they may be run all together.

For the case you described, I would refactor my input computation step into a function:

export function remapInput(definition is map) returns map
{
    // Logic goes here
    return {
        "foo" : definition.foo + 1
    };
}

...and call that function many times from an input-less feature:

annotation { "Feature Type Name" : "Run tests" }
export const runTests = defineFeature(function(context is Context, id is Id, definition is map)
    precondition {}
    {
        // Test case 1
        const input1 = {
            foo : 3
        };
        const output1 = {
            foo : 4
        };
        if (remapInput(input1) != output1)
            throw regenError(ErrorStringEnum.INVALID_INPUT);

        // Test case 2
        // Test case 3
        // etc.        
    }, {});

Now, include just the "Run tests" feature in a Part Studio. If any of the tests fail, you'll see a "throw" notice in the FeatureScript notices panel with a full stack trace.

kevin_o_toole_1

We imagine that TDD workflows can use Part Studios as unit tests.

An FS writer should use an Onshape document as container for everything associated with one feature (or a library of Features). So, a Widget feature type could be defined in one Feature Studio, and it might have a half dozen Part Studios which contain one or several Widgets, testing common cases and edge cases.

Before FeatureScript is released publicly, we intend to have inter-document references, providing an interface for anyone shared on your Widget's document (or Public) to use your feature type. If you use the workflow above, the test cases have the added benefit of providing excellent word-free documentation for your features. A user can look at how your feature is used in your document and immediately grok its intended purpose.

Now, this straightforward design pattern does rely on your eyeballs to make sure the edge cases all behave correctly. For more automated tests, you can make a feature type of the form "Create a Widget, make sure these properties are still valid, otherwise throw" (throwing inside a feature will turn that feature red in the Part Studio)

These tests could be for invariants that should be true any time you make a widget (in which case, you might consider adding said test to the standard feature to alert users of strange behavior). If you want to get more specific, you can even add a specialized test, to be used once in a Part Studio, that asserts very specific things:

annotation { "Feature Type Name" : "Widget (test 1)" }
export const widgetWithTests1 = defineFeature(function(context is Context, id is Id, definition is map)
    precondition
    {
        // Precondition copied from widget, so the UI can be the same
    }
    {
        // Make a Widget
        widget(context, id + "create", {
            // parameters
        });
        
        // Assert properties
        const volume = evVolume(context, {
               "entities" : qEverything(EntityType.BODY)
        });
        const tolerance = 0.000001 * meter^3;
        const expectedVolume = 0.05 * meter^3;
        if (abs(volume - expectedVolume) > tolerance)
            throw regenError(ErrorStringEnum.INVALID_INPUT);

        const numberOfEdges is number = size(evaluateQuery(context, qEverything(EntityType.EDGE)));
        if (numberOfEdges != 13)
            throw regenError(ErrorStringEnum.INVALID_INPUT);
        // etc.
    }, { });

The workflow for this automatic style of testing are admittedly not great right now. It's not very DRY, it makes your document more cluttered, and it's hard to know what metrics are the "right" things to be testing (even if you've been doing this a while!). For the most part, we hope for the time being that you're happy with the workflow of creating canonical use cases and manually checking them. If there are regressions in your features, you can always use the version graph and history to bisect where the behavior changed.

If your "units" for testing are smaller than the feature level, you can also make a feature wrapper that provides a test case for non-feature functions, which throw if, say, the return value of that function is incorrect. Then you can have a Part Studio with no geometry which simply includes the "Run tests" feature.

Any errors thrown will show up in the FeatureScript notices panel with a full stack trace.

dave_cowden

Hi, thanks for that answer.

I think for this problem domain, visual verification, will be OK for what I would call 'integration tests'.
I can't really see a faster way to test in many cases.

Unit tests, on the other hand, are another thing. I definitely think it should be easier than you have proposed to unit test smaller functions. Those need automated tests with non manual assertions, and there should be a junit-like framework for those.

For example, lets consider the dome feature I just made. I used the approach you proposed there:the enclosing document had a couple of uses that I visually looked at. The fact that they automatically update when the fs changes makes it pretty effective.

But, inside my feature,  I have a function that computes the beginning and ending conditions based on the inputs. There is quite a bit if logic there, so I want to write a test that asserts that all of the input permutations result in the boundaries I expect.

Because so many onshape functions operate using maps, this test is actually pretty easy. And for complex inputs , it is way, way better than doing those tests visually because I can assert all the possibilies without creating a part studio for each, which even in my simple case would be 8 parts.

Tldr; part studios will work fine for integration testing, but not for unit tests...

kevin_o_toole_1

Let me clarify what I meant in my last paragraph:

For a unit test that doesn't require a gui, you don't need to define one. Additionally, you may put many tests into a single block so that they may be run all together.

For the case you described, I would refactor my input computation step into a function:

export function remapInput(definition is map) returns map
{
    // Logic goes here
    return {
        "foo" : definition.foo + 1
    };
}

...and call that function many times from an input-less feature:

annotation { "Feature Type Name" : "Run tests" }
export const runTests = defineFeature(function(context is Context, id is Id, definition is map)
    precondition {}
    {
        // Test case 1
        const input1 = {
            foo : 3
        };
        const output1 = {
            foo : 4
        };
        if (remapInput(input1) != output1)
            throw regenError(ErrorStringEnum.INVALID_INPUT);

        // Test case 2
        // Test case 3
        // etc.        
    }, {});

Now, include just the "Run tests" feature in a Part Studio. If any of the tests fail, you'll see a "throw" notice in the FeatureScript notices panel with a full stack trace.

dave_cowden

I see. Yeah, i will try that out. Its a bit clunky, though.