IMPORTANT: You should read this document even if you are not writing tests for your code — although the real question is why are you not? — but still. This page nicely explains, with examples, how to write modular, testable and therefore good code.

1. What are unit tests?

From Wikipedia:

Unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.

In other words, unit testing is testing a component (a "unit"), in isolation. Every unit interacts with other units — getting inputs from some units and providing outputs to others. If the unit works correctly, it will, for the right inputs, correctly interact with other units and give the right outputs.

To do that, we will "mock" the dependencies of the unit that is subject to testing. Hence, the unit will think it interacts with real units, while we will only provide it facades. That way we can give the unit all possible inputs that could happen during normal operation, and then verify that, for each of the inputs, the unit responds correctly. We can do that because we mocked the dependencies, and the mocks we are using allow us to specify behavior when queried by our unit, and we can check that the right outputs were given to them. We always treat the unit under test as a black box and never test implementation details.

2. How to write code that can be unit tested

In order to be able to unit test code, it has to comply with the following criteria:

• It has to consist of many separate units. Every unit must have clearly defined interaction with other units.
• Every unit needs to have a clear interface it exposes to other units, so that it can be easily mocked
• Every unit should hold as little state as possible — see (2.1: A side note on state).
• If a unit is hard to test, it probably means it should be split in multiple components. This usually happens when a component starts holding a lot of state and/or has many possible actions and different outcomes — see (2.1: A side note on state).
• Every public interface method declaration should be preceded by the TEST_VIRTUAL macro — see (2.3: The TEST_VIRTUAL macro).

2.1. A side note on state

It is much easier to test pure functions because their output only depends on the input, while with impure functions, the output depends on the state. As this state is internal, it is harder to manipulate it - that usually involves hacks and relies on implementation details. The other possibility is to have helper methods to set up state, but if that state is not meant to be manipulated from the outside, this pollutes the code, creates a lot of ambiguity, and leads to less isolation and more bugs. We will come back to this in the 3rd paragraph of the next section.

2.2. A side note on splitting components

As a rule of thumb, most people usually create too little components, so you shouldn't be afraid of splitting code into more components. That being said, you should always think thoroughly about what a single unit is capable of doing.

Usually, a unit will either fetch the data from multiple units and feed it into other units with very little processing, or it will perform intensive processing on inputs, then return outputs based on inputs and maybe some little internal state. It should definitely not do both. If you wanted to make a component that does both, it is very easy to split it (think about it for a few second before reading on) - you just split it into a component that does routing, and another one that does the computation.

If you have a component that does a lot of processing and holds a lot of state, split it into one that does processing and one that holds state. That way, you will be able to test the processing one because you will give it inputs and verify outputs, without having to set up the state manually (because state will be an input from the other component). It will also be easier to test the one that holds state, because it will be less complicated to achieve every particular state.

2.3. The TEST_VIRTUAL macro

Every method on a unit that is a dependency for another unit (so, literally every unit) needs to have a TEST_VIRTUAL declaration in front.

So, instead of

void update(int a, int b);

you will do

TEST_VIRTUAL void update(int a, int b);

This has to be there to make sure that the unit can be mocked when other units that use it are tested. The definition of the TEST_VIRTUAL macro lives in pyxida/src/ and looks like this:

#if defined(HOOTL) || defined(ENV_TEST)
#define TEST_VIRTUAL virtual
#else
#define TEST_VIRTUAL
#endif 

What this does is declares a method virtual in testing and HOOTL but makes it non-virtual in production. Virtual functions have a bit of overhead because of the way C++ function resolving works - it involves a function pointer table (called vtable) lookup. That is why we do not want them in production code. In testing, however, the mock must be able to extend from the class and override the methods other units use, so the methods have to be virtual.

3. An example of a component and tests for it

Let's consider an example, the Armed state. It is a great example because it is derived from another component and interacts with several others.

Let us think about what the armed state is supposed to do:

• When entered, it should:
  • Start playing the armed sequence through the buzzer
  • Start logging into the circular buffer through logger
  • Send a "rocket armed" message over radio
• Every update cycle, it should:
  • Check if the rocket was launched - if yes, transition to Boost state
• On disarm over radio, it should:
  • Stop the circular buffer logging
  • Transition back to Prelaunch

Now, we can compile the list of modules Armed interacts with:

• StateMachine (for state transitions and to access components not owned by Armed)
• Estimator (for altitude and velocity change detection)
• Buzzer
• Logger
• ComHandler (sending data over radio)
• DecayingCondition (to eliminate noise during threshold detection)

3.1 Constructors

The only component that is directly owned by Armed is DecayingCondition - all the others will be accesses through the StateMachine. Therefore, the two constructors for Armed look like this:

armed.h:

class DecayingCondition;

class Armed {

	...

public:
	explicit Armed(StateMachine* sm);
	Armed(StateMachine* sm, DecayingCondition* dc);

	...

} 

armed.cpp

#include "../util/DecayingCondition.h" // might not be correct anymore

Armed::Armed(StateMachine* sm) : Armed(sm, new DecayingCondition(...)){}

Armed::Armed(StateMachine* sm, DecayingCondition* dc) : State(sm) {
    // save dc
    // do any other init
};

We need two constructors because:

• The first one is the default one, used in production code. Because the Armed completely owns DecayingCondition, it should construct it by itself - whoever constructs Armed should not have to worry about Armed's dependencies
• The second one is used by the first one and in testing: tests have to inject the mock DecayingCondition. It is the general constructor, so it should also call the State constructor, as Armed is derived from State.

NOTE: In the header file, we only use a forward declaration of DecayingCondition. We only #include "DecayingCondition.h" in the cpp file. Otherwise, a change in DecayingCondition.h would require a recompilation of "Armed.h", which would require a recompilation of all the .cpp files that include Armed - there is many - and that is unnecessary because those files do not care about DecayingCondition (if they do, they include it themselves and they will be recompiled anyways).

3.1.1 Sensor dependencies

Sensor dependencies are a different story. Sensor classes are not available at all in testing, so the default constructor won't be able to construct the sensor class unless we provide it one. Therefore, we need to pay special attention when creating a mock for the sensor -- more on this in (3.4.1: Sensormocks).

We also need to conditionally include this mock class in the .cpp implementation, as so:

#ifdef ENV_TEST
#include "../test/MockSensor.h"
#else
#include "Sensor.h"
#endif

Here, a forward declaration in the .h file of the component using the sensor is even more important, as the header file won't yet know which implementation it is getting.

3.2 Main methods made mockable

Now, let's have a look at how to make Armed mockable - it has to be mocked because it is used by other components (ComHandler in particular which calls disarm() on Armed when it receives the disarm command over radio). Therefore, we need to mark disarm() with TEST_VIRTUAL.

armed.h

public:
    TEST_VIRTUAL void disarm();
    void enterState() override; // not using the TEST_VIRTUAL macro
    
    // could also be (given that no deriving classes - there aren't any in this particular case - override enterState):
    void enterState() TEST_FINAL;

armed.cpp

void Armed::disarm(){
    // disarm the rocket
}

void Armed::enterState(){
    // play armed sequence, start logging etc.
}

• you want to make the function virtual in production code, not only for mocking
• the function is already virtual and you are overriding it (enterState() in this example).

In the second case, make sure you mark it with override and not final. We do this because the mock of this class must be able to override those methods. If you wanted to not allow subclasses to override this method, use the TEST_FINAL macro in place of final. It is defined together with the TEST_VIRTUAL macro and will mark the function final in production but override in testing. Therefore, mock will be able to override this method while child classes (in production) will not. To be precise, derived classes will also be able to override the method during testing, but not during production, so you will catch any bastards trying to override your untouchable method when compiling code for production.

3.2.1 Clarification on virtual functions and destructors.

Skip this section only if you know everything about virtual functions in C++ and destructors.

A derived class defining a method with the same name as the base class will "hide" the parent method. Hence, if the method is called on a child instance, the child's method will be called, which is right. If it is called on a child instance in a parent context, parent's method will be called, which is undesirable: we want the child to override the parent's implementation of the method completely. Let us have a look at this example.

#include <iostream>
#include <cstdio>

using namespace std;

class Base {
public:
    int foo() {
        return 0;
    }

    virtual int foo_virt() {
        return 0;
    }

    ~Base() {
        // perform cleanup for base
        cout << "Base destructor called" << endl;
    }
};

class Derived : public Base {
public:
    int foo() {
        return 42;
    }

    int foo_virt() override {
        return 42;
    }

    ~Derived() {
        // perform cleanup for base
        cout << "Derived destructor called" << endl;
    }
};


int main() {
    Base *b = new Base;
    Derived *d = new Derived;

    cout << b->foo() << endl; // 0
    cout << b->foo_virt() << endl; // 0
    cout << d->foo() << endl; // 42
    cout << d->foo_virt() << endl; // 42

    Base *p = new Derived;
    cout << p->foo() << endl; // gives 0 despite p is a pointer to a Derived instance
    cout << p->foo_virt() << endl; // gives 42 because foo_virt is a virtual function so it overrides.

    delete p; // Prints "Base destructor called", cleanup for derived not performed 
    delete b; // Prints "Base destructor called"
    delete d; // Prints "Derived destructor called" and "Base destructor called"
    return 0;
}

The desired one is the second case. We want a method of the class to behave the same independent of which context it is called from (by the way, CLion will give you a warning in the above example that "foo" hides a method in the same class - other IDEs probably as well, but I just wanted to take the chance to promote CLion).

Virtual destructors do not behave exactly like other virtual functions. If a destructor is not virtual, the child will automatically call the parent implementation. In parent's context, the parent will not call the child's implementation as is the case with other non-virtual functions. However, with destructors, this problem is even worse, because it means that one of the destructors might not be called. This leads to major memory leaks, especially if we create and destroy many object during the lifecycle of our program.

3.3 Destructors

Now, we can finally write the destructors. The procedure here is a bit different than with other functions. The destructor has to be virtual for components from which other components are derived. Therefore, it is also virtual for the child classes. This means that the destructor will not be virtual only if the component is not derived from anything and that no component extends it. The only case (afaik at least) in our code where this happens is the StateMachine.

So, if the destructor is virtual, it will be virtual in production and the mock class can easily override it - no macros needed. However, in the rare case that it does not need to be virtual (from the standpoint of production code), you can still make it virtual, or use the TEST_VIRTUAL macro. The first option is preferred because there is very little overhead of a virtual destructor for the StateMachine - it only happens after the code finishes executing = never. It is only worth it if you are really writing a component with no inheritance that is created and destroyed a lot. Additionally, if a destructor is not virtual, when it should be, this causes memory leaks because if a component is destroyed in the parent context, child's destructor is not called unless it is virtual, as we've seen in the example just above from (3.2.1: Clarification on virtual functions and destructors).

state.h

virtual ~State(); // base class destructor, important to be virtual 

armed.h

~Armed() override;

armed.cpp

Armed::~Armed() {
    // ~State will be called automatically
    delete dc; // clean up armed's dependencies
}

3.4 The mock of the component

Now that we have made the component mockable, we can write a mock for it - yay!

Our mock should live under test in the same respect as the .cpp file lives under 'src': if Armed is located at pyxida/src/states/armed.cpp, MockArmed will be located under pyxida/test/states/mock-armed.h

First, we need to pull in GoogleMock and the class declaration itself.

#include <gmock/gmock.h>
#include "../../src/states/armed.h"

Note the src in the path - we are trying to move headers into the inclkude directory but they are still in src at the time of this writing. If that is not the case anymore as you read this, amazing! Just remove ../../src/ (the include directory is in the include path by default.

The next thing: we have to declare the mock class.

class MockArmed: public Armed {

public:
    MockArmed() : Armed(nullptr, nullptr) {}

    MOCK_METHOD0(disarm, void());
    MOCK_METHOD0(enterState, void());
    
    // destructor testing
    MOCK_METHOD0(__die__, void());
    ~MockArmed() override {
        __die__();
    }
}

Destructor testing is very important because we can verify that every component performs the right cleanup. This is not to test Armed's destructor, but rather to test the destructor of components using Armed.

And that's it! We have successfully completed the mock and the class, now let's use them in a test!

3.4.1 Sensor mocks

A sensor mock serves as a full substitute for a sensor class, described in (3.1.1: Sensor dependencies). So its name must be identical to the original class name. The name of the file containing the mock should still start with Mock and be located under test/. It also does not extend from the original class, so double-check that their signatures match, as compiling tests won't verify that as is the case with other components.

#include <gmock/gmock.h>
// we don't include the sensor class header

class Sensor { // same name as the original class 
 
public:
    Sensor(Wire x, uint8_t){} // whatever the constructor signature for the sensor is.
    
    MOCK_METHOD0(bar, void());
    MOCK_METHOD1(foo, int(double));
    
    // destructor testing
    MOCK_METHOD0(__die__, void());
    ~Sensor() {
        __die__();
    }
}

3.5 The unit tests for the component

If you forgot, the (3: An example of a component and tests for it) section describes the expected behavior of Armed in detail. We will write separate tests for every feature of the component.

We will write our tests in a kebab-case named file - test-armed.cpp. In general, tests for LongComponentName will be in test-long-component-name.cpp.

3.5.1 Testing setup code

Of course, there is some code we need to write before we actually start writing the tests.

For better organization of tests, we will organize them in fixtures - each fixture will have its own file and will test one component. Each fixture needs its own class, and we will name them TestLongComponentName. You may find some of them named TestLongComponentNameFixture - this is an old naming convention. If you are editing that file, also edit the name to the new naming convention, as Fixture at the end is redundant.

Normally, when writing tests with GoogleTest, your fixture will extend ::testing::Test, but here, you should extend StateMachineFixture, located under pyxida/test. This class has some helper methods and sets up the mock for arduino hardware functions and the StateMachine mock, which will be required for virtually every test. This is only to remove boilerplate and make your life easier.

Which fixture class should I extend

Ordered by hierarchy:

• Extend ::testing::Test if you don't require hardware functions or the StateMachine. (if you are testing a utility class for example)
• Extend ArduinoTestFixture if you require hardware functions (yes, millis() is one of them) but not the StateMachine
• Extend StateMachineFixture if you require the StateMachine
• Extend another fixture that is derived from StateMachineFixture if you are testing a derived class, for example TestState)

Additionally, we have other base fixtures that are subclasses of StateMachineFixture. TestState implements some common testing behavior for testing all states. TestState serves both as a base class test fixture (base class State -> fixture TestState) and a base for fixtures of derived classes (derived class Armed -> fixture TestArmed, extends TestState);

Code

#include "TestState.h"

class TestArmed : public TestState {
protected:
    // must be protected because every single test is actually a subclass
    // behind the scenes - that's just how GoogleTest implements it -
    // and we need to make those available to the subclass.
    
    MockDecayingCondition *mockDC; // all dependencies as mocks (except StateMachine)
    
    Armed * armed; // instance under testing
    
    TestArmed() {
        mockDC = new MockDecayingCondition;
        armed = new Armed(mockSM, mockDC); // Using the general constuctor of Armed to inject mocks
    }
    ~TestArmed() override {
        // Make sure armed deletes its direct dependencies but not indirect (like StateMachine)
        EXPECT_CALL(*mockDC, __die__());
        delete armed;
    }
} 

In every test, the TestArmed fixture will be constructed and destructed at the end. Hence, we will start with a fresh Armed instance and fresh instances of all mocks.

Test that the destructor was called

EXPECT_CALL is a GoogleTest macro that helps us verify that the method __die__() was called on mockDC. This makes sure mockDC's destructor is called. __die__() does not actually refer to the destructor, but we defined the destructor of MockDecayingCondition to call __die__ as shown in (3.4: The mock of the component). For more information on EXPECT_CALL or other GoogleTest and GoogleMock macros, see (4.1: The EXPECT_CALL macro) or the useful links section at the bottom (6: Useful references and further reading). Usually, googling things will also work.

3.5.2 Test cases

So, let us begin writing the tests. The first feature of Armed:

When entered, it should:

• Start playing the armed sequence through the buzzer
• Start logging into the circular buffer through logger
• Send a "rocket armed" message over radio

For declaring the test, we will use the TEST_F macro. It is important to use this and not TEST - the latter is used if we are not using a test fixture class, which will almost never be the case.

TEST_F(TestArmed, EnterState) {
    // set expectations
    EXPECT_CALL(*mockSM->mockBuzzer, startSequence(armedSequence));
    EXPECT_CALL(*mockSM->mockComHandler, sendOverRadio("ARMED"));
    EXPECT_CALL(*mockSM->mockLogger, startCircBuffer());
    
    // perform action
    armed->enterState();
}

This is a very simple example of what a test should be doing. It sets some expectations on the mocks and then executes our action. We don't even need to verify the expectations manually; it's done by google test in the background when mocks are destructed. That is another reason why we check for mock destruction in every test (i.e. destructor of TestArmed is called after every test). See the section titled Test that the destructor was called under (3.5.1: Testing setup code).

You may ask yourself now: where is mockSM defined? Where are mockBuzzer, mockLogger, and mockComHandler defined? mockSM is declared in StateMachineFixture and the rest is defined in MockStateMachine.h. These files already make sure that when Armed (or anything else) will request the Buzzer through the StateMachine, it will get the mock at mockSM->mockBuzzer. MockStateMachine also makes sure that all of those mocks are properly destructed and our expectations verified. Your life is so much easier now because you don't have to worry about that; you just extend StateMachineFixture (or one of its derived classes) and just easily set expectations on mocks.

Important: *mockSM->mockBuzzer gives the instance of the MockBuzzer.

Explanation: EXPECT_CALL requires a mock instance and not a pointer as its first argument. mockBuzzer is a pointer, so we have to dereference it to get the instance. Additionally, mockBuzzer is a member of MockStateMachine, and mockSM is a pointer to that, so we need to use the member of pointer operator (->). -> precedes *, so *mockSM->mockBuzzer is equivalent to *(mockSM->mockBuzzer).

Read more about the EXPECT_CALL macro and all of its features just below.

4. In-depth explanation of some testing features and special cases for testing

4.1 The EXPECT_CALL macro

The second argument of EXPECT_CALL is the function name, along with parameter matchers. You can read more on parameter matchers in GoogleMock documentation. In our case, we are not even using them - rather, we are using values directly, which means that our expectations will only be satisfied if the function is called with exactly the specified parameters.

If startSequence is called with a parameter that is not armedSequence, this expectation will not be matched. If we want to match a function called with any parameters, we use ::testing::_:

EXPECT_CALL(obj, method(::testing::_));

or

// top of file
using ::testing::_;

// in the test
EXPECT_CALL(obj, method(_));

This expectation will be fulfilled if method is called with 1, -3 or 'a' (or anything else) as its parameter.

Matching is done separately for every parameter, so if we want to allow any value for the first parameter, 4 for the second one and something more than 5 for the last one, we can do

// top of file
using ::testing::_;
using ::testing::GreaterThan;

// in the test
EXPECT_CALL(obj, method(_, 4, GreaterThan(5)));

You can also specify what the method should return, or set an argument passed by reference, or anything else it should do -- described here

To just specify the action and not worry about how many times the method is called and how, use ON_CALL (used in MockStateMachine for returning mocks).

EXPECT_CALL by default matches exactly one method call. If you want to match a different number of calls, use .Times(n). n can be a number or a matcher. To verify the method is never called, use .Times(0)

EXPECT_CALL(obj, method(_)).Times(0); // test will fail if method is called with any argument

Sometimes, you may want to verify expectations before the end of the test - for example, say that the update method of a component will call method on obj every third cycle.

EXPECT_CALL(obj, method(_)).Times(0);
component->update();
EXPECT_CALL(obj, method(_)).Times(0);
component->update();
EXPECT_CALL(obj, method(_));
component->update();

EXPECT_CALL(obj, method(_)).Times(0);
component->update();
Mock::VerifyAndClearExpectations(); // verify expectations and clear them
...

4.2 Testing abstract classes

Sometimes, we need to test a functionality of an abstract class - how do we do that, if we cannot even have an instance of it?

We create a semi-mock class that extends the abstract class and has fake implementations of the pure virtual methods.

vehicle.h

class Vehicle {
public:
    void drive();
    virtual void moveForward() = 0; 
}

void Vehicle::drive() {
    if(sensor->distance() > 10) {
        moveForward();
    }
}

class VehicleForTest {
    MOCK_METHOD0(moveForward, void());
}

...
TEST_F(TestVehicle, DoNotMoveForwardIfNotFree) {
    EXPECT_CALL(*mockSensor, distance).WillOnce(Return(9));
    EXPECT_CALL(*vehicleForTest, moveForward).Times(0);
    vehicleForTest->drive();
}

TEST_F(TestVehicle, MoveForwardIfFree) {
    EXPECT_CALL(*mockSensor, distance).WillOnce(Return(11));
    EXPECT_CALL(*vehicleForTest, moveForward);
    vehicleForTest->drive();
}

5. How to run unit tests for pyxida-firmware

Actually, our great team lead Zack put together a shell script, that executes the whole below procedure -- still, see the error section (5.1: Errors and fixes). It is located at bin/runUnitTests.sh (bin is a sibling of pyxida) You can run it from bin or from the top-level folder, but not from anywhere else (for example: pyxida; bash code for that would require some dark magic not even Zack is certified for).

But, if you are feeling adventurous/want to know how it works, read on:

The full procedure

We are using cmake to build code for our tests, as it also pulls in GoogleTest and the Arduino mock. To build code with unit tests, starting in the pyxida folder:

mkdir build
cd build
cmake ..
make runUnitTests.o
./runUnitTests.o

5.1 Errors and fixes

Uninitialized git submodules

This problem has many different manifestations. They include

• cannot find MadgwickAHRS.h

• cannot find googletest

That probably means you haven't initialized git submodules, so run git submodule update --init

Error: leaked mock object

As this error tells you, you have a leaked mock object, which was not deleted. There are multiple possible reasons for this:

1. The class you are mocking does not have a virtual destructor

If the object you are mocking does not have a virtual destructor and it is deleted in the parent scope, the mock's destructor won't be called, which is very bad because no expectations on that mock can be verified.

The fix is easy:

• Make the base class destructor virtual
• Mark the mock's destructor with override

2. You forgot to delete the dependency in the code

If the object that is being mocked is a direct dependency, the component using it (under test) should delete in its destructor. Make sure:

• that destructor is called (you are deleting the component under test in the test)
• that destructor is virtual (you might be deleting the component but its destructor isn't virtual so the child one isn't being called)
• you are deleting the dependency in that destructor.

3. You forgot to delete the dependency in the test

If the mocked object is not a direct dependency of the component under test, it is not responsible for deleting it, so you should delete the mock yourself.

4. You used a mock after it has been deleted

Because of how GoogleTest operates under the hood, setting expectations on a deleted mock will not trigger any segfaults, however, GoogleTest will think that the mock wasn't deleted.

TEST_F(Fixture1, Test1) {
 
	Mock * ourMock = new Mock; // The mock is created (could also be in the test fixture constructor) at address 0x5634d2e8ca736700
	EXPECT_CALL(*ourMock, fun()); // We set expectations on the mock
 
	code->run1(); // We hope this calls ourMock->fun();
 
	delete ourMock; // Here, expectations on ourMock are verified and cleanup is performed. GoogleTest is happy.
 
	EXPECT_CALL(*ourMock, fun()); // OH NO! GoogleTest now again thinks 0x5634d2e8ca736700 is still active
 
} // after all cleanup, GoogleTest will report "Mock leaked at 0x5634d2e8ca736700 used in Fixture1.Test1", despite it was deleted on line 8.
 
 

Make sure you don't set EXPECT_CALLs after you delete the mock. Sometimes, they can hide in the test destructor, or in a mock destructor.

SIGSEGV: Segmentation fault

Oh, aren't they our favorite? The full beauty of C++ is only expressed when we encounter segmentation faults. Nevertheless, let's see some potential reasons for it:

• You dereferenced (accessed) a pointer to a chunk of memory that was already deleted
• You deleted a piece of memory that was already deleted
• You dereferenced (accessed) memory after an array that is not allocated (buffer overflow).

In code, make sure:

• You don't dereference null pointers
• You don't delete indirect dependencies
• You only delete direct dependencies in the destructor or when they are not used anymore
• You aren't deleting local variables as they are deleted when they go out of scope
• You aren't using local variables after they go out of scope

In tests, make sure:

• You don't delete a direct dependency in the test itself
• You don't pass the same direct dependency to multiple classes that will all delete it.

In general, if you debug the program, you should find the exact spot where the SEGFAULT occurs.

Not on this list

If you have an error that is not on this list, let us know and the fix will be added to the list.

6. Useful references and further reading

GoogleTest

• GoogleTest docs
• GoogleTest primer

GoogleMock

• GoogleMock docs
• GoogleMock cook book
• GoogleMock cheat sheet (a much shorter version than CookBook)