A Conceptual Concurrent Testing Framework for .Net Applications
Technological advancements in micro-processors suggest
an imminent paradigm shift in software development. The sudden shift
from old architecture to a new breed of multi–core processors dictates the use
of concurrency concepts to optimize performance and to get around synchronization
abstractions like semaphores, locks, mutexes and monitors; therefore increasing
the demand for multi-threaded testing.
In this post ill share too you some bits about my Master's thesis by taking a look at
the characteristics of concurrent software; the challenges in concurrent
development; then, present a conceptual concurrent testing framework.
What exactly is concurrent software?
Essentially, order and sequence
are the keywords that describe concurrency. Thus, by definition, a concurrent
program is a condition where two or more processes (or threads) cooperate in
performing a particular task. Each process is monolithic, executing a series of
instructions to accomplish a goal. The execution can be interleaved, simultaneous
or repeating. Using message passing or shared variables to communicate with
each other.
Hence concurrent software can be
described by the following characteristics:
·
Non-Deterministic Execution
When a concurrent
program is run twice, each having the same inputs, it is not guaranteed to
return the same output on both occasions.
·
Sequential Execution
When a concurrent
program has to perform tasks that have some form of dependency relationship, the
sequential execution model is used.
·
Repetitive Execution
When a concurrent
program has to perform tasks in a recurring manner, the repetitive execution
model is used.
·
Simultaneous Execution
When a concurrent program has to
perform tasks that are non-repetitive and non-sequential a simultaneous
execution model must be employed.
The problem with shared resource
Concurrent programs perform tasks independently. These
tasks although independent from each other need to manipulate shared resources
or global variable. This is where the problems arise.
The difficulty of finding errors caused by unexpected
interleaving of threads in concurrent programs is well known. Problems like deadlock, livelock, starvation and race condition is quite
common to concurrent programs.
What we have
Unit
testing
ensures the integrity and high quality of software being developed. Its
effectiveness is so popular that the integration of such in some agile
development method like extreme
programming is encouraged. However, limitation of this kind of automated
test is bounded to sequential programs, and to date, there is no tool currently
used for a
systematic unit test of concurrent software components.
A Conceptual Concurrent Testing Framework for .Net Applications
Our conceptual concurrent testing
framework aims to achieve the aforementioned by providing the following:
- .Net attributes for specifying the execution context
of test classes and methods
- .Net attributes for decorating the classes and methods
identified for testing
- .Net attributes for decorating test methods to be
ignored
- .Net attributes for exceptions expected from a test
method
- A static class for performing assertions inside
client code and
- A runtime engine for dynamically loading test
classes, executing test methods, generating test output.
The Runtime Engine
Like in other popular testing
frameworks, .Net attributes and reflection will play a central role this
concept. Attributes help developers add behavioral modifiers in structured code
while reflection will provide the mechanism for dynamic instantiation,
invocation and type discovery.
The Runtime Execution Engine will provide
the necessary abstractions for creating application domains, reflecting
assemblies for the presence of attributes. Getting the attributes and executing
setup, test and teardown methods.
Net attributes allow developers to embed
behavioral instructions in structured code. Structured code, as
we know it, provides a way of specifying how a piece of code is to be executed
by specifying a set of instructions to achieve something. .Net attributes
provide a way of programmatically modifying runtime behavior of structured code
without adding complexity to it.
Once metadata are inserted in the Portable Executable (PE), other .NET programs may access it
using the .NET Reflection API. The Reflection API allows programmatic
inspection of assembly metadata, including the classes exposed by the
assembly, the methods and the types found therein, the attributes assigned to
various objects, and other related information such as referenced assemblies
and assembly version numbers.
The Framework API
The Framework API is a set of .Net
attributes used to expose behavioral instructions for test code. Such
behavioral instructions are for:
- Specifying the execution context of test classes and
methods;
- Decorating the classes and methods identified for
testing
- Specifying the type of event a method aims to handle
- Decorating test methods to be ignored
- Flagging exceptions expected of a test method.
The following
is a table of attributes and their corresponding behavior implication in test
code.
|
Framework
Attributes
|
|
Attribute
|
Applies
To
|
Behavior
|
|
TestFixture
|
Classes
|
Signals
the runtime that decorated class is expected to contain test methods.
|
|
TimeOut
|
Classes,
Methods
|
When
used on a class, signals the runtime that all test methods in decorated class
must be completed within the specified time-out period. Otherwise it signals
failure.
When
used on a method, it signals the runtime to override existing class level
time-out (if applicable) .
This
allows the developer to detect a deadlock or livelock occured
|
|
Test
|
Methods
|
Signals
the runtime that decorated method is to be executed as a test method.
|
|
IsolationContext
|
Methods
|
Signals
the runtime that method must be executed in the context of a given AppDomain.
This allows the developer to choose a logical grouping for test methods
therefore allowing one test to have a combination of AsyncRepeat, Repeat and
Sequence behaviors.
|
|
Setup/FixtureSetup
|
Methods
|
Signals
the runtime that decorated method is to be executed before executing test
methods.
|
|
Teardown/FixtureTeardown
|
Methods
|
Signals
the runtime that decorated method is to be executed after executing test
methods.
|
|
ExpectedException
|
Methods
|
Signals
the runtime that decorated method is expected to throw an exception of the
specified type.
When
an exception of the specified type is caught, the runtime halts execution of
the method and flags it as a successful run.
|
|
AsyncRepeat(int numberOfTimes )
|
Methods
|
Signals the runtime that decorated method is to be invoked
repeatedly. Each invocation is run asynchronously on a new thread. Cannot be
used with methods already decorated with Repeat and Sequence attributes.
|
|
Repeat(int numberOfTimes )
|
Methods
|
Signals the runtime that decorated method is to be invoked
repeatedly. Each invocation runs synchronously on the same thread as the
previous one. Cannot be used with methods already decorated with AsyncRepeat
and Sequence attributes.
|
|
IgnoreAttribute
|
Methods
|
Signals
the runtime that decorated method is not to be executed (ignored).
|
|
SequenceAttribute( int order )
|
Methods
|
Signals
the runtime that decorated method should be invoked sequentially Cannot be
used with methods already decorated with AsyncRepeat and Repeat attributes.
|
Evaluation
As you can see,
this proposed framework supports non-deterministic, simultaneous, sequential
and repeated execution of test methods. Non-deterministic and simultaneous
execution was achieved by running test methods in an asynchronous fashion. Sequential
execution was achieved by specifying Sequence attributes in test code. Sequence
attributes forces the execution order of test methods in a developer specified
arrangement. Repeated execution was done in two ways. Developers can specify
either a AsyncRepeat attribute or a Repeat attribute. AsyncRepeat attribute
forces a repeated asynchronous execution of test methods in different threads
while Repeat attribute forces the repeated
sequential execution of test methods in a single thread.
It can detect livelock, deadlock and resource starvation via timeouts. Race
conditions can be signaled as an error. It can also be useful in simulating simultaneous
execution by multiple users in a deployment scenario.
The next step would
be to find out which extensibility model (NUnit, MbUnit) can support this
concept then implement it.