MICROSOFT PEX and TDD
PEX stands for Program Exploration which is a white box test generation tool developed at Microsoft Research that helps developers to write PUTs in .NET platform and thus generate automated test cases. For each PUT, Pex uses dynamic test-generation techniques to compute a set of input values that exercise all the statements and assertions in the analyzed program which eventually cover all reachable branches[i].
PEX generates test inputs for parameterized unit tests (PUTs). In the context of a web service, a PUT is simply a method that takes parameters, invokes a sequence of web service operations, and asserts properties of the expected behavior of the operations[ii].
HOW TO CREATE A PEX PROJECT AND START TETSING?
To work with PEX, we have to download and install Microsoft PEX white box testing framework and must have Visual Studio installed as well. Then create a test project and write a method to Test. Then right click on the method and select PEX exploration to start creating Test Axiom and Test Methods for Test Coverage for numerous possible parameter sets. If we look at the following example, it will be clear to us. We have created a method GetServicePointByNumber which accepts number as a input parameter. Microsoft PEX crteates GetServicePointByNumber Axiom and GetServicePointByNumber782 Test Method for null input parameter validation and testing. It also created a method GetServicePointByNumber547() for a valid input parameter testing. Then, if we run the test, we will see that the test results for different input parameter scenarios. Thus, test automation is possible in Microsoft PEX and could be attached to the continus interation tool for automated testing and continuous integration without spending the time to creae test cases and automating the testing and continuous interatiuon process. So, now we can answer to the question asked at the beginning of this document stating “CAN SOMEONE WRITE MY TEST?”. Now, we can see Microsoft PEX can do that for me to save that 15% - 35% development time creating test cases and it is now possible to have that confidence that all possible testing scenarios are now covered.
SAMPLE WEB METHOD TO TEST:
[WebMethod]
public ServicePointData GetServicePointByNumber(string number)
{
try
{
ServicePointData _SpData = new ServicePointData();
foreach (ServicePointData sp in myList())
{
if (sp.Number == number)
{
_SpData = sp;
}
}
return _SpData;
}
catch (Exception ex)
{
string _error = ex.Message;
return null;
}
}
public class ServicePointData
{
public ServicePointData()
{
}
public string Number
{
get;
set;
}
public string Name
{
get;
set;
}
public string Title
{
get;
set;
}
}
AUTO GENERATED TEST AXIOM:
[PexMethod]
public ServicePointData GetServicePointByNumber([PexAssumeUnderTest]ServicePoints target, string number)
{
ServicePointData result = target.GetServicePointByNumber(number);
return result;
// TODO: add assertions to method ServicePointsTest.GetServicePointByNumber(ServicePoints, String)
}
AUTO GENERATED TEST METHODS WHICH IMPLEMENTS PUT AND TEST COVERAGE:
public partial class ServicePointsTest
{
[Test]
[PexGeneratedBy(typeof(ServicePointsTest))]
public void GetServicePointByNumber782()
{
using (PexDisposableContext disposables = PexDisposableContext.Create())
{
ServicePoints servicePoints;
ServicePointData servicePointData;
servicePoints = new ServicePoints();
((MarshalByValueComponent)servicePoints).Site = (ISite)null;
disposables.Add((IDisposable)servicePoints);
servicePointData = this.GetServicePointByNumber(servicePoints, (string)null);
disposables.Dispose();
PexAssert.IsNotNull((object)servicePointData);
PexAssert.AreEqual<string>((string)null, servicePointData.Number);
PexAssert.AreEqual<string>((string)null, servicePointData.Name);
PexAssert.AreEqual<string>((string)null, servicePointData.Title);
PexAssert.IsNotNull((object)servicePoints);
}
}
[Test]
[PexGeneratedBy(typeof(ServicePointsTest))]
public void GetServicePointByNumber547()
{
using (PexDisposableContext disposables = PexDisposableContext.Create())
{
ServicePoints servicePoints;
ServicePointData servicePointData;
servicePoints = new ServicePoints();
((MarshalByValueComponent)servicePoints).Site = (ISite)null;
disposables.Add((IDisposable)servicePoints);
servicePointData = this.GetServicePointByNumber(servicePoints, "01");
disposables.Dispose();
PexAssert.IsNotNull((object)servicePointData);
PexAssert.AreEqual<string>("01", servicePointData.Number);
PexAssert.AreEqual<string>("High River", servicePointData.Name);
PexAssert.AreEqual<string>("01 - High River", servicePointData.Title);
PexAssert.IsNotNull((object)servicePoints);
}
}
}
INSTANTIATING PUT IN PEX:
[PexMethod]
public string ConvertToUpper([PexAssumeUnderTest]MyPexTest1Class target, string Input)
{
string result = target.ConvertToUpper(Input);
return result;
}
[PexMethod]
public void ConvertToUpperAgain(string Input)
{
MyPexTest1Class MyP = new MyPexTest1Class();
var First = MyP.ConvertToUpper(Input);
var Second = MyP.ConvertToUpper(First);
Assert.AreEqual(First, Second);
}
PUT AND PEX BENEFITS:
ü PUT specifies the external behavior of the methods for all test arguments and so the tests cover all possible test environment and constraints
ü PUTs can be instantiated to re-obtain traditional closed unit tests and to re-verify the result of tests
ü PUT enables automatic case analysis, which avoids writing implementation-specific unit tests
ü Using PUT in PEX, it is fairly simple to auto generate test cases and cover all possible testing scenarios, constraints and values
PEX LIMITATIONS:
ü PEX works for deterministic model of test methods and goes in cycle for non deterministic methods until hits exploration bounds
ü PEX doesn’t work in Multi-Thread model
ü PEX can analyze any .NET language but the code generation only supports C#
ü PEX uses an automatic constraint solver to determine which values are relevant for the test and the code-under-test but which is also limited [Say: floating point arithmetic cannot be reasoned precisely]
ü Lack of Tests for Oracle
[i] Automating Software Testing Using Program Analysis, Patrice Godefroid, Peli de Halleux, Aditya V. Nori, Sriram K. Rajamani, Wolfram Schulte, and Nikolai Tillmann, Microsoft Research Michael Y. Levin, Microsoft Center for Software Excellence
[ii] White-Box Testing of Behavioral Web Service Contracts with Pex, Nikolai Tillmann nikolait@microsoft.com
Jonathan de Halleux jhalleux@microsoft.com
Microsoft Research
One Microsoft Way, Redmond WA 98052, USA
Great post. I am going to the Microsoft experience center in Dayton in May to get a hands on introduction to the latest technologies from Microsoft. I can't wait!
ReplyDelete