MSTest / VSTest retry (rerun) logic
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
1
Unfortunately, there's no native test rerun logic for MStest / VStest
I'm trying to implement a custom logic like this:
Test part:
static int testNum = 1;
[TestMethod]
public void RerunTestOnce_Test()
{
testNum = testNum + 1;
Console.WriteLine("Test started");
Assert.IsTrue(testNum == 3, $"Test Failed with number {testNum}");
}
This test should fail for the first time, and pass for the second, when testNum reach value 3.
UP: This is a synthetic example to emulate fail at first run. Real tests are complex and have UI search methods and other work with a system and network, and there's no confidence that everything will be fine during a big & long test suite.
There is a special method for this - RerunTestOnce(), called in TestCleanup:
[TestCleanup]
public void TestCleanup()
{
TestHelper.RerunTestOnce(TestContext, this);
}
And here is implementation of RerunTestOnce in test helper class.
In it, using Reflection & TestContext we get names of test method and initialize method, and run them again:
public static void RerunTestOnce(TestContext testContext, object testInstance)
{
if (testContext.CurrentTestOutcome == UnitTestOutcome.Failed)
{
var type = testInstance.GetType();
if (type != null)
{
var testMethod = type.GetMethod(testContext.TestName);
var initMethod = type.GetMethods().SingleOrDefault(m=>m.CustomAttributes.SingleOrDefault(a=>a.AttributeType.Name == "TestInitializeAttribute")!= null);
var cleanupMethod = type.GetMethods().SingleOrDefault(m => m.CustomAttributes.SingleOrDefault(a => a.AttributeType.Name == "TestCleanupAttribute") != null);
Console.WriteLine($"[WARNING] Method [{testMethod}] was failed in first attempt. Trying to rerun...");
try
{
initMethod.Invoke(testInstance, null);
testMethod.Invoke(testInstance, null);
}
catch
{
Console.WriteLine($"[ERROR] Method [{testMethod}] was failed in second attempt. Rerun finished.");
}
}
}
}
Everything is ok, on the second attempt test method passes, but in the end I see Failed result and assert error message from first attempt:
Test Failed - RerunTestOnce_Test
Message: Assert.IsTrue failed. Test Failed with number 2
How and when MSTest creates test result - is it possible to update test result after second attempt to last result?
c# visual-studio mstest vstest
asked Nov 22 '18 at 11:59
VladimirVladimir
164213
|
show 1 more comment
1
Unfortunately, there's no native test rerun logic for MStest / VStest
I'm trying to implement a custom logic like this:
Test part:
static int testNum = 1;
[TestMethod]
public void RerunTestOnce_Test()
{
testNum = testNum + 1;
Console.WriteLine("Test started");
Assert.IsTrue(testNum == 3, $"Test Failed with number {testNum}");
}
This test should fail for the first time, and pass for the second, when testNum reach value 3.
UP: This is a synthetic example to emulate fail at first run. Real tests are complex and have UI search methods and other work with a system and network, and there's no confidence that everything will be fine during a big & long test suite.
There is a special method for this - RerunTestOnce(), called in TestCleanup:
[TestCleanup]
public void TestCleanup()
{
TestHelper.RerunTestOnce(TestContext, this);
}
And here is implementation of RerunTestOnce in test helper class.
In it, using Reflection & TestContext we get names of test method and initialize method, and run them again:
public static void RerunTestOnce(TestContext testContext, object testInstance)
{
if (testContext.CurrentTestOutcome == UnitTestOutcome.Failed)
{
var type = testInstance.GetType();
if (type != null)
{
var testMethod = type.GetMethod(testContext.TestName);
var initMethod = type.GetMethods().SingleOrDefault(m=>m.CustomAttributes.SingleOrDefault(a=>a.AttributeType.Name == "TestInitializeAttribute")!= null);
var cleanupMethod = type.GetMethods().SingleOrDefault(m => m.CustomAttributes.SingleOrDefault(a => a.AttributeType.Name == "TestCleanupAttribute") != null);
Console.WriteLine($"[WARNING] Method [{testMethod}] was failed in first attempt. Trying to rerun...");
try
{
initMethod.Invoke(testInstance, null);
testMethod.Invoke(testInstance, null);
}
catch
{
Console.WriteLine($"[ERROR] Method [{testMethod}] was failed in second attempt. Rerun finished.");
}
}
}
}
Everything is ok, on the second attempt test method passes, but in the end I see Failed result and assert error message from first attempt:
Test Failed - RerunTestOnce_Test
Message: Assert.IsTrue failed. Test Failed with number 2
How and when MSTest creates test result - is it possible to update test result after second attempt to last result?
c# visual-studio mstest vstest
asked Nov 22 '18 at 11:59
VladimirVladimir
164213
1
Why jump through all these hoops? "This should be done three times, succeeding the first time and failing the second" is part of the logic of your code, and should be expressed clearly as such in the test itself, not as a re-run of the test method. You can use something like Fluent Assertions to express this succinctly (.Should().[Not]Throw()) and wrap the whole thing as a method that takes anActiondelegate if you need to reuse this logic.
– Jeroen Mostert
Nov 22 '18 at 12:07
Jeroen Mostert Test expected to be passed always. This is just synthetic example to emulate fail at first run. So, if run by some reason failed, I need to rerun whole test, and only if it failed for the second time, the result should be failed.
– Vladimir
Nov 22 '18 at 12:14
2
Consider fixing the code so it doesn't fail the first time. If the tests are sufficiently brittle that they're "expected to possibly fail the first time", that's not a good thing! You should use this as an opportunity to make the code reliable enough to test (by waiting on a server to become available or somesuch), not as an incentive to change the tests. If you really need this you should probably set up an external test runner/build server that can invoke the whole set of tests (withvstest.console.exefor example) that applies this "retry logic" based on whether any test failed.
– Jeroen Mostert
Nov 22 '18 at 12:19
Again, the example is synthetic. There are not a "unit tests". Real tests are complex and have UI search methods and other work with a system and network. In a big test suite there's no 100% confidence that everything will be fine during test execution. That's why some progressive test frameworks like NUnit have implemented "retry" logic. Unfortunately, NUnit is not compatable with Coded UI tests, which we use.
– Vladimir
Nov 22 '18 at 12:40
Consider just using two test projects -- MSTest for the coded UI and NUnit or whatever else you prefer for the other. Otherwise you'll probably keep trying to stick a square peg in a round hole. You may need a bit of plumbing to get both test projects to run for one solution, but that still sounds simpler than trying to change MSTest at its core. If you're stuck with MSTest (for whatever reason), one option might be T4 templates to auto-generate scaffolding for the test classes so they can be retried without having to write code for it. You can log the extra attempts in the test context.
– Jeroen Mostert
Nov 22 '18 at 12:50
|
show 1 more comment
1
1
1
Unfortunately, there's no native test rerun logic for MStest / VStest
I'm trying to implement a custom logic like this:
Test part:
static int testNum = 1;
[TestMethod]
public void RerunTestOnce_Test()
{
testNum = testNum + 1;
Console.WriteLine("Test started");
Assert.IsTrue(testNum == 3, $"Test Failed with number {testNum}");
}
This test should fail for the first time, and pass for the second, when testNum reach value 3.
UP: This is a synthetic example to emulate fail at first run. Real tests are complex and have UI search methods and other work with a system and network, and there's no confidence that everything will be fine during a big & long test suite.
There is a special method for this - RerunTestOnce(), called in TestCleanup:
[TestCleanup]
public void TestCleanup()
{
TestHelper.RerunTestOnce(TestContext, this);
}
And here is implementation of RerunTestOnce in test helper class.
In it, using Reflection & TestContext we get names of test method and initialize method, and run them again:
public static void RerunTestOnce(TestContext testContext, object testInstance)
{
if (testContext.CurrentTestOutcome == UnitTestOutcome.Failed)
{
var type = testInstance.GetType();
if (type != null)
{
var testMethod = type.GetMethod(testContext.TestName);
var initMethod = type.GetMethods().SingleOrDefault(m=>m.CustomAttributes.SingleOrDefault(a=>a.AttributeType.Name == "TestInitializeAttribute")!= null);
var cleanupMethod = type.GetMethods().SingleOrDefault(m => m.CustomAttributes.SingleOrDefault(a => a.AttributeType.Name == "TestCleanupAttribute") != null);
Console.WriteLine($"[WARNING] Method [{testMethod}] was failed in first attempt. Trying to rerun...");
try
{
initMethod.Invoke(testInstance, null);
testMethod.Invoke(testInstance, null);
}
catch
{
Console.WriteLine($"[ERROR] Method [{testMethod}] was failed in second attempt. Rerun finished.");
}
}
}
}
Everything is ok, on the second attempt test method passes, but in the end I see Failed result and assert error message from first attempt:
Test Failed - RerunTestOnce_Test
Message: Assert.IsTrue failed. Test Failed with number 2
How and when MSTest creates test result - is it possible to update test result after second attempt to last result?
c# visual-studio mstest vstest
asked Nov 22 '18 at 11:59
VladimirVladimir
164213
Unfortunately, there's no native test rerun logic for MStest / VStest
I'm trying to implement a custom logic like this:
Test part:
static int testNum = 1;
[TestMethod]
public void RerunTestOnce_Test()
{
testNum = testNum + 1;
Console.WriteLine("Test started");
Assert.IsTrue(testNum == 3, $"Test Failed with number {testNum}");
}
This test should fail for the first time, and pass for the second, when testNum reach value 3.
UP: This is a synthetic example to emulate fail at first run. Real tests are complex and have UI search methods and other work with a system and network, and there's no confidence that everything will be fine during a big & long test suite.
There is a special method for this - RerunTestOnce(), called in TestCleanup:
[TestCleanup]
public void TestCleanup()
{
TestHelper.RerunTestOnce(TestContext, this);
}
And here is implementation of RerunTestOnce in test helper class.
In it, using Reflection & TestContext we get names of test method and initialize method, and run them again:
public static void RerunTestOnce(TestContext testContext, object testInstance)
{
if (testContext.CurrentTestOutcome == UnitTestOutcome.Failed)
{
var type = testInstance.GetType();
if (type != null)
{
var testMethod = type.GetMethod(testContext.TestName);
var initMethod = type.GetMethods().SingleOrDefault(m=>m.CustomAttributes.SingleOrDefault(a=>a.AttributeType.Name == "TestInitializeAttribute")!= null);
var cleanupMethod = type.GetMethods().SingleOrDefault(m => m.CustomAttributes.SingleOrDefault(a => a.AttributeType.Name == "TestCleanupAttribute") != null);
Console.WriteLine($"[WARNING] Method [{testMethod}] was failed in first attempt. Trying to rerun...");
try
{
initMethod.Invoke(testInstance, null);
testMethod.Invoke(testInstance, null);
}
catch
{
Console.WriteLine($"[ERROR] Method [{testMethod}] was failed in second attempt. Rerun finished.");
}
}
}
}
Everything is ok, on the second attempt test method passes, but in the end I see Failed result and assert error message from first attempt:
Test Failed - RerunTestOnce_Test
Message: Assert.IsTrue failed. Test Failed with number 2
How and when MSTest creates test result - is it possible to update test result after second attempt to last result?
c# visual-studio mstest vstest
c# visual-studio mstest vstest
asked Nov 22 '18 at 11:59
VladimirVladimir
164213
asked Nov 22 '18 at 11:59
VladimirVladimir
164213
edited Nov 22 '18 at 12:44
Vladimir
asked Nov 22 '18 at 11:59
VladimirVladimir
164213
asked Nov 22 '18 at 11:59
VladimirVladimir
164213
asked Nov 22 '18 at 11:59
VladimirVladimir
164213
164213
1
Why jump through all these hoops? "This should be done three times, succeeding the first time and failing the second" is part of the logic of your code, and should be expressed clearly as such in the test itself, not as a re-run of the test method. You can use something like Fluent Assertions to express this succinctly (.Should().[Not]Throw()) and wrap the whole thing as a method that takes anActiondelegate if you need to reuse this logic.
– Jeroen Mostert
Nov 22 '18 at 12:07
Jeroen Mostert Test expected to be passed always. This is just synthetic example to emulate fail at first run. So, if run by some reason failed, I need to rerun whole test, and only if it failed for the second time, the result should be failed.
– Vladimir
Nov 22 '18 at 12:14
2
Consider fixing the code so it doesn't fail the first time. If the tests are sufficiently brittle that they're "expected to possibly fail the first time", that's not a good thing! You should use this as an opportunity to make the code reliable enough to test (by waiting on a server to become available or somesuch), not as an incentive to change the tests. If you really need this you should probably set up an external test runner/build server that can invoke the whole set of tests (withvstest.console.exefor example) that applies this "retry logic" based on whether any test failed.
– Jeroen Mostert
Nov 22 '18 at 12:19
Again, the example is synthetic. There are not a "unit tests". Real tests are complex and have UI search methods and other work with a system and network. In a big test suite there's no 100% confidence that everything will be fine during test execution. That's why some progressive test frameworks like NUnit have implemented "retry" logic. Unfortunately, NUnit is not compatable with Coded UI tests, which we use.
– Vladimir
Nov 22 '18 at 12:40
Consider just using two test projects -- MSTest for the coded UI and NUnit or whatever else you prefer for the other. Otherwise you'll probably keep trying to stick a square peg in a round hole. You may need a bit of plumbing to get both test projects to run for one solution, but that still sounds simpler than trying to change MSTest at its core. If you're stuck with MSTest (for whatever reason), one option might be T4 templates to auto-generate scaffolding for the test classes so they can be retried without having to write code for it. You can log the extra attempts in the test context.
– Jeroen Mostert
Nov 22 '18 at 12:50
|
show 1 more comment
1
Why jump through all these hoops? "This should be done three times, succeeding the first time and failing the second" is part of the logic of your code, and should be expressed clearly as such in the test itself, not as a re-run of the test method. You can use something like Fluent Assertions to express this succinctly (.Should().[Not]Throw()) and wrap the whole thing as a method that takes anActiondelegate if you need to reuse this logic.
– Jeroen Mostert
Nov 22 '18 at 12:07
Jeroen Mostert Test expected to be passed always. This is just synthetic example to emulate fail at first run. So, if run by some reason failed, I need to rerun whole test, and only if it failed for the second time, the result should be failed.
– Vladimir
Nov 22 '18 at 12:14
2
Consider fixing the code so it doesn't fail the first time. If the tests are sufficiently brittle that they're "expected to possibly fail the first time", that's not a good thing! You should use this as an opportunity to make the code reliable enough to test (by waiting on a server to become available or somesuch), not as an incentive to change the tests. If you really need this you should probably set up an external test runner/build server that can invoke the whole set of tests (withvstest.console.exefor example) that applies this "retry logic" based on whether any test failed.
– Jeroen Mostert
Nov 22 '18 at 12:19
Again, the example is synthetic. There are not a "unit tests". Real tests are complex and have UI search methods and other work with a system and network. In a big test suite there's no 100% confidence that everything will be fine during test execution. That's why some progressive test frameworks like NUnit have implemented "retry" logic. Unfortunately, NUnit is not compatable with Coded UI tests, which we use.
– Vladimir
Nov 22 '18 at 12:40
Consider just using two test projects -- MSTest for the coded UI and NUnit or whatever else you prefer for the other. Otherwise you'll probably keep trying to stick a square peg in a round hole. You may need a bit of plumbing to get both test projects to run for one solution, but that still sounds simpler than trying to change MSTest at its core. If you're stuck with MSTest (for whatever reason), one option might be T4 templates to auto-generate scaffolding for the test classes so they can be retried without having to write code for it. You can log the extra attempts in the test context.
– Jeroen Mostert
Nov 22 '18 at 12:50
1
1
Why jump through all these hoops? "This should be done three times, succeeding the first time and failing the second" is part of the logic of your code, and should be expressed clearly as such in the test itself, not as a re-run of the test method. You can use something like Fluent Assertions to express this succinctly (
.Should().[Not]Throw()) and wrap the whole thing as a method that takes an Action delegate if you need to reuse this logic.– Jeroen Mostert
Nov 22 '18 at 12:07
Why jump through all these hoops? "This should be done three times, succeeding the first time and failing the second" is part of the logic of your code, and should be expressed clearly as such in the test itself, not as a re-run of the test method. You can use something like Fluent Assertions to express this succinctly (
.Should().[Not]Throw()) and wrap the whole thing as a method that takes an Action delegate if you need to reuse this logic.– Jeroen Mostert
Nov 22 '18 at 12:07
Jeroen Mostert Test expected to be passed always. This is just synthetic example to emulate fail at first run. So, if run by some reason failed, I need to rerun whole test, and only if it failed for the second time, the result should be failed.
– Vladimir
Nov 22 '18 at 12:14
Jeroen Mostert Test expected to be passed always. This is just synthetic example to emulate fail at first run. So, if run by some reason failed, I need to rerun whole test, and only if it failed for the second time, the result should be failed.
– Vladimir
Nov 22 '18 at 12:14
2
2
Consider fixing the code so it doesn't fail the first time. If the tests are sufficiently brittle that they're "expected to possibly fail the first time", that's not a good thing! You should use this as an opportunity to make the code reliable enough to test (by waiting on a server to become available or somesuch), not as an incentive to change the tests. If you really need this you should probably set up an external test runner/build server that can invoke the whole set of tests (with
vstest.console.exe for example) that applies this "retry logic" based on whether any test failed.– Jeroen Mostert
Nov 22 '18 at 12:19
Consider fixing the code so it doesn't fail the first time. If the tests are sufficiently brittle that they're "expected to possibly fail the first time", that's not a good thing! You should use this as an opportunity to make the code reliable enough to test (by waiting on a server to become available or somesuch), not as an incentive to change the tests. If you really need this you should probably set up an external test runner/build server that can invoke the whole set of tests (with
vstest.console.exe for example) that applies this "retry logic" based on whether any test failed.– Jeroen Mostert
Nov 22 '18 at 12:19
Again, the example is synthetic. There are not a "unit tests". Real tests are complex and have UI search methods and other work with a system and network. In a big test suite there's no 100% confidence that everything will be fine during test execution. That's why some progressive test frameworks like NUnit have implemented "retry" logic. Unfortunately, NUnit is not compatable with Coded UI tests, which we use.
– Vladimir
Nov 22 '18 at 12:40
Again, the example is synthetic. There are not a "unit tests". Real tests are complex and have UI search methods and other work with a system and network. In a big test suite there's no 100% confidence that everything will be fine during test execution. That's why some progressive test frameworks like NUnit have implemented "retry" logic. Unfortunately, NUnit is not compatable with Coded UI tests, which we use.
– Vladimir
Nov 22 '18 at 12:40
Consider just using two test projects -- MSTest for the coded UI and NUnit or whatever else you prefer for the other. Otherwise you'll probably keep trying to stick a square peg in a round hole. You may need a bit of plumbing to get both test projects to run for one solution, but that still sounds simpler than trying to change MSTest at its core. If you're stuck with MSTest (for whatever reason), one option might be T4 templates to auto-generate scaffolding for the test classes so they can be retried without having to write code for it. You can log the extra attempts in the test context.
– Jeroen Mostert
Nov 22 '18 at 12:50
Consider just using two test projects -- MSTest for the coded UI and NUnit or whatever else you prefer for the other. Otherwise you'll probably keep trying to stick a square peg in a round hole. You may need a bit of plumbing to get both test projects to run for one solution, but that still sounds simpler than trying to change MSTest at its core. If you're stuck with MSTest (for whatever reason), one option might be T4 templates to auto-generate scaffolding for the test classes so they can be retried without having to write code for it. You can log the extra attempts in the test context.
– Jeroen Mostert
Nov 22 '18 at 12:50
|
show 1 more comment
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53430550%2fmstest-vstest-retry-rerun-logic%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53430550%2fmstest-vstest-retry-rerun-logic%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Why jump through all these hoops? "This should be done three times, succeeding the first time and failing the second" is part of the logic of your code, and should be expressed clearly as such in the test itself, not as a re-run of the test method. You can use something like Fluent Assertions to express this succinctly (
.Should().[Not]Throw()) and wrap the whole thing as a method that takes anActiondelegate if you need to reuse this logic.– Jeroen Mostert
Nov 22 '18 at 12:07
Jeroen Mostert Test expected to be passed always. This is just synthetic example to emulate fail at first run. So, if run by some reason failed, I need to rerun whole test, and only if it failed for the second time, the result should be failed.
– Vladimir
Nov 22 '18 at 12:14
2
Consider fixing the code so it doesn't fail the first time. If the tests are sufficiently brittle that they're "expected to possibly fail the first time", that's not a good thing! You should use this as an opportunity to make the code reliable enough to test (by waiting on a server to become available or somesuch), not as an incentive to change the tests. If you really need this you should probably set up an external test runner/build server that can invoke the whole set of tests (with
vstest.console.exefor example) that applies this "retry logic" based on whether any test failed.– Jeroen Mostert
Nov 22 '18 at 12:19
Again, the example is synthetic. There are not a "unit tests". Real tests are complex and have UI search methods and other work with a system and network. In a big test suite there's no 100% confidence that everything will be fine during test execution. That's why some progressive test frameworks like NUnit have implemented "retry" logic. Unfortunately, NUnit is not compatable with Coded UI tests, which we use.
– Vladimir
Nov 22 '18 at 12:40
Consider just using two test projects -- MSTest for the coded UI and NUnit or whatever else you prefer for the other. Otherwise you'll probably keep trying to stick a square peg in a round hole. You may need a bit of plumbing to get both test projects to run for one solution, but that still sounds simpler than trying to change MSTest at its core. If you're stuck with MSTest (for whatever reason), one option might be T4 templates to auto-generate scaffolding for the test classes so they can be retried without having to write code for it. You can log the extra attempts in the test context.
– Jeroen Mostert
Nov 22 '18 at 12:50