Detect and list flaky scenarios in the list of not ok scenarios

Also updated the retry feature for the separation of flaky scenarios to
a separate category in the scenario counts.

Author: brasmusson

features/docs/cli/retry_failing_tests.feature

@@ -14,14 +14,14 @@ Feature: Retry failing tests
     Given a scenario "Fails-once" that fails once, then passes
     And a scenario "Fails-twice" that fails twice, then passes
     And a scenario "Solid" that passes
-    And a scenario "Fails-forever" that fails
 
   @todo-windows
   Scenario: Retry once, so Fails-once starts to pass
+    Given a scenario "Fails-forever" that fails
     When I run `cucumber -q --retry 1 --format summary`
     Then it should fail with:
       """
-      7 scenarios (5 failed, 2 passed)
+      4 scenarios (2 failed, 1 flaky, 1 passed)
       """
     And it should fail with:
       """
@@ -43,10 +43,11 @@ Feature: Retry failing tests
 
   @todo-windows
   Scenario: Retry twice, so Fails-twice starts to pass too
+    Given a scenario "Fails-forever" that fails
     When I run `cucumber -q --retry 2 --format summary`
     Then it should fail with:
       """
-      9 scenarios (6 failed, 3 passed)
+      4 scenarios (1 failed, 2 flaky, 1 passed)
       """
     And it should fail with:
       """
@@ -67,3 +68,25 @@ Feature: Retry failing tests
         Fails-twice ✗
         Fails-twice ✓
       """
+
+  @todo-windows
+  Scenario: Flaky scenarios gives exit code zero in non-strict mode
+    When I run `cucumber -q --retry 2 --format summary`
+    Then it should pass with:
+      """
+
+
+      3 scenarios (2 flaky, 1 passed)
+      """
+
+  @todo-windows
+  Scenario: Flaky scenarios gives non-zero exit code in strict mode
+    When I run `cucumber -q --retry 2 --format summary --strict`
+    Then it should fail with:
+      """
+      Flaky Scenarios:
+      cucumber features/fails_once.feature:2
+      cucumber features/fails_twice.feature:2
+
+      3 scenarios (2 flaky, 1 passed)
+      """

lib/cucumber/formatter/ansicolor.rb

@@ -60,6 +60,7 @@ module ANSIColor
       end.merge({
         'undefined' => 'yellow',
         'pending'   => 'yellow',
+        'flaky'     => 'yellow',
         'failed'    => 'red',
         'passed'    => 'green',
         'outline'   => 'cyan',

lib/cucumber/formatter/console_issues.rb

@@ -6,10 +6,17 @@ class ConsoleIssues
       include Console
 
       def initialize(config)
+        @previous_test_case = nil
         @issues = Hash.new { |h, k| h[k] = [] }
         @config = config
         @config.on_event(:test_case_finished) do |event|
-          @issues[event.result.to_sym] << event.test_case unless event.result.ok?(@config.strict?)
+          if event.test_case != @previous_test_case
+            @previous_test_case = event.test_case
+            @issues[event.result.to_sym] << event.test_case unless event.result.ok?(@config.strict?)
+          elsif event.result.passed?
+            @issues[:flaky] << event.test_case unless Core::Test::Result::Flaky.ok?(@config.strict?)
+            @issues[:failed].delete(event.test_case)
+          end
         end
       end