RSpec

To use Captain with RSpec, you need to configure your test suite to output test results to a file and then tell Captain where to find those test results.

Getting started

RSpec can output test results to a file using the --format and --out flags. Configure Captain by creating a .captain/config.yml file in the root directory of your repository:

test-suites:
  your-project-rspec:
    command: bundle exec rspec --format json --out tmp/rspec.json --format progress
    results:
      path: tmp/rspec.json

You can change your-project-rspec to any name you like, but we typically recommend using the name of your project followed by a dash followed by rspec. The command is the command you already use to run your test suite. Captain will invoke this command to run your tests. The example above shows what you might use if you use bundle exec rspec and want to store test results in tmp/rspec.json.

Once Captain is configured, you can run captain run your-project-rspec --print-summary. If you see your typical test output followed by a captain block like this:

--------------------------------------------------------------------------------
----------------------------------- Captain ------------------------------------
--------------------------------------------------------------------------------

then you've configured everything correctly! You can now supercharge your test framework's capabilities. See below for configuring each of Captain's features.

Identifying tests

Captain uses framework specific "identity recipes" to identify the tests in your suite. These recipes are order dependent components extracted from native test framework output.

We use this identity to track the executions of a test over the course of their lifetime in your suite. This enables us to do things like flake detection, quarantining, and retries.

For RSpec, Captain constructs the identity by parsing out the file and description attributes. If some of your examples have no description, see Usage with one-liner RSpec expectations.

Quarantining tests

Captain makes managing flaky tests easier than ever. When a test is identified as flaky, you can quarantine the test without modifying it, so that if only those tests fail, Captain reports a success with a 0 exit code. Unlike skipped tests, quarantined tests will continue to run, so you can still view their failure messages and see how frequently they are failing.

You can quarantine tests in OSS mode with captain add quarantine like so:

captain add quarantine your-project-rspec \
  --file ./spec/example_spec.rb \
  --description "Example is true"

See the OSS quarantining guide for more information on managing quarantined tests in OSS mode.

Retrying tests

You can configure Captain to automatically retry failed tests to help you determine if failing tests are flaky or are genuinely failing. To configure retries, update your .captain/config.yml file like so:

test-suites:
  your-project-rspec:
    command: bundle exec rspec --format json --out tmp/rspec.json --format progress
    results:
      path: tmp/rspec.json
    output:
      print-summary: true
    retries:
      attempts: 2
      command: bundle exec rspec --format json --out tmp/rspec.json --format progress {{ tests }}

Once configured, Captain will invoke your original test command, check for any failures, and retry your tests however many times you've specified (in this example, two additional times) by templating the failures into the command specified by retries.command. The output.print-summary option is not required, but we've added it for convenience in understanding the overall results after the retries have been factored in.

Partitioning

Captain can optimally partition your test suite's files into multiple groups for execution on multiple CI nodes. Captain tracks your test file runtime so that it can balance each partition.

Configure partitioning in .captain/config.yml:

test-suites:
  your-project-rspec:
    command: bundle exec rspec --format json --out tmp/rspec.json --format progress
    results:
      path: tmp/rspec.json
    output:
      print-summary: true
    partition:
      command: bundle exec rspec --format json --out tmp/rspec.json --format progress {{ testFiles }}
      globs:
        - spec/**/*_spec.rb

Captain will fill in the testFiles placeholder of your partition.command with the files resulting from expanding your configured partition.globs.

Then partition across your CI provider's parallel jobs:

# .rwx/ci.yml

tasks:
  - key: code
    call: git/clone 2.2.0

  - key: ruby
    call: ruby/install 2.0.1
    with:
      ruby-version: 3.3.0

  - key: deps
    use: [code, ruby]
    run: bundle install

  - key: captain
    call: rwx/install-captain 1.2.0

  - key: rspec
    use: [deps, captain]
    parallel: 8
    run: captain run your-project-rspec

Usage with ABQ

Captain works with ABQ to support parallel test distribution while respecting the quarantine status of any tests. To use Captain with ABQ, the Captain CLI will wrap ABQ and ABQ will wrap your test suite. For example:

test-suites:
  your-project-rspec:
    command: bash -c "abq test --worker $ABQ_WORKER --reporter rwx-v1-json=tmp/abq.json -- bundle exec rspec"
    results:
      path: tmp/abq.json

ABQ_WORKER=0 captain run your-project-rspec

Usage with one-liner RSpec expectations

Captain identifies an RSpec example by its file and full description. When an example has no description, like this one-liner:

it { is_expected.to eq(0.0) }

RSpec generates one from the example's last expectation, such as is expected to eq 0.0. Generated descriptions don't make reliable identities:

• Sibling examples that expect the same value get the same description, even when they check different things. In one context, it { expect(totals.apr).to eq(0.0) } and it { expect(totals.payment).to eq(0.0) } are both is expected to eq 0.0.
• A description that embeds a value that changes between runs, such as a generated ID in it { is_expected.to eq(record.id) }, changes with it.
• An example that's skipped, or that raises an error before its expectation runs, gets no description at all. Its identity then differs from the one it has when it passes.

When tests share an identity, retries and quarantining may not work correctly, and when a test's identity changes, Captain loses its history.

To give these examples descriptions that are unique and don't change between runs, base them on each example's source code. Add the method_source gem to the test group in your Gemfile, then add the following to your spec_helper.rb:

require "digest"
require "method_source"

# Gives each example without a description a unique, stable one for Captain.
module RSpecSourceDescription
  # Read the source here: RSpec generates descriptions while stubs are active, and reading
  # source can call a stubbed File.exist?.
  def initialize(example_group_class, description, user_metadata, example_block = nil)
    super
    return unless metadata[:description].empty? && example_block

    digest = Digest::SHA1.hexdigest(example_block.source.strip)
    duplicates = example_group_class.examples.count { |other| other.metadata[:source_digest] == digest }
    metadata[:source_digest] = digest
    metadata[:description] = "no description #{digest}"
    metadata[:description] += " (#{duplicates + 1})" if duplicates.positive?
    metadata[:full_description] += metadata[:description]
  rescue MethodSource::SourceNotFoundError
    # The block has no source file, for example because an eval defined it, so RSpec's description stays.
  end
end
RSpec::Core::Example.prepend(RSpecSourceDescription)

Each example without a description is then named no description followed by a digest of its source. Examples in one group with the same source, like those defined in a loop, are also numbered in the order they're defined.