Actionable Code Coverage

class: center, middle

# Actionable Code Coverage

???

- How code coverage works and it's limitations
  - How to turn coverage it into a helpful tool instead of an afterthought/metric
  - Actionable talk, use right now

---
  class: center, middle

# Actionable Code Coverage

github.com/grosser/ruby-coverage-talk

???

- not just a talk, but repo with runnable examples + markdown slides
  - a maintainable deep dive for new developers

---

# Michael Grosser

Senior Staff Engineer @ Zendesk<br/>
  (Hiring in SF/Madison/Dublin/CPH/Sydney)

<table>
    <tr>
      <td><img width="230" src="assets/Avatar.png"></img></td>
      <td>  </td>
      <td style="vertical-align: top">
        rubygems.org grosser<br/>
        github.com/grosser<br/>
        twitter.com/grosser<br/>
        grosser.it
      </td>
    </tr>
  </table>

???

- hiring: work-life balance + visas
  - job: help other devs, build infrastructure, make them more efficient
  - build lots of gems for testing + coverage and onboarded giant projects on them

---

# Plan

- Code Coverage Overview

???

- get everyone onto the same level

---

# Plan

- Code Coverage Overview
  - Actionable Coverage

???

- problems with current approaches
  - solutions
  - how to migrate projects piecemeal without breaking

---

# Plan

- Code Coverage Overview
  - Actionable Coverage
  - Hack Forked Coverage

???

- how parallel/forked processes can share coverage calculations

---

# Plan

- Code Coverage Overview
  - Actionable Coverage
  - Hack Forked Coverage
  - Coverage Wishlist

???

- basics for better tooling

---

# Code Coverage Overview

- builtin C library, no gems needed

---

# Code Coverage Overview

- builtin C library, no gems needed
  - Enable before loading code

---

# Code Coverage Overview

- builtin C library, no gems needed
  - Enable before loading code
  - Slows down execution

???

- slow: do not always run in tests

---

# Line Coverage

```Ruby
  require 'coverage' # -> coverage.so
  Coverage.start     # enable for newly loaded code

require_relative 'example'
  example

Coverage.result    # disable
  ```

???

- simplest form
  - .result has side-effect of disabling, can use peek_result

---

# Line Coverage

```Ruby
  {"example.rb"=>[1, 1, 0, nil, 1, nil, nil]}
  ```

???

- 0/1 = hit count
  - nil = not code (end/else/comment etc)

---

# Line Coverage

```Ruby
  {"example.rb"=>[1, 1, 0, nil, 1, nil, nil]}

def example             # 1 - Covered, defined not ran
  ```

---

# Line Coverage

```Ruby
  {"example.rb"=>[1, 1, 0, nil, 1, nil, nil]}

def example             # 1 - Covered, defined not ran
    if 1 == 2             # 1 - Covered, ran once
      "a"                 # 0 - Not-Covered
    else                  # nil
  ```

---

# Line Coverage

```Ruby
  {"example.rb"=>[1, 1, 0, nil, 1, nil, nil]}

def example             # 1 - Covered, defined not ran
    if 1 == 2             # 1 - Covered, ran once
      "a"                 # 0 - Not-Covered
    else                  # nil
      3 == 2 ? "b" : "c"  # 1 - Covered, ran once
    end                   # nil
  end                     # nil
  ```

???

- we hit example and the second branch
  - problem: in else the 3==2 has 2 outcomes, "b" never reached ... should rewrite as *next slide*

---

# Line Coverage

```Ruby
      3 == 2 ? "b" : "c"

-> Refactor

if 3 == 2           # 1 - Covered
        "b"               # 0 - Not Covered
      else                # nil
        "c"               # 1 - Covered
      end                 # nil
  ```

???

- enforce via rubocop ?
  - use branch coverage! 2.5+

---

# Branch Coverage

Ruby 2.5+

```Ruby
  Coverage.start branches: true
  ```

???

- easy right ? *next slide* NO!

---

# Branch Coverage

```Ruby
  # lines
  [1, 1, 0, nil, 1, nil, nil]

# branches
  :branches=>{
    [:if, 0, 5, 4, 5, 22]=>{
      [:then, 1, 5, 13, 5, 16]=>0,
      [:else, 2, 5, 19, 5, 22]=>1
    }
  }
  ```

---

# Branch Coverage

start line + char, end line + char => hits

```Ruby
  [..., 5, 13, 5, 16]=>0,
  ```

---

# Branch Coverage

char 4-22 = Y<br/>
  char 13-16 = N<br/>
  char 19-22 = Y<br/>

```Ruby
  3 == 2 ? "b" : "c"
  YYYYYYYYYYYYYYYYYY
           NNN   YYY
  ```

???

- enough info for simple tooling
  - slower than line

---

# Branch Coverage

```Ruby
  foo == bar || raise("wut")

-> Refactor

raise("wut") unless foo == bar
  ```

???

does not break down || or ||=

---

# Branch Coverage

```Ruby
  def foo(a = bar)
  ```

???

does not detect optional params

---

# Oneshot Coverage

Ruby 2.6+

```Ruby
  Coverage.start oneshot_lines: true
  ```

???

- for prod to find dead code prefer oneshot
  - fast: removed after execution
  - no oneshot_branches

---

# Oneshot Coverage

```Ruby
  # lines
  [1, 1, 0, nil, 1, nil, nil]

# oneshot
  {:oneshot_lines=>[1, 2, 5]}
  ```

---

# Oneshot Coverage

```Ruby
  [1, 2, 5]
  ```

```Ruby
  def example             # Covered
    if 1 == 2             # Covered
      "a"                 # ?
    else                  # ?
      3 == 2 ? "b" : "c"  # Covered
    end                   # ?
  end                     # ?
  ```

???

- bad for automation, does not show uncovered/uncoverable
  - need ruby parser to know what is uncoverable

---

# Coverage Performance

<table class="table">
    <tr><th></th><th>Lines</th><th>Branches</th><th>Oneshot</th></tr>
    
    <tr><th>Ruby</th><td>50%</td><td>100%</td><td>2%</td></tr>
    
    <tr><td colspan="4"><pre>50_000_000.times { example }</pre></td></tr>
    <tr><th>Rails</th><td>0.5%</td><td>2.4%</td><td>0%</td></tr>
    <tr><td colspan="4"><pre>app.get "/"</pre></td></tr>
  </table>

???

- to improve performance use `set_trace_func` directly via sampling approach (coverband)

---

# Coverage Performance

by @mametter

???

From 2017 RubyKaigii talk https://www.slideshare.net/mametter/an-introduction-and-future-of-ruby-coverage-library

---

# 🔄 Recap

- `Coverage.start` = simple lines
  - `branches: true` = weird but useful
  - `oneshot_lines: true` = fast, hard to use
  - `.result / .peek_result`

---

# Plan

- <del>Code Coverage Overview</del>
  - Actionable Coverage
  - Hack Forked Coverage
  - Coverage Wishlist

---

# Actionable Coverage

- Mindset
  - Problems
  - Solution
  - Migration

---

# Mindset

- Not a metric, easy to cheat on

???

- use `||` instead of if
  - only unit test

---

# Mindset

- Not a metric, easy to cheat on
  - 100% coverage != good test

???

- 100% = no stupid errors
  - not no logical errors
  - Covered != tested

---

# Mindset

- Not a metric, easy to cheat on
  - 100% coverage != good test
  - A helper/fallback: <br/>
  "Did you write a test for that ?"<br/>
  "Why does it rescue ?"<br/>
  "What happens when xyz ?"

???

- often thought I had coverage but never covered edge case
  - edge case was unreachable

---

# Problems

- Slow feedback

???

- have to make PR and wait for hook, run all tests, open browser
  - adds frustration / not actionable

---

# Problems

- Slow feedback
  - Impossible to reach 100% coverage

???

- setup code and edge-cases
  - broken window effect

---

# Problems

- Slow feedback
  - Impossible to reach 100% coverage
  - Bikeshedding about what % is ok

???

- 100% cannot be reached so we guess
  - rewrite a statement to be readable and suddenly it's not covered

---

# Problems

- Slow feedback
  - Impossible to reach 100% coverage
  - Bikeshedding about what % is ok
  - Complicated setup

???

- install webhooks
  - pay providers
  - get accounts for contributors
  - error? -> irreproducible locally

---

# Solution

???

- the solution we need is ...

---

# Solution

- quick, atomic development feedback

???

- run a single file, matches a single test, small runtime overhead, exact location, console output
  - easy to contribute when tests are easy to find
  - encourage seeking quick local feedback instead of taking a break after PRs
  - bad PRs still fail but can be reproduced

---

# Solution

- quick, atomic development feedback
  - mark intentional gaps to get 100%

???

- call out gaps explicitly, raise awareness when editing code, 100% avoid broken windows

---

# Solution

- quick, atomic development feedback
  - mark intentional gaps to get 100%
  - branch coverage

---

# Solution

- quick, atomic development feedback
  - mark intentional gaps to get 100%
  - branch coverage
  - piecemeal migration approach

???

cannot stop the world, need a place to start .. single file or static state

---

# Solution

- quick, atomic development feedback
  - mark intentional gaps to get 100%
  - branch coverage
  - piecemeal migration approach
  - simple/local/free setup

???

who wants a complicated, remote, paid setup ?

---

# SingleCov

Missing coverage on every 💚 run

```Bash
  rspec spec/foobar_spec.rb
  ......
  114 example, 0 failures

lib/foobar.rb new uncovered lines introduced (2 current vs 0 configured)",
  Uncovered lines:
  lib/foobar.rb:22:10-22:14
  lib/foobar.rb:23
  ```

???

- minitest + rspec
  - catch coverage issues before making PRs
  - makes PRs fail when coverage is missing
  - calls out exact location
  - configure allowed gaps

---

# SingleCov

- 2-5% runtime overhead on single files, compared to 10-20% for SimpleCov
  - Branch coverage on ruby 2.5+ (disable via branches: false)
  - Use with `forking_test_runner` for per test coverage

---

# SingleCov

Start with a single test, no side-effects

```Ruby
  # test/foo_test.rb
  SingleCov.covered!
  ```

???

- your work area/feature
  - security features
  - piecemeal onboarding

---

# SingleCov

Stop coverage degradation: bootstrap script

```Ruby
  ... bootstrap spec/**/*_spec.rbb

SingleCov.covered! uncovered: 42
  ```

???

- stops newly uncovered code and allows slowly working through backlog

---

# SingleCov

Inline comments

```Ruby
  SingleCov.covered! uncovered: 1

-> Refactor

SingleCov.covered!

if this_can_never_happen
    raise # uncovered
  end
  ```

---

# SingleCov

```Ruby
  SingleCov.assert_used   # all tests have coverage

SingleCov.assert_tested # all files have tests
  ```

???

- helpers to test all files have coverage+tests

---

# SingleCov

`https://github.com/grosser/go-testcov`
  ... build your own!

---

# SingleCov Demo

???

- No real demo because I'm scared
  - Randomly picked project and file, not cherry-picked because it's bad
  - Not trying to pick on maintainers, just making a point
  - you think your tests are good, they are not
  - Same is seen all over ruby land ... some accepted PRs

---

# SingleCov Demo

- clone `https://github.com/rails/rails`
  - add `gem 'single_cov'` + `bundle`
  - `cd activesupport`

???

obscure project ;)

---

# SingleCov Demo

```Ruby
  # test/abstract_unit.rb

require 'single_cov'
  SingleCov.setup :minitest, root: File.dirname(__dir__)
  ```

???

gems are in subfolders

---

# SingleCov Demo

```Ruby
  # test/cache/stores/memory_store_test.rb

SingleCov.covered! file:
    'lib/active_support/cache/memory_store.rb'
  # standard: lib/cache/stores/memory_store.rb
  ```

???

- reminder for bad  test structure
  - IDE cmd+shift+T

---

# SingleCov Demo

```Bash
  ruby -Itest test/cache/stores/memory_store_test.rb
  .............................................

Finished
  89 runs, 249 assertions, 0 failures, 0 errors, 0 skips
  lib/active_support/cache/memory_store.rb
  new uncovered lines introduced (9 current vs 0 configured)
  ...
  ```

---

# SingleCov Demo

```Bash
  Lines missing coverage:
  lib/active_support/cache/memory_store.rb:35
  lib/active_support/cache/memory_store.rb:40
  lib/active_support/cache/memory_store.rb:41
  lib/active_support/cache/memory_store.rb:42
  lib/active_support/cache/memory_store.rb:43
  lib/active_support/cache/memory_store.rb:54:13-54:39
  lib/active_support/cache/memory_store.rb:62:9-62:15
  lib/active_support/cache/memory_store.rb:107
  lib/active_support/cache/memory_store.rb:157:13-157:60
  ```

---

# SingleCov Demo

```
  # Advertise cache versioning support.
  def self.supports_cache_versioning?
    true
  end
  ```

???

Trivial ... add #uncovered

---

# SingleCov Demo

```Ruby
  def inspect # :nodoc:
    "<##{self.class.name} entries=#{@data.size}," \
      "size=#{@cache_size}, options=#{@options.inspect}>"
  end
  ```

???

Blows up when another exception tries to show the store / user debugs

---

# SingleCov Demo

```Ruby
  # Delete all data stored in a given cache store.
  def clear(options = nil)
    synchronize do
      @data.clear
      @key_access.clear
      @cache_size = 0
    end
  end
  ```

???

- We should have tests for this right ??
  - Synchronize should have some test ?

---

# SingleCov Demo

```Ruby
  # lib/active_support/cache/memory_store.rb:54:13-54:39
  delete_entry(key, options) if entry && entry.expired?

# lib/active_support/cache/memory_store.rb:157:13-157:60
  @cache_size -= cached_size(key, entry) if entry

# lib/active_support/cache/memory_store.rb:62:9-62:15
  def prune(target_size, max_time = nil)
    return if pruning?
  ```

???

- Untested behavior ...
  - added without tests and nobody noticed
  - had a test that now no longer works ?

---

# SingleCov Demo

👍 github.com/rails/rails/pulls/36028

???

To support please upvote

---

# 🔄 Recap

- Local/fast/free feedback with marked uncovered
  - `single_cov` 💖 `forking_test_runner`
  - Stop the bleeding + divide and conquer
  - Add it to your gems

---

# Plan

- <del>Code Coverage Overview</del>
  - <del>Actionable Coverage</del>
  - Hack Forked Coverage
  - Coverage Wishlist

---

# Hack forked coverage

???

- why ?

---

# Hack forked coverage

- Run all == run single

---

# Hack forked coverage

- Run all == run single
  - Pre-fork for speedup

???

Don't boot up for each test

---

# Hack forked coverage

- Run all == run single
  - Pre-fork for speedup
  - Avoid pollution + capture per file coverage

???

- Unit tests check coverage in isolation
  - addition integration tests ok
  - forking does not play nice with coverage ... *flip*

---

# Hack forked coverage

Forking resets coverage

```Ruby
  require 'coverage'
  Coverage.start
  require_relative 'example'
  example
  fork { puts Coverage.result }
  {"example.rb"=>[0, 0, 0, nil, 0, nil, nil]}
  ```

???

Makes full coverage unreachable since requires cannot be redone

---

# Hack forked coverage

```Ruby
  # ... load everything
  peek = Coverage.peek_result
  fork do
    # ... test
    result = Coverage.result.map do |file, cov|
      cov.each_with_index.map do |c, i|
        peek[file][i] ? peek[file][i] + c : c
      end
    end
  end
  {"example.rb"=>[1, 2, 1, nil, 1, nil, nil]}
  ```

???

Store prefork, test, add

---

# Hack forked coverage

```Bash
  forking-test-runner test/ --merge-coverage
  ```

???

- under the hood of forking-test-runner
  - integrates well with single_cov and makes CI coverage same as local (often more because it runs all tests)

---

# Hack forked coverage

```Bash
  forking-test-runner test/ --merge-coverage
  ```

- handles branch coverage / edge-cases
  - preloads AR fixtures / helpers etc

???

- maybe in the future oneshot

---

# Combine in parent

fork & combine coverage

```Ruby
  # ... load everything
  read, write = IO.pipe
  Process.wait(fork do
    # ... test
    Marshal.dump(Coverage.result, write)
  end)
  result = Coverage.result + Marshal.load(read)
  {"example.rb"=>[1, 2, 1, nil, 1, nil, nil]}
  ```

???

test destructive behavior in fork and combine after

---

# 🔄 Recap

- fork and coverage don't play nice
  - use peek_result to get results in-between
  - use IO.pipe to send coverage

---

# Plan

- <del>Code Coverage Overview</del>
  - <del>Actionable Coverage</del>
  - <del>Hack forked coverage</del>
  - Coverage Wishlist

???

backend tools we need to improve coverage tooling

---

# Coverage Wishlist

Oneshot usable for coverage calculations

```Ruby
  require 'coverage'
  Coverage.start oneshot_lines: :boolean
  require_relative 'example'
  example 1
  Coverage.result
  {"example.rb"=>{:oneshot_lines=>[true, false, nil, true]}}
  ```

???

Could simply grep false and report on that

---

# Coverage Wishlist

Oneshot branches

```Ruby
  require 'coverage'
  Coverage.start oneshot_branches: true
  require_relative 'example'
  example 1
  Coverage.result
  {"example.rb"=>{:oneshot_branches=>{ .. branches ... }}}
  ```

???

- production coverage (low overhead + peek_result endpoint)
  - combine all servers for coverage map

---

# Coverage Wishlist

Inherit coverage in forks

```Ruby
  require 'coverage'
  Coverage.start inherit_on_fork: true
  require_relative 'example'
  example 1
  fork do
    example 2
    Coverage.result
  end
  ```

???

avoid merge logic for branch/oneshot

---

# Coverage Wishlist

Coverage for default parameters

```Ruby
  def foo(a = default)
    puts a
  end
  ```

a was passed in <-> a was not passed in

---

# Coverage Wishlist

Coverage for logical operators

```Ruby
  # covered
  success || raise("Ooops")

# not covered
  raise("Ooops") unless success
  ```

---

# Coverage Wishlist

Path coverage

```Ruby
  a = a ? : 1 : 2
  a = a ? : 3 : 4
  ```

4 Paths through this code
  1/3 + 1/4 + 2/3 + 2/4

???

super annoying to get 100% coverage but could be interesting for security/complicated things

---

# Coverage Wishlist

Turn on for loaded code -> per request/controller coverage

```Ruby
  Coverage.start add_to_existing: true
  ```

---

# 🔄 Recap

- automatable oneshot
  - oneshot branches
  - inherit coverage on fork
  - default params coverage
  - logical operators coverage
  - path coverage
  - cover preloaded code

---

class: center, middle

# 📣 Cover a File Today!

<table>
    <tr><td>github.com/grosser/</td><td style="text-align: left">single_cov</td></tr>
    <tr><td></td><td style="text-align: left">forking_test_runner</td></tr>
    <tr><td></td><td style="text-align: left">maxitest</td></tr>
    <tr><td></td><td style="text-align: left">testrbl</td></tr>
    <tr><td></td><td style="text-align: left">ruby-coverage-talk</td></tr>
  </table>

???

- Give it a try, use for a single file
  - Slides / examples / PRs welcome