santiago stareast 2016

May 4th, 2016, Orlando, Florida Analyze, Diagnose, and Prevent Test FlakinessMay 5th, 2016, Orlando, FL

Dionny Santiago

2Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Agenda1. Automation Benefits & Challenges2. Metric Analysis3. Owning Full-Stack Testing4. UI Testing Recommendations5. Coding Recommendations

Demo:

3Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Testing Landscape• 200 engineers• 70 test engineers• 30+ million lines of code• 400,000 documented tests• Highly complex business rules• Sensitive domain

4Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Why automate?• Hands-off

regression testing• Focus on

exploratory• Reduce testing cost• Find defects

5Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Consistent

Meaningful

Quick Feedback

Ideal Automation

Detects Defects

Maintainable

Evolves with System

6Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

UI Automation Challenges

• End-to-end Complexity• Asynchronous Content• Dynamic Content• DOM Structure Evolution

7Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

UI Automation Challenges

• High test volume• Multiple platforms• Multiple browsers

8Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Inconsistent

Unstable

Delayed Feedback

“Flaky” Automation

False Negatives

Unmaintainable

9Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

“Flaky” Automation

10Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Analyze and Prevent Flakiness: MetricsPerformance Metrics Complexity Metrics

11Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Execution Performance

Failure % Over Time

Average Duration

Bug Detection Accuracy

12Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Failure % Over Time

Useful?

Failure Percentage

# Fi

xtur

es

13Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

ComplexityMetric Indicator ofTest Fixture Size Test organizationTest Method Size Test case designAssertion Count Over-testing and coupling

14Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Inheritance Complexity

0 1

2

2

6*

Depth within the inheritance hierarchy

*Difficult to refactor. May encounter unintended behavior.

15Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Cyclomatic Complexity

Start

X > 10

Y < 10

callA() callB()

callC() callD()

Edges - Nodes + 2

Edges = 9

Nodes = 8

CC = 3

Linearly independent paths through a program

*Indicator of branch/path complexity and non-determinism.

16Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Tracking and Diagnosing

Continuous Integration Tracking

Combine continuous execution with tracking software

17Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Tracking Metrics

18Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Tracking Metrics – Run Totals

19Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Unit

System

Integration

10%

20%

70%

Complexity

Runtime

Excessive UI AutomationResults in tests that are error-prone and difficult to maintain

20Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Unit

Integration

System

70%

20%

10%Complexi

ty

Runtime

Owning Full-Stack TestingQA must be more involved in lower-level testing

21Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Unit

Component

System

UI Test Breakdown• Reduce UI tests via unit tests• HTTP Mocking

[End-to-end]

22Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

23Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Demo

24Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

UI Testing Recommendations

• End-to-end Complexity• Asynchronous Content• Dynamic Content• DOM Structure Evolution

25Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

UI Testing Architecture

Test Abstraction*

Application Under Test

Application Model*Specifies tests against

Executes tests against

*Reduces end-to-end complexity*Encapsulates DOM Structure and handles dynamic content*Delegates to various platforms and browsers*Handles asynchronous content

26Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Application Modeling

Page Objects ElementsControl Objects

• Atomic• Modeled using

IDs, CSS selectors, XPath

• Runtime execution

• Containers• Promote

organization• Compile-time

• Containers• Promote DRY• Compile-time

27Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Control

Page ObjectControl

Elements

28Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

A note on element selectors…

• CSS selectors are faster than XPath on Chrome/Firefox

• Avoid presentation elements (class names)

• Avoid hierarchical paths (div > div > tr > div…)

• Avoid structural elements (div, p, table, span)

29Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Element Selectors

30Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Coding RecommendationsMaintainability Stability

31Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

MaintainabilityOrganization of tests promotes readability

Excessive nestingToo flat

32Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

MaintainabilityEstablish and adhere to a coding style standard

33Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Maintainability

Not focused and difficult to debug

Strive for short, focused test cases

34Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

MaintainabilityFavor composition over inheritance for code reuse

35Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

MaintainabilityFavor composition over inheritance for code reuse

36Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

StabilityExplicit delays may result in non-determinism and wasted execution time

37Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

StabilityFavor implicit delays over explicit delays

38Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Stability• Implicit vs. explicit delays

39Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

StabilityRaw exception handling introduces logic and affects readability

40Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

StabilityInstead, use built-in unit testing constructs

41Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

StabilityAvoid conditionals and loops

*Recall cyclomatic complexity

42Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Summary• Collect and analyze key metrics• Own full stack testing• Implement best practices• Prevent flakiness• Reduce testing cost• Find defects, faster

43Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Thank you!

Tariq KingGabriel NunezAdam CandoEduardo Pena

Robert Vanderwall

Troy ThomasJorge Martinez

Michael Mattera

Acknowledgements

Sample Project Availablegithub.com/dionny