santiago stareast 2016
TRANSCRIPT
![Page 1: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/1.jpg)
May 4th, 2016, Orlando, Florida Analyze, Diagnose, and Prevent Test FlakinessMay 5th, 2016, Orlando, FL
Dionny Santiago
![Page 2: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/2.jpg)
2Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Agenda1. Automation Benefits & Challenges2. Metric Analysis3. Owning Full-Stack Testing4. UI Testing Recommendations5. Coding Recommendations
Demo:
![Page 3: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/3.jpg)
3Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Testing Landscape• 200 engineers• 70 test engineers• 30+ million lines of code• 400,000 documented tests• Highly complex business rules• Sensitive domain
![Page 4: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/4.jpg)
4Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Why automate?• Hands-off
regression testing• Focus on
exploratory• Reduce testing cost• Find defects
![Page 5: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/5.jpg)
5Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Consistent
Meaningful
Quick Feedback
Ideal Automation
Detects Defects
Maintainable
Evolves with System
![Page 6: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/6.jpg)
6Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
UI Automation Challenges
• End-to-end Complexity• Asynchronous Content• Dynamic Content• DOM Structure Evolution
![Page 7: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/7.jpg)
7Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
UI Automation Challenges
• High test volume• Multiple platforms• Multiple browsers
![Page 8: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/8.jpg)
8Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Inconsistent
Unstable
Delayed Feedback
“Flaky” Automation
False Negatives
Unmaintainable
![Page 9: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/9.jpg)
9Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
“Flaky” Automation
![Page 10: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/10.jpg)
10Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Analyze and Prevent Flakiness: MetricsPerformance Metrics Complexity Metrics
![Page 11: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/11.jpg)
11Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Execution Performance
Failure % Over Time
Average Duration
Bug Detection Accuracy
![Page 12: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/12.jpg)
12Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Failure % Over Time
Useful?
Failure Percentage
# Fi
xtur
es
![Page 13: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/13.jpg)
13Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
ComplexityMetric Indicator ofTest Fixture Size Test organizationTest Method Size Test case designAssertion Count Over-testing and coupling
![Page 14: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/14.jpg)
14Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Inheritance Complexity
0 1
2
2
6*
Depth within the inheritance hierarchy
*Difficult to refactor. May encounter unintended behavior.
![Page 15: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/15.jpg)
15Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Cyclomatic Complexity
Start
X > 10
Y < 10
callA() callB()
callC() callD()
Edges - Nodes + 2
Edges = 9
Nodes = 8
CC = 3
Linearly independent paths through a program
*Indicator of branch/path complexity and non-determinism.
![Page 16: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/16.jpg)
16Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Tracking and Diagnosing
Continuous Integration Tracking
Combine continuous execution with tracking software
![Page 17: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/17.jpg)
17Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Tracking Metrics
![Page 18: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/18.jpg)
18Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Tracking Metrics – Run Totals
![Page 19: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/19.jpg)
19Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Unit
System
Integration
10%
20%
70%
Complexity
Runtime
Excessive UI AutomationResults in tests that are error-prone and difficult to maintain
![Page 20: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/20.jpg)
20Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Unit
Integration
System
70%
20%
10%Complexi
ty
Runtime
Owning Full-Stack TestingQA must be more involved in lower-level testing
![Page 21: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/21.jpg)
21Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Unit
Component
System
UI Test Breakdown• Reduce UI tests via unit tests• HTTP Mocking
[End-to-end]
![Page 22: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/22.jpg)
22Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
![Page 23: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/23.jpg)
23Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Demo
![Page 24: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/24.jpg)
24Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
UI Testing Recommendations
• End-to-end Complexity• Asynchronous Content• Dynamic Content• DOM Structure Evolution
![Page 25: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/25.jpg)
25Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
UI Testing Architecture
Test Abstraction*
Application Under Test
Application Model*Specifies tests against
Executes tests against
*Reduces end-to-end complexity*Encapsulates DOM Structure and handles dynamic content*Delegates to various platforms and browsers*Handles asynchronous content
![Page 26: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/26.jpg)
26Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Application Modeling
Page Objects ElementsControl Objects
• Atomic• Modeled using
IDs, CSS selectors, XPath
• Runtime execution
• Containers• Promote
organization• Compile-time
• Containers• Promote DRY• Compile-time
![Page 27: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/27.jpg)
27Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Control
Page ObjectControl
Elements
![Page 28: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/28.jpg)
28Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
A note on element selectors…
• CSS selectors are faster than XPath on Chrome/Firefox
• Avoid presentation elements (class names)
• Avoid hierarchical paths (div > div > tr > div…)
• Avoid structural elements (div, p, table, span)
![Page 29: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/29.jpg)
29Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Element Selectors
![Page 30: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/30.jpg)
30Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Coding RecommendationsMaintainability Stability
![Page 31: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/31.jpg)
31Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
MaintainabilityOrganization of tests promotes readability
Excessive nestingToo flat
![Page 32: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/32.jpg)
32Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
MaintainabilityEstablish and adhere to a coding style standard
![Page 33: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/33.jpg)
33Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Maintainability
Not focused and difficult to debug
Strive for short, focused test cases
![Page 34: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/34.jpg)
34Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
MaintainabilityFavor composition over inheritance for code reuse
![Page 35: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/35.jpg)
35Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
MaintainabilityFavor composition over inheritance for code reuse
![Page 36: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/36.jpg)
36Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityExplicit delays may result in non-determinism and wasted execution time
![Page 37: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/37.jpg)
37Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityFavor implicit delays over explicit delays
![Page 38: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/38.jpg)
38Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Stability• Implicit vs. explicit delays
![Page 39: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/39.jpg)
39Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityRaw exception handling introduces logic and affects readability
![Page 40: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/40.jpg)
40Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityInstead, use built-in unit testing constructs
![Page 41: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/41.jpg)
41Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityAvoid conditionals and loops
*Recall cyclomatic complexity
![Page 42: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/42.jpg)
42Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Summary• Collect and analyze key metrics• Own full stack testing• Implement best practices• Prevent flakiness• Reduce testing cost• Find defects, faster
![Page 43: SANTIAGO STAREAST 2016](https://reader031.vdocumento.com/reader031/viewer/2022011722/589f879b1a28abf0638b6869/html5/thumbnails/43.jpg)
43Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Thank you!
Tariq KingGabriel NunezAdam CandoEduardo Pena
Robert Vanderwall
Troy ThomasJorge Martinez
Michael Mattera
Acknowledgements
Sample Project Availablegithub.com/dionny