1807 sýtem 3 model mobiiii google

A New Model for Testing@

paul_gerrard

Paul Gerrardpaul@gerrardconsulting.com

gerrardconsulting.com

Programme Chair 2014

Helping clients transform their testing through

INNOVATION, COACHING and LEADERSHIP

Our CLIENTS– Want to be agile rather than follow Agile

dogma– Have a pragmatic approach and are focused on

delivery– Want a solution that fits, not a badly fitting

suit.

Typical tester job spec? (May 2014)• Skills & Requirements

– Thorough C# knowledge including Linq– Good .Net experience including charting and winforms

applications.– Experience building and maintaining automated test

systems– Ability to interact with the developers to anticipate future

needs– Ability to influence to ensure good test procedures are

developed and implemented• Desirable Skills

– Some knowledge of Java for mobile testing– An interest in photography. Wearable Computing

(Internet o

f Eve

rything)

Agenda• Pressure to Change• Three Innovations• Where is this Leading?• A New Model for Testing• Consequences• Shift-Left is More Important than

Agile

What's happening?• Stampede to mobile computing• Big Data• Internet of (Every)Thing, Pervasive

Computing• Continuous Delivery, DevOps• (Test) Analytics, Data-Driven

Development• Shift-left, embedded testers, no test

team• Did you really think Agile was the last

word?

Three Innovations That Are Driving

ChangeBeing agileShift LeftAnalytics

#1Agile doesn’t work

but being agile might

Agile is no longer innovative.Challenging Agile and being

agile might beIs continuous delivery for

you?

Agile doesn’t work?• It works for some• But many organisations have difficulties• Mostly a cultural problem– Requirements take months to elicit and agree– Complex stakeholder relationships– Stakeholders with ‘all the knowledge’ can’t be

spared to be the on-site customer/product owner– Lack of confidence => big system/acceptance

tests• We are Agile but ...

Pragmatism rules (again)• Pragmatic companies use the things

that work and discard those that don’t• Let's be honest: Most are doing some

form of hybrid approach:– Chaotic interfaces between Agile teams

and structured businesses– TDD, BDD, ATDD, SBE today…– Continuous Delivery, DevOps… tomorrow?

#2Shift Left

Developers in test?Testers in development?

Testing (testers?) everywhere

TestingBDD/Test-Driven)Trusted

Requirements + Stories

Shift left (redistributed testing)

Functional Requirements

Build (+little testing) but lots of rework

Testing (lots of checking + bugs)

Trusted Requirements +

StoriesTesting

SAVEDTIME

BDD/Test-Driven

Who is going to do all this?BAs, Developers or Testers?

From traditional delivery…

… to Continuous Delivery

Deployment into Production is not

the Same as ‘Going Live’

Release to production and test environments on-demand

Go Live weekly/monthly/quarterly…

#3Test Analytics

Analytics from productionAnalytics from test

An opportunity for testing

Analysis Insight Decision Development and Testing

ProductionTesting

Introducing Test Analytics

“Introducing Test Analytics”, Paul Gerrard:http://gerrardconsulting.com/index.php?q=node/630Julian Harty, "Symbiotic Relationships between Testing and SMAC":http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between-testing-and-social-mobile-analytics-cloud-computing/

“The capture, integration and analysis of test and production monitoring data to inform business and software development decision-making”.

Data silos for test analytics(an illustration)Stakeholder Requirements Assurance (Production)

ApplicationMonitoring

Production Environment Monitoring

Stakeholders Business Goals

and measures Stakeholder

Involvement/ Engagement

Risk

Requirements Stories/Feature

Descriptions Glossary of

terms and Term Usage

Processes Process Paths

(Workflows)

Manual Tests Generated Test

Code (unit, integration, system level)

Application Instrumentation

Automated Test Execution History

Test Status Feature Code

Change History

Application Process Flows

User Accesses/ Activities

Feature Calls, Response times

Feature use cases (data usage)

Application Alerts/Failures

Database Accesses

Database Content

Production failures

System assets Resource

Usage Log Performance

data System Events System Alerts/

Failures/Incidents

Outages

Where is this leading?

If I knew that, I wouldn't be here...

Three development patterns

Structured

Agile Continuous

Goal-B

ased

Hi-Process

Autonomous

There aren't three patterns

There are many

From high process to DIY process• In the past, high process was seen to be

the technical solution to the software problem

• But technical solutions cannot solve human problems

• Agile promised flexibility, but a lot of Agile is really 'high process-in-the-small'

• True agility means a team can adapt or select its process on-the-fly.

The old ways won't work in the future

We need a New Model of Testing (free from logistics)

A New Model for Testing

Don't throw stones. Yet.I'm working on a refined version (with

others).Want to help/review?

Forget Logistics(for the time being)

Document or not?Agile v waterfall?

Planned v exploratory?This technology v that

technology?

ALL Testing is Exploratory

We explore sources of knowledge ...

... to build test models ...... that inform our testing.

All Testing is Based on Models

Examples of models

Models are innate, essential, human

Judgement, exploring and testing

Testing(the system)

Our model(s) are adequate

Our model(s) are not adequate

Exploring (sources) Judgement

Creates testmodels

Uses testmodels

We explore sources of knowledge to build test models that inform our testing

BTW – Do Developers explore the same way? I think so.

Exploration process

ExplorationDefinitions specs/stories

Oracles

Sources

DesignModels

TestModels

Enquiring

Challenging

Sources:People, documents,experience, system under test

Modelling

Test Models:Can be documented

or mental models

Predicting

System under test

Testing process

TestingSystem

Under Test

Refining

Informing Applying

Interpreting

TestModels

Revise the System

Logging Revis

ing

More exploring Reporting

New Model Testing

29 page paper: http://dev.sp.qa/download/newModel

Some Consequences

Do developers and testers think the

same way?Similar processes, not thinking

the sameEasier to embed testers in

developmentDeveloper testing will improve

Covers functional and non-functional

testing?Ignoring logistics means we

can have the same foundational process for ALL

testing(I think)

Relation toTDD and BDD?

TDD is not testing BDD is modelling using stories

Testing v Checking?

I submit that a Check is "a test that can be applied by a

human or tool"But only humans can interpret

Test automation from a different

perspectiveAutomation uses different test

modelsIs that why automation efforts fail so

often?

Developer Testing

The model works (for me)

Capabilities

Enquiring, Modelling, Predicting, Challenging

Informing, Applying, Interpreting, Refining

Reporting and Logging

• Analysis, enquiry and elicitation• Modelling• Creation of custom models, using

heuristics, guesses, brainstorming, ideation, creative thinking

• Custom test design techniques• Comparison of models, value,

advantages, disadvantages, compromises

• Identification, validation and use of oracles

• Predicate logic and proof• Hypothesis and inference• Socratic method• Rapid Review and Inspection

techniques• Test case design• Test models and the meaning of

coverage• Testing as controlled experiment• Observation, Note taking,

recording

A very different skillset• Basic data analysis and statistics• Decision-making with

incomplete data• Computer forensics• Fault tree analysis• Failure diagnosis• Bug advocacy, triage processes

and negotiation• Meaningful software and test

metrics• Visual presentation of data• Reporting and presentation skills• Understanding stakeholders• Test analytics• Risk management, risk-based

testing and decision-making• Critical Thinking• Interpersonal skills• Dealing with

uncertainty/fallibility

Shift-Left is More Important than

Agile

Developers write code; testers test

because developers can’t

No they don’t

Goal: End Manual Feature Checking

Let the tools do it andfree people to do testing

please!

Testing doesn’t just apply to existing software, at the

endWe test systems, not

software.We test early, we test often but it might not be called

testing

Testers Need to Learn How to Code

Some, perhaps many will have to

Testing is about measuring

achievement, not quality

Align systems delivery with project goals and risks and you’ve got a seat on the

board

Testers don’t own testing anymore

Be a TestMaster"Assurance through the

lifecycle"

New Model Testing changes what

testers & developers need to

learnRoles of developers/testers become

blurredPerhaps they are not different disciplines

after all?Why have a tester certification?

The phase after development is REWORK, not

TestingTesting must align with

development, not compete with it or rescue it

A New Model for Testinghttp://dev.sp.qa/download/newModel

@paul_gerrard

Paul Gerrardpaul@gerrardconsulting.com

gerrardconsulting.com

Programme Chair 2014