Introduction
This is a pretty standard scenario from the quality
engineering perspective where all the organization are trying to move towards DevOps.
Organization have conventionally been doing automated functional and regression
testing but with DevOps a change in approach towards continuous testing across DevOps
phases is very important. In this small write-up I am trying to present an
approach how teams/testing organization can move towards enabling continuous
testing in DevOps.
Change of approach and mindset
I strongly believe that this change is driven more by
approach and mindset rather than technological changes required as teams have
been automating since many year but with emergence of DevOps we need to look at
things in a different way. In my opinion below six mindset changes are import
to be successful for CT enablement:
1.
Engage QE
teams early: This is an important practice where QE teams are involved from
the planning phase. This is done by following combination of processes and
tools such as: TDD, BDD, static code analysis, code walkthrough, pair
programming, mandatory unit test coverage etc.
2.
Be ready
to discard: This is another important lesson where we need to change our
mindset. In conventional automation approach we look at the ROI in the long run
whereas that might not be a case here. In the fast agile there are instances
where you may or may not use automation for longer time but at the same time is
import to use automated testing to validate scenarios. For example validating
huge data files, some intermediate changes to data which may not be there in
further releases. In such scenario we should be ready to adapt mindset of
developing, using and discarding scripts quickly. It does not always mean that
everything we have built needs to be discarded but discard quickly as a
possibility needs to be taken into consideration.
3.
Follow
testing pyramid: Testing pyramid needs to be followed religiously thereby
cutting down no of UI scenarios to minimum. This would help to create a more
robust automation suite with less maintenance and would provide a better
coverage and faster testing feedback to developers.
4.
Automate
early: Focus needs to be on more automation. With increasing complexity and
amount of data, it makes sense to automate test cases. It also helps to test
features often and quickly. Other important thing is to automate early. This is
where practices such as TDD and BDD comes into pictures and test are developed
along with development and in some cases even before development has started.
5.
Service
Virtualization: Virtualization helps to test component more comprehensively
even when other system is not available or is not developed yet. This would
help speed up testing giving more confidence to developers.
6.
Test more
often and quickly: Test as often as possible – with every local build, with
every new changes added, and in different environments. Including multiple
quality gates and having more test coverage helps to build more reliable
products and reduces chances of defect slippage in a higher environment.
Testing activities in DevOps Phases
Next step would be to understand different testing
activities which are required to be performed in different DevOps phases. These
can be broadly categorized as below:
1.
Planning
Phase: This phase would involve creation of user stories, UACs, and coming
up with solutions for the user stories. Static testing techniques such as
review can be used to provide feedback in planning phase.
2.
Development
and Integration: This is the phase where developers would develop features
and integrate the code. Validation such as unit tests and checkstyles need to
be performed at this stage to make sure that code changes are not breaking
anything existing or have any conflict with any other developer who is also
working on the same branch. Developers
also use this phase to perform component testing of their development and
virtualization can play a key role to help developers to test their changes and
get more confidence.
3.
Building
the code: As soon as developer checks in his/her changes based on check-in
or feature branch merge the build would be triggered. This build trigger is according
to the branching strategy deployed. At this point changes might either be
pushed to a container image or to an environment. At this point it becomes
critical to ensure that changes pushed in are not breaking anything and unit
test coverage is used as a build breaker to ensure that at least at the code
level things are good.
4.
Deploy
and deliver: Once build is done, it is deployed in the environment and
functional changes are required to be tested. All the conventional test phases
comes into picture in this phase where functional tests and regression tests
are performed to ensure new features are working fine and old features are
intact. Functional/regression tests are used in this phase to ensure build is
fine and ready to be deployed. After tests are good, build is deployed to
higher environment.
You can see from above phases that same phases are automated
using orchestration tools such as Jenkins and other automation tools are used
to perform different activities in different phase. As the process is
automated, testing is all the more important to give confidence before
deploying a build and automation testing will play a critical role to perform
validations across different phases in a seamless manner.
Create a continuous testing maturity model
Start with creating a basic model to represent continuous
testing maturity. This will help you assess and standardize your projects and
come up with a roadmap. For example: have different levels such as: Beginner,
Intermediate, Advances and have set of questions associated with each level.
Questions may be:
a.
Do you have automated testing in place for
functional testing?
b.
Have you automated your API tests?
c.
Do you engage QE early in the development cycle?
d.
Are you following agile methodology?
e.
Do you plan to adapt DevOps? If so, where are
you in the journey right now?
f.
Do you automate unit tests?
g.
Do you use checkstyles in your code?
h.
Do you follow TDD/BDD etc?
Based on answers of above questions, you may evaluate your
projects and come up with a maturity level. By no means above list is
exhaustive or the levels I have defines are sufficient. But this can be a good
starting point to come up with your own maturity model for continuous testing.
Approach towards continuous testing in DevOps
1. Assess current status: Use maturity
model defined above to evaluate state of the project and come up with a
maturity for each project.
2. Identify gaps: Based on maturity level
come up with a gap analysis report to help you understand the current state.
3. Create transformation roadmap and strategy:
Based on the gaps create a transformation roadmap which will help you to
define strategy how you can make changes and integrate to DevOps cycle thereby
provide value.
4. Implement transformation strategy: Time
to implement the strategy which you have defined in step 3.
I have tried to list down the approach at a very high level
for transformation. I would love to hear what you think about this.
