Posted on

Automation Plan – Keep it simple

Nowadays, all companies started moving towards more and more automation. It is essential to have a plan in place; otherwise, automation may fail.

Here’s a guide (by Ray Claridge) to making automation a success.

  • Business buy in – Before starting to automate, make sure you’ve got buy in from line managers and developers. Remember, automating is time consuming and will cost your company money to get off the ground.
  • Plan – Don’t just start automating random functionality, have a plan and document explaining the approach and how long each test will take to develop. Remember, get sign off from all parties involved.
  • Identify high risk areas – Automating a fully fledged system is going to take a long time. So do some analysis to identify the high risk areas such as: most used, high volume, security or transactional sections and focus on them first.
  • Identify areas less likely to change – Maintaining automation test scripts is not a five minute job, so don’t start on areas that are likely to change. Equally, don’t assume that functionality less likely to change doesn’t need testing. Past experience has taught me never to assume.
  • Document your tests – You need to this so that it’s clear to others what exactly the tests cover. Also handy if your automated product is not available or your tests are falling over.
  • Keep track of your test runs – Keeping a chart of all you your tests and tracking automated vs manual effort, gives visibility that you’re saving your company money. Also handy when trying to get buy in.
  • Keep it simple – Remember, tests should be simple so they can be re-used again and again. This keeps down the costs when maintaining and allows others to pick them up in the future, especially if you’ve got a contractor in to write the tests.
  • And lastly one for all the Product Mangers, Development Managers and Business Units –
    Don’t assume that because you’ve got someone writing automated tests that all your code quality issues are over. Remember – automation is only as good as the tests written!
Posted on

Security Testing Taxonomy in 7 Steps

360logica provides a foolproof testing solution by adopting interesting ways and using a range of neat tools. Here is the 7 step process to test security of any application.

Step 1 – Discovery: The step involves analyzing system based on its scope, proposed and making a checklist of possible threats at each stage.

Step 2 – Vulnerability Scan: The system is run against prepared vulnerabilities to define risk level using automated tools.

Step 3 – Vulnerability Assessment: Considering existing and potential risks in the framework and tracking it to the environment under test mandate.

Step 4 – Security Assessment: Broadly assessing vulnerabilities and manually verifying it confirm exposure. It also involves assessing system response, file logs, codes, error messages, and broad coverage to check system defects.

Step 5 – Penetration Test: Using SQL injection and cross site scripting techniques to simulate malicious attack. This helps in identifying system’s ability to resist unauthorized access, data integrity,seamless operation, consistency, and problem solving abilities.

Step 6 – Security Audit: Specifying risk functions, control issues, compliance difficulties, areas reported during the security testing.

Step 7 – Security Review: Detailed analysis and information validation in such a way that ensures that the security standards are implemented and work in a seamless way through gap analysis, review of code and design documents, and evaluation of architecture diagrams.

Posted on

Tips to Baseline the Performance of the Old System

If your project is replacing an existing system, it is immensely valuable to establish a baseline for the new system based on the existing system. Start by analyzing the usage patterns of the existing site. What operations are most common? What paths are users following through the site? How many users are accessing the system at various times throughout the day? Wherever possible, this data should come from system logs rather than assumptions and guesswork. Then start designing your test:

  1. Create a mix of scenarios that account for 70-80% of the site usage
  2. Add any other scenarios that are known, or suspected, of having poor performance or affecting the overall performance of the system.
  3. Run a test that simulates various load levels corresponding to the load levels seen throughout a typical day
  4. Verify that the results agree with actual performance of the system during real-world use. If it does not, then adjust the test design and scenario mix to better match the real-world usage. Return to step 3 and try again

Having an accurate benchmark of the existing system yields several benefits:

  1. Performance measurements of the existing system can be compared objectively to the new system
  2. Research on the usage patterns of existing users will greatly improve the estimates of usage of the new system – improving the design and accuracy of tests on the new system

Reference: webperformance.com

Posted on

Selenium – Python – Get all child elements

There are various ways to find child elements:

  • find_elements_by_name
  • find_elements_by_xpath
  • find_elements_by_link_text
  • find_elements_by_partial_link_text
  • find_elements_by_tag_name
  • find_elements_by_class_name
  • find_elements_by_css_selector

For instance, consider this page source:

<html>
<body>
<form id=”loginForm”>
<input name=”username” type=”text” />
<input name=”password” type=”password” />
<input name=”continue” type=”submit” value=”Login” />
</form>
</body>
<html>

The form element can be located like this:

login_form = driver.find_element_by_id(‘loginForm’)

More references here

Posted on

Performance Testing Interview Questions – Part 1 (Scenario Based included)

Q: What is Endurance Testing & Spike Testing?

  • Endurance Testing – It is a type of performance testing where the testing is conducted to evaluate the behavior of the system when a significant workload is given continuously.
  • Spike Testing – It is a type of performance testing that is performed to analyze the behavior of the system when the load is increased substantially.

Q: What are the top 3 Response KPIs for Performance Testing?

  • Average response time – time to first byte or last byte
  • Peak response time – tells your longest cycle
  • Error rate – percentage of problems compared to all requests

Q: What Is Distributed Load Testing?

  • Distributed load testing is the process of simulating a very high workload of an enormous number of users using multiple systems. As a single system cannot generate a large number of threads (users), multiple systems are used for load testing. It helps to distribute the tests and the load.

Q: What is the difference between front-end and back-end performance testing? Which one is more important?

  • Front-end performance is concerned with how quickly text, images, and other page elements are displayed on a user’s browser page.
  • Back-end performance is concerned with how quickly these elements are processed by the site’s servers and sent to the user’s machine upon request. Remember: Front-end performance is the part of the iceberg above the water line, and back-end performance is everything underneath that you can’t see.
  • Both are important because they can both determine whether a user continues to use your application.

Q: You run a load test against a server with 4GB RAM and the results show an average response time of 30 seconds for a particular request. The production server has been allocated 8GB RAM. What would you expect the average response time of the same request to be in production given the same load?

  • While you may be tempted to answer that the response time would be halved to 15 seconds, reality is rarely that convenient. Response times are a factor of so much more than memory. Things like CPU utilisation, network throughput, latency, load balancing configuration, and just application logic are always going to influence load tests. You can’t assume a linear progression in response time just because you’ve upgraded one part of the hardware. This is why it’s important to load test against an environment that is as production-like as possible.

Q: After running a load test, the results show that 300 transactions were executed with a PASSED status and returning responses of HTTP 200. However, the DBA tells you that there were no records written to the database as a result of the transactions. How would you troubleshoot this?

  • Its important to add various counters and monitors DB servers too. + Its very important to add assertions to make sure a request is successful. In the above mentioned case, it can be a bug or it can be due to poor scripting too. Adding right assertions, and counters will help in troubleshooting.

Q: Have you heard about New Relic or Datadog?

  • These are the monitoring tools.

Q: What are the benefits of monitoring tools like – New Relic , Datadog, etc?

  • Monitoring tools lets you monitor all your infrastructure in one place by creating monitors that will continuously check the health of your infrastructure. Access everything you need in one dashboard and see the data as it comes in straight from your systems.
  • Establishes end-to-end application performance and stability, by identifying and leveraging preventative tasks on-demand.
  • Creation of active metrics associated with data response times, CPU usage, load-balance alarms, exceptions etc.

Stay Tuned For Part 2..

Posted on

Challenges with Traditional Testing and the adoption of Continuous Testing

Organizations now expect software development teams to deliver more, and more innovative, software within shorter delivery cycles. To meet these demands, teams have turned to lean approaches, such as Agile, DevOps, and Continuous Delivery, to try to speed up the systems development life cycle (SDLC). After accelerating other aspects of the delivery pipeline, teams typically find that their testing process is preventing them from achieving the expected benefits of their SDLC acceleration initiative. Testing and the overall quality process remain problematic for several key reasons:

[[Also Read: Understanding the Continuous Testing in DevOps]]

  • Traditional testing processes are too slow. Iteration length has changed from months to weeks or days with the rising popularity of Agile, DevOps, and Continuous Delivery. Traditional methods of testing, which rely heavily on manual testing and automated GUI tests that require frequent updating, cannot keep pace. At this point, organizations tend to recognize the need to extend their test automation efforts.
  • Even after more automation is added to the existing test process, managers still lack adequate insight into the level of risk associated with an application at any given point in time. Understanding these risks is critical for making the rapid go/no go decisions involved in Continuous Delivery processes. If tests are developed without an understanding of what the business considers to be an acceptable level of risk, it is possible to have a release candidate that passes all the available tests, but which the business leaders would not consider to be ready for release. For the test results to accurately indicate whether each release candidate meets business expectations, the approach to designing tests must be based on the business’s tolerance for risks related to security, performance, reliability, and compliance. In addition to having unit tests that check code at a very granular bottom-up level, there is a need for a broader suite of tests to provide a top-down assessment of the release candidate’s business risk.
  • Even if testing is automated and tests effectively measure the level of business risk, teams without a coordinated end-to-end quality process tend to have trouble satisfying the business expectations within today’s compressed delivery cycles. Trying to remove risks at the end of each iteration has been shown to be significantly slower and more resource-intensive than building quality into the product through defect prevention strategies such as development testing.
  • Organizations adopt Continuous Testing because they recognize that these problems are preventing them from delivering quality software at the desired speed. They recognize the growing importance of software as well as the rising cost of software failure, and they are no longer willing to make a tradeoff between time, scope, and quality.

Continuous testing has evolved to become an important phase in modern application development and delivery. Its very important for today’s testers to understand the Shift Left and Continuous Testing practices. It is not possible achieve Shift Left / Continuous Testing without enhancing the skill set to SDET/Full Stack QE

Posted on

Understanding the Continuous Testing in DevOps

Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests. Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

[[Also Read: Challenges with Traditional Testing and the adoption of Continuous Testing]]

Following Key Points will help in understanding the Continuous Testing:

  • Continuous Testing’s primary goal is assessing business risk coverage
  • Continuous Testing provides instant insight on whether a release candidate is too risky to proceed through the delivery pipeline
  • Continuous Testing establishes a safety net that helps the team protect the user experience in accelerated development processes and avoid software failure headlines
  • Continuous Testing expects testing to be embedded within the development process, not tacked on at the end
  • Continuous Testing is seamlessly integrated into the software delivery pipeline and DevOps toolchain
  • Continuous Testing expects a stable test environment with valid test data to be available for each and every test run
  • Continuous Testing embraces everything from “shift left” (unit, component, coverage…) to “shift right” (monitoring/ APM, Testing in Production)
  • Continuous Testing involves executing the right set of tests at the right stage of the delivery pipeline—without creating a bottleneck
  • Continuous Testing delivers actionable feedback appropriate for each stage of the delivery pipeline
  • Continuous Testing evaluates each layer of a modern architecture at the appropriate stage of the delivery pipeline
  • Continuous Testing includes end-to-end tests that realistically assess the end-user experience across all associated technologies (front-end and back-end)
  • Continuous Testing’s tests must be broad enough to detect when an application change inadvertently impacts functionality that users have come to rely on
  • Continuous Testing reduces false positives by prioritizing robust, flexible modern test frameworks over brittle scripts
  • Continuous Testing involves continuously reviewing and optimizing the test suite to eliminate redundancy and maximize business risk coverage

source

Posted on

Mutation testing with Stryker

Test your tests with mutation testing.

What is mutation testing?

Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

It’s that simple.

Are you still confused? Why not take a look at our example page and try it out yourself?

But wait, what about code coverage?

Well… code coverage doesn’t tell you everything about the effectiveness of your tests. Think about it, when was the last time you saw a test without an assertion, purely to increase the code coverage?

Imagine a sandwich covered with paste. Code coverage would tell you the bread is 80% covered with paste. Mutation testing, on the other hand, would tell you it is chocolate paste and not… well… something else.

Meet: Stryker

Sounds complicated? Don’t worry! Stryker has your back. It uses one design mentality to implement mutation testing on three platforms. It’s easy to use and fast to run. Stryker will only mutate your source code, making sure there are no false positives.

Checkout this example which demonstrates why code coverage is not sufficient and why there is a need for tools like Stryker – https://stryker-mutator.io/example

Stryker is an open source tool. Give this a try: https://stryker-mutator.io/

Posted on

Continuous Testing Practices

  • Testing should be a collaboration of Development, QA, and Operations—aligned with the priorities of the line of business—within a coordinated, end-to-end quality process.
  • Tests should be logically-componentized, incremental, and repeatable; results must be deterministic and meaningful.
  • All tests need to be run at some point in the build pipeline, but not all tests need be run all the time.
  • Eliminate test data and environment constraints so that tests can run constantly and consistently in production-like environments.
  • To minimize false positives, minimize test maintenance, and more effectively validate use cases across modern systems with multi-tier architectures, teams should emphasize API testing over GUI testing.