This post will give you enough ideas to improve software delivery performance (and Automation Plays a key role here) and how to measure software delivery performance using statistical methods. I would like to share industry proven and practical tips which helps you speed up the Software Delivery Performance.
Accelerate classified the capabilities into 4 categories:
Technology & Automation capabilities:
Product and Process Capabilities:
Lean Management and Monitoring Capabilities:
Cultural Capabilities:
In next couple of articles, we will go over these capabilities one by one.
The purpose of the test pyramid is to provide a structure for designing and executing tests. It is also the cornerstone of testing at Google.
The test pyramid has four layers – unit tests, service tests, integration tests, end-to-end or acceptance tests. By dividing up the different types of tests into these levels, it becomes easier to decide what type of testing needs to be done when building and maintaining software.
At the bottom are unit tests. Unit tests are “white box” tests because they interact directly with product code. They typically perform operations on functions, methods, and classes. Unit tests should be concise, to the point, and focused on a single object/variation. They should be devoid of external dependencies and should instead rely on mocks/monkey-patching. These tests took less time to execute.
Integration tests are located in the center. Integration tests examine the point at which two distinct entities come together. They should be “black box” in nature, interacting with actual instances of the product being tested, rather than with code. Integration tests include service call tests (REST, SOAP, and others).
End-to-end tests are at top level. End-to-end tests trace a system’s path. They could be argued to be a multi-step integration test, and they should also be “black box.” Typically, they interact with the product in the same way that a real user would. Web UI tests are an example of integration tests, as they require the entire stack to run.
And manual testing is the cherry on top. Certain types of tests, such as exploratory or usability testing, cannot be automated, but we should always strive to minimize the number of manual tests.
You should be aware that as we progress up the pyramid, three factors begin to increase:
Costs associated with the development and maintenance of tests
Time frame for execution
Possibility of erroneous negatives
Key aspects relating to the software testing pyramid include the following:
Avoid conducting duplicate tests at multiple levels. There is no reason to conduct the same tests at various levels. For instance, if boundary values were validated in unit tests, they should not be repeated in the GUI – you will not obtain any new data.
One should strive to reduce the difficulty of tests whenever possible. For example, you can verify interest calculation on a negative sum at a “medium level” or even at the unit test level. Therefore, why should you do it at the UI level? Lower-level test automation is more efficient; it enables you to identify defects earlier and saves you time and money.
This is a list of awesome Containerization/Docker Interview Questions for Testers. These are shared with in one of our groups. Please email us at i.m@fullstackqe.org, we will give the credit to the awesome person who prepared this list.
To that end, Google employs a four-stage testing process for changes to the search engine, consisting of:
Testing by dedicated, internal testers (Google employees)
Further testing on a crowdtesting platform
“Dogfooding,” which involves having Google employees use the product in their daily work
Beta testing, which involves releasing the product to a small group of Google product end users
Facebook: Developer-driven testing
Facebook employs a wide variety of automated testing solutions. The tools that are used range from PHPUnit for back-end unit testing to Jest (a JavaScript test tool developed internally at Facebook) to Watir for end-to-end testing efforts.
Amazon: Deployment comes first
The feeling at Amazon is that its development and deployment processes are so mature (the company famously deploys software every 11.6 seconds!) that there is no need for elaborate and extensive testing efforts. It is all about making software easy to deploy, and, equally if not more important, easy to roll back in case of a failure.
Spotify: Squads, tribes and chapters
Testing at Spotify is taken very seriously. Just like programming, testing is considered a creative process, and something that cannot be (fully) automated. Contrary to most other companies mentioned in this article, Spotify heavily relies on dedicated testers that explore and evaluate the product, instead of trying to automate as much as possible.
Microsoft: Engineers and testers are one
Microsoft’s ratio of testers to developers is currently around 2:3 (SDETs)
In this 6 Hrs long masterclass video, Tariq discusses how we can innovate and make our testing better through smarter automation and the use of artificial intelligence.
Tariq King is the senior director and engineering fellow for quality and performance at Ultimate Software. With more than fifteen years’ experience in software testing research and practice, Tariq heads Ultimate Software’s quality program by providing technical and people leadership, strategic direction, staff training, and research and development in software quality and testing practices. His primary research interest is engineering autonomous self-testing systems. He is cofounder with Jason Arbon of the Artificial Intelligence for Software Testing Association.
[Must check Speakers and Agenda, even if you don’t want to register. You will love the topics].
Register Now & Get Flat 20% Discount on tickets. Use code [STT20]
India: INR 1200/-
Other Countries: USD 16
TestCon 2020 is the Software Testing Conference that brings together hundreds of Test Professionals, who seek to improve their skills to fit new market requirements and stay tuned with the latest trends!
Parallel testing is an automated testing process that developers and testers can launch multiple tests against different real device combinations and browser configurations simultaneously. The goal of parallel testing is to resolve the constraints of time by distributing tests across available resources.
For example, if 20 test cases take a total of 100 minutes to complete, then 10 parallel execution could run 2 test cases each and bring the total testing time down to 10 minutes. Ideally, if you have sufficient resources, say 20 real mobile devices and real browsers for simultaneous execution of 20 test cases, then you’ll be able to significantly shrink the runtime to 5 minutes.
Benefits of Parallel Testing:
Speed: Sequential testing is time-consuming, while parallel testing will allow you to divide invested time by the number of environments. In order to test your application against ten devices, all you need to do is write only ONE script and run it against all your target devices, cutting your testing time by ten times.
Cost-Efficiency: The building, maintaining and keeping your own test environment up-to-date can burn a hole in your pocket. When it comes to parallel testing, maintenance isn’t a headache anymore — in fact, you lease the needed testing environment, always updated. Plus, cloud-based testing grids allow you to run tests at high concurrency, making the cost per test significantly lower.
Better Coverage: It’s always a good idea to run your application through as many platform-device-browser combinations as possible so that no bug sneaks in. Parallel testing will take your test coverage to the next level, giving you a significant ROI boost.
Optimization of Your CI/CD Processes: Parallel testing is the best friend of continuous integration and delivery. By testing in parallel, you can run tests as soon as developers submit new code updates throughout the entire SDLC. Timely reporting and quick feedback in parallel testing will also facilitate better communication between various departments.
Improvement of Testing Practices: Parallel testing improves the QA routine in your company. The reason is crystal clear: by testing at high speed, you can test more. This gives your QA team a chance to improve their testing practices and pinpoint bugs faster.
Automated tests are a key component of CI (continuous integration) pipelines. They provide confidence that with newly added check-ins, the build will still work as expected. In some cases, the automated tests have the additional role of gating deployments upon failure.
With such a critical responsibility, it’s important that automated tests are developed to meet the needs of continuous integration.
In this video, Angie Jones (Applitools) and Jessica Deen (Microsoft) discussed key factors that should be considered when writing tests that will run as part of CI.
Angie and Jessica also showed a live demo of integrating frontend tests to your pipeline within an Azure DevOps build — so we can see all those theories come to life.
Use the same devices your customers use: Run tests and interact with a large selection of physical devices. Unlike emulators, physical devices give you a more accurate understanding of the way users interact with your app by taking into account factors like memory, CPU usage, location, and modifications made by manufactures and carriers to the firmware and software. We are always adding devices to the fleet.
Reproduce and fix issues faster: Manually reproduce issues and run automated tests in parallel. Device Farm collect videos, logs, and performance data so you can dive deep and solve problems quickly. For automated tests, it will identify and group issues so you can focus on the most important problems first.
Simulate real-world environments: Fine-tune your test environment by configuring location, language, network connection, application data, and installing prerequisite apps to simulate real-world customer conditions.
Choose the tests that work for you Run our built-in test suite (no scripting required) or customize your tests by selecting from open-source test frameworks like Appium, Calabash, and Espresso (see supported frameworks). You can also perform manual tests with Remote Access.
Integrate with your development workflow: Integration with CI/CD pipeline: You can use AWS CodePipeline to incorporate mobile app tests configured in Device Farm into an AWS-managed automated release pipeline. Jenkins plugin for AWS Device farm: https://github.com/awslabs/aws-device-farm-jenkins-plugin
Setup your own private device lab in the cloud Device Farm’s private device lab offering lets you choose iOS and Android devices for your exclusive use. Device Farm provisions these devices with the exact configurations you need, and lets you persist settings between sessions. Since these devices are exclusively for your use, you don’t have to wait for other users to finish using them.
Simple retry and wait strategy, no need to graduate from any test-automation university to understand the difference between “implicit waits”, “explicit waits” and “fluent waits” 🙂
In STC 2019 conference, pCloudy hosted a contest “What can disrupt mobile app testing in 2025?”. I participated and won the contest. Below are my thoughts on mobile app testing in 2025:
Intelligent Digital Mesh:
Intelligent Digital Mesh is the mesh of people, devices, digital content and services. It will transform the mobile businesses and apps.
1. Intelligent apps use AI and machine learning to interact in a more intelligent way with people and surroundings. A recent example is Google’s Motion Sense feature in pixel 4 (radar-based technology developed by Google) where you can control apps on mobile by moving hand/fingers in air without touching the mobile screen.
Soon this will be adopted by mobile apps and the challenge will be how to test such features. Next is what if we implement AI/ML in Motion Sense tech where system learns the user gestures.
2. Conversational platforms (like Alexa, Google Assistant, Siri) when integrated with digital twins or AR/VR and mixed reality are changing the way that people perceive and interact with the digital world and devices.
3. Block chain, event driven when combined with AI/ML and Conversational platforms will make the testing further complicated.
We need to build strong AI/ML automated algo for mobile testing to test all the above mentioned stack.
Note – Intelligent Digital Mesh is in the Gartner’s Top 10 Strategic Technology Trends for future from last couple of years.
A couple of years back, the focus of Testing was “Shift Left”. It is still the focus for the companies who are not able to adopt the Shift Left practices.
Now after Shift Left / CI/CD, Automation tool companies are now looking how to bring intelligence in Test automation which can help in better test coverage and delivering quality at speed.
In this post, we will go over some intelligent features of Functionize and how they are using AI in bringing intelligence in automation and efficiency in Quality and Delivery.
Artificial intelligence ( Machine learning, Computer Vision & Natural language processing) can help in speeding up the automation test scripting, analysis and maintenance.
Below are some of the examples of how Functionize is using AI which helps in delivering Quality@Speed:
Faster test creation: Write tests in plain English (no, not Gherkin) and the framework with identify the objects by analyzing the DOM and perform actions. Example below:
Open xyz.com
Enter Username “abc”
Enter password “asdf123”
Navigate to Contact module.
Click in person record “Braidy”
Self Healing Scripts: Framework identifies the script failure reasons(due to change in object properties) and give your suggestions on fixing the scripts or fix the script by itself?
Autonomous Testing: All you need to do is – place a their tracking widget in your web app. Functionize will automatically create new test cases based on how live users interact with your site.
I am sure this short post will give you some ideas – how AI/ML can make our life easy and can help in delivering fast with quality.
TestOps is the reclusive member of the DevOps family. Reports of sightings have been few and far between. Although we’ve heard stories, most of us don’t know anyone who’s actually seen it in the wild. I’ve turned to my personal experience as a tester to search for clues about what TestOps is. We’ll talk about what testers have to offer in a DevOps environment and look at the magical things that happen when the worlds of testing and operations collide.
There will be real-life stories, including: simulating an environment migration under production-like traffic (+ screwing it up and learning from it)choosing an AWS architecture with actual test data to compare performance for different instance types busting the myth that you can mitigate a DOS threat by scaling horizontally. Finally, we’ll dare to look into the future at what can happen when intentionally applying a TestOps mentality to our job and what regular teams can do to get there.
Key takeaways: How getting testers involved in DevOps brings actual value.Why applying a TestOps mentality to your work can open the door to a whole new type of testing.Getting there is not as hard as you think!
A Full Stack Quality Engineers (a.k.a. Testers) is the opposite of a traditional black box tester. In contrary, a Full Stack Tester, creates business value by giving fast feedback loops on all layers of a technical process. Full Stack Testers can provide the information about the software quality faster by automating the testing activities of each layer. They are adaptive, collaborative, responsive and culturally fit in as an agile team member.
Full Stack QEs are professionals, or “nerds”, with knowledge of operating systems, databases, web servers, server side codes, browsers and client side code. These skills and flexibility allow the tester to deliver quickly and efficient.
Full Stack Testers also go deeper and specialize in automated testing with usage of different tools e.g. Tosca, Selenium, more examples. Therefore, a base knowledge in programming languages like Java is very normal within this chapter.
Their focus is on building quality in the software rather. They gave priority to Defect Prevention over Defect detection.
Continuous improvement – retrospect your practices, process being followed at set intervals and Seek for better solutions.
REMEMBER: Being technical is not enough. Understanding the need for Continuous Delivery, Continuous Integration, Continuous Testing, Its about using the right skills at right time. Technical skills can be gained easily, but change in mindset and ownership are the difficult things to adopt.
In upcoming posts, we will learn in detail about – Skills required for Full Stack Quality Engineer and How to become a Full Stack Quality Engineer.
Contract testing for API is about making sure that the API behaves according to the contract that has been agreed upon, and it can be done using different tools and frameworks.
Here’s an example of contract testing using Postman:
# Import the collection from a file
pm.test("status code is 200", function () {
pm.response.to.have.status(200);
});
pm.test("response should be a JSON", function () {
pm.response.to.have.header("Content-Type", "application/json");
});
pm.test("response should be not be empty", function () {
pm.expect(pm.response.json().data).to.not.be.empty;
});
This code is a test script written in JavaScript, it check if response status is 200, if the response type is JSON and if the JSON response data is not empty. The test will fail if the API does not adhere to the contract, and the developers can use the failure message to identify and fix the issue.
To make a successful transition from a Software Test Lead to a Testing Manager, you will need a combination of technical, leadership, and project management skills. Some key skills that are important for a Testing Manager include:
Extensive knowledge and experience with software testing processes and methodologies
The ability to develop and manage a team of software testers
Strong agile project management skills, including the ability to create and manage budgets and schedules
Excellent communication and interpersonal skills, including the ability to communicate complex technical concepts to non-technical stakeholders
The ability to identify and mitigate project risks
The ability to foster collaboration and teamwork among team members
Strong problem-solving and critical thinking skills
In addition to these technical skills, it’s also important for a Testing Manager to have strong leadership skills, such as the ability to motivate and mentor team members, provide constructive feedback, and foster a positive and productive work environment.
To invoke the Microsoft Edge browser in Selenium, you need to use the EdgeDriver class. Here’s an example of how you can do this:
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.edge.EdgeDriver;
public class EdgeTest {
public static void main(String[] args) {
// Set the path to the Microsoft Edge driver
System.setProperty("webdriver.edge.driver", "path/to/MicrosoftWebDriver.exe");
// Create a new instance of the Edge driver
WebDriver driver = new EdgeDriver();
// Open a new browser window and navigate to a website
driver.get("http://www.example.com");
}
}
Note that you need to download and install the Microsoft WebDriver server in order to use the EdgeDriver class. You can find the download link and instructions on how to install the WebDriver server on the Selenium website.
First, API testing is much faster than UI testing. This is because API tests do not have to wait for a page to load, as they interact directly with the application’s backend.
Second, API tests are more reliable than UI tests. This is because UI tests are subject to the vagaries of the user interface, such as changes in the layout or position of elements. API tests, on the other hand, are not affected by such changes.
Third, API tests are easier to create and maintain than UI tests. This is because UI tests often require a lot of code to simulate user interactions, whereas API tests only need to make calls to the application’s API.
Fourth, with API testing, one can find bugs related to functionality, reliability, performance, and security early in the SDLC and hence those are cheaper to fix.
In conclusion, API testing has many advantages over UI testing, and should be used whenever possible.
The future of software testing is likely to be more automated and focused on artificial intelligence (AI) and machine learning. This is due to the increasing complexity of software systems and the need for faster and more accurate testing.
AI and machine learning can help to automate the testing process by generating test cases and test data. They can also help to identify errors and potential areas of improvement. In addition, AI and machine learning can be used to create virtual environments for testing, which can be more realistic and representative of the real world.
As the use of AI and machine learning in testing becomes more widespread, it is likely that the role of human testers will change. They will become more focused on overseeing the automated testing process and interpreting the results.
The future of software testing is likely to be more efficient and effective as a result of the increasing use of AI and machine learning. This will benefit both developers and users by reducing the time and cost of testing, and by improving the quality of software products.
TestFlix 2022 (virtual conference) is happening on October 8th & 9th 2022.In this virtual conference Testers from top companies (across the globe) share their knowledge through Atomic Talks.
Atomic talks by definition are meant to be small, yet Powerful and that’s the whole idea this time, it has 60+ Global Speakers & 14 Themes
Topics: Automation, Testing, AI ML, Web 3.0, Performance, Leadership/Management, API & lot more
Recording Access: Practical demonstration based teaching, clear ‘how-tos’ (only to those who registered)I highly recommend you guys to register for this conference and attend the sessions which are relevant to you.
