Showing posts with label ruby. Show all posts
Showing posts with label ruby. Show all posts

Wednesday, July 6, 2016

Any browsermob-proxy users facing issues with some requests not getting fired?

Is there anyone using browsermob-proxy who is having issues with some requests not getting fired?

I have integrated browsermob-proxy with my protractor tests. This works wonderfully when I run my tests from Mac (against local environment, or any other test environment).

However, when I run my tests from CI (agent is SUSE Enterprise 11.4) - my tests fail. 

I narrowed down the problem to the following scenario:

On some specific user action in the UI, there are a lot (>100) of requests fired from the browser in parallel (batches). There are a couple of scenarios like this in my application - and the test fails in all these cases.

Here is a screenshot of what the captured HAR file shows -



The same test works when I run this locally from Mac

Any idea how to fix this? Thank you in advance!

See this issue for more details - (https://github.com/lightbody/browsermob-proxy/issues/492)


Tuesday, July 5, 2016

Any WAAT (Web Analytics Automation Testing Framework) users out there?

It has been over 2 years since any update to WAAT - Java or Ruby. Over the years, I have realised, and also received a lot of thoughts / feedback from users of WAAT around where it helps, and what challenges exist. 

Also, given the widespread IoT & Big Data based work going on around the world, (Web) Analytics now plays a much bigger role in guiding business take better decisions. 

WAAT (again) fits in the grand scheme of things very nicely as a framework to automate the validation of correct reporting of tags to any Web Analytics solution provider.

Hence, its a no-brainer for me - it is high time I work on some of the feedback and limitations of WAAT to make it usable again!

At the recently concluded Selenium Conference 2016 held in Bangalore, India, I got an idea of how to overcome a lot of challenges (listed below) and pain in using WAAT. 


What's next?

To implement my new idea, this does mean a couple of things:

  • Existing plugins have limited use - and needs to be deleted.
  • A new plugin would need to be created - which may mean different set of APIs, and also different way to specify the test data.

Questions for you

Before I go ahead making these changes - I would like to get answers to the below questions (please add your answers directly in the comments):
  • Is anyone currently using WAAT? If yes - 
    • which version (Java / Ruby)?
    • which plugin
    • Using HTTP / HTTPS?
    • Which Web Analytic solution are you using? (ex: Google Analytics, WebTrends, etc?)
  • Would you be interested in using the new WAAT? If yes - 
    • Which language? Java / Ruby / JavaScript / Python / etc?
  • Would you like to contribute to implementing this new WAAT? If yes - contact me! :)
-----------------------------------------------------------------------------------------------------------------

Current plugins available in WAAT:

  • Omniture Debugger (WAAT-Java)
    • Pros:
      • OS independent
      • Run using regular-test-user 
    • Cons:
      • Browser dependent - need to implement ScriptRunner for the UI-driver in use
      • Web-Analytic solution dependent - only for Adobe Marketing Clout / Omniture SiteCatalyst
  • HTTPSniffer (WAAT-Java, WAAT-Ruby)
    • Pros
      • Web-Analytic solution independent
      • Browser independent
      • UI-Driver independent
    • Cons
      • 3rd party libraries are OS dependent
      • HTTPs is not supported out-of-the-box
      • Run tests as "root"
  • JSSniffer (WAAT-Java, WAAT-Ruby)
    • Pros
      • Web-Analytic solution independent
      • Browser independent
      • HTTPs supported out-of-the-box
      • No 3rd party library dependency
    • Cons
      • Need to write JavaScript to get the URL from the browser context
      • UI-Driver dependent
-----------------------------------------------------------------------------------------------------------------

Sunday, November 29, 2015

Patterns in Test Automation - Framework, Data, Locators at Agile Noida

On 28th November 2015, I spoke in Agile Noida on "Patterns in Test Automation - Framework, Data, Locators".

I had spoken on the same topic (Patterns in Test Automation) in vodQA Hyderabad - but that was a Testing conference and I knew the attendees were Testing and Test Automation focused. Here, I was skeptic about how this topic would be received by the attendees - given that the conference was focused on Agile, and this topic was core Testing related, and to add to the fact - very technical where I was showing various Java / Ruby code samples.

My skepticism was, thankfully, unfounded. There was a good number of attendees that came to the talk and based on conversations after the talk, I realized that I was able to get the message across to the attendees.

Below is the abstract, slides and video from the talk.

Abstract

Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.
However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.
Design Patterns play a big role in helping achieve this goal of building a good and robust framework. 
In this talk, we will talk about, and see examples of various types of patterns you can use for:
  1. Build your Test Automation Framework
  2. Test Data Management
  3. Locators / IDs (for finding / interacting with elements in the browser / app)
Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!

Slides


Video


Saturday, August 22, 2015

Patterns in Test Automation

I spoke in vodQA Hyderabad on Sat, 22nd August 2015 about Patterns in Test Automation - Frameworks, Data & Locators.

The slides are available on SlideShare:


The video is available on YouTube:



Abstract

Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.

However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.

Design Patterns play a big role in helping achieve this goal of building a good and robust framework.

In this talk, we will talk about, and see examples of various types of patterns you can use for:

  • Build your Test Automation Framework
  • Test Data Management
  • Locators / IDs (for finding / interacting with elements in the browser / app)
Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!

Learning outcome


  • Patterns for building Test Automation Framework
  • Patterns for Test Data Management, with pros and cons of each
  • Patterns for managing locators / IDs for interaction with UI



Monday, May 11, 2015

vodQA Geek Night in ThoughtWorks, Hyderabad - Client-side Performance Testing Workshop

I am conducting a workshop on "Client-side Performance Testing" in vodQA Geek Night, ThoughtWorks, Hyderabad from 6.30pm-8pm IST on Thursday, 14th May, 2015.

Visit this page to register!

Abstract of the workshop:

In this workshop, we will see the different dimensions of Performance Testing and Performance Engineering, and focus on Client-side Performance Testing. 

Before we get to doing some Client-side Performance Testing activities, we will first understand how to look at client-side performance, and putting that in the context of the product under test. We will see, using a case study, the impact of caching on performance, the good & the bad! We will then experiment with some tools like WebPageTest and Page Speed to understand how to measure client-side performance.

Lastly - just understanding the performance of the product is not sufficient. We will look at how to automate the testing for this activity - using WebPageTest (private instance setup), and experiment with yslow - as a low-cost, programmatic alternative to WebPageTest.

Venue:

ThoughtWorks Technologies (India) Pvt Ltd.
3rd Floor, Apurupa Silpi,
Beside H.P. Petrol Bunk (KFC Building),
Gachibowli,
Hyderabad - 500032, India 

Saturday, November 22, 2014

To Deploy or Not to Deploy - decide using Test Trend Analyzer (TTA) in AgilePune 2014

I spoke on the topic - "To Deploy or Not to Deploy - decide using Test Trend Analyzer (TTA)" in Agile Pune, 2014.

The slides from the talk are available here, and the video is available here.



 

Below is some information about the content.


The key objectives of organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality!
In order for these organizations to to understand the quality / health of their products at a quick glance, typically a team of people scramble to collate and collect the information manually needed to get a sense of quality about the products they support. All this is done manually.


So in the fast moving environment, where CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury, how can teams take decisions if the product is ready to be deployed to the next environment or not?


Test Automation across all layers of the Test Pyramid is one of the first building blocks to ensure the team gets quick feedback into the health of the product-under-test.

The next set of questions are:
  • How can you collate this information in a meaningful fashion to determine - yes, my code is ready to be promoted from one environment to the next?
  • How can you know if the product is ready to go 'live'?
  • What is the health of you product portfolio at any point in time?
  • Can you identify patterns and do quick analysis of the test results to help in root-cause-analysis for issues that have happened over a period of time in making better decisions to better the quality of your product(s)?
The current set of tools are limited and fail to give the holistic picture of quality and health, across the life-cycle of the products.
 

The solution - TTA - Test Trend Analyzer
 
TTA is an open source product that becomes the source of information to give you real-time and visual insights into the health of the product portfolio using the Test Automation results, in form of Trends, Comparative Analysis, Failure Analysis and Functional Performance Benchmarking. This allows teams to take decisions on the product deployment to the next level using actual data points, instead of 'gut-feel' based decisions.
 
There are 2 sets of audience who will benefit from TTA:
1. Management - who want to know in real time what is the latest state of test execution trends across their product portfolios / projects. Also, they can use the data represented in the trend analysis views to make more informed decisions on which products / projects they need to focus more or less. Views like Test Pyramid View, Comparative Analysis help looking at results over a period of time, and using that as a data point to identify trends.

 
2. Team Members (developers / testers) - who want to do quick test failure analysis to get to the root cause analysis as quickly as possible. Some of the views - like Compare Runs, Failure Analysis, Test Execution Trend help the team on a day-to-day basis.
 
NOTE: TTA does not claim to give answers to the potential problems. It gives a visual representation of test execution results in different formats which allow team members / management to have more focussed conversations based on data points.

Some pictures from the talk ... (Thanks to Shirish)








Sunday, September 7, 2014

Perils of Page-Object Pattern

I spoke at Selenium Conference (SeConf 2014) in Bangalore on 5th September, 2014 on "The Perils of Page-Object Pattern".

Page-Object pattern is very commonly used when implementing Automation frameworks. However, as the scale of the framework grows, there is a limitation on how much re-usability really happens. It inherently becomes very difficult to separate the test intent from the business domain.
 

I want to talk about this problem, and the solution I have been using - Business Layer - Page-Object pattern, which has helped me keep my code DRY.

The slides from the talk are available here. The video is available here

Video taken by professional:


Video taken from my laptop:


Slides:




If you want to see other slides and videos from SeConf, see the SeConf schedule page.


Monday, June 23, 2014

To Deploy or Not to Deploy - decide using Test Trend Analyzer (TTA)

[UPDATE: 18th July 2014] I spoke on the same topic - "To Deploy or Not to Deploy - decide using Test Trend Analyzer (TTA)" at Unicom's World Conference on Next Generation Testing in Bangalore on 18th July 2014. The slides are available here and the video is available here. In this talk, I also gave a demo of TTA.
 
I spoke in 3 conferences last week about "To Deploy or Not to Deploy - decide using Test Trend Analyzer (TTA)"

You can find the slides here and the videos here:

Here is the abstract of the talk:

The key objectives of organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality. To understand the quality / health of their products at a quick glance, typically a team of people scramble to collate and collect the information manually needed to get a sense of quality about the products they support. All this is done manually. So in the fast moving environment, where CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury, how can teams take decisions if the product is ready to be deployed to the next environment or not? Test Automation across all layers of the Test Pyramid is one of the first building blocks to ensure the team gets quick feedback into the health of the product-under-test.
 

The next set of questions are:
    •    How can you collate this information in a meaningful fashion to determine - yes, my code is ready to be promoted from one environment to the next?
    •    How can you know if the product is ready to go 'live'?
    •    What is the health of you product portfolio at any point in time?
    •    Can you identify patterns and do quick analysis of the test results to help in root-cause-analysis for issues that have happened over a period of time in making better decisions to better the quality of your product(s)?
 

The current set of tools are limited and fail to give the holistic picture of quality and health, across the life-cycle of the products.
 

The solution - TTA - Test Trend Analyzer
 

TTA is an open source product that becomes the source of information to give you real-time and visual insights into the health of the product portfolio using the Test Automation results, in form of Trends, Comparative Analysis, Failure Analysis and Functional Performance Benchmarking. This allows teams to take decisions on the product deployment to the next level using actual data points, instead of 'gut-feel' based decisions.

Monday, May 19, 2014

WAAT at StarEast2014

I was speaking about "Build the 'right' regression suite using Behavior Driven Testing (BDT)" at StarEast 2014 and met Marcus Merrell who was speaking about "Automated Analytics Testing with Open Source Tools". I figured out deep into our conversation that he has used WAAT, and a few others at the table also were aware of and had used WAAT before. Felt great!

Thursday, April 10, 2014

Sample test automation framework using cucumber-jvm

I wanted to learn and experiment with cucumber-jvm. My approach was to think of a real **complex scenario that needs to be automated and then build a cucumber-jvm based framework to achieve the following goals:
  • Learn how cucumber-jvm works
  • Create a bare-bone framework with all basic requirements that can be reused
Once you know the basics and fundamentals of building a scalable and maintainable Test Automation frameworks, it was really easy to apply my past learning and experiences to learn cucumber-jvm and build a framework from scratch.

So, without further ado, I introduce to you the cucumber-jvm-sample Test Automation Framework, hosted on github. 

Following functionality is implemented in this framework:

  • Tests specified using cucumber-jvm
  • Build tool: Gradle
  • Programming language: Groovy (for Gradle) and Java
  • Test Data Management: Samples to use data-specified in feature files, AND use data from separate json files
  • Browser automation: Using WebDriver for browser interaction
  • Web Service automation: Using cxf library to generate client code from web service WSDL files, and invoke methods on the same
  • Take screenshots on demand and save on disk
  • Integrated cucumber-reports to get 'pretty' and 'meaningful' reports from test execution
  • Using apache logger for storing test logs in files (and also report to console)
  • Using aspectJ to do byte code injection to automatically log test trace to file. Also creating a separate benchmarks file to track time taken by each method. This information can be mapped separately in other tools like Excel to identify patterns of test execution.

Feel free to fork and use this framework on your projects. If there are any other features you think are important to have in a Test Automation Framework, let me know. Even better would be to submit pull requests with those changes, which I will take a look at and accept if it makes sense.

** Pun intended :) The complex test I am talking about is a simple search using google search.

Friday, March 28, 2014

WAAT Java v1.5.1 released today

After a long time, and with lot of push from collaborators and users of WAAT, I have finally updated WAAT (Java) and made a new release today. 

You can get this new version - v1.5.1 directly from the project's dist directory.

Once I get some feedback, I will also update WAAT-ruby with these changes.

Here is the list of changes in WAAT_v1.5.1:

Changes in v1.5.1

  • Engine.isExpectedTagPresentInActualTagList in engine class is made public
  • Updated Engine to work without creating testData.xml file, and directly sending exceptedSectionList for tags
    Added a new method
        Engine.verifyWebAnalyticsData(String actionName, ArrayList
    expectedSectionList, String[] urlPatterns, int minimumNumberOfPackets)
  • Added an empty constructor for Section.java to prevent marshalling error
  • Support Fragmented Packets
  • Updated Engine to support Pattern comparison, instead of String contains
Do let me know if you see any problems / issues with this update.

Thanks.

Thursday, February 27, 2014

Automate across Platform, OS, Technologies with TaaS

[Updated - link to slides, audio, experience report added]

The talk at Agile India 2014 went really well. A few things happened:
  • My talk was the 2nd last talk on a Saturday. My hopes of having a decent sized audience was low. But I was very pleasantly surprised to see the room almost full.
  • Usually in conferences I have spoken at, the ratio of technical / hands-on people Vs leads / managers is around 20:80. In this case, that ratio was almost inverted. There were about 70-80% technical / hands-on people in the audience.
  • Due to the higher technical audience, there were great questions all along the way - which resulted in me not able to complete on time ... I actually went over by 5-6 minutes and that too had to really rush through the last few sections of my presentation, and was not able to do a complete demo.
  • Almost everyone was able to relate to the challenges of integration test automation, the concept and the problem TaaS solves - which was a great validation for me!
  • Unfortunately I had to rush to the airport immediately after the talk, which prevented me from networking and talking more specifics with folks after the talk. Hopefully some of them will be contacting me to know more about TaaS!

The slides from the talk are available here on slideshare. You can download the audio recording from the talk from here. The pictures and video should be available from Agile India 2014 site soon. I will update the links for the same when that becomes available.

--------

After what seems to be a long time, I am speaking in Agile India 2014 in Bangalore on  "Automate across Platform, OS, Technologies with TaaS".

I am changing the format of the talk this time and am hoping to do a good demo instead of just showing code snippets. Hopefully no unpleasant surprises crop up in the demo!! 

Other than that, really looking forward to interacting with a lot of fellow-enthusiasts at the conference.

Slides and experience report to follow soon.

Saturday, October 12, 2013

vodQA Pune - Faster | Smarter | Reliable schedule announced

A very impressive and engrossing schedule for vodQA Pune scheduled for Saturday, 19th October 2013 at ThoughtWorks, Pune has now been announced. See the event page for more details.

I am going to be talking about "Real-time Trend and Failure Analysis using Test Trend Analyzer (TTA)"

Sunday, October 6, 2013

Offshore Testing on Agile Projects


Offshore Testing on Agile Projects …
Anand Bagmar

Reality of organizations

Organizations are now spread across the world. With this spread, having distributed teams is a reality. Reasons could be a combination of various factors, including:

Globalization
Cost
24x7 availability
Team size
Mergers and Acquisitions
Talent

The Agile Software methodology talks about various principles to approach Software Development. There are various practices that can be applied to achieve these principles. 

The choice of practices is very significant and important in ensuring the success of the project. Some of the parameters to consider, in no significant order are:

Skillset on the team
Capability on the team
Delivery objectives
Distributed teams
Working with partners / vendors?
Organization Security / policy constrains
Tools for collaboration
Time overlap time between teams
Mindset of team members
Communication
Test Automation
Project Collaboration Tools
Testing Tools
Continuous Integration

** The above list is from a Software Testing perspective.

This post is about what practices we implemented as a team for an offshore testing project.

Case Study - A quick introduction

An enterprise had a B2B product providing an online version of a physically conducted auction for selling used-vehicles, in real-time and at high-speed. Typical participation in this auction is by an auctioneer, multiple sellers, and potentially hundreds of buyers. Each sale can have up to 500 vehicles. Each vehicle gets sold / skipped in under 30 seconds - with multiple buyers potentially bidding on it at the same time. Key business rules: only 1 bid per buyer, no consecutive bids by the same buyer.

Analysis and Development was happening across 3 locations – 2 teams in the US, and 1 team in Brazil. Only Testing was happening from Pune, India.

“Success does not consist in never making mistakes but in never making the same one a second time.”

We took that to heart and very sincerely. We applied all our learning and experiences in picking up the practices to make us succeed. We consciously sought to creative, innovative and applied out-of-the-box thinking on how we approached testing (in terms of strategy, process, tools, techniques) for this unique, interesting and extremely challenging application, ensuring we do not go down the same path again.

Challenges

We had to over come many challenges for this project.
  • Challenge in creating a common DSL that will be understood by ALL parties - i.e. Clients / Business / BAs / PMs / Devs / QAs
  • All examples / forums talk using trivial problems - whereas we had lot of data and complex business scenarios to take care of.
  • Cucumber / capybara / WebDriver / ruby do not allow an easy way to do concurrency / parallel testing
  • We needed to simulate in our manual + automation tests for "n" participants at a time, interacting with the sale / auction
  • A typical sale / auction can contains 60-500 buyers, 1-x sellers, 1 auctioneer. The sale / auction can contain anywhere from 50-1000 vehicles to sell. There can be multiple sales going on in parallel. So how do we test these scenarios effectively?
  • Data creating / usage is a huge problem (ex: production subset snapshot is > 10GB (compressed) in size, refresh takes long time too,
  • Getting a local environment in Pune to continue working effectively - all pairing stations / environment machines use RHEL Server 6.0 and are auto-configured using puppet. These machines are registered to the Client account on the RedHat Satellite Server.
  • Communication challenge - We are working from 10K miles away - with a time difference of 9.5 / 10.5 hours (depending on DST) - this means almost 0 overlap with the distributed team. To add to that complexity, our BA was in another city in the US - so another time difference to take care of.
  • End-to-end Performance / Load testing is not even a part of this scope - but something we are very vary of in terms of what can go wrong at that scale.
  • We need to be agile - i.e. testing stories and functionality in the same iteration.

All the above-mentioned problems meant we had to come up with our own unique way of tackling the testing.

Our principles - our North Star

We stuck to a few guiding principles as our North Star:
  • Keep it simple
  • We know the goal, so evolve the framework - don't start building everything from step 1
  • Keep sharing the approach / strategy / issues faced on regular basis with all concerned parties and make this a TEAM challenge instead of a Test team problem!
  • Don't try to automate everything
  • Keep test code clean

The End Result

At the end of the journey, here are some interesting events from the off-shore testing project:
  • Tests were specified in form of user journeys following the Behavior Driven Testing (BDT) philosophy – specified in Cucumber.
  • Created a custom test framework (Cucumber, Capybara, WebDriver) that tests a real-time auction - in a very deterministic fashion.
  • We had 65-70 tests in form of user journeys that covers the full automated regression for the product.
  • Our regression completed in less than 30 minutes.
  • We had no manual tests to be executed as part of regression.
  • All tests (=user journeys) are documented directly in Cucumber scenarios and are automated
  • Anything that is not part of the user journeys is pushed down to the dev team to automate (or we try to write automation at that lower level)
  • Created a ‘special’ Long running test suite that simulates a real sale with 400 vehicles, >100 buyers, 2 sellers and an auctioneer.
  • Created special concurrent (high speed parallel) tests that ensures even at highest possible load, the system is behaving correctly
  • Since there was no separate performance and load test strategy, created special utilities in the automation framework, to benchmark “key” actions.
  • No separate documentation or test cases ever written / maintained - never missed it too.
  • A separate special sanity test that runs in production after deployment is done, to ensure all the integration points are setup properly
  • Changed our work timings (for most team members) from 12pm - 9pm IST to get some more overlap, and remote pairing time with onsite team.
  • Setup an ice-cream meter - for those that come late for standup.

Innovations and Customizations

Necessity breeds innovation! This was so true in this project.

Below is a table listing all the different areas and specifics of the customization we did in our framework.

Dartboard

Created a custom board “Dartboard” to quickly visualize the testing status in the Iteration. See this post for more details: “Dartboard – Are you on track?

TaaS

To automate the last mile of Integration Testing between different applications, we created an open-source product – TaaS. This provides a platform / OS / Tool / Technology / Language agnostic way of Automating the Integrated Tests between applications.

Base premise for TaaS:

Enterprise-sized organizations have multiple products under their belt. The technology stack used for each of the product is usually different – for various reasons.

Most of such organizations like to have a common Test Automation solution across these products in an effort to standardize the test automation framework.

However, this is not a good idea! If products in the same organization can be built using different / varied technology stack, then why should you pose this restriction on the Test Automation environment?

Each product should be tested using the tools and technologies that are “right” for it.

TaaS” is a product that allows you do achieve the “correct” way of doing Test Automation.

See my blog for all information related to TaaS.

WAAT - Web Analytics Automation Testing Framework

I had created the WAAT framework for Java and Ruby in 2010/2011. However this framework had a limitation - it did not work products what are configured to work only in https mode.

For one of the applications, we need to do testing for WebTrends reporting. Since this application worked only in https mode, I created a new plugin for WAAT  - JS Sniffer that can work with https-only applications. See my blog for more details about WAAT.