Friday, April 8, 2011

WAAT - Web Analytics Automation Testing Framework

[UPDATE]  See my post about how you can get WAAT here (http://essenceoftesting.blogspot.com/2011/04/waat-is-alive.html).

Problem statement:

On one of the projects that I worked on, I needed to test if Omniture reporting was done correctly.

The client relied a lot on Omniture reports to understand and determine the direction of their business. They have a bunch of Omniture tags reported for a lot of different actions on the site. Manual testing was the only way this functionality could be done verified. But given the huge number of tags, it was never possible to be sure that all tags were being reported correctly on a regular basis.

So I came up with a strategy to remove this pain-point.

Approach:
I created a framework in our existing automation framework to do Omniture testing. The intention of creating this framework was:
1. There is minimal impact on existing tests.
2. There should be no need to duplicate the tests just to do Omniture testing.
3. Should be easy to use (specify Omniture tags for various different actions, enable testing, etc.)

How it helped us?
1. We provided a huge and reliable safety net to the client and the development team by having Omniture testing automated.
2. Reduced the manual testing effort required for this type of testing, and instead got some bandwidth to focus on other areas.

Next Steps:
I am making this into a generic framework - a.k.a. WAAT - Web Analytics Automation Testing Framework to enable others doing Omniture testing to easily automate this functionality. This project will be hosted on github.

Phase 1 of this implementation will be for Omniture Debugger and input data provided in xml format. This framework will be available as a jar file.

Phase 2 also now complete includes support for any Web Analytic tool. I have tested this with Google Analytics as well as Omniture (NOT using Omniture Debugger). This uses a generic mechanism to capture packets from the network layer and processes them appropriately. Given this generic approach to work with any Web Analytic tool, the framework does become OS dependent.

Watch this space for more information (instructions, links to github, etc). Also, please do contact me with ideas / suggestions / comments about the same.

27 comments:

  1. I would be very interested to see how this turns out, since on our project we use Google Analytics heavily, with nothing other than manual testing. Please keep us up to date as this progresses!

    ReplyDelete
  2. What OS do you use for automation? Also, what programming language do you use in this project?

    WAAT will be available as a Java .jar file for Windows and Linux platforms.

    I should be able to get something hosted on github hopefully next week.

    Anand

    ReplyDelete
  3. Anand,

    Two questions for you, how does this differ from what companies such as ObservePoint offer with their site audits?

    Secondly, will your solution work with more than Omniture (ie, Comscore, Nielsen, etc.)?

    -S

    ReplyDelete
  4. WAAT is a Automation Test Framework addin. It will work as part of your Java functional testing framework (Ruby port is in the works).

    What Web Analytics solution providers offer is the analysis of the usage of your website. You have to program your site to send certain "specific" data to these providers, so they can provide "custom" reports to you.

    What WAAT does is validate the correct tags are being reported from the client browser as the site is in development / or also in production to the Web Analytics solution of your choice with the help of test automation.

    WAAT has been tested with Omniture and Google Analytics. I have not tested with ObservePoint, Comscore nor Neilson - but as long as the means of reporting the tags is via http, WAAT will be able to test it.

    Hope this helps.

    ReplyDelete
  5. Are you interested in porting this for a .Net project ?

    ReplyDelete
  6. I have plans to port this for Ruby and .Net - depending on interest and requirement.

    ReplyDelete
  7. We heavily use Omniture for all our web applications (Java and Ruby). We have ~50 applications where business demands omniture usage reports. As you have rightly said, it becomes nightmare for our testers to validate all patterns of user activity. Looking forward for more information.

    ReplyDelete
  8. Excellent pieces. Keep posting such kind of information on your blog. I really impressed by your blog.

    ReplyDelete
  9. Hi Anand,

    Do you have the java docs for the classes you have created.This would be greatly useful to customize this tool for our testing purpose.

    Thanks.
    Shruti

    ReplyDelete
    Replies
    1. Hi Shruti,
      The detailed WAAT documentation is available here: https://github.com/anandbagmar/waat/wiki
      The source code is available here: https://github.com/anandbagmar/WAAT.
      There is also some documentation I have created in https://github.com/anandbagmar/WAAT

      Is there something specific and different you are looking to find out from the Java docs which is not available in the links mentioned above? I can try and get that info for you.

      Delete
  10. Thanks for sharing the post. This post having some nice links shared about the testing methodologies. Really nice link shared. Thanks for sharing the post..

    Loadrunner Services

    ReplyDelete
  11. Hi Anand,

    I'm trying to associate WAAT with Eclipse, but unfortunately only errors are displayed. Can you please provide detailed instruction how to associate newly downloaded WAAT with newly created eclipse java project.

    Thank you,
    And

    ReplyDelete
    Replies
    1. Hi,
      What errors are you seeing? There should really be no difference in using WAAT with any IDE - as long as the correct libraries are setup. Can you list what you have tried doing? Better yet, can you share a sample project with me?

      Delete
    2. Hi,

      Please see my steps below:
      1) Unzip WAAT
      2) Create new Java project in Eclipse
      3) Associate all JARs from WAAT/lib folder (as well as other folders located in WAAT/lib) with new Java project in Eclipse via Properties -> Java Build Path -> Add External JARs

      Later when I'm trying to import relevant packages for WAAT using JSSniffer combination, but Eclipse can't find them:
      /*
      Import packages
      */
      import com.thoughtworks.webanalyticsautomation.common.BROWSER;
      import com.thoughtworks.webanalyticsautomation.plugins.WebAnalyticTool;
      import static com.thoughtworks.webanalyticsautomation.Controller.getInstance;
      import static com.thoughtworks.webanalyticsautomation.common.Utils.currentDirectory;
      import static com.thoughtworks.webanalyticsautomation.common.Utils.fileSeparator;

      I've tried to link classes from WAAT/src/com.thoughtworks/* with new Java project in Eclipse via Properties -> Java Build Path -> Add External Class folder -> move classes under my main project src folder and rename packages accordingly, but it looks incorrect

      Thank you,
      And

      Delete
    3. Hi,

      Seems to be on the right track. Haven't used Eclipse before so do not know the specific details of Eclipse configuration. If this was IntelliJ Idea - then if the jars are loaded correctly in the classpath, then Idea would tell you what is wrong in the import. Does Eclipse tell that to you?

      Delete
  12. Hi Anand,
    I am new to web analytics testing, i have explored WAAT framework but not able to configure it.
    Is there any post or link where i can find the step by step process to set up framework for web analytics testing.
    Thanks!

    ReplyDelete
    Replies
    1. Hi,
      Did you look at the WAAT-wiki (https://github.com/anandbagmar/WAAT-ruby/wiki, or https://github.com/anandbagmar/WAAT/wiki?

      Let me know what you have done so far.

      Thanks. Anand

      Delete
    2. As Per the WAAT read me doc, So far, I am done with the following activities.
      1. Created a new project with eclipse
      2. Unziped the WAAT file in the lib folder of created project, Associated all Jars from WAAT/lib folder
      3. Copied and pasted the seleniumscriprunner.java file interface in 'src' folder
      4. Test code changes is pending - shall i create a new class file in same project with the code base provided by you??

      Thanks for your help!

      Delete
    3. Hi,
      Ideally you would have your own test automation project (already configured in Idea / Eclipse / ..) and also running from command-line. In this project, you would import WAAT jars and update your existing tests, or create new tests to enable web analytics validation. Does that help?

      I will also create a new sample project how you can incorporate web analytics testing using WAAT and put it on github sometime next week.

      Thanks, Anand

      Delete
    4. HiAnand,

      Great work, please help me with the sample project, i am in a middle of a poc and i am not able to proceed,kindly help.

      Thanks

      Delete
  13. Hi Anand, did you get chance to create sample project ,i am not able to configure WAAT in eclipce ,Could provide detailed step by step process.

    Thanks

    ReplyDelete
  14. Hi, Keep an eye out on the waat-java-sample project on github (https://github.com/anandbagmar/waat-java-sample) for the sample project to use WAAT. Note that this is still created using IntelliJ IDEA. I welcome contributions to a sample project using Eclipse - however I do not have time right now to make that happen - sorry.

    ReplyDelete
  15. Hi Anand ,

    After setting up the Waat framework using the JS Sniffer approach, i am getting the following exception

    org.openqa.selenium.WebDriverException: window.WebTrends.prototype is undefined
    Command duration or timeout: 63 milliseconds
    Build info: version: '2.40.0', revision: 'fbe29a9', time: '2014-02-19 20:54:28'
    System info: host: 'NOISSI11601495', ip: '10.203.100.160', os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.7.0_02'
    Session ID: 79766c3d-b29b-499b-802a-a55eb30db098
    Driver info: org.openqa.selenium.firefox.FirefoxDriver
    Capabilities [{platform=XP, acceptSslCerts=true, javascriptEnabled=true, browserName=firefox, rotatable=false, locationContextEnabled=true, version=17.0.1, cssSelectorsEnabled=true, databaseEnabled=true, handlesAlerts=true, browserConnectionEnabled=true, nativeEvents=true, webStorageEnabled=true, applicationCacheEnabled=true, takesScreenshot=true}]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:193)
    at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:573)
    at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:482)
    at com.thoughtworks.webanalyticsautomation.plugins.JsSnifferSampleTest.extractUrlFromJavascript(JsSnifferSampleTest.java:85)
    at com.thoughtworks.webanalyticsautomation.plugins.JsSnifferSampleTest.captureAndVerifyDataReportedToWebAnalytics_JsSniffer_Webtrends_Selenium_Firefox(JsSnifferSampleTest.java:71)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
    at org.testng.TestRunner.privateRun(TestRunner.java:767)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    at org.testng.SuiteRunner.run(SuiteRunner.java:240)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    at org.testng.TestNG.run(TestNG.java:1057)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)


    Can u please help and also explain the meaning of the variable used in your javascript such as WT,DCS etc...

    what do these variables capture.

    Any help in this regard will be highly appreciated.

    Thanks,
    Simran

    ReplyDelete
  16. hi anand,

    Hi Anand ,

    After setting up the Waat framework using the JS Sniffer approach, i am getting the following exception

    org.openqa.selenium.WebDriverException: window.WebTrends.prototype is undefined
    Command duration or timeout: 63 milliseconds
    Build info: version: '2.40.0', revision: 'fbe29a9', time: '2014-02-19 20:54:28'
    System info: host: 'NOISSI11601495', ip: '10.203.100.160', os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.7.0_02'
    Session ID: 79766c3d-b29b-499b-802a-a55eb30db098
    Driver info: org.openqa.selenium.firefox.FirefoxDriver
    Capabilities [{platform=XP, acceptSslCerts=true, javascriptEnabled=true, browserName=firefox, rotatable=false, locationContextEnabled=true, version=17.0.1, cssSelectorsEnabled=true, databaseEnabled=true, handlesAlerts=true, browserConnectionEnabled=true, nativeEvents=true, webStorageEnabled=true, applicationCacheEnabled=true, takesScreenshot=true}]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:193)
    at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:573)
    at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:482)
    at com.thoughtworks.webanalyticsautomation.plugins.JsSnifferSampleTest.extractUrlFromJavascript(JsSnifferSampleTest.java:85)
    at com.thoughtworks.webanalyticsautomation.plugins.JsSnifferSampleTest.captureAndVerifyDataReportedToWebAnalytics_JsSniffer_Webtrends_Selenium_Firefox(JsSnifferSampleTest.java:71)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
    at org.testng.TestRunner.privateRun(TestRunner.java:767)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    at org.testng.SuiteRunner.run(SuiteRunner.java:240)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    at org.testng.TestNG.run(TestNG.java:1057)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)


    Can u please help and also explain the meaning of the variable used in your javascript such as WT,DCS

    what do these variables capture.

    Any help in this regard will be highly appreciated.

    Thanks,
    Simran

    ReplyDelete
  17. Hi Anand ,

    After setting up the Waat framework using the JS Sniffer approach, i am getting the following exception

    org.openqa.selenium.WebDriverException: window.WebTrends.prototype is undefined
    Command duration or timeout: 63 milliseconds
    Build info: version: '2.40.0', revision: 'fbe29a9', time: '2014-02-19 20:54:28'
    System info: host: 'NOISSI11601495', ip: '10.203.100.160', os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.7.0_02'
    Session ID: 79766c3d-b29b-499b-802a-a55eb30db098
    Driver info: org.openqa.selenium.firefox.FirefoxDriver
    Capabilities [{platform=XP, acceptSslCerts=true, javascriptEnabled=true, browserName=firefox, rotatable=false, locationContextEnabled=true, version=17.0.1, cssSelectorsEnabled=true, databaseEnabled=true, handlesAlerts=true, browserConnectionEnabled=true, nativeEvents=true, webStorageEnabled=true, applicationCacheEnabled=true, takesScreenshot=true}]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:193)
    at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:573)
    at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:482)
    at com.thoughtworks.webanalyticsautomation.plugins.JsSnifferSampleTest.extractUrlFromJavascript(JsSnifferSampleTest.java:85)
    at com.thoughtworks.webanalyticsautomation.plugins.JsSnifferSampleTest.captureAndVerifyDataReportedToWebAnalytics_JsSniffer_Webtrends_Selenium_Firefox(JsSnifferSampleTest.java:71)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
    at org.testng.TestRunner.privateRun(TestRunner.java:767)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    at org.testng.SuiteRunner.run(SuiteRunner.java:240)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    at org.testng.TestNG.run(TestNG.java:1057)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)


    Can u please help and also explain the meaning of the variable used in your javascript such as WT,DCS

    what do these variables capture.

    Any help in this regard will be highly appreciated.

    Thanks,
    Simran

    ReplyDelete
  18. Has anyone tried this setup with Espresso framework?

    ReplyDelete
    Replies
    1. Not sure. But don't see a reason why this would not work - as long as via the Espreso tests the events get triggered, you would be able to test it

      Delete