Definition of the Testing Phase It is very difficult to define "the testing phase". In order to illustrate the idea of this article, "the testing phase" will refer to the phase in which comprehensive testing begins to be carried out on the development products (usually by designated testers). For discussion purposes, comprehensive testing begins from the integration stages and up to the system test. Many refer to this stage as the "QA stage". Because of the complexity of the concept "QA" and vast description, the article will adhere to the concept "testing phase". Objectives of the Testing Phase Before we examine the contribution of the testing phase to the project, it is important to understand the declared objectives of this phase. The accepted answers to this question are as follows: • "To allow interested parties in the product to receive a measure of its quality and compliance to its requirements" • "To prove that the software does what it is supposed to do" (partial and problematic definition) • "To find the gaps between what the software is supposed to do/not do to what it actually does/does not do" (a slightly more complete definition) This is indeed the main purpose of the testing phase - an important and critical phase for the success of the project. However, is it the exclusive purpose? Is the job of the testers summed up in this way? In most cases the answer is positive without much hesitation. His focused task to mark a "V" on the quality of the product as it is perceived by many, results in lack of administrative attention. Most of the focus in the project, as well as administrative enthusiasm, is in the development stage (in its traditional sense: the world of software developers). In view of this, managers tend to put less emphasis to planning of the testing phase in the planning stages of the project and many encourage more attention to be paid to this stage. Many of the project managers recall the testing phase only at the last minute. Few involve and integrate them still in the planning stages and only the "meticulous" refer to the planning of the testing as an integral part of the project planning itself.
Testing Tools – Test Execution
If you work in web application testing, you probably know that Selenium is one of the best tool for automating web applications. Though it can be used to test various aspects of web application, sometime you need to find your way around when using Selenium, one such situation is testing flash component using Selenium. Developers use Flash to beautify their websites, but it makes test automation a challenging task because Selenium can not read or record any actions on flash objects.
There are two main issues that needs to be addressed while automating web applications involving flash objects –
Verifying a certain value is set or it exists or not or in simple terms you need to ‘get’ certain properties of the object and compare it against your benchmarks
Performing actions on an object or ‘set’ certain values.
These two issues could easily be handled by Selenium for most of the website objects except for Flash.
In this article we will describe how to automate testing of Flash based components of web applications. The basic steps involved in any automation are:
Observe and analyse
Draw a common pattern
Use this pattern for automation.
These three steps are very important while understanding the automation of Flash based components for web applications . Most of the time Flash is used in the web applications for –
To play static content in the form of movie. The reason I am saying static content is that the contents of the movie stay the same.
To render the dynamic contents using Flash such as rendering the contents for a strip of images or a strip of clips using Flash or displaying the title of any page using Flash. The reason I am mentioning these as dynamic content , is that Flash is just picking up this data from some system and displaying it in the Flash format. The contents may vary depending on the context of the page. For example, Flash can be used to display the a strip of pictures uploaded for a particular program for a media website. In this case the behaviour of the Flash is the same for all the programmes, only the content it is displaying is changing.
Both these cases could be automated in a different manner depending upon the information available in the DOM (Document Object Model). We are going to take second case first and explain how we used the DOM information for our automation and addressed one of the two issues (i.e. verifying a certain values). We used the 3 steps mentioned earlier for automating the Flash based component of a media website. If you have a website where dynamic Flash information is read in ‘flashvars’ then you could also follow the same steps.
So how do you know whether the Flash components of your website are using the ‘flashvars’ information or not? Well its not that difficult to find it out. All you need to do is:
Install firebug plugin for Firefox, if you do not have it already.
Search for the ‘flash’ in the firebug you will get some div tag informing about the flash content and containing an object tag for the flashplayer.
In this object tag search for ‘param’ mostly it is called ‘flashvars’ and the value contains all the information that flash will display.
Alternatively you could search directly for ‘flashvars’ 🙂 As displayed below
Looking at this you could easily get its xpath ( just browse you mouse over the element displayed on top of firebug near edit) or use another Firefox plugin, XPath checker.
After having the xpath you could get the value of this ‘param’ or ‘flashvars’ in a string using getattribute() function of Selenium.
This string would appear a bit strange on first look but on analyzing this string you would find a common pattern and it would be easy for you to then parse this string to get the required values.
This technique is very useful in at least verifying certain values. you could always search for ‘flash’ in the firebug and see if there is any information in its parameters or other tags that could be exploited for automation. Let us apply this technique on http://www.youtube.com because we know all the videos on you tube are displayed through Flash. A very interesting case of applying this technique could be testing whether the id of the video played on youtube is the same as the one displayed for embedding or not. Here is the screen grab of a random video on youtube.
As you can see we have followed the same steps mentioned in this article and searched for ‘flashvars’ we found that flashvars contains some words as ‘video_id’ in the string and then after the ‘=’ sign it contains the value of the id. Yes we got it we could parse this string for value of ‘video_id’ and compare it with the value specified in the url !!!
In the technique presented above, there is nothing which is only applicable for only Flash objects. You can follow the three steps mentioned above for any kind of automation and might come up with similar technique for something else as well based on your context. In the next article we will discuss a Flash specific technique for the Selenium using flash-selenium.jar to automate more advanced interactions with Flash using Selenium.
Any IT professional is sure to know the different phases of the Software Development Life Cycle or SDLC, namely Feasibility Study, Requirement Analysis, Design, Construction or Coding, Testing, Implementation, Maintenance & Support – the activities carried out in each phase & their significance. But, very few agree on the importance of Software Testing phase.
“Software implementation is a cozy bonfire, warm, bright, a bustle of comforting concrete activity. But beyond the flames is an immense zone of darkness. Testing is the exploration of this darkness.” – extracted from the 1992 Software Maintenance Technology Reference Guide
Testing is often considered as a thankless job.
While developers say with pride: “Wow!! My code is running in production”, testers usually don’t say “Wow!! The code that I tested is running in production”!!! This attitude can also be justified if we consider some examples of the usual talk that goes on among colleagues/peers/friends in the IT circle, like:
Mr. A: Which project are you working on?
Mr. B (Tester): Currently. I’m in a Testing project.
Mr. A: Oh…Umm…OK…
Mr. A: Mr. C, how about you?
Mr. C (Programmer/Developer): A Development & Maintenance project
Mr. A: Oohh?? What technology? Which platform? What’s the project all about?? … And so on
Even though there’s no denying the fact that Construction/Coding is a very significant phase in the life cycle of any software product, the role of Testing as an activity should be given its due importance, because of the main reason, among others, that its the phase of SDLC just before a software product goes to production; i.e.; when your software product/application actually starts functioning in real world.
Therefore, it’s the only phase where you can ensure & gain a reasonable level of confidence that that you are delivering quality products to the customer.
This does not mean that quality assurance starts in the Testing phase. Ensuring & Maintaining quality is a continuous process which should be practiced right from Day 1 of the SDLC cycle.
In other words, during testing phase, you can “bang” your product in different ways, by which I refer to different kinds of testing like functional testing, stress testing, performance testing & so on, and validate whether the software product works fine.
Needless to say, delivering quality software delights customers & helps in getting more business. That said about the importance of Testing, we will now proceed to look into some perceptions or “myths” about software testing which prevails, in general, among the IT community
Testing is done to demonstrate that there are no errors/bugs/defects in the software product being developed
No. Not at all!! Though this is one way of conveying the meaning, it takes us away from
the main objective of testing
Testing is the process of uncovering defects… The objective of any tester should be to
try his best to crack the code. This should not be seen as a destructive activity which points fingers at or find faults with the developers. In fact, testing should be considered as a healthy feedback mechanism to the developer community so that they can make maximum use of the defects found during testing, analyze them, find the root cause & devise appropriate preventive mechanisms. Actually, the defects found during testing helps improving the quality of the code!
A tester should never work under the assumption that the system works!! The software product should be tested with the intent of finding errors. The test, which when executed reveals a defect in the software, is a success. Though Testing, by itself, does not improve the quality of the software product, it is an indicator of the quality. Rather that considering Testing
as a separate phase in the SDLC, it should be an activity which is integrated into each & every phase. The intent should be to Analyze & Test, Design & Test, Develop & Test, Fix Defects & Test more…!!
Testing is done to ensure that the software/application does what it is supposed to do
True enough!! But it’s not just that…The software should also not do what it is not supposed to do. Good test cases are the only way to establish a reasonable level of confidence on the software product. The Tester should think of all possible scenarios based upon the test requirements or test plan. Though there are various methods to prepare a large number of test cases like Equivalence Partioning, Boundary Value Analysis and so on, a lot depends on one’s own intuition to come up with some good cases which reveals defects in the software
Testing is easy…!!!
Sometime back, I conducted a short class on Software Testing to a group of new recruits. While we were discussing about different type of projects, one participant said that he is very interested in a testing career. When asked the reason for his interest, prompt came the answer: “Testing is very easy, that’s why!!” This holds good only in some situations.
Though it’s simple to prepare straightforward test cases, at times testing can be a real challenging task.
Any production issues will, in many cases, backfire first to the testing teams. Why was this scenario not covered in the test plan??
Therefore, a Tester should develop the capability to look or think beyond the requirements mentioned in the test plan or specifications.
This is very important in case of System Testers who are responsible for ensuring that the software product works appropriately from “end-to-end “.
Testing does not offer any opportunities for career growth.
There are a wide range of roles that one can take up, if opting for a Testing career.
Pursuing a testing career offers more scope for improving Business/Domain knowledge.
It enables one to adopt a holistic approach of the entire software system instead of concentrating on just a unit or module.
A good number of testing certifications are offered by reputed institutes, which helps you attain a strong foundation in this career path.
This does not mean that you don’t have all these opportunities at all if you are a Developer.
It can be said that a Testing career has its own plus points. It entirely depends on one’s own interests!
Testers do not perform well in development projects (or Testers are poor in coding)
Tester or not, the expertise on developing quality code depends upon one’s own programming skills and constant or continuous learning in that area.
Some people, though in testing projects, take time out of their work to improve their programming skills by contributing to coding efforts or taking up projects voluntarily.
Thus, being a Tester does not prevent you in any way from being an expert Programmer or Vice Versa!!!
One way to narrow the communication gap between Tester & Developer community is to include the Testing teams, right from the Requirements/Design Stage meetings so that everyone involved in the life cycle of developing a software product can take part in the discussion & offer valuable suggestions.
This is evident from the below lines, which wonderfully describes the importance of Testing
More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded – indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest. -Boris Beizer, Software Testing Techniques, “Creating a Software Engineering Culture” by Karl Eugene Wiegers”
No formal training is needed to work in a Testing project… Anyone can be a Tester!
True enough, anyone can be a Tester…but, only a good Tester can come up with quality Test cases (just like how an expert Developer can write quality Code).
It is essential that proper training is imparted to everyone joining Testing projects. This would not only helps one to understand the importance of Testing, but also tune one’s mind to the requirements of becoming a good Tester, which would greatly contribute to a good career in Testing
There is a lot of confusion about how the concepts of Runtime Objects and Test Objects are used in QTP. The general idea is usually clear enough, but when it comes down to the actual details, sometimes even the experts do not quite know if using the GetROProperty method on a TO will work.
This article will guide you through the general concepts, and explain all the nitpicking involved in all the commands and practices relevant to the matter.
The QTP Test Object Shell
Under their fancy name, Test Objects refer to your everyday QTP entities. VBCheckbox, Window, WebTable, Browser – these are all QTP constructed test objects, which represent real-world application objects “out there” (i.e., in our application under test).
Whenever we record a real-world object (also known as a Runtime object), QTP creates a corresponding Test Object and stores it in the Object Repository. The Test Object contains QTP specific properties and methods which allow us both to identify the relevant runtime object during playback, as well as to perform standard operations on it and with it.
For example, a Window test object contains properties which allow QTP to identify it in the “real world”: text, height, nativeclass, etc.; on top of these, it also contains methods to operate it: Close, Click, Maximize. We can easily see these methods and properties via the object spy’s Test Object tab.
It is important to understand that the test object’s properties and methods are not really “out there” in the application’s runtime object, but are only derived from it. So for example, a SwfComboBox Test Object has the property “all items”, while the actual .net application combobox runtime object has no such property. QTP builds the “all items” property out of the combobox runtime object’s available items collection, so even though the property does not exist in the runtime object, it is derived from it.
Test Objects roles
Test objects play several roles in QTP, the most immediate one is object identification. By using a window test object with a text property of “Notepad”, we tell QTP to look for a runtime window object with text = “Notepad”.
Another important role is standardization. While the application runtime objects might be extremely complex, the corresponding QTP test objects are standard and very simple in comparison. The test object’s simple properties and methods act as a bridge, sparing us the need to nitpick our way through the runtime object’s unknown commands, events and properties. QTP will translate the test object’s simple commands to the runtime object’s commands required to perform the task at hand.
For example, selecting a value in a .Net combobox usually involves changing several properties (SelectedIndex, SelectedItem), and raising several events (SelectionChanged, ItemSelected, TextChanged, etc.). However, the corresponding SwfComboBox test object provides a much more simple and standard “Select” method which does the trick for you.
Working with Test Objects
How can we work with test objects in our scripts?
The most straight-forward way is to simply use them. Every time we maximize a window, select a value from a VBComboBox or click a WebElement, we do it through the QTP test objects. QTP will then locate the corresponding “real” runtime object, and perform the relevant action on it. We can either use prerecorded test objects from the object repository, or create instant test objects via Descriptive Programming.
Aside from straightforwardly using our test objects to perform actions, there are 4 special commands which allow us to access the core of QTP’s test object mechanism: GetTOProperty, GetTOProperties, SetTOProperty and GetROProperty.
GetTOProperty allows us to retrieve the value of a test object property. For example, this command will print out “Notepad”:
The GetTOProperty command will retrieve the value as it was originally recorded (or created via DP). It does not matter if the corresponding runtime object exists, or if that value was updated in “the real world” since the object was recorded.
GetTOProperties returns a collection with all the test object properties. Like GetTOProperty, the GetTOProperties command has nothing to do with the actual runtime object in our application, only the test object which represents it.
GetROProperty allows us to get the current value of a test object property. This means that unlike the GetTOProperty and GetTOProperties commands, GetROProperty requires the test object’s corresponding runtime object to exist. QTP will touch the runtime object, and retrieve the value of the property from it. For example, if the text of the Notepad window changed from when that object was recorded, this command will retrieve the updated value:
Unlike GetTOProperty and GetTOProperties, it does not even matter if the test object originally recorded the relevant property. In the following example, our test object has only one property – nativeclass, but GetROProperty will still be able to retrieve the value of the window’s text property (where GetTOProperty would return an empty result).
It is important not to be confused by the “RO” in the command name. Even though it has RO (Runtime Object) in it, the command still retrieves only test object properties. So even though a .Net combobox runtime object does not have an “all items” property, you could still execute the following command, and it will print out all the available items:
Msgbox SwfWindow(“X”).SwfCombobox(“Y”).GetROProperty(“all items”)
Last but not Least, SetTOProperty allows us to change a test object’s property for the duration of the test-run. Any changes made with SetTOProperty will be reset once the test-run had finished, or even when you go in and out of actions. It is important to note that changing a test object’s properties might cause QTP not to identify its corresponding runtime object. So for example:
Call Window(“Notepad”).SetTOProperty(“nativeclass”, “incorrect value”)
Window(“Notepad”).Close ‘this command will fail
The second command will fail as QTP will search for an object with nativeclass = “incorrect value”, and it will not find any object matching that description.
From Test Objects to Runtime Objects
Test objects are the cornerstone of all QTP object operations, but they are sometimes too limited for the task at hand. There are two situations in which we might prefer working with the actual runtime objects of the application, as opposed to the QTP test objects:
There are some objects (especially .Net custom controls), which QTP fails to recognize correctly. For example, some combo-boxes might appear as SwfObjects, or generic ActiveX controls. As QTP fails to recognize the object for what they are (in this example, combo-boxes), the test objects it creates do not have the properties or methods that are required to operate the objects (like “all items” and Select).
Another case is when we are required to perform some non-standard operation with an object. Here it makes no difference if QTP succeeds to correctly recognize the object, as the corresponding test object will only provide us with a standard set of commands and properties. A good example for this case is trying to find the font style of a WebElement. The WebElement test object simply does not carry the information we need, and so we have to resort to unusual measures such as working with the WebElement’s runtime object.
Working with Runtime Objects
You can see an object’s runtime properties and methods in the RO tab of the object spy. If it is a .net object, you can also use the .Net form spy to get a more detailed look of those properties that contain inner complex objects.
Not all test objects support working with their corresponding runtime objects, but if they do, it can be done via their .Object property. This property exposes all the properties and methods of the runtime object “hiding” behind our test object. For example:
Print Browser(“X”).Page(“Y”). WebList (“z”).Object.selectedIndex
This command will find the runtime object corresponding to the WebList’s test object, access its selectedIndex property, and print its value. Of course that in order for this to work the runtime object must exist when we execute the command. It is important to note that WebList test object does not have a selectedIndex property, only a derived property called “selected item index”. So in this case, these commands will print the same value:
Print Browser(“X”).Page(“Y”). WebList (“z”).Object.selectedIndex
Print Browser(“X”).Page(“Y”). WebList (“z”).GetROProperty(“selected item index”)
The true benefit of working with runtime objects lies with properties and methods that are not available through the QTP test object. So for example, we can find out the color of a WebList’s items text:
Print Browser(“X”).Page(“Y”). WebList (“z”).Object.CurrentStyle.Color
We can even use the runtime object’s methods, and perform operations that are otherwise unavailable to us:
Print Browser(“X”).Page(“Y”). WebList (“z”).Object.focus
By accessing the runtime world, we can perform almost any operation on our objects, regardless of QTP’s ability to recognize them for what they truly are. So even if our .net combobox is identified as a SwfObject, we’ll still be able to select an item from it, by overwriting the runtime property of the selected item’s index.
SwfWindow(“X”).SwfObject(“Y”).Object.SelectedIndex = 1
With all their benefits, it is important to remember that working directly with runtime objects can have significant downsides. Unlike the standard and simple test objects, it is unclear how to operate the object, and exactly what properties and methods it has (CurrentStyle.Color was hard to come by). Even when we find the right properties or methods, using them “from behind the scenes” may cause application errors and invoke hidden bugs.
If you had to chance to work extensively with QTP 10 debugger, you must have noticed that it isn’t as robust as QTP 9.5 debugger. Many users have encountered a wide array of problems, from breaking the PDM.DLL hack, to disappearing complex values (occurs sometimes when working with external objects in the debugger). Some of these problems stem from a build in latency in the new .Net framework, which times out the debuggers request before receiving the answer from an external object).
These problems have been an inconvenience to some, and a deal-breaker to others – and they will be addressed in an upcoming patch from HP.
It has been told that the patch is undergoing though massive QA, to make sure all the critical issues are resolved before releasing it to HP’s software update site.
You could expect the patch to come out in the near future. Of course we will keep you updated on any further developments.
Monitoring the system memory (RAM) is not usually helpful in identifying memory performance problems. A better indicator will be monitoring paging activities to the page/swap file. Memory paging activities will be covered in another article. Most current OS have virtual memory comprising of the actual (real) system memory using physical RAM chips, and one or more page/swap files on the system disks. Processes that are currently running are operating in real memory. The OS can take pages from any of the processes currently in real memory and swap them out to disk. This is known as paging. Paging leaves free space in real memory to allocate to other processes that need to bring in a page from disk. Obviously, if all the processes currently running can fit into real memory, there is no need for the system to swap out any pages. However, if there are too many processes to fit into real memory, paging allows the system to free up system memory to run more processes.
Paging affects system performance in many ways. One, is that if a process moved some some pages to disk and the process becomes runnable, the OS has to retrieve the pages from the disk before that process can be run which results to delays in performance. In addition, both CPU and the disk I/O spend time doing the paging, reducing available processing power and increasing the load on the disks. This cascading effect involving both the CPU and I/O can degrade the performance of the whole system in such a way that it maybe difficult to even recognize that paging is the problem. The extreme version of too much paging is thrashing, in which the system is spending too much time moving pages around that it fails to perform any other significant work. (The next step is likely to be a system crash).
A little paging of the system does not affect the performance enough to cause concern. In fact, some paging can be considered good which indicates that the system’s memory resources are fully utilized. But at the point where paging becomes a significant overhead, the system is overloaded. To monitor paging activities is easy. On UNIX, the utilities vmstat and iostat provide details as to the level of paging, disk activity and memory levels. On Windows, the performance monitor has categories to show these details, such as Pages/sec, Page faults/sec and % Page files usage.
If there is more paging than optimal, the system’s RAM is insufficient or processes are too big. To improve this situation, you need to reduce the memory being used by (a) reducing the number of processes or (b) the memory utilization of some processes. Alternatively, you (c) can add RAM. If the problem is caused by a combination of your application and others, you can partially address the situation by using process priorities. By issuing priorities to the processes, you need to be aware that using this option reduces the amount of RAM available to all other processes, which can make overall system performance worse. Therefore measure the trade-offs before deciding the resolution.
(Source: “Java Performance Tuning”)
General look and feel: QTP has departed from the old Tab layout, and into the more modern settings-tree layout (similar to Office, EMule, Adobe, and pretty much every other program). It’s nice, but nothing as groundbreaking as the transformation in QTP 9.0.
Bitmap Checkpoint improvements: These include presenting the size of the selected area when choosing to check only a part of an Image, as well as the ability to plug your own custom DLL for comparing images. I suppose that there are some projects for which this can actually be quite revolutionary. Another great addition is the ability to see the difference between the expected and actual bitmaps in a separate tab.
API changes: QTP Automation API will receive several upgrades, the most noteworthy of which is the ability to read and write that code of the test you’ve opened. Writing the code will not effect an ongoing run-session (there goes my ambitious try-catch implementation for QTP), but it still opens the door for some creative tweaks and hacks…
Saving a test with all linked resources: For those who’re working with QC, this is a real blessing. Up until QTP 10, copying a QC saved test to your local system was a hellish procedure of changing each and every resource and action reference to your local file-system. QTP and QC Atlantis offer a one-click solution for copying a test and all its resources to your local file-system, and automatically changing all the references accordingly.
Dynamic Action Call: I’ve saved the best for last. QTP 10 presents a new native command – LoadAndRunAction. It allows you to load external actions during the run-session, without having to associate them with your test beforehand. All the run-time debug abilities will be available for these dynamically added actions, so you’re not giving anything up by using this feature. I think it’s a long awaited feature, and a well executed one.
Other miscellaneous tweaks and improvements will be presented in the final release.
QTP Atlantis’ feature set can be described both as long-overdue, and as quite revolutionary.
For those who’re working with QC, the source control features will blow your mind. The intellisense features answer most of the pressing problems with QTP’s IDE, although some critical omissions (such as debug and VBScript Classes intellisense) hamper the overall effect. On top of all these, the new reporter functionality, dynamic action calls and API changes offer valuable improvements both to first-steppers and to advanced QTP programmers.
Beside the actual functionality, it seems that HP has paid attention to the UI as well. The features seem to be conveniently accessible, Simplified (though not over-simplified), and the whole look-and-feel seems to hit the spot. It’s nice to see that QTP 9.5 attention to UI has effected the long-term thinking of the development team.