Blogs from Kiran Badi

While browsing the Netbean site, I came across the below picture, which neatly shows the various libraries that are part of Java Development Kit,so thought let me have this in my blog.Having more information about JDK sometimes help in understanding various aspects of technologies involved and sometimes aids in doing research in the right direction.

One among many skills required to be good Performance Engineer is to know, understand, interpret and analyze the logs of the backend servers and trace back those events to the load test executed during that known time period. Many times it so happens that in spite of having perfect script, perfect workload model, response times of the tests do not meet the service level agreements or they are just beyond the range as expected by the business owners or application developers. There could be many reasons for this, but to pin point the exact cause and come up with conclusive data, it becomes extremely necessary that we understand importance of logging and analyzing the server logs for the backend servers. So in this post I would be focusing my effort in explaining the importance of HTTP ERR logs which is the part of IIS logging.

Since IIS handles applications which are HTTP based, errors originating from the HTTP API of the IIS are logged in HTTP ERR logs. The most common types of errors logged here are network drop connections, connection dropped by the application pool, Service Unavailable errors, bad client request etc. Best part of this log is that it gives you the precise information as which URL or application pool is responsible for incorrect behavior, something which is very valuable in case if you have 10s or 100’s of sites sitting on the same IIS box or sharing the same application pool. All these information one can easily correlate with load test results to build a strong case that the application under test has some issues which needs some attention from the concerned team and may possibly have performance impact.

HttpErr logs comes with standard fields as shown below.It has all the information that requires for troubleshooting any fatal issue that occurs during performance testing. One thing I would like to mention here is that in case if you have IIS server sitting behind the  load balancers, then it might happen that the IP what you see in these logs might belong to the load balancer rather than end client users.Again this could be due to your infrastructure/environment design.

Another good thing about HTTP ERR logs is that they can be queried exactly the way we query IIS Access logs by using log parser.

Below is one such example query, we can use to query HTTP ERR logs is as below,

LogParser‘SELECT TO_STRING(date,'YYYY-MM-DD'),TO_STRING

(time,'hh:mm:ss'),c-ip,c-port,s-ip,s-port,cs-version,cs-method,

cs-uri,sc-status,s-siteid,s-reason,s-queuename

 From \\YOUR PATH TO HTTP ERROR LOG’

Most often the HTTPERR logs are written in SYSTEMROOT\SYSTEM32\Log Files\HTTPERR Folder of the IIS box, however since this location is configurable  so at times we might need to confirm the exact location with Server administrators.Also in some rare cases , these logs might also have been disabled due to various reasons, so in those cases we might need to check this with Server administrators and get those enabled though I feel no one really disables it for any reason.

Once you query these logs with log parser you should be able to get the details as shown below.

As you can see the wealth of the information lies in the s-reason codes which helps us in identifying the issues with the application and aids a lot in troubleshooting some of the hardest performance bottlenecks.

Below is the Microsoft definition of the reason codes that comes as part of HTTP ERR logs,

AppOffline: A service unavailable error occurred (an HTTP error 503). The service is not available because application errors caused the application to be taken offline.

AppPoolTimer: A service unavailable error occurred (an HTTP error 503). The service is not available because the application pool process is too busy to handle the request.

AppShutdown : A service unavailable error occurred (an HTTP error 503). The service is not available because the application shut down automatically in response to administrator policy.

Bad Request : A parse error occurred while processing a request.

Client_Reset : The connection between the client and the server was closed before the request could be assigned to a worker process. The most common cause of this behavior is that the client prematurely closes its connection to the server.

Connection_Abandoned_By_AppPool : A worker process from the application pool has quit unexpectedly or orphaned a pending request by closing its handle.

Connection_Abandoned_By_ReqQueue : A worker process from the application pool has quit unexpectedly or orphaned a pending request by closing its handle. Specific to Windows Vista and Windows Server 2008.

Connection_Dropped : The connection between the client and the server was closed before the server could send its final response packet. The most common cause of this behavior is that the client prematurely closes its connection to the server.

ConnLimit : A service unavailable error occurred (an HTTP error 503). The service is not available because the site level connection limit has been reached or exceeded.

Connections_Refused : The kernel NonPagedPool memory has dropped below 20MB and http.sys has stopped receiving new connections.

Disabled : A service unavailable error occurred (an HTTP error 503). The service is not available because an administrator has taken the application offline.

EntityTooLarge : An entity exceeded the maximum size that is permitted.

FieldLength : A field length limit was exceeded.

Forbidden : A forbidden element or sequence was encountered while parsing.

Header : A parse error occurred in a header.

Hostname : A parse error occurred while processing a Hostname.

Internal : An internal server error occurred (an HTTP error 500).

Invalid_CR/LF : An illegal carriage return or line feed occurred.

N/A : A service unavailable error occurred (an HTTP error 503). The service is not available because an internal error (such as a memory allocation failure) occurred.

N/I : A not-implemented error occurred (an HTTP error 501), or a service unavailable error occurred (an HTTP error 503) because of an unknown transfer encoding.

Number : A parse error occurred while processing a number.

Precondition : A required precondition was missing.

QueueFull : A service unavailable error occurred (an HTTP error 503). The service is not available because the application request queue is full.

RequestLength : A request length limit was exceeded.

Timer_AppPool:The connection expired because a request waited too long in an application pool queue for a server application to de queue and process it. This timeout duration is ConnectionTimeout. By default, this value is set to two minutes.

Timer_Connection Idle :The connection expired and remains idle. The default ConnectionTimeout duration is two minutes.

Timer_EntityBody :The connection expired before the request entity body arrived. When it is clear that a request has an entity body, the HTTP API turns on the Timer_EntityBody timer. Initially, the limit of this timer is set to the ConnectionTimeout value (typically 2 minutes). Each time another data indication is received on this request, the HTTP API resets the timer to give the connection two more minutes (or whatever is specified in ConnectionTimeout).

Timer_HeaderWait :The connection expired because the header parsing for a request took more time than the default limit of two minutes.

Timer_MinBytesPerSecond :The connection expired because the client was not receiving a response at a reasonable speed. The response send rate was slower than the default of 240 bytes/sec. This can be controlled with the MinFileBytesPerSec metabase property.

Timer_ReqQueue : The connection expired because a request waited too long in an application pool queue for a server application to dequeue. This timeout duration is ConnectionTimeout. By default, this value is set to two minutes. Specific to Windows Vista and Windows Server 2008.

Timer_Response : Reserved. Not currently used.

URL : A parse error occurred while processing a URL.

URL_Length : A URL exceeded the maximum permitted size.

Verb : A parse error occurred while processing a verb.

Version_N/S : A version-not-supported error occurred (an HTTP error 505).

So next time if you are load testing any .Net Web Application installed on IIS Web Server, make sure that you ask for access to IIS HTTP ERR logs as well in addition to the IIS Access logs.It will help you identifying issues in the fraction of seconds compared to any other methods used quite often.

Recently one of the business executive from Catchpoint posted the link in Performance Specialist list in LinkedIn about the potential issue with IE Browser network capture tools where in network capture tool in IE was showing high response time for one of resource when in fact the response time for that resource was in fraction of milliseconds.I found this case very interesting given that given that I have seen lot many performance Engineers and Users using these tools to identify the performance issues. Catch point had made its point very clear, that there are certain cases where in the browser network capture tools do not show us the correct network capture time and the time shown by these tools are impacted by elements of the page.

Given that they had also provided with the test case as how to reproduce this issue,I thought that let me try and see if I can reproduce the issue, so I pulled out their script and loaded it in my Dreamweaver to check this.I was curious to know and understand as what factors of the page were responsible for this abnormal behavior.

I made some changes to the script and pointed the IFrame source to my local Test PHP page hosted on my local Apache server.The point of making this change was to ensure that there is no actual network issues impacting the test case.In addition to this I changed the loop frequency in order to check at what level of DOM manipulation will this issue surface and has major impact on the network capture feature of the browsers.

        }

The above piece of code is taken from Catch Point’s test case and only changes I made was to change value of the loop i.e, I ensure that its creating 10,20,50,1000,5000 and 10000 span elements on the fly and they are aligned inline without styles applied to it.After making changes, I ensured that I am taking the network capture with browser’s developers tools for IE9 ,Firefox 9,Chrome 15, and Safari 5.1.

For Creating 10 Span elements, response time reported by IE Capture tools is 32ms.

For Creating 20 Span elements, response time reported by IE Capture tools is 31ms.

For Creating 100 Span elements, response time reported by IE Capture tools is 46ms.

For Creating 1000 Span elements,response time reported by IE Capture tools is 250ms.

For Creating 5000 Span elements,response time reported by IE Capture tools is 820ms.

For Creating 10000 Span elements,response time reported by IE Capture tools is 1.66 sec.

As you see, more the span elements we create ,more the Iframe’s response time. Since the test case was executed in the local environment with absolutely nothing in network to download, we would have expected the response time of IFrame to be same in all cases.Unfortunately it was not.So I think its fair to conclude that IE 9 network capture tool cannot be used to measure the response time at least for sure iframe elements and possibly for those elements which generate the network requests in the heavily DOM operated environments.

I would like to also add that this test case is not about writing perfect javascript but more of knowing and understanding whether Browser’s capture tools are in fact independent of its rendering and parsing architecture.Finally with this test case we can conclude that at least in IE 9, browser’s network capture tool do depend on its rendering and parsing architecture which at times might give users a incorrect picture of network latency.

Hope some one from Microsoft take a note of this behavior and fix this behavior.

In addition to this there are also many other interesting findings which I came across, maybe sometimes later I will write a note on that in another post.

Finally thanks to Catchpoint for highlighting this issue.

Sometimes back I had written about some interesting issues encountered during the load testing application integrated with CA Siteminder. Load testing application integrated with Siteminder has its own challenges and limitations with regard to environment and scope of testing. Normally I have observed that most projects do not test Siteminder integration/approach/solutions as an independent component, but rather they club it with applications and prefer to test this with End to End approach as most people believe that it saves some time or they believe that it’s of little importance. I would say its incorrect approach or in cases a incomplete approach, if you have Siteminder integrated with your applications and you have 10’s of application with 1000’s of users, and then probably with performance testing Siteminder integration individually as a separate project might give you a lot of mileage in long run. Siteminder is a complex product and it does deserve performance testing individually given that it handles some of the critical functions of the business namely Access Management.

The purpose of my writing this post was to highlight one of the interesting bottlenecks we had encountered/investigated and fixed during Siteminder load testing for one of the projects. During load testing one of the applications we were constantly getting 401 errors and nature of the 401 errors were such that it was redirecting itself N number of times and in some cases going into infinite redirection loops. Reproducing this kind of error via browser is a harder job as browser often hides the redirection events and you just see progress activity bar in the browser status bar. With HTTP Debugging tool you can clearly see the redirection calls happening behind the scene, but again I believe browsers don’t really redirect exactly N number of times. So in browsers I have seen max redirection depth of 40 to 50 per tab, after that it use to stay still, may be browsers stops further redirections events. The redirection events looked like below in fiddler trace.

In above image,we can clearly see that siteminder agent installed on the server is redirecting the request URL N number of times.Now during Load testing the issue was looking like below,

We had number of calls with CA team and after we gave them this precise description of the  issue along with our vuser logs and fiddler traces we were seeing during our tests, they came back with below finding and fix,

One out-of four policy server had a bad value for the encryption key; this resulted in request sent to the bad Policy Serve to fail with “invalid key in use” – Looping observed in the fiddler traces for request failing to authorize.

Once we fixed in the encryption key in the sm.registry file all request processed as expected – no more looping (re-authentication process).

I am writing the exact text as given by the CA R &D team in case if any one faces this issue,they can try out this one as well.However please ensure that you see the same behavior as we have seen it and ensure that you do not have any issues with SM cookie collectors and they are doing their job fine.In addition to this make sure your users do not have access issues.

Sometimes during performance testing it becomes necessary our script send the correct user-agent string to the server so that server identifies and responds correctly to the script.Most of the load testing tools available in market today provide a way to customize the user-agent string. So in case if you want to know and customize the user-agent string, then below JavaScript function can help you in getting this information. All you need to do is copy and paste the below piece of code in your browser and press enter key. Modal window box will be displayed showing you the user agent string of that browser.

javascript: alert (navigator.userAgent);

In fact with this function, you can get lot of information like version of framework involved, type of platforms, OS version etc. etc. Most browsers today send around at least 2 to 3 lines of information as part of user agent string and they send it voluntarily and send it for every request they serve.

Sometimes I really think is it necessary and correct for browsers to attach unnecessary information on every request they serve? Isn’t there any other better way to track the browser’s usage/OS Penetration rather than choking the network with extra characters that very few people care about?

TypePerf is the command line tool which writes the performance data to the log file or to the console window .I feel this tool can be useful to performance engineers who wants to monitor few stats and to those who wants to maintain a some distance from PerfMon. This tool basically queries the same performance objects as PerfMon does, so data retrieved by this tool is exactly the same as retrieved by PerfMon. I feel this tool is easy to use compare to PerfMon for certain situations where in we just need a very few counters to monitor. We can also automate monitoring using this tool exactly the way we do with PerfMon. In order to start using this tool, one needs to go to CMD Prompt and then type typeperf.

C:\>typeperf /?

Microsoft r TypePerf.exe (6.1.7600.16385)

Typeperf writes performance data to the command window or to a log file. To stop Typeperf, press CTRL+C.

Usage:

typeperf { <counter [counter ...]> | -cf <filename> | -q [object] | -qx [object] } [options]

Parameters:

<counter [counter ...]> Performance counters to monitor.

Options:

-? Displays context sensitive help.

-f <CSV|TSV|BIN|SQL> Output file format. Default is CSV.

-cf <filename>            File containing performance counters to monitor, one per line.

-si <[[hh:]mm:]ss>      Time between samples. Default is 1 second.

-o <filename>             Path of output file or SQL database. Default is STDOUT.

-q [object]                   List installed counters (no instances). To

                                   list counters for one object, include the

                                   object name, such as Processor.

-qx [object]                   List installed counters with instances. To

                                     list counters for one object, include the

                                     object name, such as Processor.

-sc <samples>              Number of samples to collect. Default is to

                                     sample until CTRL+C.

-config <filename>       Settings file containing command options.

-s <computer_name>   Server to monitor if no server is specified

                                     in the counter path.

-y                                  Answer yes to all questions without prompting.

Note:

Counter is the full name of a performance counter in

“\\<Computer>\<Object>(<Instance>)\<Counter>” format,

such as \\Server1\Processor(0)\% User Time.

Below are the some sample queries I ran successfully to retrieve performance stats on  my windows 7 box,

C:\>typeperf “\Processor(_Total)\% Processor Time”

C:\>typeperf “\Memory\Available bytes” “\processor(_total)\% processor time”

C:\>typeperf “\Processor(*)\% Processor Time”

Finally in case if you need more information about this command,I suggest you to visit Microsoft site and do a quick search for typeperf.

http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/nt_command_typeperf.mspx?mfr=true

I feel this tool is supported by all version of windows operating system starting from Windows XP to Windows 7.

HTML 5 has some fantastic changes to the form elements which I feel is surely going to make the life of the developers somewhat easy as now they might have to write less code to do some of form validation work. Use of these HTML Tags as part of form means you write less code and do less amount of testing since the real implementation and validation is now done by the browsers.

In all there should be around 10 to 15 new form HTML tag elements which I feel is going to be implemented as part of HTML 5 Specs. Below are the some of the new form elements which I feel are going to be introduced in HTML 5,

· search – This is going to be used for Search text field.

· tel – This is going be used as text field which will accept telephone numbers. It also has pattern attribute to customize according to one’s requirement.

· url – This field is going to accept the url of the site.

· email – This field will accept only email addresses. It has the attribute which can accept multiple email addresses in the same field.

· color – Shows up the color picker control.

· number – This field designed to accept the numerical input. It supports the pattern wherein the user can be restricted to give only numbers in the field.

As of now these HTML 5 elements are not fully supported by browsers, but I have tested this in latest version of Google chrome and Firefox, they do support these tags.

Below is the sample code which I tried in chrome and Firefox,

<!DOCTYPE HTML>
<html>
<head>
<meta charset="utf-8">
<title>HTML5 Form Elements</title>
</head>

<body>
<form method="post" enctype="multipart/form-data" name="form1">
    <p>
        <label for="search">Search:</label>
        <input type="search" name="Search" id="idsearch">
    </p>
    <p>
        <label for="tel">Phone:</label>
        <input type="tel" name="Phone" id="idphone">
    </p>
   <p>
        <label for="url">Website Details:</label>
        <input type="url" name="website" id="idurl">
    </p>
    <p>
        <label for="email">Email:</label>
        <input type="email" name="email" id="idemail" multiple>
    </p>
    <p>
        <label for ="number"> Lucky Number : </label>
        <input type="number" name="luckynumber" pattern="int">
    </p>
    <p>
        <label for="color">Select color:</label>
        <input type="color" name="colorn" id="idcolor">
    </p>
    <p>
        <input type="submit" name="send" id="send" value="Submit">
    </p>
</form>

</body>
</html>

Don’t me believe, copy paste this code in notepad and save it as html document and open this in Firefox and chrome and use your testing mind and see for yourself how much testing and development efforts you can save when you use these tags as part of your development practice.

I am also one of those guys who year after year had some resolutions and then year after year has broken it for various reason.Now that I am maintaining my own blog, I feel its good thing that I should note down  the list of tasks which I believe I should focus on in 2012.There are number of things which I want to do in 2012 both at personal and professional level.

Some of the stuffs I will be focusing on in 2012 will be ,

• For some time now, I had a desire of building something like Facebook or Twitter ,though the skeleton is somewhat ready, I need to focus more on dressing the skeleton appropriately.I really wish to spend more time on this in 2012.
• I think I should work more hard this year in understanding CSS/HTML/HTML5/Javascript and how these things are rendered by browsers.I think this is one of the most interesting areas which every performance engineer should focus on at some part of their career.
• I think I should be writing more on the Performance Engineering topics which has always been my interest.I will also be writing about as how to use/identify/verify/fix performance issues with LoadRunner/Silk Performer.These are my Favorite tools and will continue to be.I just love the way these tools work and have been designed.
• My blog  needs some attention from me,lot of people and spammers have complained that I don’t maintain “About me” and overall my site sucks in terms of look and feel.So I will working on to improve my site. I will see if I can convert this site to full fledged knowledge repository site.
• On personal level, I feel I need to practice more on controlling the emotions and concentrate more on my health.2011 was one of most bad year for me when I think about it in terms of health issues.
• I will be watching more movies, first day first show and spend lot of time with my family even if this means doing less work and giving up projects.

Finally I am thankful to all friends, colleagues for making my last year 2011 a good one.

In Recent times I have seen a kind of trend where in performance testers use tools like fiddler/yslow/httpwatch for scripting purpose in spite of having fully licensed version of the LoadRunner or other performance tools.I find this approach to be somewhat immature and prone to many errors if done by someone who do not understand performance engineering concepts or someone who does not understand the semantics of the web communication. I also find this kind of trend bit disturbing and harmful to the load testing industry for many reasons. I will list down some of the reasons why I feel this is a very bad idea and why one should not take this way for scripting purpose for preparing their load testing scripts.

I have encountered many times in my career where in the application in spite of working with supported protocol do not get recorded by the load testing tool. On further analysis of the issue, I did come across certain findings that either the setting which we use for recording the applications were incorrect or we were recording in the unsupported environment or it was just that firewall or DEP or some antivirus software setting were blocking the recording. So for sake of this post I am not writing here anything on as how I troubleshoot recording problems but will surely take some time later on to write on this as well. It’s interesting topic and deserves a separate post. So let me go back to original purpose of this post and write some of my thoughts why one should not use fiddler or any other HTTP Debugging tool to script a load testing script,

1. Tools like Fiddler/Yslow are the http debugging tools and not the scripting tools. They show http requests as resources are downloaded and interpreted by the browsers. Since page contains many resources, it becomes hard to tag these resources to the main request unless you are page author. Since the requests captured by these tools are context less and are not orderly distributed so I feel it cannot be used for scripting purpose and so they cannot replace the load testing tools.

2. Building the post requests from these tools is somewhat harder compare to get requests though not completely impossible depending upon the type and complexities of applications.

3. Since requests are displayed context less or in uneven order, correlation or capturing dynamic values is somewhat hard and prone to lot of errors resulting in many hours being wasted.

4. These types of tools follow the web standards and to know and understand information generated by these tools in real sense one needs to know and understand various RFC’s involved in the web communication. I have seen most of the Performance testers are unaware of these standards.At least this is my observation. It has taken years for me to understand as how web communicates.

5. From project finance prospective, we should not forget that we are paying five to six figure amounts to the licensed load testing tool vendors for using their product, so it makes more sense to ask them to fix these recording issues rather than using HTTP Debugging tools for scripting purpose. Also it looks bit odd when we use Open source HTTP Debugging tools to script when we have licensed version of the load testing tool. If you have program management office or PMO, who are auditing your projects, then they are surely going to scream on you for wasting so much of amount on load testing tools and not using it to their full potentials or capabilities.

Finally I would like to mention here is that this post is more about using right tools for the right job. HTTP Debugging tools are for debugging the web http communication, so I suggest let’s use these tools for debugging and load testing tools for preparing load testing scripts. In case for any reasons, if you are unable to record the script in spite of having supported protocol, then by all means you have all the rights to shout at the vendor and get the issue resolved.

If you are doing performance testing, then I assume that you for sure know as what correlation means and how to do it. So I am not going to write anything on this. I know for sure that many times it happens performance script which runs perfectly fine in local machine, often fails miserably during the load test often throwing error saying that correlated value could not be found or parameter size is insufficient so increase the parameter size. Having worked with number of various protocols/technologies, I feel there are many reasons as why your correlation can fail ,so I will try to list down some of the reasons which I feel one needs to investigate in case his correlation is failing,

1. We have specified left and right boundaries incorrect. Sometimes it so happens that we miss to specify the special characters like trailing slash, comma etc. in boundaries. It is one of simple error which quite often we do while scripting.

2. We have not placed the function which does correlation at the right place that is right before the request which generates the dynamic value for the first time. This is also one of most common mistakes I have seen people doing.

3. We have not used the attributes of the function correctly. For example, value is getting generated in headers and we make the function search for value in body part of the response.

4. We have used left and right boundaries that are too generic and there are many instances of such boundaries in the server response for that request. In other words boundaries are not unique For example, if you have a form with many fields and if you give boundary as below, then it is bound to fail.

   1: web_reg_save_param (“LB= value”,”RB=,”, “Ord = 1”, LAST);

5. If the page returned as a server response is different than what we were expecting during replay, then obviously our correlation is going to fail. Since response page content will be different, function will not be able to find boundaries it requires to succeed.

6. If the size of the parameter where we are storing the dynamically captured value is less than actual size of the parameter returned by the server, then our correlation is going to fail. For example, let’s say we have parameter A and we allocate 1000 bytes to store the value and if the actual size of A is more than 1000 bytes, then our correlation is going to fail. Another good example of this is view states correlation.

7. Sometimes it so happens that we use incorrect data in our tests and due to this data, we do not get the page what we were expecting, and this error on our side also shows up as correlation failure in the test run. Sometimes we try to correlate client side generated values. I have made this mistake at some point of my career.

8. We really have performance bottleneck in the system. Quite often it happens that test runs fine for certain duration and then suddenly we see correlation failures for some or all of our functions. This is also one of area we need to investigate in case you see correlation failure after a certain period of time during the test.

9. We have not correctly replaced the parameters captured in the script.

10. Finally the function in itself has bug and due to this bug it might not be behave as it should. Failure to capture correctly view states by most tools during early days of ASP.Net is good example of this.

These are some reasons which I can recollect now due to which correlation can fail. I am sure that these reasons applies for all tools which are used for doing performance testing in all domains and all technologies.

Hope this helps.