IntelliTrace User Manual

Configuring Dynamic Instrumentation

To configure dynamic instrumentation, go to Configuration -> Configure Dynamic Instrumentation in the top menu. Here, you can deploy or undeploy the profiling agent, set up Dynamic Instrumentations and optionally configure Method Profiling.

 

Enable Agents on Server

To ensure deployed profiling and monitoring functions correctly, the Applicare Profiling Agent must be activated on the app server. You can enable it from the IntelliTrace Deployment tab by selecting the server from the list on the right hand side.

Disable Dynamic Profiling

Choosing this option will turn off all instrumentation across all attached agents.

Standard Profiling

This option activate HTTP, SQL and JDBC connection profiling on the app servers or JVMs where the agent is enabled.

Advanced Mode

In advanced mode, you can choose and deploy specific aspects of profiling. For instance, if you only want to detect JDBC Connection leaks, you can optimize performance by excluding other aspects you're not interested in.

Here are the available profiling aspects.

1. HTTP Profiling
2. HTTP Error Code Profiling
3. SQL Profiling
4. JDBC Connection Profiling
5. Business Transactions
6. IntelliTrace profiling - (Deploying or Undeploying IntelliTrace profiling requires the agent servers to be restarted for the changes to take effect.)
7. Method Profiling
8. External WebServices
9. End User Experience Profiling
10. End Client Process Monitoring (Slow Calls only)
11. End Client Process Monitoring (Every Call)
12. JNDI Lookup
13. External JMS Messages
14. LDAP Calls Profiling
15. Mail Monitoring
16. Collections Monitoring
17. Network Connection Profiling
18. Heap Tracker (BETA)

Please read section InelliTrace Configuration (for Applicare 6.9.0 onwards) & Enabling IntelliTrace on Production Servers for using IntelliTrace in production servers.

Deploy Aspects In Advanced Mode

1. Select the aspects you want to deploy from the Available Aspects list and click the Add button.
2. Selected Aspects will be moved to the Deployed Aspects list.
3. Click Save button to save the configuration.

Deploying or Undeploying IntelliTrace profiling requires the agent servers to be restarted for the changes to take effect.

Undeploy Aspects In Advanced Mode

1. Choose the aspects you wish to undeploy from the Deployed Aspects list and click the Remove button.
2. The selected Aspects will be removed to the Deployed Aspects list.
3. Click Save button to save the configuration.

 

Per Server Advanced Mode

This mode allows each server to have its own unique set of profiling aspects.

Activating IntelliTrace for Specific Applications or Application sets

Starting with Applicare 6.9.0, you can enable dynamic IntelliTrace for specific applications or sets of applications. This feature allows you to focus instrumentation on key applications, minimizing overhead.

To enable this, add ",perappinstrumentation=true,perappinstapps=/JPetStore:/qos" to the Applicare -javaagent argument when starting the app server. In this example, JPetStore and qos are the applications that will be instrumented by the Applicare Profiling Agent. Note that the application names are case-sensitive and should match exactly as they appear in the app server deployment. 

Depending on how you provide the Applicare startup arguments, you may need to modify the setAgentEnv.sh or setAgentenv.bat file in the scripts directory, or adjust the method you use to pass Applicare arguments to your app server. for instance, If you are using WebLogic and specifying the startup arguments via 'Server Start' tab under configuration then you should make the change there.

-javaagent:$ARCTURUS_HOME/applicare-aspects.jar=verbosity=0,logfile=false,perappinstrumentation=true,perappinstapps=/JPetStore:/qos -Darcturus_home=$ARCTURUS_HOME "

Static IntelliTrace Instrumentation for specific applications has been available in Applicare from the start, but it required modifying and redeploying application JARs. With the new pre-app dynamic instrumentation, you no longer need to instrument the JARs and redeploy them; instead, you can enable instrumentation dynamically without these additional steps.

Intellitrace Configuration

IntelliTrace configuration offers detailed control over which packages are instrumented and which are excluded. Excessive instrumentation of packages or classes can impact performance, so it's advisable to instrument only the essential packages. This feature was introduced in Applicare 6.9.0 version and is not available in earlier versions.

Diagnostic mode enables you to see which packages and how many classes would be instrumented by the current settings, without actually applying the instrumentation. This is particularly useful for app server with numerous packages or applications, helping to optimize server to startup time and performance.

When Diagnostic mode is enabled, no classes are instrumented, but all the data displayed on IntelliTrace Configuration tab is collected. Use this mode to review which packages or classes would be instrumented and to refine the list of packages that are required to ensure you achieve the desired results.

By default, the packages listed in the "trace-exclusions.txt" file located in the Applicare home directory are considered as the exclusion list. These packages are not transformed. If you modify the package details on the IntelliTrace Configuration tab and click "apply", changes a new exclusion file named  "trace-exclusions_$ServerName" Applicare home directory containing the updated exclusion package details. These changes will take effect once the app server is started.

Threshold limit: This limits the # of classes instrumented by Applicare. This is designed to prevent prolonged startup times for app server with numerous deployed application. Classes are transformed until the threshold limit is reached and after that no classes are transformed. A message is shown in IntelliTrace and BTM Summary tab for each server where this limit is reached.

Diagnostic mode: When diagnostic mode is enabled, no classes will be transformed. If diagnostic mode is enabled, packages not on the exclusion list will be transformed until the threshold limit is reached.

Instrumentation count details: This displays the count of both transformed and untransformed classes/packages.

Packages included in instrumentation: This shows the list of packages which "included in instrumentation".

Packages excluded from instrumentation: This shows the list of packages which "excluded from instrumentation".

Right arrow: This moves the selected packages from "included packages from instrumentation" to "excluded packages from instrumentation".

Left arrow: This moves the selected packages from "excluded packages from instrumentation" to "included packages in instrumentation".

If no packages are avaialable in exclude packages list by default the following packages will be excluded:
     "javax.", "org.apache.commons.", "weblogic.", "com.bea.", "bea.", "jrockit.", "com.certicom.", "org.quartz.", "com.arcturus.", "kodo.", "com.rsa.", "org.omg.", "com.octetstring.", "com.google.", "org.snmp4j.", "org.nsclient4j.", "com.tangosol."

Note:

i) All changes applied by clicking the "Apply" button in the bottom right of the page take effect after app server restart.

ii) If you remove the file "trace-exclusions_$ServerName.txt" in the applicare home directory, then the packages in the file "trace-exclusions.txt" will be considered as exclusion list.

iii) Below packages are not allowed to be added to inclusion list/included in instrumentation:
   "javax.", "org.apache.commons.", "org.apache.struts", "com.bea.", "bea.", "jrockit.", "com.certicom.", "org.quartz.", "com.arcturus.", "kodo.", "com.rsa.", "org.omg.", "com.octetstring.", "com.google.", "org.snmp4j.", "org.nsclient4j.", "com.tangosol.", "weblogic."

 

 

Method Profiling Configuration

Select the Method Profiling Configuration tab to create / modify the filter criteria for Method Profiling Aspect.

Method Type: Select the type of methods to filter (All, public, private or protected)

Return Type: Return type of the methods (eg. java.lang.String ). Use * to match all types including void.

Package/Class/Interface: Expression specifying the package, class or interface name
Eg.
org.apache..* - Match all classes and method names in org.apache package and sub packages
org.apache.*.* - Match all classes and method names in org.apache package
org.apache.commons.StringUtils.escape - Match methods named 'escape' in org.apache.commons.StringUtils class.

Parameter Type(s): Parameter types of the methods. Use .. to match all parameter types
Eg. java.lang.String, java.lang.Long � Match methods accepting String and a Long as a parameter

Inverse Filter: When selected, adds a logical NOT condition to the filter

Add the condition you built by clicking the + button. You can build a complex filter by adding multiple conditions as shown in the screen shot. Once all the conditions are added to the list, save the configuration by clicking the Save button.

Also remember to deploy the Method Profiling Aspect from the Profiling Deployment tab, if it is not already deployed.

 

IntelliTrace Analyzer

IntelliTrace Analyzer contains separate views to display data collected by IntelliTrace aspects. Click Server Analyzer -> IntelliTrace to open IntelliTrace Analyzer.

SQL View

SQL Statement Profiling provides real-time performance statistics of database requests. You can view what statements are taking the most time and quickly diagnose performance impacting SQL/s.

Errors

Errors view displays any SQL, JNDI, HTTP, LDAP and WebServices errors that occured on the selected server for the selected time period. Error detection is enabled when respective profiling is enabled. Selecting a SQL in the table displays the stack trace of the code that invoked the SQL. SQL Error detection allows you to quickly track down SQL issues to the exact line in the code plus the reason for the failure.

JDBC Connection View

JDBC Connection Profiling feature allows you to detect leaked JDBC connections, connections held open for long periods of time and code responsible for connection leaks. You can view the connections that are currently in open state and clicking on an open connection will display the call stack which requested the connection. Statistics displayed are for the selected server in the UI.

HTTP View

HTTP Profiling feature allows you to easily view the performance statistics of real-time HTTP requests on-the-fly. Statistics displayed are for the selected server in the UI.

IntelliTrace View

IntelliTrace decides what methods to track dynamically and decides the depth of the call stack based on invocation counts and time spent on each individual method to minimize the overhead of tracing. IntelliTrace does what the real experts do with the profiling data - iliminate data that has little impact but it does it in real time. This way no time/resources are spent on gathering data that doesn't have meanigful impact on performance.

The Tree view displays the invocations as a call graph and shows the invocation count of each method and the total time spent within the methods. HTTP requests and SQLs are displayed inline with the URL and SQL prefixes respectively.

The three pie charts below displays following data:

1. Children Methods - Time Spent: Absolute Time spent in each child method below the selected point in call graph.
2. Children Methods - Invocation Count: Total invocation count for each child method below the selected point in call graph.
3. Cumulative Time (Code vs Database): Total time spent in Java code and Database call below the selected point in call graph.

Initial IntelliTrace view charts show time spent on selected calls e.g. 94.04 % time went into viewCategory and 5.535 % in view Item.

Selecting the Hotspots tab in IntelliTrace view displays performance data of all children methods below the selected point in call graph. This is useful in identifying the most expensive calls in a call graph without drilling down through the tree view. Hotspot view shows percentile data.

 

IntelliSense View

IntelliSense is a feature that works when IntelliTrace is enabled to automatically detect transaction executions that deviate considerably from the normal execution times and saves a complete call graph of the execution for later analysis.

 

Method Profiling

Method Profiling feature allows you to easily view the performance statistics of real-time method executions on-the-fly. This Profiling Aspect needs to be configured and deployed before you can view performance data. See Method Profiling Configuration for more information on configuration.

Parameter Value Masking

Parameter Value masking feature allows you to mask the value of HTTP parameters in Applicare. This way you can hide user's sensitive information.

Enabling IntelliTrace on Production Servers

IntelliTrace was designed to run on production servers with very minimal overhead so that the end users don't experience a noticeable performance degradation. However on large applications where lot of third party libraries are used, it is recommended to enable IntelliTrace on a test server and perform the following optimization before enabling IntelliTrace on production servers.

Basically you can tune IntelliTrace to ignore third party libraries altogether by adding the package names of the third party code to trace-exclusions.txt file in Applicare home directory. In most cases you are not interested in the internal working of these third party libraries and excluding them from tracing will reduce the overhead significantly.

Enable IntelliTrace on the test server (requires restart) and run through few different transactions or run a load script. Now in IntelliTrace window if you expand the transactions to view call graphs you may see third party classes/methods in the call graph.

Open trace-exclusions.txt file in a text editor and you can see that we have already added most of the common libraries such as Spring framework and Apache Commons in this file. So if you see some third party classes/methods in you IntelliTrace window that you are not concerned about, you should add the base package name of that library to trace-exclusions.txt in a new line.

e.g. To disable tracing all classes in org.apache and its sub packages, you should add the following line

org.apache.

Remember to copy the modified file to all agent Applicare directories on other machines too and you will have to restart application servers for the changes to take effect.

Now you can use the modified file in production Applicare installation to get best performance while IntelliTrace is enabled. Also you can send the packages you want excluded to Arcturus Tech so they will be included in trace-exclusions.txt file by default in future versions.

Enable or Disable IntelliTrace Profiling and Business Transaction Tracing Without Restarting

By default enabling / disabling IntelliTrace Profiling and Business Trasaction tracing requires affected servers to be restarted for the changes to take effect. If for some reason you need to apply these changes without restarting, you can do so by editing the relevant agent startup script in Applicare/scripts/ folder (e.g. for Tomcat servers on Windows, setTomcatAgentEnv.bat) and change norestart-intellitrace=false to norestart-intellitrace=true. Enabling this flag is not recommended for production servers. On a loaded server with this flag set to true when you enable IntelliTrace the server may stop responding for few minutes. This happens because on enabling IntelliTrace Applicare Profiling agent would start instrumenting classes and that has impact on server performance.

Click here for Custom HTTP Tags help and here for Business Transaction User Guide.