Configuring Dynamic Instrumentation
To configure dynamic instrumentation, select Configuration -> Configure Dynamic Instrumentation on the top menu. This page will allow you to deploy/undeploy profiling agent, configure Dynamic Instrumentations and Method Profiling (optional).
Enable Agents on Server
Applicare Profiling agent should be enabled on the app server for deployed profiling/monitoring to work. This can be done on IntelliTrace Deployment tab by selecting the server from the right hand server list.
No Dynamic Profiling
Selecting this option will disable all instrumentation on all the attached agents.
This will enable HTTP, SQL and JDBC connection profiling on the agent enabled app servers/JVMs.
In advanced mode, you can select and deploy individual aspects. For example you maybe only interested in detecting JDBC Connection leaks and might want to improve performance by taking out the other aspects you don't want.
Following are the available aspects.
1. HTTP Profiling
2. HTTP Error Code Profiling
3. SQL Profiling
4. JDBC Connection Profiling
5. Business Transactions
6. Intellitrace profiling - Deploying / Undeploying IntelliTrace profiling requires the agent servers to be restarted for the changes to take effect.
7. Method Profiling
8. External WebServices
9. End User Experience Profiling
10. End Client Process Monitoring (Slow Calls only)
11. End Client Process Monitoring (Every Call)
12. JNDI Lookup
13. External JMS Messages
14. LDAP Calls Profiling
15. Mail Monitoring
16. Collections Monitoring
17. Network Connection Profiling
18. Heap Tracker (BETA)
Note: Please read section InelliTrace Configuration (for Applicare 6.9.0 onwards) & Enabling IntelliTrace on Production Servers for using IntelliTrace in production servers.
Deploy Aspects In Advanced Mode
1. Select the aspects you want to deploy from the Available Aspects list and click the Add button.
2. Selected Aspects will be moved to the Deployed Aspects list.
3. Click Save button to save the configuration.
Note: Deploying / Undeploying IntelliTrace profiling requires the agent servers to be restarted for the changes to take effect.
Undeploy Aspects In Advanced Mode
1. Select the aspects you want to undeploy from the Deployed Aspects list and click the Remove button.
2. Selected Aspects will be removed to the Deployed Aspects list.
3. Click Save button to save the configuration.
Per Server Advanced Mode
In this mode each server can have its own set of aspects.
Starting with Applicare 6.9.0 dynamic IntelliTrace can be enabled for an app or a particular set of apps. This helps only focus intrumentation on important applications and keep the overhead extremly low.
This is enabled by adding ",perappinstrumentation=true,perappinstapps=/JPetStore:/qos" to the Applicare -javaagent argument on app server start. In this example JPetStore and qos are the applications that will be instrumented by Applicare Profiling Agent. Since app servers allow you to deploy two different apps as JPetStore and jpetstore, application name for this should be same case as application name for deployment in app server. Application name here is case sensitive.
Depending upon how you pass Applicare startup arguments either you will need to modify setAgentEnv.sh /setAgentenv.bat file in scripts directory or changes will need to be made to however you pass Applicare arguments to your app server. If you are using WebLogic and passing the startup arguments via 'Server Start' tab under configuration then you will make this change there.
-javaagent:$ARCTURUS_HOME/applicare-aspects.jar=verbosity=0,logfile=false,perappinstrumentation=true,perappinstapps=/JPetStore:/qos -Darcturus_home=$ARCTURUS_HOME "
Static IntelliTrace instrumention on a particular app has been available on Applicare since beginning but that requires instrumenting the application jars and redeploying them. With this perapp dynamic instrumentation there is no need to instrument the jar and redeploy the instrumented jar.
IntelliTrace configuration allows very fine-grained control on what packages are instrumented and what packages should be excluded from instrumentation. Instrumenting too many packages/classes does have a performance impact and only necessary packages should be instrumented. This feature is added in Applicare 6.9.0 version and is not available in prior versions.
Diagnostic mode allows finding out what packages and how many classes would be instrumented by the current settings without actually instrumenting classes. This is helpful for app servers where many packages/apps are deployed and server startup time and performance needs to be tuned.
When Diagnotic mode is enabled, no classes are instrumented but all the data shown on IntelliTrace Configuration tab is gathered. You should use this mode to find out what packages/classes are instrumented and fine tune the list of packages that are required for desirable results.
By default the packages in the file "trace-exclusions.txt" in the applicare home directory are considered as the exclusion list. These packages are not transformed. If you update the package details here on IntelliTrace Configuration tab and "apply" changes, then a new exclusion file is created in the applicare home directory with the name "trace-exclusions_$ServerName" and exclusion package details are persisted in this file. These changes take effect when the app server is restarted.
Threshold limit: This limits the # of classes intrumented by Applicare. This is meant to prevent app server start taking too long in case many apps are deployed on it. Classes are transformed until the threshold limit is reached and after that no classes are transformed. A message is shown in IntelliTrace and BTM Summary tab for each server where this limit is reached.
Diagnostic mode: If diagnostic mode is enabled then no classes will be transformed. If it is not enabled, then the packages which are not in exclusion list are transformed until the threshold limit is reached.
Force apply: It helps to re-instrument the classes without required to restart the app server.
Instrumentation count details: This shows the count of transformed/untransformed classes/packages.
Packages included in instrumentation: This shows the list of packages which "included in instrumentation".
Packages excluded from instrumentation: This shows the list of packages which "excluded from instrumentation".
Right arrow: This moves the selected packages from "included packages from instrumentation" to "excluded packages from instrumentation".
Left arrow: This moves the selected packages from "excluded packages from instrumentation" to "included packages in instrumentation".
If no packages are avaialable in exclude packages list by default the following packages will be excluded:
"javax.", "org.apache.commons.", "weblogic.", "com.bea.", "bea.", "jrockit.", "com.certicom.", "org.quartz.", "com.arcturus.", "kodo.", "com.rsa.", "org.omg.", "com.octetstring.", "com.google.", "org.snmp4j.", "org.nsclient4j.", "com.tangosol."
i) All changes applied by clicking the "Apply" button in the bottom right of the page take effect after app server restart. But if "Foce Apply" checkbox is enabled, then it will take effect without app server restart.
ii) In case if the changes are applied (without Force Apply) when app server is down, then app server will run with older changes when it starts. Otherwise if the changes are applied with Force Apply when app server is down, then the app server will run with the latest changes when it starts.
iii) If you remove the file "trace-exclusions_$ServerName.txt" in the applicare home directory, then the packages in the file "trace-exclusions.txt" will be considered as exclusion list.
iv) Below packages are not allowed to be added to inclusion list/included in instrumentation:
"javax.", "org.apache.commons.", "org.apache.struts", "com.bea.", "bea.", "jrockit.", "com.certicom.", "org.quartz.", "com.arcturus.", "kodo.", "com.rsa.", "org.omg.", "com.octetstring.", "com.google.", "org.snmp4j.", "org.nsclient4j.", "com.tangosol.", "weblogic."
Select the Method Profiling Configuration tab to create / modify the filter criteria for Method Profiling Aspect.
Method Type: Select the type of methods to filter (All, public, private or protected)
Return Type: Return type of the methods (eg. java.lang.String ). Use * to match all types including void.
Package/Class/Interface: Expression specifying the package, class or interface name
org.apache..* - Match all classes and method names in org.apache package and sub packages
org.apache.*.* - Match all classes and method names in org.apache package
org.apache.commons.StringUtils.escape - Match methods named 'escape' in org.apache.commons.StringUtils class.
Parameter Type(s): Parameter types of the methods. Use .. to match all parameter types
Eg. java.lang.String, java.lang.Long ï¿½ Match methods accepting String and a Long as a parameter
Inverse Filter: When selected, adds a logical NOT condition to the filter
Add the condition you built by clicking the + button. You can build a complex filter by adding multiple conditions as shown in the screen shot. Once all the conditions are added to the list, save the configuration by clicking the Save button.
Also remember to deploy the Method Profiling Aspect from the Profiling Deployment tab, if it is not already deployed.
IntelliTrace Analyzer contains separate views to display data collected by IntelliTrace aspects. Click Server Analyzer -> IntelliTrace to open IntelliTrace Analyzer.
SQL Statement Profiling provides real-time performance statistics of database requests. You can view what statements are taking the most time and quickly diagnose performance impacting SQL/s.
Errors view displays any SQL, JNDI, HTTP, LDAP and WebServices errors that occured on the selected server for the selected time period. Error detection is enabled when respective profiling is enabled. Selecting a SQL in the table displays the stack trace of the code that invoked the SQL. SQL Error detection allows you to quickly track down SQL issues to the exact line in the code plus the reason for the failure.
JDBC Connection Profiling feature allows you to detect leaked JDBC connections, connections held open for long periods of time and code responsible for connection leaks. You can view the connections that are currently in open state and clicking on an open connection will display the call stack which requested the connection. Statistics displayed are for the selected server in the UI.
HTTP Profiling feature allows you to easily view the performance statistics of real-time HTTP requests on-the-fly. Statistics displayed are for the selected server in the UI.
IntelliTrace decides what methods to track dynamically and decides the depth of the call stack based on invocation counts and time spent on each individual method to minimize the overhead of tracing. IntelliTrace does what the real experts do with the profiling data - iliminate data that has little impact but it does it in real time. This way no time/resources are spent on gathering data that doesn't have meanigful impact on performance.
The Tree view displays the invocations as a call graph and shows the invocation count of each method and the total time spent within the methods. HTTP requests and SQLs are displayed inline with the URL and SQL prefixes respectively.
The three pie charts below displays following data:
1. Children Methods - Time Spent: Absolute Time spent in each child method below the selected point in call graph.
2. Children Methods - Invocation Count: Total invocation count for each child method below the selected point in call graph.
3. Cumulative Time (Code vs Database): Total time spent in Java code and Database call below the selected point in call graph.
Initial IntelliTrace view charts show time spent on selected calls e.g. 94.04 % time went into viewCategory and 5.535 % in viewItem.
Selecting the Hotspots tab in IntelliTrace view displays performance data of all children methods below the selected point in call graph. This is useful in identifying the most expensive calls in a call graph without drilling down through the tree view. Hotspot view shows percentile data.
IntelliSense is a feature that works when IntelliTrace is enabled to automatically detect transaction executions that deviate considerably from the normal execution times and saves a complete call graph of the execution for later analysis.
Method Profiling feature allows you to easily view the performance statistics of real-time method executions on-the-fly. This Profiling Aspect needs to be configured and deployed before you can view performance data. See Method Profiling Configuration for more information on configuration.
Parameter Value masking feature allows you to mask the value of HTTP parameters in Applicare. This way you can hide user's sensitive information.
IntelliTrace was designed to run on production servers with very minimal overhead so that the end users don't experience a noticeable performance degradation. However on large applications where lot of third party libraries are used, it is recommended to enable IntelliTrace on a test server and perform the following optimization before enabling IntelliTrace on production servers.
Basically you can tune IntelliTrace to ignore third party libraries altogether by adding the package names of the third party code to trace-exclusions.txt file in Applicare home directory. In most cases you are not interested in the internal working of these third party libraries and excluding them from tracing will reduce the overhead significantly.
Enable IntelliTrace on the test server (requires restart) and run through few different transactions or run a load script. Now in IntelliTrace window if you expand the transactions to view call graphs you may see third party classes/methods in the call graph.
Open trace-exclusions.txt file in a text editor and you can see that we have already added most of the common libraries such as Spring framework and Apache Commons in this file. So if you see some third party classes/methods in you IntelliTrace window that you are not concerned about, you should add the base package name of that library to trace-exclusions.txt in a new line.
e.g. To disable tracing all classes in org.apache and its sub packages, you should add the following line
Remember to copy the modified file to all agent Applicare directories on other machines too and you will have to restart application servers for the changes to take effect.
Now you can use the modified file in production Applicare installation to get best performance while IntelliTrace is enabled. Also you can send the packages you want excluded to Arcturus Tech so they will be included in trace-exclusions.txt file by default in future versions.
By default enabling / disabling IntelliTrace Profiling and Business Trasaction tracing requires affected servers to be restarted for the changes to take effect. If for some reason you need to apply these changes without restarting, you can do so by editing the relevant agent startup script in Applicare/scripts/ folder (e.g. for Tomcat servers on Windows, setTomcatAgentEnv.bat) and change norestart-intellitrace=false to norestart-intellitrace=true. Enabling this flag is not recommended for production servers. On a loaded server with this flag set to true when you enable IntelliTrace the server may stop responding for few minutes. This happens because on enabling IntelliTrace Applicare Profiling agent would start instrumenting classes and that has impact on server performance.