Technology and Community

Jim O'Neil

Subscribe to Jim O'Neil: eMailAlertsEmail Alerts
Get Jim O'Neil: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Apache Web Server Journal

Apache Web Server: Article

EAServer Problem Analysis & Troubleshooting

Part art, part science Part 2

Dumping the Threads
If you have narrowed down a particular performance or stability issue to the threading level, some low-level diagnostic tools can be used to help determine the source of the problem. However, these tools require some rather sophisticated knowledge of both the operating system and EAServer internals to be of benefit. Typically, at this stage in the process, you will have enlisted the aid of Sybase technical support.

Microsoft Windows (NT, 2000, XP)
userdump is part of the OEM Support Tools package available from Microsoft's web site. After it's installed, userdump can monitor running applications for exceptions or produce a dump file of a stalled or hung process. The dump file can be analyzed with a post-mortem debugger such as WinDBG (also available from Microsoft). With the debugger, you can determine the status of every thread in the process to gain insight into the cause of the failure.

On Solaris you can use the /usr/proc/pstack command to dump the call stack for each lightweight process (kernel thread) in your process. The output includes method names and memory offsets that can help identify the cause of an explicit failure or stalled server.

truss traces system calls executed and faults encountered. Using the -p switch, you can attach truss to an existing process to turn on the trace in the specific area of execution that you want to analyze rather than starting the trace when EAServer starts up.

AIX 5 incorporates the truss command described for Solaris. Earlier versions of the AIX operating system include a trace daemon that runs asynchronously and captures the various user and system events requested when the command is run. Because it's a daemon process, you can start tracing whenever needed and stop tracing by issuing the trcstop command.

tusc is a truss-like tool that can be used on HP-UX 11. It's not currently an official HP product, but it can be downloaded from an HP FTP site. More recent versions of tusc can run it in truss emulation mode, providing some level of compatibility with the Solaris command.

Database Access
Database interaction is likely to represent a big part of your n-tier application, so you need options to trace and diagnose issues related to database operations. In a distributed architecture, the database "client" is really the application server itself rather than an end user. This environment can be more complex than in a client/server application because the application server (re)uses just a few persistent physical database connections to service disparate requests across multiple end users. In terms of tracing database operations executed in PowerBuilder components, you can use a PowerBuilder-specific tracing option as well as specific tracing tools and options provided by the vendor of the back-end database.

PowerBuilder Database Tracing
The PowerBuilder database trace you're familiar with in the client/server environment can also be used with EAServer running on the Windows NT, 2000 and XP platforms. To kick the trace off, you must preface the database driver specification in your Transaction object's DBMS property with the characters TRA followed by a space. The output of this trace will appear by default in the root Windows directory, typically c:winnt. You can change the location of the file generated by putting a file called PB.INI in the bin directory of your EAServer installation. For example, include the following two lines in %JAGUAR%/bin/pb.ini to redirect the log file to c:logseaspb.log (assuming that c:logs is a valid directory).

Database Access
[Database] DBTraceFile=c:logseaspb.log

This trace is useful in diagnosing SQL errors for specific situations; however, because the log can grow quickly and the logging functionality is not thread-safe, it's less useful in busy multiple-user scenarios.

Vendor-Specific Database Tracing
You can gain insight into databases from publicly available client-tracing tools. Most enterprise database-management systems also include rather sophisticated tracing capabilities done on the database server itself. Examining those proprietary techniques exceeds our scope here; furthermore, they generally involve your DBA, who will be more equipped to provide the specific tracing information you desire. For our purposes, we will confine our discussion to client traces that you can set up easily.

ODBC Trace
ODBC API tracing lets you track the individual API calls made to the ODBC driver manager libraries, including the input and output parameters and API return values. You have to be a bit familiar with the ODBC API to fully understand the contents of the trace, but documentation available on the Microsoft support site can be of assistance here. The tracing option captures all ODBC activity, regardless of the data source, so the trace can grow quite large. Besides, output for all ODBC connections is routed to a single file, so it can be difficult to analyze one specific connection in a busy server environment.

Microsoft Windows Platforms (NT, 2000, XP) ODBC tracing is enabled via the ODBC Administrator interface accessed from the Control Panel as shown in Figure 5. To enable tracing, just specify the desired trace file and press the Start Tracing Now button. If an ODBC data source is already loaded, you may find that the trace file is empty. If this happens, shut down all running ODBC applications and restart EAServer to initiate the trace.

Unix Platforms On Unix platforms the ODBC driver manager implementation is provided by DataDirect Technologies in the $JAGUAR/intersolv directory. EAServer's default startup environment sets the ODBCINI environment variable to the odbc.ini file located in this directory; therefore, to enable tracing, you should set the properties in the [ODBC] section of this file as detailed in Table 17.5.

OCI Tracing
The Oracle network layer, commonly called SQLNet, Net8 or Net9 depending on the version, provides both a logging and a tracing option. The logging option is used to record information about network failures and can help in diagnosing errors such as ORA-12541 No Listener. Tracing provides a good deal of detail about the communications between the connection cache and the database server, and can be used to diagnose incorrect application performance not accompanied by an actual Oracle error.

Logging Oracle automatically logs all the client network errors that may occur. Although you can't turn this option off, you can specify where the log file should be written via two parameters in the network/ADMIN/sqlnet.ora file located in the Oracle Home directory on the EAServer machine. In lieu of editing this file manually, you can use Oracle Network Manager or the $ORACLE_HOME/bin/netasst command on Unix hosts to specify the following parameters:

  • LOG_FILE_CLIENT: Name of log file (the default is sqlnet.log)
  • LOG_DIRECTORY_CLIENT: Directory where the log file is written (the default is the current working directory)
To trace the communications between the Oracle connection cache and the Oracle server, set the following parameters in the sqlnet.ora file. As with logging, these parameters can be set manually or with the graphical utilities supplied by Oracle.
  • TRACE_DIRECTORY_CLIENT: Directory where the log file will be written (the default is the current working directory)
  • TRACE_FILE_CLIENT: Name of the client log file (the default is sqlnet.trc)
  • TRACE_UNIQUE_CLIENT: When set to ON, creates a separate trace file for each client connection, which is useful when tracing multiple connections
  • TRACE_LEVEL_CLIENT: Value or constant specifying level of tracing, with the higher values yielding more information.
OFF = 0
USER = 4

The information contained in the trace can be a bit obtuse because it was designed to be analyzed by Oracle technical staff rather than users; nevertheless, it may provide some help in troubleshooting anomalies that may not be detected with other troubleshooting techniques.

JDBC Tracing
For tracing JDBC connections, two main options can be supplemented by vendor-specific tracing capabilities. The next sections cover the use of high-level connection cache tracing through an EAServer-specific property and more detailed statement-level tracing via a third-party JDBC tracing tool called JDBCSpy. For vendor-specific tracing tools, consult the OCI and OpenClient tracing sections if you're using Oracle or Sybase servers. For other database servers, consult the vendor's documentation.

More Stories By Jim O'Neil

Jim is a Technology Evangelist for Microsoft who covers the Northeast District, namely, New England and upstate New York. He is focused on engaging with the development community in the area through user groups, code camps, BarCamps, Microsoft-sponsored events, etc., and just in general serve as ambassador for Microsoft. Since 2009, Jim has been focusing on software development scenarios using cloud computing and Windows Azure. You can follow Jim on Twitter at @jimoneil

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.