Technology and Community

Jim O'Neil

Subscribe to Jim O'Neil: eMailAlertsEmail Alerts
Get Jim O'Neil: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: Apache Web Server Journal

Apache Web Server: Article

EAServer Problem Analysis & Troubleshooting

Part art, part science Part 2

log4j
When dealing with Java-based components such as EJBs or JSPs, another option is to use the log4j package. log4j is a thread-safe Java package sponsored by the Apache Jakarta project. It's designed for Java developers to include as many as five levels of logging in their code and activate that logging code declaratively based on a properties file.

Besides hierarchical loggers, log4j includes the concept of appenders, which allow the output to be directed to a variety of targets, including files, consoles, the Windows Event Log or even a Java Message Service implementation. In conjunction with an appender, you format the output using a layout such as PatternLayout, which allows a variety of format masks similar to those available in the C printf function.

Although log4j has its roots in Java, the implementation has been ported to other languages such as C and C++. Given PowerBuilder's roots in C++, it's possible to integrate this functionality in PowerBuilder components via external function calls or as a redistributable PBNI extension. log4c, log4cpp and log4cplus are three open source projects available at www.sourceforge.net that you can leverage if you want to pursue such an architecture.

Threading in EAServer
A thread is a single execution path through a given process such as EAServer. Generally each client request to EAServer executes on a distinct thread in the process. EAServer itself operates on a native-threaded Open Server kernel and, therefore, relies solely on the operating system to create and manage threads, including balancing them across the available CPUs.

As just another program running on the host operating system, EAServer has a limited number of threads at its disposal (a little more than 2,000 on Windows operating systems), making them critical to scalability. Besides, the server's need to context-switch the given threads across many different tasks in the server can expose points of failure that might never have occurred in a simpler client/server version of the same application.

The cardinal rule of threads in EAServer is not to start your own! The EJB specification itself prohibits using user-spawned threads because of the complexity and concurrency issues that they can introduce. If you are using other component models in EAServer and need threading functionality, use EAServer's ThreadManager API. Do not use java.lang.Thread in Java components, CreateThread in C/C++ components or shared objects in PowerBuilder components because the threads created that way operate outside of EAServer's auspices often leading to stability issues.

EAServer Component Settings
Each component exposes two specific properties that control how EAServer handles threads executing in instances of those components. The first of these properties specifies whether threads are disassociated from idle component instances; the second indicates whether EAServer can safely execute multiple threads concurrently.

Bound Threads
The com.sybase.jaguar.component.bind.thread component property controls the thread assignment associated with a component instance when the instance is created.

When this property is set to true, a component instance is associated with a given operating system thread when it's created and that thread is not disassociated or available for reuse until the component instance is destroyed. That means that all instances of components with this setting retain the thread on which they were created, even if the component instance has been pooled and is currently inactive.

When com.sybase.jaguar.component.bind.thread is set to a value of false, the thread associated with the incoming client request is used to issue the requested method call and is relinquished when the method completes. In this case, component instances don't retain their thread when pooled; they potentially become associated with many distinct threads as they are activated and deactivated during their lifetime.

NOTE: Set com.sybase.jaguar.component.bind.thread to true for components implemented with PowerBuilder version 8.0.2 or earlier that use DataStore objects and are deployed to EAServer operating on Windows platforms. You must also set this property to true for ActiveX components in EAServer, as well as for any component that makes use of thread-local storage (such as the TlsAlloc, TlsFree, TlsGetValue and TlsSetValue methods in the Windows API).

Using bound threads unnecessarily limits the scalability of your application, especially as the number of pooled component instances grows. Not setting this property when it's required causes stability issues such as crashes, server hangs and other behaviors that are difficult to diagnose.

Concurrency (Thread Safety)
The com.sybase.jaguar.component.thread.safe component property indicates whether multiple threads can simultaneously execute through instances of that component.

If the component is also marked as shared (com.sybase.jaguar.component.sharing= true), multiple threads can execute simultaneously in that single instance. To safely allow this scenario, your component implementation must be thread-safe. Because of their reliance on the PowerBuilder VM, PowerBuilder components can't be shared or be concurrent by definition. Versions of EAServer earlier than 3.6.1 didn't enforce this rule and can source of instability in older installations.

If the component isn't marked as shared, the thread.safe property specifies whether multiple instances can execute simultaneously, with each instance executing on its own thread. You generally want this scenario; otherwise your throughput will suffer because any client request to execute a method on a particular component will be handled sequentially, not concurrently.

NOTE: Set com.sybase.jaguar.component.thread.safe to true if your shared component implementation isn't thread-safe, if it uses external function calls to libraries that aren't thread-safe libraries, or if it requires serialized access to some other resource.

Unnecessarily setting this property to true will degrade the performance of your application. However, incorrectly setting it to false will generally cause stability issues, including crashes, server hangs and other behavior that's difficult to diagnose.

Binding EAServer to a CPU
Occasionally an application just crashes or hangs for no apparent reason, even though you feel you've addressed the primary threading concerns just discussed. If you suspect a problem with the way threads are being handled in EAServer, one way to narrow the behavior is to bind the EAServer process to a single CPU. This isn't a solution, but it may let you keep the system relatively stable while you track down the root cause of the failure.

Microsoft Windows (NT, 2000, XP)
You can bind any running process to one or more processors via Task Manager. On a multiprocessor, the Set Affinity option seen in Figure 3 is provided in the pop-up context menu. When selected, this yields the Modal dialog box shown in Figure 4, letting you select the processor to bind to the process. From that dialog box you can see that the user has elected to bind the EAServer process to CPU 1.

When specified via Task Manager, this setting is transient, so the next time EAServer is started, it will be eligible to use all of the CPUs in the machine. For a more permanent solution, you can modify the EAServer executable image via the imagecfg program. This utility is provided in the support folder on the NT 4.0 CD-ROM and the Windows 2000 Server Resource Kit. The syntax is as follows:

imagecfg -a affinity_mask path_to_jagsrv

The affinity_mask is a hexadecimal number indicating the processor(s) to which the executable should be bound. Specifically, each CPU is associated with one bit of this 32-bit value. For instance, 0x1 = CPU0, 0x2 = CPU1, 0x4 = CPU2, and 0x8 = CPU3. The command for the binding shown in Figure 3 would be:

imagecfg -a 0x2 "c:program filessybaseeaserverinjagsrv.exe "

Solaris
On Solaris use the psrinfo command for the identifiers of the CPUs that are available and online. Follow that with the pbind command to bind the EAServer process to one of those CPUs. Note that the PSR column value output by the ps -P command can be used to show whether the process is currently bound to a CPU. You can also use the -u switch of the pbind command to disassociate the process from a specific CPU.

By defining processor sets on Solaris, you can bind EAServer to multiple processors. A super-user can create a processor set using the psrset command with the -c switch. Executing that command returns a processor set ID that you can provide to the psrset -b command to bind individual processes that the newly created processor set. When psrset is used, only processes bound to the defined processor set will execute on the processors in that set. All other unbound processes will be limited to running only on CPUs not associated with a processor set.

AIX
Use the bindprocessor command in AIX to determine the available processors and bind the process and its threads to a CPU. The -q switch enumerates the processors, and the ps -obnd command displays the CPU to which a process is bound.

HP-UX
HP-UX doesn't offer a command-line option for binding a process to a CPU; however, you can write a wrapper program in C that executes the mpctl system command with the MPC_SETPROCESS_FORCE flag to bind a given process to a CPU. Consult the manual page for mpctl on an HP-UX system for more information on how you might build such a utility.


More Stories By Jim O'Neil

Jim is a Technology Evangelist for Microsoft who covers the Northeast District, namely, New England and upstate New York. He is focused on engaging with the development community in the area through user groups, code camps, BarCamps, Microsoft-sponsored events, etc., and just in general serve as ambassador for Microsoft. Since 2009, Jim has been focusing on software development scenarios using cloud computing and Windows Azure. You can follow Jim on Twitter at @jimoneil

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.