Sunday, May 17, 2009

In-Depth: Load-Balancing E-Business Suite Environments

As I watched blood drain from my arm, my thoughts turned to load-balancing and system redundancy. My phlebotomist had just cheerfully informed me the lab's primary testing PC had failed that morning, so there would be an unusual delay in getting blood test results until a replacement arrived.


Increasing Fault Tolerance at Lower Cost

You can use load-balancing routers (LBRs) to protect your E-Business Suite from similar types of system failures. Load-balancers increase your environment's fault-tolerance and scalability by distributing load across a pool of application servers like this:

Generic Apps Load-balancing:

Besides fault-tolerance and scalability, another appealing benefit is that you can use load-balancing to substitute expensive SMP boxes with clusters of inexpensive Linux-based commodity servers.

Linux Load-Balancing on Oracle's Global Single Instance

In fact, this is what we've done at Oracle: our own E-Business Suite environment is a Global Single Instance running on about 58 Linux-based application servers. This has reportedly saved us millions in operating costs.

If you're joining us at this year's OpenWorld conference, make sure you attend Bret Fuller's ever-popular session on how we run the E-Business Suite internally; some of the statistics he shows on our transactional volumes are mind-blowing.

Supported Load-Balancing Methods

The E-Business Suite supports the following types of load-balancing:
I'll cover only the first two methods in this article.

HTTP Layer Load-Balancing

HTTP Layer load-balancing is the most common method used in E-Business Suite environments.

HTTP Layer Load-Balancing:

In this configuration, end-users navigate to a specific Web Entry Point that represents your E-Business Suite's domain name. An HTTP Layer load-balancer routes all subsequent traffic for a specific user to a specific Web Node.

HTTP Layer load-balancers may use heartbeat checks for node death detection and restart, and sophisticated algorithms for load-balancing.

DNS-Based Load-Balancing

When an end-user's browser attempts to access your E-Business Suite environment, your local Domain Name Server (DNS) can direct that user to a specific application server in a pool based on available capacity:

DNS-Based Load-balancing:

Traffic for that user's session will be handled by the application server 10.10.10.10, while other users' traffic may be directed to other application servers in the pool. Like HTTP layer load-balancers, many DNS-based load-balancers use heartbeat checks against nodes and sophisticated algorithms for load-balancing.

Business Continuity ("Disaster Recovery")

Our larger enterprise-class customers combine DNS-based and HTTP layer load-balancers to support their business continuity plans. In the event of a disaster, end-users are directed via a DNS-based load-balancer from the primary E-Business Suite environment to an offsite standby site.

Disaster Recovery using DNS + HTTP LBRs:

Minimum Requirement: Session Persistence

Remember that although Oracle doesn't certify specific load-balancers or networking hardware with the E-Business Suite, we do support their use generically.
In other words, we've designed the E-Business Suite to be able to use load-balancers in general.

The minimum requirement is that a load-balancer support session
persistence
, where a client's initial HTTP connection is directed to a
particular application server, then subsequent HTTP requests from that
client are directed to the same server. As long as a
load-balancer is able to handle session persistence (also referred to
as "stickiness"), it's likely to work with the E-Business Suite.

Happy Learning !

Optimizing R12 Performance via OC4J Load-Balancing

Oracle Application Server provides features that allow customers to load balance their middle tier deployments. OC4J Clustering in OracleAS 10g is one such deployment that is widely used in load balanced configurations.

Starting with Oracle E-Business Suite Release 12.0.2 (Release Update Pack 2), we support OC4J Clustering as part of AutoConfig's load balancing deployment options. This configuration option is supported for the OC4J instance running out of R12's 10.1.3 ORACLE_HOME.

Clustering Model

Oracle Application Server 10g 10.1.3 supports various models for OC4J clustering:
  • Dynamic node discovery
  • Static discovery servers
  • Cross topology gateways
  • Manual node configuration.
E-Business Suite Release 12 uses the manual node configuration model, which is also referred to as static node-to-node communication. Node and port information is manually specified in this mode. Configuration details are managed by Autoconfig, which handles all necessary settings in the R12 configuration files.

Figure-1 static node-to-node communication model

static-communication:

Deployment Options


Release 12 supports the following deployment options with OC4J clustering:
  1. Single Web Entry Point
  2. Multiple Web Entry Points

Prerequisites

  1. Oracle Applications Technology Stack Patch Set 2 5917601 or higher.
    • This Patch Set is included in the R12.ATG_PF.A.DELTA.2 5917344 and the Release 12.0.2 (Release Update Pack 2)
  2. A correctly-configured hardware load balancer

Configuring OC4J Load Balancing

Configuring both deployment options requires:
  1. Changing the E-Business Suite Context file using the Context File Editor
  2. Running AutoConfig
  3. Restarting the application tier server processes.
Single Web Entry Point

In this deployment option, there is single web entry point for the OC4J
applications - oacore, forms and oafm services are configured to run on all or some
of the application tier nodes.

Figure-2: Deployment model with Single Web Entry point and OC4J application running on all the nodes

SingleWebEntry: Architecture diagram showing an E-Business Suite environment with two application tier servers and a single webentry point

Multiple Web Entry Points

In this deployment option, there are multiple web entry points with OC4J applications running on selected application tier nodes.

Figure-3: Deployment model with Multiple Web Entry points and OC4J applications running on selected nodes

multipleweb: Architecture diagram showing a multinode Apps environment

References
Happy Learning !

Understanding JDBC Connections From the eBusiness Middle Tier

In this article I will describe the basics of configuring and monitoring JDBC connections between the eBusiness Web Tier and the Database, then cover some common issues to help identify root causes for both Release 11i and Release 12 of eBusiness Suite.

11i Architecture:

Brief overview

In general, whenever a functional page requires data from the database, it makes a call to a lower code level (the Java layer in the Application Object Library, also known as AOL/J) which handles the database connectivity. The AOL/J code provides a JDBC connections to the database through the Database Connection Pool.

You should note that the Java Connection Pool mechanism for eBiz is completely provided by eBiz code and does not use the mechanisms provided to pool connections through Java language directly, nor through the Application Server configuration.

Configuring JDBC connection pooling

The JDBC connection pool is created using the settings in the DBC file. The name, location and contents of this DBC file is controlled through AutoConfig. To modify any of the JDBC connection pool parameters, you should therefore use the techniques described in the following document to ensure changes are maintained in a supportable and consistent way:

The "s_dbc_file_name" variable in the CONTEXT.xml file provides the dbc filename and is located in the $FND_TOP/admin/<>_ directory.

JDBC connection pool parameters and their use are covered in the following documents:

When considering any changes, you should also take into account that every JVM has its own JDBC connection pool. For example, if you have one Web Node with three OACoreGroup JVMs plus one XmlSvcsGrp JVM configured, then you will have a total of four JDBC connection pools with connections to your eBiz database

Monitoring the JDBC connection pool

It is always a good idea to understand how your environment looks when things are going well, to give you a baseline to compare against if you need to investigate any issues.

You will most certainly need to review the JDBC connection data if you are experiencing issues.

Monitoring JDBC Connections through Oracle Applications Manager (OAM)

Login to OAM directly or via the "System Administration" responsibility.
  1. Select the "JServ Usage" under the Monitoring section in OAM
  2. Click the "Expand all" link to list the Servers and all the JServ processes for OACoregroup. This shows memory usage, connections (including "Potentially Leaked") and Application Module information. You can click the "Add to support cart" to capture this page if Oracle Support are assisting your investigation.
  3. If there are any non zero values for "Potentially Leaked" then click on this number to drill down into the details
  4. Select "Yes" for the "Filter by Potentially Leaked" option and click "Go" button
  5. Click "Show all details" to display the Java Thread dump for all the potentially leaks Java connections

mz_jdbc_oam:

The "old" way of gathering this data was to use the URL http://host.domain:port/OA_HTML/jsp/fnd/AoljDbcPoolStatus.jsp but this will only give data for the one JVM you happen to connect to, so may not be so useful in multi-JVM environments.

Run SQL scripts to monitor database connections

Using SQL scripts will not give so much information as OAM, but can be used to provide useful summary information on a periodic basis. For example you may wish to include the information from the SQL below as part of your baseline data:

REM
REM START OF SQL
REM
REM Connections by machine and instance
select s.machine, s.username, s.module, s.inst_id, count(*) how_many
from (select distinct PROGRAM, PADDR, machine, username, module, inst_id from gV$SESSION) s,
gv$process p
where s.paddr = p.addr
and p.inst_id = s.inst_id
group by s.machine,s.username, s.module, s.inst_id
/
REM
REM END OF SQL
REM

NOTE - when looking at V$SESSION, the STATUS of JDBC connections tend to show as INACTIVE, this is normal and does not indicate a specific problem

Where Could It Go Wrong?

Issues with the JDBC connection pool tend to be of a nature whereby the number of database connections increase over time and do not seem to be released again. If this continues unchecked, then you may find the database will run out of sessions/processes and/or the Middle Tier JVM will run out of connections or memory.

A "quick fix" would normally be to restart Apache, but the underlying cause would need to be investigated further.

Issues can potentially occur at five different levels:

  1. Core Java Code
  2. AOL/J JDBC Code
  3. OA Framework
  4. Functional Code
  5. External Influences

I'll discuss each of these areas next.

Core Java code

Although eBiz does not use native Java connection pooling methods, we do rely on the underlying Java APIs generally. Any issues at this level will generally require the latest Java version to be installed

If you need to upgrade your Java version, see:

AOL/J JDBC code

As this is the code that handles JDBC connection, it is often the first area to be blamed, but the JDBC connection pool can only drop database connections where the calling application has released the JDBC connection it holds in the pool, so it often turns out to be an issue higher up the code stack.

Number of JDBC connections increase after ATG Rup 5 because jdbc parameters are lower case (Metalink Note 459072.1) describes a known issue with Apps 11i.

It is prudent to be on the latest JDBC driver patch, but should have at least applied one of the following patches:-

The latest JDBC patch can be found in:

You should note that the JDBC driver version has no relation to the Database version, as it is installed on the eBiz Middle Tier. For example, the latest version of JDBC drivers provided by patch 4899697 (9.2.0.8) is the same patch for all RDBMS versions.

OA Framework (OAF)

OA Framework calls AOL/J when it needs a database connection, and it is up to OAF to release any such connection when it has finished with it. There is an added complexity, in that OAF also has its own pooling mechanism for the OAF Pages, which is the "Application Module pool" (AM pool). This means that although a user may have finished with a page, the page and its associated database connection are retained for a period of time.

The AM pool is controlled by profile options, which are described in :-

Issues at this code level would tend to be either:

1. Issue with AM Pooling

You can test the effect of disabling AM pooling by setting the profile option "FND: Application Module Pool Enabled" to "No". Use this technique with caution if considering this for a production environment.

2. Specific bug where database connection is not released.

This would generally require patching.

Functional Code

Issues at this code level would tend to be a specific bug where a connection is not released.

External influences

Firewall timeouts are known to potentially cause an issue for JDBC connections. For details, see:

Although this note is for Apps 11i, the technical issue can also apply to Release 12 as well

Configuring eBiz to minimize JDBC connections

If investigating issues with JDBC connections increasing, it may be useful to minimise the database connections as much as possible by de-tuning the JDBC pool. This may reduce end user performance, so should be used with caution if considering this for a production environment.

To do so, you need to do both these steps:

1. Disable Application Module (AM) Pooling

This is necessary as the AM objects hold a JDBC connection whilst they are alive, even if they are not currently used.

Set the profile option "FND: Application Module Pool Enabled" (AMPOOL_ENABLED) at SITE level to a value of "No"

2. Set the JDBC connection pool parameters to release connections:

FND_JDBC_BUFFER_DECAY_INTERVAL=180
FND_JDBC_BUFFER_MIN=0
FND_JDBC_BUFFER_MAX=0
FND_JDBC_BUFFER_DECAY_SIZE=50

Conclusion

Identifying issues with JDBC connections can sometimes be a frustrating process, as the investigations may need to consider multiple failure points and complex architectures. I hope this article has given you a better understanding of JDBC Pooling and where to start looking for issues.

Happy Learning !