Oracle SOA, AIA BPEL ESB and OSB knowledge base

Friday, January 23, 2009

Oracle Service Bus : Send Email

The first step is to create an SMTP Server configuration (it is the SMTP server that actually sends the email). To do this, you need to do the following:

- Click on the System Administration link in the left hand navigation in the service bus console

- Click on Create in the Change Center to create a new session in which you can make changes

- Click on Add and enter the details for your SMTP Server, e.g.

smtpserver

- Click Save.

Now we have an SMTP Server configuration we can use multiple times within Oracle Service Bus without entering this information again.

The next step is to create an email business service that uses this SMTP Server configuration:

- Click on the Project Explorer link in the left hand navigation

- Select (or create) the project and folder you want to create the email business service in

- From the Create Resource drop down select Business Service

- Enter a name and description for your business service and ensure you select the Service Type as Messaging Service:

emailBusinessService

- Click Next

- Select Text as the Request Message Type, leave the Response Message Type as none (sending email is a one-way service) and then click Next.

- On the next screen, select the Protocol as email and set the Endpoint URI to be mailto: and click Add:

emailBusinessService2

- Click Next

- Select your SMTP Server from the drop down list of SMTP Servers available and configure the rest of the settings on this screen appropriately, e.g.

emailBusinessService3

- Click Next, review the summary and then click Save.

- Click Activate to enforce these changes to Oracle Service Bus.

- Enter a description and click Submit.

You have now successfully created an email business service which you can use to send email from Oracle Service Bus. If you click on the bug icon in the Actions column alongside it, you can use the Test Console to prove this service does indeed send an email to the address you specified, with the settings you configured and the payload you specified in the Test Console.

In the second post of this set I will explain how a proxy service can make use of this email business service and some tips for customising the email sent.

Sunday, January 18, 2009

FTP Debatching

With the release of patchset 10.1.3.3 for Oracle SOA Suite 10.1.3, some cool new features have been introduced that I would like to bring to your attention. Besides the long-awaited error-policy framework, we now also get the option to de-batch large XML files when polling them from a file or FTP location. Debatching has been an option for the File and FTP adapters for some time now, but was always restricted to native (flat) files (typically record-based layouts). Large XML files have always been a challenge to BPEL, because internally XML is loaded in a DOM tree, which takes a considerable amount of memory compared to the original size of the document in its text form.

What does de-batching do?

Debatching polls for incoming files, and when it finds a file and starts reading it, it doesn't start a single new instance of a BPEL process and delivers the file contents, but instead it chops up the file in "batches" of configurable size (typically 1), and starts a new instance for each batch. So for a file of 1000 records (lines) and a batch size of 1, you will end up with 1000 new instances. There is no guarantee that these 1000 instances will be executed in order of the records in the original file, mind you. These instances will be processed by the pool of engine threads just like any other active instance. For XML, as part of the patch install instructions for the 10.1.3.3 patchset, you install a "stream-based" XML parser called STaX, which is well-known in the Java community. Instead of reading the whole file into main memory, the STaX parser just "browses" like a lexical scanner through the XML file and does not build large trees of objects in memory.

How it works

To activate the de-batching behaviour of the adapter, you have to add the "PublishSize" attribute to the jca:operation tag in the adapter's wsdl:

<jca:operation

PhysicalDirectory="D:\"

ActivationSpec="oracle.tip.adapter.file.inbound.FileActivationSpec"
DeleteFile="true"

IncludeFiles=".*\.xml"
PublishSize="1"
PollingFrequency="10"
MinimumAge="5"
OpaqueSchema="false"
>

What it will do at runtime is take your XML file, preserve the root tag, and for each child of the root, generate a separate inbound document containing the child with the root tag again wrapped around it:

<dcs320 xmlns="http://www.customer.com/schemas/dcs320/1.0">
<salesline>
<messagetype>DCS320</messagetype>
<messageid>487235</messageid>
<messagelinenumber>1</messagelinenumber>
<datamutationtype>0</datamutationtype>
<dcaddressnumber>0000410</dcaddressnumber>
</salesline>
<salesline>
<messagetype>DCS320</messagetype>
<messageid>487235</messageid>
<messagelinenumber>2</messagelinenumber>
<datamutationtype>0</datamutationtype>
<dcaddressnumber>0000411</dcaddressnumber>
</salesline>
<salesline>
<messagetype>DCS320</messagetype>
<messageid>487235</messageid>
<messagelinenumber>3</messagelinenumber>
<datamutationtype>0</datamutationtype>
<dcaddressnumber>0000412</dcaddressnumber>
</salesline>
</dcs320>

The above document will launch instances that get documents as input that look like:

<dcs320 xmlns="http://www.customer.com/schemas/dcs320/1.0">
<salesline>
<messagetype>DCS320</messagetype>
<messageid>487235</messageid>
<messagelinenumber>1</messagelinenumber>
<datamutationtype>0</datamutationtype>
<dcaddressnumber>0000410</dcaddressnumber>
</salesline>
</dcs320>

And so on, one for each <salesline> child. The ability to de-batch large XML files gives us the option to process large XML documents (that have a repeating structure) in BPEL, one piece at a time.

Process Control Challenges

The main challenge I've always had with debatching or other "parallel processing" patterns is to stay in control regarding error handling and monitoring progress & completion. Especially for non-transactional protocols like File and FTP, this can be tricky. Another neat feature that has been built into the 10.1.3.3 adapter is a batch notification mechanism. This means that at certain points in its processing, the adapter will post meta-data events about its progress to an event handler, typically another BPEL process. You configure this notification behaviour in the activationAgent section of the bpel.xml in the debatching process. Below an example of such a bpel.xml:

<?xml version = '1.0' encoding = 'UTF-8'?>
<BPELSuitcase>
<BPELProcess id="XMLDebatcher" src="XMLDebatcher.bpel">
<partnerLinkBindings>
<partnerLinkBinding name="debatch">
<property name="wsdlLocation">debatch.wsdl</property>
</partnerLinkBinding>
</partnerLinkBindings>
<activationAgents>
<activationAgent className="oracle.tip.adapter.fw.agent.jca.JCAActivationAgent" partnerLink="debatch">
<property name="batchNotificationHandler">bpel://default|BatchMgmtProcess</property>
<property name="portType">Read_ptt</property>
</activationAgent>
</activationAgents>
</BPELProcess>
</BPELSuitcase>

Above, the batch notification events are sent to a BPEL process called "BatchMgmtProcess", deployed in the "default" domain on the same BPEL server. The wsdl of this BatchMgmtProcess must define a number of messageTypes and operations. This is what it should look like (never mind the correlation stuff for now):

<definitions
name="BatchManagerInterface"
targetNamespace="http://xmlns.oracle.com/pcbpel/batching"
xmlns="http://schemas.xmlsoap.org/wsdl/"
xmlns:bpws="http://schemas.xmlsoap.org/ws/2003/03/business-process/"
xmlns:tns="http://xmlns.oracle.com/pcbpel/batching"
xmlns:plnk="http://schemas.xmlsoap.org/ws/2003/05/partner-link/"
xmlns:pns1="http://xmlns.oracle.com/BatchMgmtProcess/correlationset"
xmlns:batch="http://xmlns.oracle.com/pcbpel/batching/types"
>
<import namespace="http://xmlns.oracle.com/BatchMgmtProcess/correlationset" location="BatchMgmtProcess_Properties.wsdl"/>
<types>
<schema attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://xmlns.oracle.com/pcbpel/batching/types"
xmlns:tns="http://xmlns.oracle.com/pcbpel/batching/types" xmlns="http://www.w3.org/2001/XMLSchema">
<element name="batchReadInitiateElement" type="tns:batchReadInitiateType"/>
<complexType name="batchReadInitiateType">
<sequence>
<element name="batchId" type="string"/>
<element name="batchMetaData" type="string"/>
<element name="batchDescription" type="string"/>
<element name="process" type="string"/>
<element name="domain" type="string"/>
</sequence>
</complexType>
<element name="batchReadCompleteElement" type="tns:batchReadCompleteType"/>
<complexType name="batchReadCompleteType">
<sequence>
<element name="batchId" type="string"/>
<element name="batchMetaData" type="string"/>
<element name="batchDescription" type="string"/>
<element name="batchExpectedSize" type="long"/>
<element name="process" type="string"/>
<element name="domain" type="string"/>
</sequence>
</complexType>
<element name="batchProcessCompleteElement" type="tns:batchProcessCompleteType"/>
<complexType name="batchProcessCompleteType">
<sequence>
<element name="batchId" type="string"/>
<element name="batchMetaData" type="string"/>
<element name="batchDescription" type="string"/>
<element name="batchFinalSize" type="long"/>
<element name="process" type="string"/>
<element name="domain" type="string"/>
</sequence>
</complexType>
<element name="batchReadFailureElement" type="tns:batchReadFailureType"/>
<complexType name="batchReadFailureType">
<sequence>
<element name="batchId" type="string"/>
<element name="batchMetaData" type="string"/>
<element name="batchDescription" type="string"/>
<element name="batchPartialSize" type="long"/>
<element name="process" type="string"/>
<element name="domain" type="string"/>
</sequence>
</complexType>
</schema>
</types>
<message name="batchReadInitiateMessage">
<part name="event" element="batch:batchReadInitiateElement"/>
</message>
<message name="batchReadCompleteMessage">
<part name="event" element="batch:batchReadCompleteElement"/>
</message>
<message name="batchProcessCompleteMessage">
<part name="event" element="batch:batchProcessCompleteElement"/>
</message>
<message name="batchReadFailureMessage">
<part name="event" element="batch:batchReadFailureElement"/>
</message>
<portType name="BatchManagerInterface">
<operation name="onBatchReadStart">
<input message="tns:batchReadInitiateMessage"/>
</operation>
<operation name="onBatchReadComplete">
<input message="tns:batchReadCompleteMessage"/>
</operation>
<operation name="onBatchProcessComplete">
<input message="tns:batchProcessCompleteMessage"/>
</operation>
<operation name="onBatchReadFailure">
<input message="tns:batchReadFailureMessage"/>
</operation>
</portType>
<plnk:partnerLinkType name="BatchManagerInterfacePartnerLinkType">
<plnk:role name="BatchManagerInterfaceRole">
<plnk:portType name="tns:BatchManagerInterface"/>
</plnk:role>
</plnk:partnerLinkType>
<bpws:propertyAlias propertyName="pns1:batchId" messageType="tns:batchReadCompleteMessage" part="event"
query="/batch:batchReadCompleteElement/batch:batchId"/>
<bpws:propertyAlias propertyName="pns1:batchId" messageType="tns:batchReadInitiateMessage" part="event"
query="/batch:batchReadInitiateElement/batch:batchId"/>
<bpws:propertyAlias propertyName="pns1:batchId" messageType="tns:batchReadFailureMessage" part="event"
query="/batch:batchReadFailureElement/batch:batchId"/>
</definitions>

The value of the batchId field is generated at runtime by the adapter, and can for example be used to initiate a correlationSet. You can then later on receive (e.g. using a Pick with correlation) a batchReadComplete or batchReadFailure message for the same file in the same monitoring instance. Here’s a fragment of what it could look like:

<correlationSets>

<correlationSet name="CorrelationSet_1" properties="ns3:batchId"/>

</correlationSets>

<sequence name="main">

<pick name="Pick_1" createInstance="yes">

<onMessage portType="ns1:BatchManagerInterface"

operation="onBatchReadStart"

variable="OnMessage_onBatchReadStart_InputVariable"

partnerLink="BatchManagerInterface">

<correlations>

<correlation initiate="yes" set="CorrelationSet_1"/>

</correlations>

Thursday, January 15, 2009

BPEL Deployment Framework

One of the coolest features in BPEL 10.1.3.4 is the deployment framework which makes the job of moving processes between dev, test and production environments much easier.  Modifying the bpel.xml file or altering descriptors through the console provide a degree of customisation for different environments, but it is all done on a single property at a time basis and requires a lot of work for each environment, especially when it is considered that individual WSDL files may also need to be updated. There is a better way in SOA Suite 10.1.3.4; the deployment framework.

The deployment framework combines the BPEL suitcase with a deployment plan that updates multiple files in the BPEL suitcase with the correct values for the deployment environment. Different deployment plans can be created and maintained for each deployment plan.

DepoymentPlan

It is possible to generate a template BPEL deployment plan from a BPEL suitcase which can then be customised and used with the base BPEL suitcase at deployment time to update the various URLs and properties.

The steps to customise the BPEL Suitcase for each environment are as follows.

  • Create a deployment template from the BPEL project or suitcase which will be used as the basis for the deployment plans.
  • Create a deployment plan based on the template for each target environment.
  • Attach the appropriate deployment plan to the BPEL suitcase or project prior to deploying in the target environment.
  • Deploy the BPEL process into the target environment.

image

Creating a Deployment Plan Template

To create a deployment template we need to add the following commands to the build.xml ant file that is built by JDeveloper and found in the Resources folder of our BPEL project.

<target name="generate_plan_from_project">
<generateplan planfile="${process.dir}/planfile.xml"
verbose="true"
overwrite="true"
descfile="${process.dir}/bpel/bpel.xml"/>
</target>
<target name="generate_plan_from_suitcase">
<generateplan planfile="${process.dir}/planfile.xml"
verbose="true"
overwrite="true"
suitecase="${process.dir}/output/bpel_${BPELSuitcase.BPELProcess(id)}_${rev}.jar"/>
</target>


This creates two new ant targets, generate_plan_from_project and generate_plan_from_suitcase. These both create a template deployment plan, either directly from the project files, or from a generated suitcase. A BPEL suitcase is only generated when a BPEL project is deployed or when the project is “made”, so if using the option to generate from a suitcase it is necessary to ensure that the suitcase has previously been created. If the suitcase generated is a revision other than 1.0 then it is necessary to set the revision property in the build.properties file that is found in the Resources folder of the BPEL project in JDeveloper.



# Change below if deploying with process revision other than 1.0
rev = 1.1


The generateplan command in ant uses four attributes




  • suitecase or descfile is the source information for the deployment plan.


  • planfile is the location of the planfile template to be generated.


  • overwrite will replace any existing planfile of the same name.


  • verbose turns up the level of reporting.




A sample deployment plan template is shown below. Note the use of two elements:




  • <replace> is used to replace the value of a property within a specific part of the bpel.xml


  • <searchReplace> is used to <search> for a string in WSDL and XSD files and <replace> it with another string.




<?xml version="1.0" encoding="UTF-8"?>

<BPELDeploymentPlan xmlns="http://schemas.oracle.com/bpel/deployplan">

<BPELProcess id="SimpleFileProcess">

<configurations/>

<partnerLinkBindings>

<partnerLinkBinding name="ReadNewCustomerFileService">

<property name="wsdlLocation">

<replace>ReadNewCustomerFileService.wsdl</replace>

</property>

</partnerLinkBinding>

<partnerLinkBinding name="WriteNewCustomerDBService">

<property name="retryInterval">

<replace>60</replace>

</property>

<property name="wsdlLocation">

<replace>WriteNewCustomerDBService.wsdl</replace>

</property>

</partnerLinkBinding>

<partnerLinkBinding name="CardValidatorService">

<property name="wsdlLocation">

<replace>CardValidatorService.wsdl</replace>

</property>

<property name="location">

<replace>http://w2k3/chapter18/CardValidatorServiceSoapHttpPort</replace>

</property>

</partnerLinkBinding>

</partnerLinkBindings>

</BPELProcess>

<wsdlAndSchema name="CardValidatorService.wsdl|DBAdapterOutboundHeader.wsdl|fileAdapterInboundHeader.wsdl|NewCustomerFile.xsd|ReadNewCustomerFileService.wsdl|WriteNewCustomerDBService.wsdl|WriteNewCustomerDBService_table.xsd">

<searchReplace>

<search/>

<replace/>

</searchReplace>

</wsdlAndSchema>

</BPELDeploymentPlan>



Creating a Deployment Plan



Having created a template we can use this to create deployment plans for each specific environment. We do this by creating a copy of the deployment plan by selecting Save As from the file menu in JDeveloper and then editing the <search> and <searchReplace> tags to match our target environment.



We will search and replace all instances of our local development machine hostname, w2k3, with the name of our test server, testserver, across wsdl and xsd files. To do this we modify the search and replace elements as shown below.



<wsdlAndSchema name="*">
<searchReplace>
<search>w2k3</search>
<replace>testserver</replace>
</searchReplace>
</wsdlAndSchema>


This will cause the BPEL process manager to search all WSDL and schema files “*” in the suitcase at deployment time and replace the string “w2k3” with the string “testserver”. Note that it is possible to have multiple <searchReplace> elements.



Attaching a Deployment Plan to a BPEL Suitcase



Having created and saved a deployment plan specific for one or more specific environments we will want to deploy our process into an environment. Before doing this we must first attach the specific deployment plan to the BPEL suitcase. We do this using the following ant command.



<target name="attach_plan">
<attachplan planfile="${planfile}" verbose="true"
overwrite="true"
suitecase="${process.dir}/output/bpel_${BPELSuitcase.BPELProcess(id)}_${rev}.jar"/>
</target>


This will create a file bpeldeployplan.xml in the BPEL suitcase. This is the deployment plan that will be used at deployment time by the BPEL process manager. Note that the name of the deployment plan to use is encoded as an Ant property planfile that must be set in the build.xml. Once attached the deployment plan will be executed when the BPEL suitcase is deployed.



Modifying Ant to Use Deployment Plan



In addition to adding the two tasks above to the build.xml file, it is possible to add the attachment of the plan file as part of the regular deploy process by modifying the dependencies of the process-deploy task by adding the attach_plan dependency after the compile dependency.



<target name="process-deploy"
depends="validateTask, compile, attach_plan, deployProcess,
deployTaskForm, deployDecisionServices" />


Now when building and deploying with ant the deployment plan will be attached to the suitcase before the suitcase is deployed to the target environment. This allows us to provide a different deployment plan for each environment, which can then be deployed from the command line in a reproducible fashion.



Summary



The deployment plans allow a lot of automation in the move from development to production, and because they can be used with editable files and command lines they are easy to include in the version control systems and automated build environments.  Congratulations to the BPEL developers on a useful piece of functionality that appeals to those responsible for ensuring consistency of environments rather than developers.

Oracle SOA Suite: High Availability Resources

Oracle has post on Oracle Technology Network as nice page that contains very practical documents (PDF) on how to configure Oracle SOA Suite or part of the SOA Suite in a high availability network (AKA HA or clustered environment. It covers a FAQ, RAC, WebLogic, OWSM. http://www.oracle.com/technology/tech/soa/ha/index.html

FEEDJIT Live Traffic Map

My Blog List