MuleSoft Technical Guides
Scatter Gather in Mule 4
Scatter Gather Pattern helps MuleSoft developers distribute different tasks to achieve parallel processing of tasks/messages/events & then finally aggregate the responses as a single response.
- The Scatter-Gather component receives a Mule event and sends a reference of this Mule event to each processing route.
- Each of the processing routes starts executing in parallel then returns a Mule event. This is because the Scatter-Gather component executes each route in parallel, not sequentially.
- After all processing routes have finished execution, the Scatter-Gather component creates a new Mule event that combines all resulting Mule events from each route. It then passes the new Mule event to the next component in the flow.
Scatter Gather Example:
In Scatter-Gather flow, we have three routes: Flow Reference Component(Flow Reference routes the Mule event to another flow or sub-flow). One set Payload Transformer(The Set Payload component lets you update the payload of the message).
Both sub-flow contains a Database select query operation which gives us data from the Database. This transform message component converts the data from the Database into JSON format and a logger which will track the logs.
1) Route 1 contains a Flow reference of a sub-flow (db-flow1)
Configuration of select query operation –
Configuration XML:
<sub-flow name="db-flow1" doc:id="bf4acf02-bd0d-4156-9980-6aad5b1e908b" > <db:select doc:name="Select" doc:id="38eff10a-1141-4b16-aeb4-c3ed78eeae66" config-ref="Database_Config1"> <db:sql ><![CDATA[select * from product]]></db:sql> </db:select> <ee:transform doc:name="Transform Message" doc:id="d3590e43-a0c1-42e9-b6d5-562db1bec17d" > <ee:message > <ee:set-payload ><![CDATA[%dw 2.0 output application/json --- payload]]></ee:set-payload> </ee:message> </ee:transform> <logger level="INFO" doc:name="Logger" doc:id="342df3a5-b836-4a4c-a775-06bde82712d4" message="#[payload]"/> </sub-flow>
2) Route 2 contains a Flow reference of a sub flow (db-flow2)
Configuration of Select Query :
Configuration XML:
<sub-flow name="db-flow2" doc:id="f88987da-ea4c-44f5-bc39-0096d78eaa72" > <db:select doc:name="Select" doc:id="930efe51-8e73-40f2-9184-8174be3e9d01" config-ref="Database_Config1"> <db:sql ><![CDATA[select * from product1]]></db:sql> </db:select> <ee:transform doc:name="Transform Message" doc:id="ec45dbb2-92eb-43e2-836b-84029b0f4290" > <ee:message > <ee:set-payload ><![CDATA[%dw 2.0 output application/json --- payload]]></ee:set-payload> </ee:message> </ee:transform> <logger level="INFO" doc:name="Logger" doc:id="b100cf5e-809e-4d09-bfba-05f8114b748d" message="#[payload]"/> </sub-flow>
3) Route 3 contains a Set Payload component
Configuration of Set Payload:
Configuration XML:
<route > <set-payload value="scatter-gather flow payload" doc:name="Set Payload" doc:id="f8c0d364-044e-4e59-b31d-3fb4e8805466" /> <logger level="INFO" doc:name="Logger" doc:id="62563f04-a79c-4cef-9ab9-5eea9e17637a" message="#[payload]"/> </route>
Tables in Database:
- Product Table
2. Product1 Table
Flow XML:
<?xml version="1.0" encoding="UTF-8"?> <mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core" xmlns:db="http://www.mulesoft.org/schema/mule/db" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd http://www.mulesoft.org/schema/mule/db http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd"> <http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="dc87c934-3f63-4e54-a7a6-57b9852c6e2a" > <http:listener-connection host="0.0.0.0" port="8081" /> </http:listener-config> <db:config name="Database_Config1" doc:name="Database Config" doc:id="99af9304-0aa1-484d-986c-be9022aa3d21" > <db:my-sql-connection host="localhost" port="3306" user="root" password="Caelius@123" database="muletrainingdb" /> </db:config> <flow name="scatter-gather" doc:id="b7ea3fed-f402-4359-8a0d-c7a952903010" > <http:listener doc:name="Listener" doc:id="8e71879a-1711-4c3d-b4a1-1effa23cdec0" config-ref="HTTP_Listener_config" path="/test"/> <scatter-gather doc:name="Scatter-Gather" doc:id="f79ea679-6ffc-432c-a8e5-c56c9a6ff025" > <route > <flow-ref doc:name="db-flow1" doc:id="da378a2d-6584-4682-be11-746cb1b443a6" name="db-flow1"/> </route> <route > <flow-ref doc:name="db-flow2" doc:id="765de27e-c4ac-4c0b-bced-58f49f2bc460" name="db-flow2"/> </route> <route > <set-payload value="scatter-gather flow payload" doc:name="Set Payload" doc:id="f8c0d364-044e-4e59-b31d-3fb4e8805466" /> <logger level="INFO" doc:name="Logger" doc:id="62563f04-a79c-4cef-9ab9-5eea9e17637a" message="#[payload]"/> </route> </scatter-gather> <ee:transform doc:name="Transform Message" doc:id="df9303a4-c1dd-4eec-a731-a9358859ed18"> <ee:message> <ee:set-payload><![CDATA[%dw 2.0 output application/json --- payload]]></ee:set-payload> </ee:message> </ee:transform> </flow> <sub-flow name="db-flow1" doc:id="bf4acf02-bd0d-4156-9980-6aad5b1e908b" > <db:select doc:name="Select" doc:id="38eff10a-1141-4b16-aeb4-c3ed78eeae66" config-ref="Database_Config1"> <db:sql ><![CDATA[select * from product]]></db:sql> </db:select> <ee:transform doc:name="Transform Message" doc:id="d3590e43-a0c1-42e9-b6d5-562db1bec17d" > <ee:message > <ee:set-payload ><![CDATA[%dw 2.0 output application/json --- payload]]></ee:set-payload> </ee:message> </ee:transform> <logger level="INFO" doc:name="Logger" doc:id="342df3a5-b836-4a4c-a775-06bde82712d4" message="#[payload]"/> </sub-flow> <sub-flow name="db-flow2" doc:id="f88987da-ea4c-44f5-bc39-0096d78eaa72" > <db:select doc:name="Select" doc:id="930efe51-8e73-40f2-9184-8174be3e9d01" config-ref="Database_Config1"> <db:sql ><![CDATA[select * from product1]]></db:sql> </db:select> <ee:transform doc:name="Transform Message" doc:id="ec45dbb2-92eb-43e2-836b-84029b0f4290" > <ee:message > <ee:set-payload ><![CDATA[%dw 2.0 output application/json --- payload]]></ee:set-payload> </ee:message> </ee:transform> <logger level="INFO" doc:name="Logger" doc:id="b100cf5e-809e-4d09-bfba-05f8114b748d" message="#[payload]"/> </sub-flow> </mule>
Output Combining all routes:
{
"0": {
"inboundAttachmentNames": [],
"exceptionPayload": null,
"inboundPropertyNames": [],
"outboundAttachmentNames": [],
"payload": [
{
"original_price": 1000.0,
"product_id": 1,
"name": "Hp Pavilion laptop",
"offer_valid_until": "2016-06-27T10:45:56",
"description": "Hp Laptop ",
"brand_name": "HP",
"offer_price": 1000.0
},
{
"original_price": 3000.0,
"product_id": 2,
"name": "Macbook Pro laptop",
"offer_valid_until": "2016-06-27T10:45:57",
"description": "Apple Laptop ",
"brand_name": "Apple",
"offer_price": 3000.0
},
{
"original_price": 2000.0,
"product_id": 3,
"name": "Mac Book laptop",
"offer_valid_until": "2016-06-27T10:45:57",
"description": "Apple Laptop ",
"brand_name": "Apple",
"offer_price": 2000.0
},
{
"original_price": 4000.0,
"product_id": 4,
"name": "IBM laptop",
"offer_valid_until": "2016-06-27T10:45:57",
"description": "IBM Laptop ",
"brand_name": "IBM",
"offer_price": 4000.0
},
{
"original_price": 1000.0,
"product_id": 5,
"name": "MotoX Mobile",
"offer_valid_until": "2016-06-27T10:45:57",
"description": "Motorola Mobile ",
"brand_name": "Motorola",
"offer_price": 1000.0
},
{
"original_price": 5000.0,
"product_id": 6,
"name": "Samsung Note 5",
"offer_valid_until": "2016-06-27T10:45:57",
"description": "Samsung Mobile ",
"brand_name": "Samsung",
"offer_price": 5000.0
}
],
"outboundPropertyNames": [],
"attributes": null
},
"1": {
"inboundAttachmentNames": [],
"exceptionPayload": null,
"inboundPropertyNames": [],
"outboundAttachmentNames": [],
"payload": [
{
"Name": "hp",
"id": 1001
},
{
"Name": "asus",
"id": 1002
},
{
"Name": "dell",
"id": 1003
},
{
"Name": "apple",
"id": 1004
}
],
"outboundPropertyNames": [],
"attributes": null
},
"2": {
"inboundAttachmentNames": [],
"exceptionPayload": null,
"inboundPropertyNames": [],
"outboundAttachmentNames": [],
"payload": "scatter-gather flow payload",
"outboundPropertyNames": [],
"attributes": {
"headers": {
"user-agent": "PostmanRuntime/7.28.0",
"accept": "*/*",
"postman-token": "01f2bd2d-5eaa-4d9f-bd6a-164ab590be81",
"host": "localhost:8081",
"accept-encoding": "gzip, deflate, br",
"connection": "keep-alive"
},
"clientCertificate": null,
"method": "GET",
"scheme": "http",
"queryParams": {},
"requestUri": "/test",
"queryString": "",
"version": "HTTP/1.1",
"maskedRequestPath": null,
"listenerPath": "/test",
"localAddress": "/127.0.0.1:8081",
"relativePath": "/test",
"uriParams": {},
"rawRequestUri": "/test",
"rawRequestPath": "/test",
"remoteAddress": "/127.0.0.1:54523",
"requestPath": "/test"
}
}
}
All the MuleSoft Anypoint platform users can use this guide to implement scatter-gather pattern in their Mule projects.
Find more MuleSoft technical guides at Caelius Consulting Resource Centre.
Recent Blogs
NetSuite Integration using MuleSoft
NetSuite is a cloud-based enterprise resource planning (ERP) software suite that...…
Learn how MuleSoft & Salesforce helps businesses create incredible digital experiences. Speak with one of our consultants today
Let's Work Together