Scatter-Gather Router

Mulesoft

The Scatter-Gather component is a routing event processor that processes a Mule event through different parallel processing routes that contain different event processors. 

Each route receives a reference to the Mule event and executes a sequence of one or more event processors. Each of these routes uses a separate thread to execute the event processors, and the resulting Mule event can be either the same Mule event without modifications or a new Mule event with its own payload, attributes, and variables. The Scatter-Gather component then combines the Mule events returned by each processing route into a new Mule event that is passed to the next event processor only after every route completes successfully.
The routing message processor Scatter-Gather sends a request message to multiple targets concurrently. It collects the responses from all routes, and aggregates them into a single message.

Scatter Gather in MuleSoft

  • The ScatterGather component executes each route in parallel.
  • Each route receives a reference to the Mule event and executes a sequence of one or more event processors.
  • Each of these routes uses a separate thread to execute the event processors, and the resulting Mule event can be either the same Mule event without modifications or a new Mule event with its own payload, attributes, and variables.The Scatter-Gather component then combines the Mule events returned by each processing route into a new Mule event that is passed to the next event processor only after every route completes successfully
  • Scatter-Gather component to have at least two routes; otherwise, your Mule application throws an exception and does not start.
  • Use Try scope in each route of a ScatterGather component to handle any errors that might be generated by a route’s event processors

The following diagram details the behavior of the Scatter-Gather component:

scatter gather in mule

Scatter Gather Component:

  1. Scatter-Gather router receives a mule event and sends a reference/copy of mule event to each route simultaneously.
    • In our case, Scatter-Gather sends mule events to route-1, route-2 and route-3 parallel.
  2. All the routes start processing the mule event in parallel. After all routes finish the processing then returns a mule event.
    • The event can be a same mule event or a new event created by the processors in the route.
  3. After all the routes finish their processing, the Scatter-Gather router combines the all resulting Mule events from each route and create a new mule event then passes the mule event to the next processor.

Example

     Drag and drop HttpListener and Configure

        Global configuration

http-listener-config

  General configuration

http general config

Step 2: Add a Set Variable and set a variable and name it.set variable

Scatter-Gather router config:

http requester config

Route-1:

  • Add HTTP Requester node and refer to the global HTTP Requester configuration and then configure the General parameters.
get user1 details
  • Add Set Variable and override the variable the we set before the Scatter-Gather router.
set variable scatter gather

Route-2:

  • Add HTTP Requester node and refer to the global HTTP Requester configuration and then config the General parameters.
get user2 details
  • Add Set Variable and override the variable the we set before the Scatter-Gather router.
set user2 variable

The Scatter-Gather router should look like:

scatter gather router

Step 4: Add Transform Message and simply transform the object to JSON model.

object to json

Step 5: At last add the logger and log the variable “name“.

log variables

That’s it. Now run the mule application and trigger a request to http://localhost:8089/scatterGather.

Scatter-Gather returns the payload as an Object. The sample output would look like:

{

    “0”: {

        “inboundAttachmentNames”: [],

        “exceptionPayload”: null,

        “inboundPropertyNames”: [],

        “outboundAttachmentNames”: [],

        “payload”: {

            “userId”: 1,

            “id”: 1,

            “title”: “delectus aut autem”,

            “completed”: false

        },

        “outboundPropertyNames”: [],

        “attributes”: {

            “headers”: {

                “date”: “Sun, 21 Feb 2021 05:42:24 GMT”,

                “content-type”: “application/json; charset=utf-8”,

                “content-length”: “83”,

                “connection”: “keep-alive”

            },

            “reasonPhrase”: “OK”,

            “statusCode”: 200

        }

    },

    “1”: {

        “inboundAttachmentNames”: [],

        “exceptionPayload”: null,

        “inboundPropertyNames”: [],

        “outboundAttachmentNames”: [],

        “payload”: {

            “userId”: 1,

            “id”: 2,

            “title”: “quis ut nam facilis et officia qui”,

            “completed”: false

        },

        “outboundPropertyNames”: [],

        “attributes”: {

            “headers”: {

                “date”: “Sun, 21 Feb 2021 05:42:24 GMT”,

                “content-type”: “application/json; charset=utf-8”,

                “content-length”: “99”

            },

            “reasonPhrase”: “OK”,

            “statusCode”: 200

        }

    }

}

Note: We have removed few headers to keep the response simple.

The above response contains the attributes and the payload but most of the times the attributes are not required.

In such cases, to extract the only payload you can use flatten function to merge the both the responses.

To do that, replace the above dataweave script as shown below:

%dw 2.0

Output application/json

Flatten (payload..payload)

The response would look like after the above transformation:

flatten payload response

Point to Rememeber:

  • Scatter-Gather returns the mule event as an Object.
  • Use flatten function to merge the both responses.

Variable Propagation in Scatter-Gather

Now let’s see what happens to variables when we override the variables inside Scatter-Gather router.

Every route starts with the same initial variable values. Modifications to a variable within a specific route do not affect other routes. So, if a variable is added or modified in one route, then, after aggregation, the value is defined by that route. If a variable is added or modified by more than one route, the value is added to a list of all the values defined for that variable within all the routes.

So, if a variable is added or modified in one route, then, after aggregation, the value is defined by that route.

variable propagation

Concurrency Control

By default the Scatter-Gather process the mule event in parallel but sometimes we may need to process the mule event sequentially.

To achieve the sequential process, we can use two methods:

  1. Using maxConcurrency element
  2. Using Try scope

Follow Me

If you like my post please follow me to read my latest post on programming and technology.

Instagram

Facebook

Leave a Reply

Your email address will not be published. Required fields are marked *