Mulesoft Batch Processing Made Easy: Unleashing the Power of DataWeave’s pluck Function

Martin Larizzate – Mulesoft Batch Processing Made Easy: Unleashing the Power of DataWeave’s pluck Function

In the world of data transformation, efficiency is key, especially when dealing with large arrays. Today, we’re diving into a nifty DataWeave function that allows you to group your data into manageable batches. Let’s take a closer look at this code snippet:

%dw 2.0
output application/json

var inputArray = [
  {
    "attr1": "value1",
    "discounts": [
      { "arratt": "value2" },
      { "arratt": "value3" }
    ]
  },
  {
    "attr1": "value3",
    "discounts": [
      { "arratt": "value4" },
      { "arratt": "value5" },
      { "arratt": "value6" }
    ]
  },
  {
    "attr1": "value5",
    "discounts": []
  },
  {
    "attr1": "valueX",
    "discounts": [
      { "arratt": "valueX" },
      { "arratt": "valueY" },
      { "arratt": "valueZ" }
    ]
  },
  {
    "attr1": "value7",
    "discounts": [
      { "arratt": "value8" },
      { "arratt": "value9" }
    ]
  },
  {
    "attr1": "value9",
    "discounts": [
      { "arratt": "value10" }
    ]
  },
  {
    "attr1": "value11",
    "discounts": [
      { "arratt": "value12" },
      { "arratt": "value13" },
      { "arratt": "value14" },
      { "arratt": "value15" },
      { "arratt": "value16" }
    ]
  },
  {
    "attr1": "value12",
    "discounts": [
      { "arratt": "value13" }
    ]
  },
  {
    "attr1": "value14",
    "discounts": []
  },
  {
    "attr1": "value16",
    "discounts": [
      { "arratt": "value17" },
      { "arratt": "value18" }
    ]
  }
]

// Function to group the array into batches of size batchSize
fun groupByBatchSize(array, batchSize) =
    array 
    map ((item, index) -> {
        "index": floor(index / batchSize),
        "item": item
    })
    groupBy ((item) -> item.index)
    pluck ((value, key) -> value.*item)

var itemsPerLot = 2
---

 groupByBatchSize(inputArray, itemsPerLot)

 

Understanding the Magic Behind the Code

This function groupByBatchSize takes two parameters: an array and a batch size. It groups the input array into smaller batches based on the specified size. Here’s a breakdown of how it works:

1. Mapping with Index: The code first maps over the array, pairing each item with its index. This index is calculated in a way that it essentially divides the index by the batch size, grouping items accordingly.
2. Grouping: The mapped items are then grouped by the computed index, allowing you to create batches.
3. Using pluck: Finally, the pluck function comes into play. This function is like a powerful extractor, taking the values from the grouped data and returning a simpler array structure. In this case, it helps us retrieve just the items from each batch, discarding the index information we no longer need.

Why Use pluck?

The beauty of the pluck function lies in its ability to simplify complex data structures. It allows you to focus on the data you care about—here, the items in each batch—without worrying about the overhead of the metadata (like indices) that might clutter your output.

In a nutshell, this DataWeave snippet demonstrates how easy it can be to manage data in batches, thanks to the straightforward yet powerful capabilities of DataWeave. So, the next time you find yourself handling large arrays, remember this function and how pluck can make your life easier!

_- Martin Larizzate -_ Java and Mulesoft Certified Developer, Salesforce Ranger and Salesforce Solution Architect