Is the For task supposed to be always parallel or sequential?

[rojkov]

In the pre v1.x specification there was the ForEach state which could be marked either as parallel or as sequential (with parallel being the default). The current spec doesn't mention how the For loop iterations should run, in parallel or sequentially. Is it just an implementation detail left to an implementor to decide or sequential mode is implied?

[ricardozanini]

The tasks are supposed to run sequentially, not in parallel.

To run in parallel, you can do:

document:
  dsl: '1.0.2'
  namespace: test
  name: for-example
  version: '0.1.0'
do:
  - checkup:
      for:
        each: pet
        in: .pets
        at: index
      while: .vet != null
      do:
        - brachTask:
              fork: 
                branches:
                  - task1:
                  - task2:

Inside your for loop, you do a fork. Simply!

[rojkov]

Thanks! That's smart :)

[rojkov]

Sorry for bothering again, but I guess either I was too quick to accept or I am simply too slow at understanding.

Looks like the suggested solution doesn't serve the case when the number of parallel branches isn't fixed, but depends on the size of a collection (.pets in this case). So, here we do two parallel checkup activities for each pet sequentially. We run task1 and task2 in parallel, then synchronize, then take the next pet and iterate. But what about parallel checkups for each pet at same time?

Consider this typical "map-reduce" problem solved with parallel for loops:

1. split a huge text file in chunks where the number of chunks is an input parameter;
2. for each chunk fork a task calculate_histogram
3. when all the histograms are calculated, merge them into one to get the original text file's histogram.

[ricardozanini]

For this to work you would have to take another approach, and not the for loop. You'd have to spawn one task for each item in the collection. I'd personally would run all of those async and wait for them to process in a listen task.

A practical example would be to spawn one event per chunk, another workflow/task to listen to it (can be a subworkflow in the same context), process, and generate a reply event. Then in your original workflow you would listen to those callbacks, synchronize, and exit.

But I wouldn't do this in a bigdata scenario, obvioulsy. In this case, I don't think this is the right tool for the job. Unless, yoru eventing architecture is local to the processing. Let's say a local broker in the same machine/process instance that can spawn and listen to local events.

[rojkov]

Thanks! I've been thinking about the same approach, but hoped the syntax could be more familiar to the users with the HPC background. Yes, this necessitates event processing being local to the state of the loop, but in my case it's already there (even Fork is implemented asynchronously in my system, but this detail is hidden from end users).