Thoughts on new DAG Grid

[tagoad]

Just updated to 2.3 and was surprised by the replacement of the tree view with the grid view, and i wanted to get some conversation going about pros and cons of it. While i appreciate the cleanness of it, I do have some issues with it:

• Task Relationships: This is my major issue with it, there isn't anything to show the relationship between tasks. Maybe some grouping based on task groups could be added? Alternates colors similar to banded table rows to show difference in task groups?
• The run information is right aligned which leaves a ton of whitespace between the task names and their run info. Literally have my task names far left and then the info far right, not great for ease of reading.

I do really like the details panel, and I think that opens up a lot of possibilities for at a glance info for individual task you select.

[tagoad]

Looking at the new docs i do see that there is some functionality with Task Groups and the grid view so i am going to looking into that.

[bbovenzi]

Thanks for the feedback. Tree -> Grid was a big change, but I think it opens up interesting possibilities.

• Task Groups: You can expand/collapse them just like in the Graph view:
• There is also a "crosshair" highlight when you hover over a box to see what task and run it belongs to. But maybe we can make that more clear.
• Task relationships: Beyond groups and the order tasks are displayed, we don't show too much of the relationship. This is something I want to improve on. One idea is to add the Graph into the details panel.

[ashb]

@argibbs Could you share a screenshot of how the Tree view looked for your dag for comparison?

Cos one of the problems with the tree view showing dependencies was that it very quickly collapsed into a) a mess (with the same task in multiple places) or b) an incredibly deep tree

[ashb]

A preview in the details pane of the relationships for a specific task would be really nice. Overall I do like it.

We have thought about bringing some (all?) of the Graph view into the new grid view -- we will explore ways of doing this so when you click on a Task/TaskInstance of showing something useful. (Right now I can't think of an easy way of showing it without a click without the drawbacks of the tree view)

[argibbs]

@argibbs Could you share a screenshot of how the Tree view looked for your dag for comparison?

Absolutely. Here you go (very slightly redacted, sorry). This is the same dag as before, only without the red for the latest dag run, as that was caused by #23411 and I've subsequently rolled back to get it to go green.

[argibbs]

I totally agree with @ashb - the old tree rapidly descends into a mish-mash of duplicated nodes. We have some dags that wear the scroll wheel out they're that long. However, it does provide useful info if you are used to it.

My personal opinion on what the grid "needs" is:

1. One-click fully expanded (ideally can make this by default).
2. Some indication of the node relationships. For me, I mostly care because I want to know if I clear the node what's going to be impacted. I think it would be enough to "glow" all immediate children, but I'd need to use it to be sure I don't need all descendants of any depth, sorry.

---

Edit: thinking on it some more, yes, what I want to see is the immediate children. My use case is usually (as you can see from the failed dag runs above) "quickly find the failed nodes, assess how bad the rot is, clear the fails and retry". I do make heavy use of the future/past/upstream/failed options when clearing. Tree view does not scale and turns into a mélange of node, but what it does almost always do is 'given a node, show you the immediate children' which helps immensely with the above task. Of course, you could do this by simply showing them in the info pane on the right, rather than trying to do highlights.

[potiuk]

Edit: thinking on it some more, yes, what I want to see is the immediate children.

That was also my intuition - that users need to see the "immediate" children and possibly "immediate parents" of task they are focusing on at any given moment, rather than full graph.

And taking into account that in the future we will have versioning likely, I'd say this relation should be property of particular task execution not the whole task. I.e. - if you click (or hover over) particular execution or particular tasks, then you should. see (and be able to quickly navigate to) immediate parents and children in tree-like view for this particular task execution.

Now. The difficul part is how to do it fast and intuitive :)

[KevinYang21]

Bringing the discussion in this PR here:

The thing missing for me is the task depedency (I can collect more use cases from our internal users). Not the topology for the entire DAG, but a couple layers of the upstream/downstream for a couple of tasks. For those upstream/downstream tasks, what I do mostly are check the task status, check task log and clear task.

One example of my operation is:

• want to check the progress of my DAG
• go into tree view and see if the root task I care the most finished or not
• found it has not started and start to panic
• quickly eyeball the next a couple of rows (internally we view the depedency from downstream->upstream) to figure out which upstream tasks are still running
• found the upstream task stuck in running state because we ran out of EMR resources
• rush to our EMR team and scream for more resources

Edit 0:
Adding a quick use case from an internal client: they own a big DAG with a couple thoudsands of tasks, they don't use task group and thus use root=task_id to view only the task and it's upstreams. Maybe at least partial view with a root task can be done in grid view?

[argibbs]

found it has not started and start to panic

LMAO. It me.

[ashb]

Adding a quick use case from an internal client: they own a big DAG with a couple thoudsands of tasks, they don't use task group and thus use root=task_id to view only the task and it's upstreams. Maybe at least partial view with a root task can be done in grid view?

@KevinYang21 The Grid view respects the ?root= parameter as before, and there's a "Filter Upstream" button.

[KevinYang21]

@ashb that's good to know, thanks!

[bbovenzi]

From all the discussion here are a few ideas I have:

1. Visually distinguish group summaries from their actual children tasks. I'm thinking something like below: (A previous version of the grid view used a background color but it got too noisy visually) PR: Visually distinguish task group summaries #23488

2. We have a "Filter upstream" to see just the upstream of a task. Maybe we should have a filter downstream too. Also, we can include a mini-graph in the details panel to show immediate parents and children.

3. If a DAG has task groups, add an expand all/collapse all button. PR: Expand/collapse all groups #23487

4. We're adding more filters soon in Grid view status filters #23392. We can add filters for task instances that will also expand task groups to show that original failing task.

5. Once we update all the task action endpoints Move Grid webserver endpoints to REST API #23412. Instead of just a text dump of what task instances are affected, we visually highlight the affected tasks in the Grid view.

[argibbs]

Is the absolute size of the mini graph fixed? We have some biiiig graphs, if they shoved the task actions pane off the bottom I would cry.

[tagoad]

This WIP is just showing the whole dag but centering on the selected task.

The grid view reordering: that's probably a bug. Let me look into it.

Do you want me to create an issue for it?

[bbovenzi]

Is the absolute size of the mini graph fixed? We have some biiiig graphs, if they shoved the task actions pane off the bottom I would cry.

I want to keep the task actions in view. Right now, I'm exploring it with a fixed height and no zoom (but maybe a minimap). And adding a button to center on the task in question.

[bbovenzi]

@tagoad yes, feel free to make an issue!

[potiuk]

Suggestion: I am not sure if it's easy/difficult to do, but for huge graphs there should be IMHO an option to just show direct upstream/downstream deps of selected task. in the case visible above you cannot directly see the upstream/downstream tasks and you loose a lot of the visualisation 'strength' this way.

[potiuk]

1. LGTM. Maybe another option is some icon indicating task group? But this is not strong. All the rest - cool.

[bbovenzi]

The up/down chevron is already that icon, no?

[potiuk]

right. the bottm border + bold is cool :)