Fixing CRUD cascading operations to work on raw data rows not visible rows.

Also revising menu management to regenerate menus for tabs and their descendents based on update and delete events.

Author: ddooley

docs/README_developer.md

@@ -85,6 +85,74 @@ After each merge step, if the resulting entry lacks a `name` field, the slot/att
 
 ---
 
+## CRUD Operations (`AppContext.js`)
+
+DataHarmonizer's 1-to-many mode links multiple Handsontable tabs in a parent–child hierarchy driven by foreign-key annotations in the schema. `AppContext.js` implements all CRUD coordination across those tabs.
+
+### Relation map (`crudGetRelations`)
+
+At load time, `crudGetRelations(schema)` walks every class's `attributes` and looks for slots annotated with `foreign_key: OtherClass.slot_name`. For each such slot it builds a symmetric entry in `this.relations`:
+
+```
+relations[childClass].parent[parentClass][childSlot]  = parentSlot
+relations[childClass].dependent_slots[childSlot]      = { foreign_class, foreign_slot }
+relations[parentClass].child[childClass][parentSlot]  = childSlot
+relations[parentClass].target_slots[parentSlot][childClass] = childSlot
+```
+
+`relations[cls].dependents` is then populated by `crudGetDependents`, which walks the child graph breadth-first to collect every transitive descendant of a class.
+
+### Dependent-row state (`dependent_rows`)
+
+`this.dependent_rows` is a `Map` keyed by class name. Each entry records the current cascade context for that class:
+
+| Field | Meaning |
+|-------|---------|
+| `fkey_vals` | Foreign-key slot values used to filter this class's rows |
+| `fkey_status` | 0 = unsatisfied, 1 = partial, 2 = no FKs, 3 = fully satisfied |
+| `key_vals` | The class's own key slot values (used by children as their FK values) |
+| `key_changed_vals` | Which key slots are changing and to what new value |
+
+### Refreshing tab display (`refreshTabDisplay`)
+
+Called whenever the user selects a row in any tab. It calls `crudGetDependentRows` to recompute `dependent_rows` for the selected class and all its descendants, then:
+
+- Calls `setDHTabStatus` on each descendant — enabling or disabling the tab based on `fkey_status`.
+- Calls `tabFilter` on each descendant — hiding rows whose FK values do not match the cascade context.
+
+`tabFilter` uses Handsontable's HiddenRows plugin so that hidden rows remain in the source data and are not deleted.
+
+### Cascade update (`getChangeReport` + `beforeChange` hook)
+
+When a user edits a key slot in a tab's row, `DataHarmonizer.js`'s `beforeChange` hook fires. It calls `getChangeReport` which in turn calls `crudGetDependentRows` with the pending change. If any descendant rows would be affected, a confirmation dialog lists them. On confirmation, each affected row in each descendant tab has its FK slot updated via `setSourceDataAtCell` (physical row index, reaches hidden rows).
+
+#### How cascade FK values are built for dependents
+
+`crudGetDependentRows` iterates the `dependents` map of the changed class in dependency order. For each dependent class it derives `fkey_vals` **purely from the cascade chain already recorded in `dependent_rows`** — not from the user's current tab selection. This ensures the cascade scope is determined only by the record being changed:
+
+- If `start_name = 'Schema'` and Schema.name is being renamed, every descendant (Class, Slot, UniqueKey, Enum, etc.) is searched using only `{schema_id: oldName}`. A currently selected Class row does **not** narrow the search to that class's rows.
+- If `start_name = 'Class'` and Class.name is being renamed, descendants are searched using `{class_id: oldName, schema_id: <value from Schema's cascade entry>}`.
+
+Null FK values (parents not in the cascade chain) are excluded from the row search, so only the keys that actually flow from the changed record are applied as filters.
+
+`key_vals` stored for intermediate classes in `dependent_rows` contains only FK-chain values (no user-selection values), preventing user selections from leaking into downstream FK lookups.
+
+### Cascade delete (`getChangeReport` + right-click Delete)
+
+The same `getChangeReport` machinery is used. On confirmation, dependent rows are removed via `crudDeleteRowsByPhysical`, which filters the source data array and calls `hot.loadData()` — removing all matching rows (visible or hidden) without needing visual-row index conversion.
+
+### Row search (`crudFindRowByKeyVals` / `crudFindAllRowsByKeyVals`)
+
+Searches are done on physical (source) row indices using `getSourceDataAtCell`, so hidden rows are always included. Null values in the search filter are treated as "match any" by being excluded from the filter before the search.
+
+### Current scope and future directions
+
+The cascade system currently operates in **single-hierarchy** mode: a rename or delete on a given class propagates to all descendants of that class using only the changed record's key values as the filter. The user's selections in other tabs do not influence the cascade scope.
+
+A future option could support **multi-focus cascading**, where the user first narrows one or more intermediate tabs (e.g., selects a specific Class row) and the cascade is then scoped to that focused subset — for example, renaming only the UniqueKey rows that belong to the selected class, rather than all UniqueKey rows for the schema. This would require extending `crudGetDependentRows` to optionally incorporate user-selection values from non-start-class tabs into the `fkey_vals` chain.
+
+---
+
 ## `tabular_to_schema.py` — Legacy Schema Build Tool
 
 `script/tabular_to_schema.py` was the previous way of assembling a complete DataHarmonizer `schema.yaml` from spreadsheet-based source files. It combines three inputs:

lib/AppContext.js

@@ -1463,10 +1463,44 @@ export default class AppContext {
 
     // For each dependent
     this.relations[start_name]?.dependents?.keys().forEach((class_name) => {
-      // 1) Assemble the needed key_vals for this start class and its 
-      //    dependents (via foreign keys).
-      let [key_vals, fkey_vals, fkey_status] = 
-        this.crudGetDependentKeyVals(class_name);
+      // Build fkey_vals purely from the cascade chain already recorded in
+      // dependent_rows.  Do NOT call crudGetDependentKeyVals() here, because
+      // that function reads values from the user's currently selected rows in
+      // each tab — which would over-narrow the cascade search filter.
+      // E.g. if the user has "GRDISample" selected in the Class tab, calling
+      // crudGetDependentKeyVals for UniqueKey would inject class_id="GRDISample"
+      // into the filter, missing every other class's unique keys.
+      let fkey_vals = {};
+      let fkey_status;
+      let found_fkey = false;
+
+      if (this.relations[class_name].foreign_key_count === 0) {
+        fkey_status = 2; // no FKs — satisfied
+      } else {
+        fkey_status = 3; // assume satisfied until a null FK is found
+        for (let [slot_name, slot_link] of Object.entries(this.relations[class_name].dependent_slots)) {
+          let parent_dep = this.dependent_rows.get(slot_link.foreign_class);
+          let foreign_value = null;
+          if (parent_dep) {
+            foreign_value = parent_dep.key_vals[slot_link.foreign_slot] ?? null;
+            if (foreign_value === '') foreign_value = null;
+          }
+          fkey_vals[slot_name] = foreign_value;
+          if (foreign_value !== null) {
+            found_fkey = true;
+          } else {
+            fkey_status = 0;
+          }
+        }
+        if (fkey_status === 0 && found_fkey) fkey_status = 1; // partial
+      }
+
+      // key_vals for cascade: only FK-chain values, no user-selection values.
+      // When a downstream dependent reads this class's key_vals to get its own
+      // FK value, it will see null (not the user's selected row value), so it
+      // won't over-narrow its own search filter.
+      let key_vals = {...fkey_vals};
+
       let dependent_rows_obj = {
         fkey_vals: fkey_vals,
         fkey_status: fkey_status,
@@ -1509,10 +1543,15 @@ export default class AppContext {
       */
 
       if (fkey_status > 0) {
-        // Issue: special case: Field / slot table has slots with no class name,
-        // as well as class name = one in fkey_vals.  Allow empty key fields - 
-        // leave that to validation to catch.
-        let rows = this.crudFindAllRowsByKeyVals(class_name, fkey_vals);
+        // Exclude null FK values from the search: a null value means the
+        // corresponding parent has no selection, so treat it as "match any".
+        // This ensures multi-FK tables (Slot, SlotGroup, UniqueKey, EnumValue)
+        // are included in the cascade report even when only the top-level
+        // ancestor (e.g. Schema) is selected.
+        let search_vals = Object.fromEntries(
+          Object.entries(fkey_vals).filter(([k, v]) => v !== null)
+        );
+        let rows = this.crudFindAllRowsByKeyVals(class_name, search_vals);
         if (rows.length) {
 
           dependent_rows_obj['count'] = rows.length;   
@@ -1705,6 +1744,24 @@ export default class AppContext {
   }
   */
 
+  /**
+   * Delete rows from a DH tab by physical (source) row index.
+   * Copies the source data, removes the specified rows, and bulk-reloads the
+   * table via hot.loadData().  This approach works on all rows regardless of
+   * whether they are currently hidden by the tab filter.
+   * Note: loadData() resets hidden-row plugin state; re-apply tabFilter if needed.
+   *
+   * @param {String} class_name  DH tab name
+   * @param {Array}  physical_rows  Array of physical (source) row indices to delete
+   */
+  crudDeleteRowsByPhysical(class_name, physical_rows) {
+    const dh = this.dhs[class_name];
+    if (!dh || !physical_rows.length) return;
+    const rowSet = new Set(physical_rows);
+    const newData = dh.hot.getSourceData().filter((_, idx) => !rowSet.has(idx));
+    dh.hot.loadData(newData);
+  }
+
   /**
    * Search for key_vals combo in given dh handsontable.
    * @param {Object} dh DataHarmonizer instance
@@ -1734,11 +1791,12 @@ export default class AppContext {
   crudFindRowByKeyVals(dh, key_vals, row = 0) {
     const total_rows = dh.hot.countSourceRows();
     while (row < total_rows) {
-      if ( // .every() causes return on first break with row 
+      if ( // .every() causes return on first break with row
         Object.entries(key_vals).every(([slot_name, value]) => {
-          const col_value = dh.hot.getDataAtCell(row, dh.slot_name_to_column[slot_name]);
-          // Allow empty values by HandsonTable?  But adding "col_value === null" triggers infinite loop?
-          return value == col_value; //|| (slot_name === 'class_id' && col_value === '')
+          // Use getSourceDataAtCell (physical row index) to search all rows,
+          // including those hidden by the tab filter.
+          const col_value = dh.hot.getSourceDataAtCell(row, dh.slot_name_to_column[slot_name]);
+          return value == col_value;
         })
       ) {
         break;

lib/DataHarmonizer.js

@@ -278,10 +278,11 @@ class DataHarmonizer {
                   )
                 ) {
                   // User has seen the warning and has confirmed ok to proceed.
-                  for (let [dependent_name, dependent_obj] of Object.entries(change_report)) { 
+                  for (let [dependent_name, dependent_obj] of Object.entries(change_report)) {
                     if (dependent_obj.rows?.length) {
-                      let dh_changes = dependent_obj.rows.map(x => [x,1]);
-                      self.context.dhs[dependent_name].hot.alter('remove_row', dh_changes);    
+                      // rows are physical (source) indices. Delete via source-data reload
+                      // so hidden rows are also removed without needing toVisualRow().
+                      self.context.crudDeleteRowsByPhysical(dependent_name, dependent_obj.rows);
                     }
                   };
                   self.hot.alter('remove_row', selection); 
@@ -579,13 +580,14 @@ class DataHarmonizer {
                       let dependent_dh = self.context.dhs[dependent_name];
                       self.hot.suspendExecution(); // Advantageous to do this?
                       dependent_dh.hot.batchRender(() => {
-                        for (let dep_row in dependent_obj.rows) {
+                        for (let dep_row of dependent_obj.rows) {
                           Object.entries(dependent_obj.key_changed_vals).forEach(([dep_slot, dep_value]) => {
                             let dep_col = dependent_dh.slot_name_to_column[dep_slot];
-                            // While setDataAtCell triggers this beforeUpdate() again, 'batch_updates' event is trapped without further ado above.
-                            dependent_dh.hot.setDataAtCell(dep_row, dep_col, dep_value, 'batch_updates');
+                            // dep_row is a physical (source) row index from crudFindRowByKeyVals.
+                            // setSourceDataAtCell works on physical rows and reaches hidden rows.
+                            dependent_dh.hot.setSourceDataAtCell(dep_row, dep_col, dep_value);
                           })
-                        } 
+                        }
                       }); // End of hot batch render
                       self.hot.resumeExecution();
                     }