Localapi feature support

README.md

@@ -4,7 +4,43 @@ Home Assistant SunPower Integration using the local installer ethernet interface
 
 Original Integration is [https://github.com/krbaker/hass-sunpower](https://github.com/krbaker/hass-sunpower)
 
-* If this is a fork, please add what's different here and fix up the badges below
+## 🆕 What's Different in This Fork
+
+This fork adds automatic LocalAPI support for newer PVS firmware (build >= 61840) while maintaining full backwards compatibility with legacy CGI endpoints.
+
+### Key Enhancements
+- Automatic API Detection: Queries firmware version and automatically selects the appropriate API
+- LocalAPI for Newer Firmware: Uses the more efficient Varserver FCGI endpoints with authentication
+- Direct LAN Access: LocalAPI works via the standard LAN IP - no proxy or special network setup required
+- Legacy CGI Fallback: Maintains support for older firmware using traditional CGI endpoints
+- Improved Performance: Caching mechanism reduces API calls and improves response times
+- Automatic Credential Management: Serial suffix auto-fetched from PVS, no manual configuration needed
+
+### How It Works
+1. During setup, the integration queries `/cgi-bin/dl_cgi/supervisor/info` to check firmware version
+2. If firmware build >= 61840: Uses LocalAPI with session-based authentication
+3. If older firmware: Uses legacy CGI endpoints (same as before)
+4. Serial suffix (last 5 characters of PVS serial) is auto-fetched when possible
+5. Field name mapping: LocalAPI returns camelCase field names (e.g., `pMppt1Kw`), which are automatically converted to snake_case (e.g., `p_mppt1_kw`) to match legacy CGI format, ensuring identical data structures for backwards compatibility
+6. No configuration changes needed - it just works!
+
+### Benefits
+- Faster response times with caching (caches variable paths, not values - data is always fresh)
+- More reliable session management
+- Better error handling and retry logic
+- Reduced load on PVS system
+- Cache resets on Home Assistant restart to ensure optimal performance
+
+### Recent Bug Fixes
+- **Fixed LocalAPI reliability issues**: Simplified /vars endpoint calls by removing cache parameter. While LocalAPI supports caching, it adds complexity and the performance benefit is negligible at 120-second polling intervals. This change improves reliability and eliminates potential cache state issues.
+- **Fixed KeyError crashes**: Added defensive checks to prevent crashes when device data is temporarily unavailable
+- **Improved error logging**: Added detailed diagnostics to help troubleshoot connection issues
+- **Better resilience**: Integration now gracefully handles missing PVS, inverter, or meter data
+
+### LocalAPI Documentation
+For technical details on the LocalAPI implementation, see:
+- [LocalAPI Documentation](https://github.com/SunStrong-Management/pypvs/blob/main/doc/LocalAPI.md) - Comprehensive guide to Varserver FCGI endpoints
+- [pypvs Project](https://github.com/SunStrong-Management/pypvs) - Python library for PVS LocalAPI access
 
 [![GitHub Release][releases-shield]][releases]
 [![GitHub Activity][commits-shield]][commits]
@@ -80,19 +116,23 @@ removed (This addition thanks to [@CanisUrsa](https://github.com/CanisUrsa))
 
 ## Options (available from 'configure' once integration is setup)
 
+**To change polling intervals:** Settings → Devices & Services → SunPower → Configure
+
 ### Solar data update interval (seconds)
 
+**Default: 120 seconds** | **Minimum: 60 seconds**
+
 This sets how fast the integration will try to get updated solar info from the PVS.
-The lowest "safe" rate looks like about 120 seconds.  I am concerned some PVSs may fail
-to work properly over time and I'm guessing it might be request or error logging filling
-their memory.  I am running with 300 seconds right now as I went through a heck of a time
-with a PVS that began to fail pushing to Sunpower's cloud.
+
+**For LocalAPI (firmware >= 61840):** The official documentation recommends polling "once every few seconds" to avoid overloading the PVS CPU. The default 120 seconds is conservative and safe for long-term reliability.
+
+**For Legacy CGI (older firmware):** The PVS takes a very long time to return data. The lowest "safe" rate is about 120 seconds. Some PVSs may fail to work properly over time with aggressive polling, possibly due to request or error logging filling their memory.
 
 ### Energy storage update interval (seconds)
 
-Should evenly divide into Solar data update interval or be an even multiple of it (this is due to the
-currently silly way polling is handled through one timer).  The original author of the ESS addon
-[@CanisUrsa](https://github.com/CanisUrsa) had it as low as 20 seconds (see warning above)
+**Default: 60 seconds** | **Minimum: 20 seconds**
+
+Should evenly divide into Solar data update interval or be an even multiple of it (this is due to the currently silly way polling is handled through one timer). The original author of the ESS addon [@CanisUrsa](https://github.com/CanisUrsa) had it as low as 20 seconds (see warning above)
 
 ## Network Setup
 
@@ -260,6 +300,29 @@ Power Output.
 If you file a bug one of the most useful things to include is the output of
 > curl <http://172.27.153.1/cgi-bin/dl_cgi?Command=DeviceList>
 
+### LocalAPI Authentication Issues
+
+If you have newer firmware (build >= 61840) and see authentication errors:
+
+1. Check firmware version: 
+   ```bash
+   curl http://172.27.153.1/cgi-bin/dl_cgi/supervisor/info
+   ```
+   Look for the `BUILD` number in the response.
+
+2. Verify serial suffix: The integration automatically fetches the serial suffix (last 5 characters of PVS serial) from the PVS during initialization. If auto-fetch fails, the integration will raise an error. The serial suffix is used as the password for LocalAPI authentication.
+
+3. Check logs: Look in Home Assistant logs for connection errors or authentication failures during integration setup.
+
+### API Type Detection
+
+The integration automatically detects and uses the appropriate API based on firmware version:
+- Firmware build >= 61840: Uses LocalAPI automatically
+- Older firmware: Uses legacy CGI endpoints automatically
+- No configuration needed - the integration handles this transparently
+
+Note on Firmware Upgrades: API detection occurs when Home Assistant starts or when the integration reloads. If SunPower remotely upgrades your PVS firmware while Home Assistant is running, the integration will continue using the current API until you restart Home Assistant. After restart, it will automatically detect the new firmware and switch to LocalAPI if supported.
+
 ### Missing solar production. Appears that the Sunpower meter has disappeared from the device list
 
 Run the debugging command and look for the METER entries.

custom_components/sunpower/__init__.py

@@ -73,7 +73,13 @@ def create_vmeter(data):
     freq_avg = sum(freq) / len(freq) if len(freq) > 0 else None
     volts_avg = sum(volts) / len(volts) if len(volts) > 0 else None
 
-    pvs_serial = next(iter(data[PVS_DEVICE_TYPE]))  # only one PVS
+    # Check if PVS device exists before trying to access it
+    pvs_devices = data.get(PVS_DEVICE_TYPE)
+    if not pvs_devices:
+        _LOGGER.warning("PVS device not found in data, skipping virtual meter creation")
+        return data
+
+    pvs_serial = next(iter(pvs_devices))  # only one PVS
     vmeter_serial = f"{pvs_serial}pv"
     data.setdefault(METER_DEVICE_TYPE, {})[vmeter_serial] = {
         "SERIAL": vmeter_serial,
@@ -98,8 +104,33 @@ def create_vmeter(data):
 def convert_sunpower_data(sunpower_data):
     """Convert PVS data into indexable format data[device_type][serial]"""
     data = {}
+
+    # Log total device count
+    total_devices = len(sunpower_data.get("devices", []))
+    _LOGGER.info(f"Processing {total_devices} devices from API")
+
     for device in sunpower_data["devices"]:
-        data.setdefault(device["DEVICE_TYPE"], {})[device["SERIAL"]] = device
+        device_type = device.get("DEVICE_TYPE", "UNKNOWN")
+        serial = device.get("SERIAL", "UNKNOWN")
+        data.setdefault(device_type, {})[serial] = device
+
+    # Log device types found for debugging
+    device_types = list(data.keys())
+    device_counts = {dt: len(data[dt]) for dt in device_types}
+    _LOGGER.info(f"Device types found: {device_counts}")
+
+    # Check for expected device types
+    if not data.get(PVS_DEVICE_TYPE):
+        _LOGGER.error(f"CRITICAL: PVS device type '{PVS_DEVICE_TYPE}' not found!")
+        _LOGGER.error(f"Available device types: {device_types}")
+        # Log ALL devices to see what we're getting
+        for i, dev in enumerate(sunpower_data["devices"][:10]):  # First 10 devices
+            _LOGGER.error(f"Device {i+1}: TYPE='{dev.get('DEVICE_TYPE')}', SERIAL={dev.get('SERIAL')}, MODEL={dev.get('MODEL')}")
+
+    if not data.get(INVERTER_DEVICE_TYPE):
+        _LOGGER.warning(f"No inverter devices found - this is normal for some PVS configurations")
+        _LOGGER.info(f"Available device types: {device_types}")
+        _LOGGER.info("Inverter data may be aggregated in meter readings or unavailable via LocalAPI")
 
     create_vmeter(data)
 
@@ -117,24 +148,40 @@ def convert_ess_data(ess_data, data):
     sunvault_power_inputs = []
     sunvault_power_outputs = []
     sunvault_state = "working"
+
+    # Ensure device types exist in data
+    if BATTERY_DEVICE_TYPE not in data:
+        _LOGGER.warning("BATTERY_DEVICE_TYPE not found in data, skipping battery ESS data conversion")
+        data[BATTERY_DEVICE_TYPE] = {}
+    if ESS_DEVICE_TYPE not in data:
+        _LOGGER.warning("ESS_DEVICE_TYPE not found in data, skipping ESS data conversion")
+        data[ESS_DEVICE_TYPE] = {}
+    if HUBPLUS_DEVICE_TYPE not in data:
+        _LOGGER.warning("HUBPLUS_DEVICE_TYPE not found in data, skipping HubPlus data conversion")
+        data[HUBPLUS_DEVICE_TYPE] = {}
+
     for device in ess_data["ess_report"]["battery_status"]:
-        data[BATTERY_DEVICE_TYPE][device["serial_number"]]["battery_amperage"] = device[
+        serial = device["serial_number"]
+        if serial not in data[BATTERY_DEVICE_TYPE]:
+            _LOGGER.warning(f"Battery {serial} not found in PVS data, skipping")
+            continue
+        data[BATTERY_DEVICE_TYPE][serial]["battery_amperage"] = device[
             "battery_amperage"
         ]["value"]
-        data[BATTERY_DEVICE_TYPE][device["serial_number"]]["battery_voltage"] = device[
+        data[BATTERY_DEVICE_TYPE][serial]["battery_voltage"] = device[
             "battery_voltage"
         ]["value"]
-        data[BATTERY_DEVICE_TYPE][device["serial_number"]]["customer_state_of_charge"] = device[
+        data[BATTERY_DEVICE_TYPE][serial]["customer_state_of_charge"] = device[
             "customer_state_of_charge"
         ]["value"]
-        data[BATTERY_DEVICE_TYPE][device["serial_number"]]["system_state_of_charge"] = device[
+        data[BATTERY_DEVICE_TYPE][serial]["system_state_of_charge"] = device[
             "system_state_of_charge"
         ]["value"]
-        data[BATTERY_DEVICE_TYPE][device["serial_number"]]["temperature"] = device["temperature"][
+        data[BATTERY_DEVICE_TYPE][serial]["temperature"] = device["temperature"][
             "value"
         ]
-        if data[BATTERY_DEVICE_TYPE][device["serial_number"]]["STATE"] != "working":
-            sunvault_state = data[BATTERY_DEVICE_TYPE][device["serial_number"]]["STATE"]
+        if data[BATTERY_DEVICE_TYPE][serial]["STATE"] != "working":
+            sunvault_state = data[BATTERY_DEVICE_TYPE][serial]["STATE"]
         sunvault_amperages.append(device["battery_amperage"]["value"])
         sunvault_voltages.append(device["battery_voltage"]["value"])
         sunvault_temperatures.append(device["temperature"]["value"])
@@ -155,75 +202,89 @@ def convert_ess_data(ess_data, data):
             sunvault_power_inputs.append(0)
             sunvault_power_outputs.append(0)
     for device in ess_data["ess_report"]["ess_status"]:
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["enclosure_humidity"] = device[
+        serial = device["serial_number"]
+        if serial not in data[ESS_DEVICE_TYPE]:
+            _LOGGER.warning(f"ESS {serial} not found in PVS data, skipping")
+            continue
+        data[ESS_DEVICE_TYPE][serial]["enclosure_humidity"] = device[
             "enclosure_humidity"
         ]["value"]
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["enclosure_temperature"] = device[
+        data[ESS_DEVICE_TYPE][serial]["enclosure_temperature"] = device[
             "enclosure_temperature"
         ]["value"]
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["agg_power"] = device["ess_meter_reading"][
+        data[ESS_DEVICE_TYPE][serial]["agg_power"] = device["ess_meter_reading"][
             "agg_power"
         ]["value"]
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["meter_a_current"] = device[
+        data[ESS_DEVICE_TYPE][serial]["meter_a_current"] = device[
             "ess_meter_reading"
         ]["meter_a"]["reading"]["current"]["value"]
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["meter_a_power"] = device[
+        data[ESS_DEVICE_TYPE][serial]["meter_a_power"] = device[
             "ess_meter_reading"
         ]["meter_a"]["reading"]["power"]["value"]
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["meter_a_voltage"] = device[
+        data[ESS_DEVICE_TYPE][serial]["meter_a_voltage"] = device[
             "ess_meter_reading"
         ]["meter_a"]["reading"]["voltage"]["value"]
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["meter_b_current"] = device[
+        data[ESS_DEVICE_TYPE][serial]["meter_b_current"] = device[
             "ess_meter_reading"
         ]["meter_b"]["reading"]["current"]["value"]
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["meter_b_power"] = device[
+        data[ESS_DEVICE_TYPE][serial]["meter_b_power"] = device[
             "ess_meter_reading"
         ]["meter_b"]["reading"]["power"]["value"]
-        data[ESS_DEVICE_TYPE][device["serial_number"]]["meter_b_voltage"] = device[
+        data[ESS_DEVICE_TYPE][serial]["meter_b_voltage"] = device[
             "ess_meter_reading"
         ]["meter_b"]["reading"]["voltage"]["value"]
     if True:
         device = ess_data["ess_report"]["hub_plus_status"]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["contactor_position"] = device[
-            "contactor_position"
-        ]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["grid_frequency_state"] = device[
-            "grid_frequency_state"
-        ]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["grid_phase1_voltage"] = device[
-            "grid_phase1_voltage"
-        ]["value"]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["grid_phase2_voltage"] = device[
-            "grid_phase2_voltage"
-        ]["value"]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["grid_voltage_state"] = device[
-            "grid_voltage_state"
-        ]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["hub_humidity"] = device[
-            "hub_humidity"
-        ]["value"]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["hub_temperature"] = device[
-            "hub_temperature"
-        ]["value"]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["inverter_connection_voltage"] = device[
-            "inverter_connection_voltage"
-        ]["value"]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["load_frequency_state"] = device[
-            "load_frequency_state"
-        ]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["load_phase1_voltage"] = device[
-            "load_phase1_voltage"
-        ]["value"]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["load_phase2_voltage"] = device[
-            "load_phase2_voltage"
-        ]["value"]
-        data[HUBPLUS_DEVICE_TYPE][device["serial_number"]]["main_voltage"] = device[
-            "main_voltage"
-        ]["value"]
+        serial = device["serial_number"]
+        if serial not in data[HUBPLUS_DEVICE_TYPE]:
+            _LOGGER.warning(f"HubPlus {serial} not found in PVS data, skipping")
+        else:
+            data[HUBPLUS_DEVICE_TYPE][serial]["contactor_position"] = device[
+                "contactor_position"
+            ]
+            data[HUBPLUS_DEVICE_TYPE][serial]["grid_frequency_state"] = device[
+                "grid_frequency_state"
+            ]
+            data[HUBPLUS_DEVICE_TYPE][serial]["grid_phase1_voltage"] = device[
+                "grid_phase1_voltage"
+            ]["value"]
+            data[HUBPLUS_DEVICE_TYPE][serial]["grid_phase2_voltage"] = device[
+                "grid_phase2_voltage"
+            ]["value"]
+            data[HUBPLUS_DEVICE_TYPE][serial]["grid_voltage_state"] = device[
+                "grid_voltage_state"
+            ]
+            data[HUBPLUS_DEVICE_TYPE][serial]["hub_humidity"] = device[
+                "hub_humidity"
+            ]["value"]
+            data[HUBPLUS_DEVICE_TYPE][serial]["hub_temperature"] = device[
+                "hub_temperature"
+            ]["value"]
+            data[HUBPLUS_DEVICE_TYPE][serial]["inverter_connection_voltage"] = device[
+                "inverter_connection_voltage"
+            ]["value"]
+            data[HUBPLUS_DEVICE_TYPE][serial]["load_frequency_state"] = device[
+                "load_frequency_state"
+            ]
+            data[HUBPLUS_DEVICE_TYPE][serial]["load_phase1_voltage"] = device[
+                "load_phase1_voltage"
+            ]["value"]
+            data[HUBPLUS_DEVICE_TYPE][serial]["load_phase2_voltage"] = device[
+                "load_phase2_voltage"
+            ]["value"]
+            data[HUBPLUS_DEVICE_TYPE][serial]["main_voltage"] = device[
+                "main_voltage"
+            ]["value"]
     if True:
         # Generate a usable serial number for this virtual device, use PVS serial as base
         # since we must be talking through one and it has a serial
-        pvs_serial = next(iter(data[PVS_DEVICE_TYPE]))  # only one PVS
+        # Check if PVS device exists before trying to access it
+        pvs_devices = data.get(PVS_DEVICE_TYPE)
+        if not pvs_devices:
+            _LOGGER.warning("PVS device not found in data, skipping SunVault virtual device creation")
+            return data
+
+        pvs_serial = next(iter(pvs_devices))  # only one PVS
         sunvault_serial = f"sunvault_{pvs_serial}"
         data[SUNVAULT_DEVICE_TYPE] = {sunvault_serial: {}}
         data[SUNVAULT_DEVICE_TYPE][sunvault_serial]["sunvault_amperage"] = sum(
@@ -283,6 +344,10 @@ def sunpower_fetch(
     except (ParseException, ConnectionException) as error:
         raise UpdateFailed from error
 
+    if not sunpower_data or "devices" not in sunpower_data:
+        _LOGGER.error("Invalid PVS data structure: %s", sunpower_data)
+        raise UpdateFailed("PVS returned invalid data structure - missing 'devices' key")
+
     data = convert_sunpower_data(sunpower_data)
     if ESS_DEVICE_TYPE in data:  # Look for an ESS in PVS data
         use_ess = True
@@ -331,7 +396,12 @@ async def async_setup_entry(hass: HomeAssistant, entry: ConfigEntry):
     entry_id = entry.entry_id
 
     hass.data[DOMAIN].setdefault(entry_id, {})
-    sunpower_monitor = SunPowerMonitor(entry.data[SUNPOWER_HOST])
+    # Create monitor in executor since __init__ makes blocking calls
+    sunpower_monitor = await hass.async_add_executor_job(
+        SunPowerMonitor,
+        entry.data[SUNPOWER_HOST],
+        None,  # Auto-fetch from PVS
+    )
     sunpower_update_invertal = entry.options.get(
         SUNPOWER_UPDATE_INTERVAL,
         DEFAULT_SUNPOWER_UPDATE_INTERVAL,