PoC: Modify single IV to accept nans #1296
Conversation
|
This PR is stale because it has been open for 60 days with no activity. |
|
thanks for adding this @AshwinParanjape (and for your patience--I somehow missed this PR). Can you add a test case for this based on the referenced issue? Have also added an inline comment on informing the user about missing data. |
| y0_z = np.mean(data[self._target_estimand.outcome_variable[0]][instrument == 0]) | ||
| x1_z = np.mean(data[self._target_estimand.treatment_variable[0]][instrument == 1]) | ||
| x0_z = np.mean(data[self._target_estimand.treatment_variable[0]][instrument == 0]) | ||
| y1_z = np.nanmean(data[self._target_estimand.outcome_variable[0]][instrument == 1]) |
There was a problem hiding this comment.
I understand the motivation, but sometimes the user may not be aware of the missing data (especially when it may be due to a bug in their data processing code).
For this reason, can we issue a warning message as this nanmean operation is done?
the code structure can be: try np.mean operations, if it fails, then we issue a warning that "The dataset contains missing data, proceeding to ignore it", and then using nanmean as a fallback.
for warning, you can use, self.logger.warning("message")
I modified the binary and continuous single IV case to see how it be made to work with partial data.
For binary IV, np.mean -> np.nanmean is sufficient
For continuous valued IV, I'm using np.ma (masked array) to mask the NaNs from covariance computations