Skip to content

Conversation

@avolkov-intel
Copy link

Changes

Added implementation of profiler to collect and compare activations value of openVino model to investigate cause of accuracy degradation.

Reason for changes

Related tickets

162317

Tests

Copy link
Contributor

@ljaljushkin ljaljushkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution, Anatoly!
Please consider adding visualization functionality and enhancing the scalability

>>> profiler = NNCFProfiler(pattern=r'attn', dataset=ds, num_samples=100)
>>> acts_fp32 = profiler.collect_activations(model_fp32)
>>> acts_int8 = profiler.collect_activations(model_int8)
>>> comparison = profiler.compare_activations(acts_fp32, acts_int8)
Copy link
Contributor

@ljaljushkin ljaljushkin Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, it would be great to have visualization functionality as in the original scripts:
https://github.com/andreyanufr/compression_validation/blob/aanuf/activation_cmp/utils/cmp_activations/compare_model_outputs.py

Secondly, most likely there will be a need to add other comparisons and plots and keep it in one profiler class could be not extensible.
Could you think of registering comparators and visualizations inside profiler and delegate this responsibility to them?

For instance,

@comparator("mean_diff")
def mean_diff(a, b): ...
 
@visualizer("hist")
def hist_plot(...): ...

def compare():
    ...
    for metric in metrics:
        if metric not in COMPARATORS:
            raise ValueError(f"Unknown comparator '{metric}'. "
                             f"Available: {list(COMPARATORS)}")
        row[metric] = COMPARATORS[metric](a, b)
    
def plot(self, plot_type, *args, **kwargs):
        if plot_type not in VISUALIZERS:
            raise ValueError(f"Unknown plot type '{plot_type}'. "
                             f"Available: {list(VISUALIZERS)}")
        return VISUALIZERS[plot_type](*args, **kwargs)
        
cmp = profiler.compare(acts_fp, acts_int, metrics=["mean_diff", "relative_mean_diff"])

profiler.plot("hist", acts_fp, acts_int, layer="layer_7")
profiler.plot("mean_std", acts_fp)
profiler.plot("metric", cmp, metric="relative_diff")
# or alternatively have a different methods for plotting
profiler.plot_metric(cmp, metric="relative_mean_diff")
profiler.plot_hist(acts_fp, acts_int, layer="layer_10.attn")
profiler.plot_mean_std(acts_fp, act_type="in")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants