Improve image processing utils #451

msluszniak · 2025-07-17T07:22:49Z

Description

This PR make ImageProcessing module cleaner and more readable.

Needs to be tested - implement tests and run them

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update (improves or adds clarity to existing documentation)
Improving code style

Tested on

iOS
Android

Testing instructions

Screenshots

Related issues

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly - No changes needed
My changes generate no new warnings

Additional notes

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.h

msluszniak · 2025-08-25T18:03:09Z

I'll wait for writing tests for this one till merging #438

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp

packages/react-native-executorch/common/rnexecutorch/tests/TestRunner.cpp

packages/react-native-executorch/common/rnexecutorch/tests/LogTest.cpp

packages/react-native-executorch/common/rnexecutorch/tests/ImageProcessingTest.cpp

msluszniak · 2025-09-01T21:30:03Z

I had a problem with adding ImageProcessing.cpp to testing setup. I cannot force cmake to find executorch/extension/tensor/tensor.h I got the same error:

[ 13%] Built target gtest
[ 26%] Built target gtest_main
[ 33%] Building CXX object CMakeFiles/RNExecutorchTests.dir/ImageProcessingTest.cpp.o
In file included from /react-native-executorch/packages/react-native-executorch/common/rnexecutorch/tests/ImageProcessingTest.cpp:2:
/react-native-executorch/packages/react-native-executorch/common/rnexecutorch/tests/../data_processing/ImageProcessing.h:3:10: fatal error: 'executorch/extension/tensor/tensor.h' file not found
    3 | #include <executorch/extension/tensor/tensor.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/RNExecutorchTests.dir/ImageProcessingTest.cpp.o] Error 1
make[1]: *** [CMakeFiles/RNExecutorchTests.dir/all] Error 2
make: *** [all] Error 2

I left one dummy test for this file and commented lines that adds the file to testing via cmake. If someone wants to play with it and potentially fix the issue just uncomment these commented lines in CMakeLists.txt in tests and give it a try ;)

…DME.md

…essing/ImageProcessing.cpp

…essing/ImageProcessing.h

…essing/ImageProcessing.cpp

packages/react-native-executorch/common/rnexecutorch/data_processing/Numerical.cpp

packages/react-native-executorch/common/rnexecutorch/tests/ImageProcessingTest.cpp

packages/react-native-executorch/common/rnexecutorch/tests/LogTest.cpp

packages/react-native-executorch/common/rnexecutorch/tests/TestRunner.cpp

.gitmodules

mkopcins · 2025-09-17T09:06:01Z

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp

-  for (size_t i = 0; i < numChannels; ++i) {
-    backgroundScalar[i] = cvFloor(backgroundScalar[i]);
-  }
+  cv::Scalar backgroundScalar = computeBackgroundScalar(corners);


Suggested change

cv::Scalar backgroundScalar = computeBackgroundScalar(corners);

const cv::Scalar backgroundScalar = computeBackgroundScalar(corners);

mkopcins · 2025-09-17T10:01:51Z

packages/react-native-executorch/common/rnexecutorch/models/classification/Classification.cpp

+  auto imageAsMatrix = image_processing::readImageToMatrix(imageSource);
+  const auto tensorDims = getAllInputShapes()[0];
+  image_processing::adaptImageForTensor(tensorDims, imageAsMatrix);
  auto inputTensor =
-      image_processing::readImageToTensor(imageSource, getAllInputShapes()[0])
-          .first;
+      image_processing::getTensorFromMatrix(tensorDims, imageAsMatrix);


Not quite sure why we can't have a single function for this like we did before? Maybe restore readImageToTensor and place those four lines inside?

Because original original scope of function readImageToTensor is not getting tensor from matrix, but also resizing, converting to accurate format etc. My intention by splitting this was to get more self-explanatory code. The second thing is that original function returns ugly std::pair and one value is often unused. Having code splitted internally inside image_processing::readImageToTensor might be a way to go

mkopcins · 2025-09-17T10:12:03Z

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp

+  for (int i = 0; i < pixelCount; ++i) {
    int row = i / mat.cols;
    int col = i % mat.cols;


Suggested change

for (int i = 0; i < pixelCount; ++i) {

int row = i / mat.cols;

int col = i % mat.cols;

for (size_t i = 0; i < pixelCount; ++i) {

size_t row = i / mat.cols;

size_t col = i % mat.cols;

mkopcins · 2025-09-17T10:12:38Z

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp

+  for (int i = 0; i < pixelCount; ++i) {
+    const int row = i / matSize.width;
+    const int col = i % matSize.width;



Suggested change

for (int i = 0; i < pixelCount; ++i) {

const int row = i / matSize.width;

const int col = i % matSize.width;

for (size_t i = 0; i < pixelCount; ++i) {

const size_t row = i / matSize.width;

const size_t col = i % matSize.width;

Look that below I use cv::Mat::at(int x, int y) which will introduce implicit conversion from size_t to int:

I will think about changing it to size_t and static casting in cv::Mat::at to int

mkopcins · 2025-09-17T10:15:54Z

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp

+  constexpr int kMinCornerPatchSize = 1;
+  constexpr int kCornerPatchFractionSize = 30;
  int cornerPatchSize =


Suggested change

constexpr int kMinCornerPatchSize = 1;

constexpr int kCornerPatchFractionSize = 30;

int cornerPatchSize =

constexpr uint32_t kMinCornerPatchSize = 1;

constexpr uint32_t kCornerPatchFractionSize = 30;

uint32_t cornerPatchSize =

msluszniak · 2025-09-17T10:40:21Z

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp

+static cv::Mat handleBase64Data(const std::string &imageURI) {
+  std::stringstream uriStream(imageURI);
+  std::string stringData;
+  std::size_t segmentIndex{0};
+  while (std::getline(uriStream, stringData, ',')) {
+    ++segmentIndex;
+  }
+  if (segmentIndex != 1) {
+    throw std::runtime_error("Read image error: invalid base64 URI");
+  }
+  auto data = base64_decode(stringData);
+  cv::Mat encodedData(1, data.size(), CV_8UC1,
+                      static_cast<void *>(data.data()));
+  return cv::imdecode(encodedData, cv::IMREAD_COLOR);
+}
+
+static cv::Mat handleLocalFile(const std::string &imageURI) {
+  auto url = ada::parse(imageURI);
+  return cv::imread(std::string{url->get_pathname()}, cv::IMREAD_COLOR);
+}


TODO: handle these via anonymous namespaces, not static functions

msluszniak marked this pull request as draft July 17, 2025 07:22

msluszniak commented Jul 17, 2025

View reviewed changes

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp Outdated Show resolved Hide resolved

msluszniak commented Jul 17, 2025

View reviewed changes

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.h Outdated Show resolved Hide resolved

msluszniak commented Aug 25, 2025

View reviewed changes

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp Outdated Show resolved Hide resolved

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp Outdated Show resolved Hide resolved

msluszniak force-pushed the @ms/improve_image_processing_utils branch from 408c1ae to b380ba8 Compare August 30, 2025 16:28

msluszniak commented Sep 1, 2025

View reviewed changes

packages/react-native-executorch/common/rnexecutorch/data_processing/ImageProcessing.cpp Outdated Show resolved Hide resolved

msluszniak commented Sep 1, 2025

View reviewed changes

msluszniak marked this pull request as ready for review September 1, 2025 21:26

Mateusz Sluszniak and others added 17 commits September 2, 2025 09:39

Improve numerical utils

80d893f

Improve testing using CMakeLists.txt

055cff2

Add newlines

6d9d07a

Small fixes

4204076

Add more tests and clear implementation

5ba583d

Update packages/react-native-executorch/common/rnexecutorch/tests/REA…

243578a

…DME.md

Change z-normalization to norm normalization

8936ef0

Add corrections in ImageProcessing module

db9ae69

Update packages/react-native-executorch/common/rnexecutorch/data_proc…

f42a8c5

…essing/ImageProcessing.cpp

Update packages/react-native-executorch/common/rnexecutorch/data_proc…

13d3abe

…essing/ImageProcessing.h

Remove artifacts from rebase

ce8ffa1

More post-rebase clearance

41180aa

Add fixes to other files

4b98d1c

Remove redundant function and split function into two

d62b45c

Update packages/react-native-executorch/common/rnexecutorch/data_proc…

2fae648

…essing/ImageProcessing.cpp

Further refactor

b3c5834

Make minimal setup for tests

41b915d

msluszniak force-pushed the @ms/improve_image_processing_utils branch from 61c58dd to 41b915d Compare September 2, 2025 07:43

msluszniak commented Sep 2, 2025

View reviewed changes

Apply fixes after rebase

ff66e27

mkopcins changed the title ~~Improve image processing utils~~ chore: Improve image processing utils Sep 16, 2025

mkopcins requested changes Sep 17, 2025

View reviewed changes

msluszniak commented Sep 17, 2025

View reviewed changes

msluszniak self-assigned this Oct 6, 2025

msluszniak changed the title ~~chore: Improve image processing utils~~ Improve image processing utils Oct 6, 2025

msluszniak added chore PRs that are chores style improvement labels Oct 6, 2025

	cv::Scalar backgroundScalar = computeBackgroundScalar(corners);
	const cv::Scalar backgroundScalar = computeBackgroundScalar(corners);

Improve image processing utils #451

Are you sure you want to change the base?

Improve image processing utils #451

Uh oh!

Conversation

msluszniak commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

Uh oh!

Uh oh!

msluszniak commented Aug 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msluszniak commented Sep 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkopcins Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

mkopcins Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

msluszniak Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

mkopcins Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

mkopcins Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

msluszniak Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkopcins Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

msluszniak Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

msluszniak commented Jul 17, 2025 •

edited

Loading

msluszniak Sep 17, 2025 •

edited

Loading