feat(#6669): barcode scanner #8684

latin-panda · 2023-11-06T02:17:49Z

Description

Nb. This will be delivered in 4 different PRs:

Search by barcode (this PR)
Enketo widget to read barcode
Autoselect if search by barcode returns 1 result
Automation test coverage

This work uses the native Barcode Detector API from the browser by using a hidden input type file to natively open the camera in Android devices. This avoids custom code in the CHT Android and allows more accessibility since the feature can be used in CHT Android, PWA, and Android browsers.

See Figma design

It works the following way:

The user clicks on the barcode scanner icon
Prompt is displayed by the OS to take a picture or pic from the gallery
User takes a picture
CHT loads the image and then uses the Barcode Detector API to read the barcode.
CHT enters that code into the search input field and triggers the search.

More considerations:

It works in Android Chrome, PWA, and CHT Android app.
This feature is enabled by using can_use_barcode_scanner permission
This feature isn't supported on desktops and in browsers older than Chrome/Wedview v90.
When clicking on the barcode icon to scan, the OS will open an option to take a picture or pick from the gallery. This is fully managed by the OS, and we can't limit these options (like preventing from selecting from the gallery)
The user needs to grant permission to use the camera, sometimes it's already granted, but they can remove it. We can't force it, and we can't know if they have removed the camera access since we are using the native browser APIs.
Some browsers might not support all the barcode types. Supported barcodes are printed in the console from the browser's developer tools.
The DB Admin user doesn't have access to this feature. Otherwise, they will always have access to this.

Telemetry:

search_by_barcode:open -> When a user clicks on the barcode scanner icon
search_by_barcode:not_supported -> When the feature isn't supported because the device is a desktop or the Barcode Detector API isn't available in the browser.
search_by_barcode:scan -> When the user has taken the picture.
search_by_barcode:trigger_search -> When Barcode Detector gets the code from the image and triggers the search.
search_by_barcode:barcode_no_detected -> When Barcode Detector couldn't detect the code from the image.
search_by_barcode:failure -> Any system failure when the Barcode Detector tries to get the code from the image.

Snackbar messages:

On error when the feature is enabled and the picture is loaded but cannot read the barcode: Failed to read the barcode. Retry

Videos

Happy path: https://github.com/medic/cht-core/assets/66472237/17bfa6a9-f1ae-4632-b213-83d80a329899
Errors: When a picture is bad quality (distorted) and it can’t detect the barcode: see snackbar message: https://github.com/medic/cht-core/assets/66472237/8354f530-b82f-4bff-9f53-4ac276dc8d27
Error: Barcode Detector API no supported / Desktop machine (see snack bar and absence of icon to scan): https://github.com/medic/cht-core/assets/66472237/e9c3f442-7070-4c4f-ae6a-2e94afd25bfc

#6669

Code review checklist

Readable: Concise, well named, follows the style guide, documented if necessary.
Documented: Configuration and user documentation on cht-docs
Tested: Unit and/or e2e where appropriate
Internationalised: All user facing text
Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.

Compose URLs

If Build CI hasn't passed, these may 404:

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

api/resources/translations/messages-en.properties

latin-panda · 2023-11-08T15:00:52Z

Hi @m5r, @lorerod, @Benmuiruri

This is ready for an initial review. Can you please have a look and provide feedback? 🙂

I've added videos and details in this PR description.
Please ignore the translation files for now. We're still deciding on the text to display. I'll let you know once that's ready.
I have a PR that should fix this CI, I'm just waiting for the review, and then I can update this one

Most questions and feedback are related to a place in the code or file, I prefer those as comments in the file instead of the "PR timeline" because in the "PR timeline" I can't reply just under the comment. So the file flows better because of the treads.😬

Thanks!

lorerod

Thank you, @latin-panda, for the unit test coverage!
I left some questions and minor suggestions. None of them are blockers, though.

config/covid-19/app_settings.json

webapp/src/ts/components/search-bar/search-bar.component.html

webapp/tests/karma/ts/components/search-bar/search-bar.component.spec.ts

garethbowen · 2023-11-08T22:02:01Z

@latin-panda It looks like the barcode scanning API is not supported in Firefox which is a browser we support. We might need to consider polyfilling with a library?

…into 6669-search-by-barcode

latin-panda · 2023-11-09T03:00:30Z

It looks like the barcode scanning API is not supported in Firefox which is a browser we support. We might need to consider polyfilling with a library?

@garethbowen I think we're fine for now in this Feature Release.

We are turning off the feature on desktop machines.
We are also turning off the feature If it isn't supported in the browser. The user is notified with the snackbar and the error is logged.
The adoption of Firefox in mobile devices is about 0.5% of the global market.
If the partner faces this issue, the options are to use PWA, CHT Android (Webview) or to install Chrome.

garethbowen · 2023-11-09T03:17:10Z

PWA includes Firefox.

It's fine if you want to roll with this for the Feature Release but it can't be merged to master until it supports Firefox.

api/resources/translations/messages-fr.properties

latin-panda · 2023-11-13T11:25:25Z

@m5r just wondering if you have some time for a quick review of this draft :)

m5r

That's pretty advanced for a draft PR 😁

Well done Jenni! I've left a couple change requests

api/resources/translations/messages-fr.properties

webapp/src/ts/components/search-bar/search-bar.component.ts

webapp/src/css/inbox.less

webapp/src/ts/components/search-bar/search-bar.component.html

m5r

All good! 🙌

Benmuiruri

Looks good and working as expected!

lorerod · 2023-11-15T16:37:47Z

webapp/tests/karma/ts/components/search-bar/search-bar.component.spec.ts

+      expect(telemetryService.record.calledWith('search_by_barcode:failure')).to.be.true;
+    }));
+
+    it('should record telemetry when barcode is clicked.', fakeAsync(async () => {


Nice! Thanks!

lorerod · 2023-11-16T19:36:15Z

Hi @latin-panda I tested this today. I will continue tomorrow. If you want something specific for me to try, let me know.

Tested 6669-search-by-barcode branch on my local machine using cht-couchdb.yml and cht-core.yml

Phones:

Sony Xperia
Android WebView 95.0.4638.74
Version de Android 5.1.1
Cht android v1.1
Samsung Galaxy J7
Android WebView 118.0.5993.112
Version de Android 8.1
Cht android v1.1
Samsung Galaxy A01 Core
Version de Android 10
Android WebView 115.0.5790.166
Version de Android 10
Cht android v1.0.1

Desktop:
macOs Sonoma 14
Chrome 119.0.6045.159

Admin users using desktop

Login as admin
Enabled can_use_barcode_scanner for every role
Login as offline user
Go to people
Barcode scanner icon doesn't show on search bar

Admin users using mobile

Login as admin on phone
Go to people
Barcode scanner icon doesn't show on search bar

Offline users using desktop

Login as offline user on desktop
Go to people
Barcode scanner icon doesn't show on search bar

Offline users using phones

Login on the three phones using three different offline users
Go to people
Barcode scanner icon was visible on the People tab search bar

Images:

Using admin user on desktop, disabled can_use_barcode_scanner for chw
On phones, after sync, the barcode scanner disappeared from the People tab search bar.

Image:

Using admin user on desktop, enabled can_use_barcode_scanner for chw
On phones the barcode scanner appeared again on the People tab search bar
Tap the barcode scanner icon, and the menu to choose Camera or Files is open.

Image:

Tried to scan a foggy barcode: the three phones showed the error Failed to read the barcode. Retry

Image:

Tried to scan an incomplete barcode (cutting the beginning of the barcode): 2 phones showed the error Failed to read the barcode. Retry, 1 phone (the Samsung Galaxy A01 Core) interpret a code that do not correspond ⚠️
Tried again to scan an incomplete barcode (cutting the beginning and end): The 3 phones showed the error Failed to read the barcode. Retry
Disabled wifi on three phones to start testing offline.
Tested with a barcode that gives a lot of results

Image:

Tested with a barcode that gives no results

Image:

Tested with a barcode that gives one result

Image:

Tested on each phone with different kinds of barcodes and QR codes generated with this page. (Code-128, aztec, code-39, QR code). It worked with each phone for black and white codes.
Tested on each phone with background green and code blue and it failed with each phone. Also with background colored and black code. Showed the error Failed to read the barcode. Retry ⚠️

Image:

Tested taking the picture with the phones in horizontal. It worked with every phone.
With admin user un desktop changed the config can_view_old_filter_and_search enabled for chw.
Sync on phones and old filter and search shows, and barcode scanner icon disappeared.

Image:

22 With admin change config can_view_old_filter_and_search back to disabled for chw.
23. Sync on phones the barcode scanner appeared again on the People tab search bar
24. Tested using files instead of camera worked on three phones if file was an image of a barcode.
25. Failed on three phones if the file was an image of a QR code with quiet zone 0 ⚠️

Image:

Worked if the file was an image of a QR code with quiet zone 10

Image:

latin-panda · 2023-11-17T08:29:47Z

Thanks a lot, @lorerod. All these scenarios are great! You even covered some I didn't think about, like green/blue codes and quiet zones (I had to google the meaning 😅). I think you covered all I can think of so far.

I tested using the following:

Chrome v100 and v119, PWA Chrome, Webview 119, and CHT Android v1.1.
Nokia Android 9, Samsung Android 10, Samsung Android 12, and Samsung Android 13 - That's what I have at home.
It's working as expected for Desktop, mobile, offline, and online users.
Michael did some simple tests in Tecno Android 7 - and successfully scanned the partner's printed barcode (code_128).

I struggled to scan a white barcode with a transparent background because of the lack of contrast. It's probably the same issue with green/blue barcodes. I think it's not a blocker for a Feature Release; the partner's barcode is black and white, type code_128; we can observe if someone uses something with low contrast (green/blue, white/transparent)

The cases with quiet zone 0 or incomplete barcodes/QR codes are probably because the tool's algorithm doesn't know with certainty the start and end of the barcode 🤔

lorerod · 2023-11-17T17:52:34Z

I agree that this is working well. I will work on adding an e2e test to cover the happy path scenario. Thanks Jenny

search by barcode

725e80a

latin-panda changed the base branch from master to 4.4.1-FR-barcode November 6, 2023 02:21

latin-panda added 8 commits November 6, 2023 16:35

translations and telemetry

5129937

coverage

1c513fd

feedback

4638ccb

more unit tests

e1af2a0

trigger e2e

8658c6b

unit

0245303

unit

f4b9b1c

prettier code and more telemetry

50bb163

latin-panda commented Nov 8, 2023

View reviewed changes

api/resources/translations/messages-en.properties Outdated Show resolved Hide resolved

Improving checks

bc82414

latin-panda requested review from m5r, lorerod and Benmuiruri November 8, 2023 14:51

lorerod approved these changes Nov 8, 2023

View reviewed changes

Merge branch '4.4.1-FR-barcode' of https://github.com/medic/cht-core …

61dea99

…into 6669-search-by-barcode

latin-panda added 4 commits November 9, 2023 13:27

Feedback: translations and format

b28ea19

Feedback: unit tests

05aa0d9

Feedback: new unit test

5dad80f

sonar

773b30e

latin-panda commented Nov 9, 2023

View reviewed changes

api/resources/translations/messages-fr.properties Outdated Show resolved Hide resolved

sonar

c78882b

latin-panda mentioned this pull request Nov 10, 2023

Barcode scanning using device camera #6669

Open

latin-panda added 2 commits November 10, 2023 17:49

Hiding from reports, fixing race condition in input type file

f976ed5

alignments

51cbbf7

m5r suggested changes Nov 13, 2023

View reviewed changes

feedback: fr translation, better html and css

0b7f30b

latin-panda requested review from m5r and lorerod November 14, 2023 15:21

latin-panda marked this pull request as ready for review November 14, 2023 15:21

m5r approved these changes Nov 14, 2023

View reviewed changes

Tiny improve for click target

f65a9f3

Benmuiruri approved these changes Nov 15, 2023

View reviewed changes

lorerod approved these changes Nov 15, 2023

View reviewed changes

latin-panda merged commit 59a1dbd into 4.4.1-FR-barcode Nov 16, 2023

latin-panda deleted the 6669-search-by-barcode branch November 16, 2023 03:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(#6669): barcode scanner #8684

feat(#6669): barcode scanner #8684

latin-panda commented Nov 6, 2023 •

edited

Loading

latin-panda commented Nov 8, 2023

lorerod left a comment

garethbowen commented Nov 8, 2023 •

edited

Loading

latin-panda commented Nov 9, 2023

garethbowen commented Nov 9, 2023

latin-panda commented Nov 13, 2023

m5r left a comment

m5r left a comment

Benmuiruri left a comment

lorerod Nov 15, 2023

lorerod commented Nov 16, 2023 •

edited

Loading

latin-panda commented Nov 17, 2023

lorerod commented Nov 17, 2023

feat(#6669): barcode scanner #8684

feat(#6669): barcode scanner #8684

Conversation

latin-panda commented Nov 6, 2023 • edited Loading

Description

Code review checklist

Compose URLs

License

latin-panda commented Nov 8, 2023

lorerod left a comment

Choose a reason for hiding this comment

garethbowen commented Nov 8, 2023 • edited Loading

latin-panda commented Nov 9, 2023

garethbowen commented Nov 9, 2023

latin-panda commented Nov 13, 2023

m5r left a comment

Choose a reason for hiding this comment

m5r left a comment

Choose a reason for hiding this comment

Benmuiruri left a comment

Choose a reason for hiding this comment

lorerod Nov 15, 2023

Choose a reason for hiding this comment

lorerod commented Nov 16, 2023 • edited Loading

latin-panda commented Nov 17, 2023

lorerod commented Nov 17, 2023

latin-panda commented Nov 6, 2023 •

edited

Loading

garethbowen commented Nov 8, 2023 •

edited

Loading

lorerod commented Nov 16, 2023 •

edited

Loading