Skip to content

Platform Acceptance Test for Windows

Jump to bottom
rc-swag edited this page Nov 16, 2023 · 20 revisions

Warning

These acceptance tests are now deprecated as the source for formal testing. Formal testing is done against regression tests stored in a test suite managed by the test team. Developers can update acceptance tests here on the wiki, however, if the developer recommends a test needs to be added to the formal tests contact the test team.

Keyman for Windows Acceptance Test Procedures

  1. These test procedures are to be run before moving from alpha to beta, or beta to stable, or before PRs are merged into stable branches.

  2. Results are annotated with results, use > at the start of a new line under the checkbox/Test to note the result.

User Testing

Gather Assets for Testing

  • Windows 10 (x64) - VM or physical okay
  • Windows 11 (x64) - VM or physical okay
  • Keyboards for Caps Lock test sets - The test keyboard layouts are found in the keyman repo at app/windows/src/test/manual-tests/caps-lock-stores.
  • Download the 64-bit version of the basic Text Editor this is built as part of the Test Artifacts found in the PR.

Download_Text_Editor

Setup Steps

  1. Uninstall previous versions of Keyman Desktop and Keyman Developer on the system
  2. Restart system, install updates
  3. Make sure all additional Windows languages are removed
  4. Make sure no debug flags are set in HKCU/Software/Keyman/Debug
  5. Place the Basic Text editor executable in a location on the test machine. Run it where mentioned throughout the acceptance test procedure. e.g the Downloads folder, Documents folder.

SUITE_BASELINE: Keyman for Windows Base Line Tests

  • GROUP_WIN10
  • GROUP_WIN11

Test cases

click to expand

  • TEST_INSTALL:

    • Install test build keyman-a.b.c.exe
    • Make sure Keyman is started and visible in the Notification area

  • TEST_INSTALLING_OFFLINE

  1. Open a command prompt and change to the directory where the installation package is
  2. Type: keyman-a.b.c.exe -o
  3. Observe the installation begin, opening a window for installation with the correct version number;
  4. Continue and ensure installation is successful

  • TEST_KEYBOARD_INSTALLATION_REMOTE:

    • Start Keyman Configuration
    • Click Download Keyboard, select a keyboard from the download dialog and install it
    • If the package includes a readme file, verify that it appears in the install window.
    • If the package includes a welcome file, verify that it appears after installation.
    • Click OK to close Keyman Configuration
    • Start the Text Editor
    • Verify that the newly installed keyboard appears in the Keyman menu and can be selected

  • TEST_INSTALL_PKG_DISK:

    • Install a package from disk ##
    • Locate a .kmp file on disk and double-click to install it
    • If the package includes a readme file, verify that it appears in the install window.
    • If the package includes a welcome file, verify that it appears after installation.
    • Start Text Editor
    • Verify that the keyboard appears in the Keyman menu

  • TEST_KEYBOARD_OUTPUT:

    • Start Text Editor and select a Keyman keyboard (testing Legacy integration)
    • Verify that various test sequences are output correctly
    • Start Word and select a Keyman keyboard (testing TSF integration)
    • Verify that various test sequences are output correctly

  • TEST_ON_SCREEN_KEYBOARD:

    • Click Keyman icon in Notification area, select On Screen Keyboard
    • Verify that OSK displays correctly for selected Keyman keyboard
    • Verify that typing characters by clicking on OSK produces expected output

SUITE_CAPSLOCK:

  • GROUP_WIN10
  • GROUP_WIN11

Caps Lock

The test keyboard layouts are found in the keyman repo at app/windows/src/test/manual-tests/caps-lock-stores. There is a project file for the 3 keyboards used in this test. The project file can be used to build the keyboard packages, but you can conveniently use the .kmp file zipped and included respectively below.

The test cases below expect the usage of the capslock.kmp.zip keyboard. That keyboard outputs pass or fail if following the test cases.

Prerequisites before each test

  • System keyboard layout is en-US
  • Install a keyboard that doesn't use any of the caps lock stores, e.g. capslock.kmp.
  • CapsLock is currently on
  • Currently active keyboard is the capslock.kmp keyboard

Test cases

click to expand

  • TEST_CAPSLOCK-1: uppercase with virtual key

    • press and release a

    Expected result:

    • pass. (with other keyboards uppercase A)

  • TEST_CAPSLOCK-2: lowercase with virtual key

    • press and hold 'Shift'
    • press and release b
    • release Shift

    Expected result:

    • pass. (with other keyboards lowercase b)

  • TEST_CAPSLOCK-3: capslock ignored for numbers

    • press and hold 'Shift'
    • press and release 3
    • release Shift

    Expected result:

    • pass. (with other keyboards #)

  • TEST_CAPSLOCK-4: uppercase

    • press and release c

    Expected result:

    • pass. (with other keyboards uppercase C)

  • TEST_CAPSLOCK-5: lowercase

    • press and hold 'Shift'
    • press and release d
    • release Shift

    Expected result:

    • pass. (with other keyboards lowercase d)

CapsAlwaysOff

For these tests, use a keyboard with the caps_always_off.kmp.zip store set. We call this keyboard capsalwaysoff below.

Any keyboard with that store set will work; if you don't have one at hand you can use the caps_always_off.kmp keyboard. The caps_always_off.kmp keyboard will prevent switching caps lock on. As a sanity check to verify that Keyman is actually active, pressing the key a will output ncaps_little_a, and Shift+a will output ncaps_shift_A.

Note: When testing in a virtual machine, use an on-screen keyboard (in VirtualBox: Input/Keyboard/Soft Keyboard) and observe the caps lock indicator of the on-screen keyboard. Using the hardware keyboard might show side effects with caps lock.

Prerequisites before each test

  • Install a keyboard that has CapsAlwaysOff store set, e.g. caps_always_off.kmp.
  • CapsLock is currently off
  • Currently active keyboard is a non-Keyman keyboard

Test cases

click to expand

  • TEST_CAPSOFF-1: sanity check

    • switch to capsalwaysoff keyboard
    • press and release a

    Expected result:

    • output: ncaps_little_a

  • TEST_CAPSOFF-2: caps lock stays off

    • switch to capsalwaysoff keyboard
    • press and release CapsLock key
    • press and release a

    Expected result:

    • caps lock indicator stays turned off
    • output: ncaps_little_a

  • TEST_CAPSOFF-3: no caps lock while holding capslock key

    • switch to capsalwaysoff keyboard
    • press and hold CapsLock key
    • press and release a
    • release CapsLock key

    Expected result:

    • output: ncaps_little_a

  • TEST_CAPSOFF-4: no caps lock while holding capslock key

    • switch to capsalwaysoff keyboard
    • press and hold CapsLock key
    • press and hold Shift key
    • press and release a
    • release CapsLock and Shift keys

    Expected result:

    • output: ncaps_shift_A

  • TEST_CAPSOFF-5: switching turns off caps lock

    • turn on caps lock
    • switch to capsalwaysoff keyboard
    • press and release a

    Expected result:

    • caps lock indicator turned off
    • output: ncaps_little_a

SHIFT: CapsOnOnly/ShiftFreesCaps

For these tests, use a keyboard with the CapsOnOnly and ShiftFreesCaps stores set. We call this keyboard shift_frees_caps below.

Any keyboard with these stores set will work; if you don't have one at hand you can use the shift_frees_caps.kmp.zip keyboard.

The shift_frees_caps.kmp keyboard will enable caps lock by pressing the CapsLock key, and will turn capslock off by pressing the Shift key. The keyboard outputs pass or fail if following the test cases.

Note: When testing in a virtual machine, use an on-screen keyboard (in VirtualBox: Input/Keyboard/Soft Keyboard) and observe the caps lock indicator of the on-screen keyboard. Using the hardware keyboard might show side effects with caps lock. Except for TEST_CAPSONLY-5 which can only be reliably tested on a hardware keyboard on host OS (not a VM). For windows 10 and windows 11 with a virtual box vm-onscreen keyboard, the following happens. The VM soft keyboard does NOT actually send the Shift Shift Key Stroke through but rather will change the keys pressed for example if an a is pressed the soft keyboard itself will change that key to a A. This means we can't Test TEST_CAPONLY-5 on a soft keyboard.

Prerequisites before each test

  • Install a keyboard that has the CapsOnOnly and ShiftFreesCaps stores set, e.g. shift_frees_caps.kmp.
  • CapsLock is currently off
  • Currently active keyboard is shift_frees_caps keyboard

Test cases

click to expand

  • TEST_CAPSONLY-1: no caps

    • press and release 1

    Expected result:

    • output: pass.

  • TEST_CAPSONLY-2: caps

    • press and release CapsLock
    • press and release 2

    Expected result:

    • caps lock indicator turned on
    • output: pass.

  • TEST_CAPSONLY-3: caps doesn't toggle

    • press and release CapsLock
    • press and release CapsLock
    • press and release 6

    Expected result:

    • caps lock indicator turned on
    • output: pass.

  • TEST_CAPSONLY-4: shift turns off

    • press and release CapsLock
    • press and hold Shift
    • press and release 3
    • release Shift

    Expected result:

    • caps lock indicator turned off
    • output: pass.

  • TEST_CAPSONLY-5: shift by itself turns off Be aware of limitations when testing this on virtual machines as noted above.

    • press and release CapsLock
    • press and release Shift

    Expected result:

    • caps lock indicator turned off
    • (no output)

SUITE_TSF_APP:

Background: Some windows applications use the windows TSF these include: MS Word, Firefox, Fieldworks and the windows search field on the taskbar.

Setup Text Services Framework(TSF) testing with Keyman for Windows

  1. Install Keyman Developer version 15 or greater.

  2. Open the Keyman Developers Test page in Firefox. It must be Firefox Browser

    i. By default this is at localhost:8008 so you could type this location in to the firefox address bar.

    ii. You can also access it by pressing the Test package on web button found in the Compile tab for any keyboard package.

  3. On the Keymand Developer Keyboard Test Page use the Keyboard dropdown to select system keyboard

  4. Use the Keyman for Windows icon located in the System Tray to select the desired Keyman for the Windows keyboard. system_tray

  5. Typing text into the input field will now use the TSF with the keyboard selected in step 4.

Test cases: double processing

click to expand

To run these tests the EuroLatin (SIL) Keyboard from here.

  • GROUP_WIN10

  • GROUP_WIN11

  • TEST_DOUBLE_PROCESSING_FIREFOX:

    1. Follow steps 1-4 from Setup Text Services Framework(TSF) testing with Keyman for Window above selecting the EuroLatin Keyboard at step 4.
    2. Select the EuroLatin Keyboard
    3. Type in to the windows search bar a b c d Backspace
    4. Expected result is abc

  • TEST_DOUBLE_PROCESSING_SEARCHBAR:

    1. Start Keyman
    2. Select the EuroLatin Keyboard
    3. Type in to the windows search bar a b c d Backspace
    4. Expected result is abc

    Search_bar

  • TEST_DOUBLE_PROCESSING_TEXTEDITOR:

    1. Open Text Editor
    2. Start Keyman
    3. Select the EuroLatin Keyboard
    4. Type a sequence of letters a b c d
    5. Expected result: abcd

SUITE_IMX_KEYBOARDS

Test IMX keyboards still work after core integration. see tests suite on #5936 and also #6187

This is the keyboard to be tested. imsample.zip If an IMTest keyboard is already installed you will need to uninstall it first, restart windows and then install the Imsample keyboard attached to this PR.

I have copied the tests from #5936 and made a few modifications.
The overall test we are aiming for is that in doing the tests below the keyboard does not cause a keyman or app crash.

  • GROUP_WIN10
  • GROUP_WIN11

Test cases

click to expand

IMSAMPLE_KEYBOARD_INLINE_MENU:

This keyboard uses the letters aeom to allow IMX input. This keyboard is a bit flakey being a demo and not polished. It also takes some setting up the following registry keys need to be added and set once installed. The location of the registry keys are at. Computer\HKEY_CURRENT_USER\SOFTWARE\Keyman\Keyman Engine\Active Keyboards\imsample and are of type REG_DWORD The keys are ShowIMWindow and ShowIMWindowAlways they both need to be set to 0.

In this mode IM window is not shown rather the app text is updated with a menu when either aeom is pressed. For example when type he the text will become h[1ɛ 2ɜ 3ə 4e 5ɘ] then you type the number 3 the text displayed should now be hə

  • TEST_IMSAMPLE_INPUT:

    • Install Imsample keyboard attached to this PR.
    • Select the keyboard its name is IMTest
    • Type some keys that are not the special keys e.g. t r y
      • Type the space key then a then type 1 to get the 1st option
    • Expected Result: try æ
    • continue to the next test

  • TEST_IMSAMPLE_INPUT_CONT: ... contd from above

    • Type 'm' and type 3 to get ### option
    • Expected Result: try æ###

  • TEST_IMSAMPLE_BACKSPACE: Using Text Editoror Libreoffice

    • Type f
    • Type e
    • The screen should have f[1ɛ 2ɜ 3ə 4e 5ɘ]
    • Now press Backspace. the whole option menu including the square brackets [ ] should be deleted.
    • Expected result f

IMSAMPLE_KEYBOARD_IM_WINDOW:

This keyboard uses the letters aeom to allow IMX input. This time a IM window should display. image

The first time you use the keyboard with an application the registry keys will be created and by default set to 1.

If the IM window is not appearing you will need to verify the value of two registry keys. The location of the registry keys are at. Computer\HKEY_CURRENT_USER\SOFTWARE\Keyman\Keyman Engine\Active Keyboards\imsample and are of type REG_DWORD The keys are ShowIMWindow and ShowIMWindowAlways they both need to be set to 1.

Note for the IM window you need to use the mouse pointer to select the options, you can't type the number.

  • TEST_IMSAMPLE_INPUT_IMW:

    • Install Imsample keyboard attached to this PR (if not done already)
    • Select the keyboard its name is IMTest
    • Type some keys that are not the special keys e.g. t r y
    • Type the space key then a then click the 1st option
    • Expected Result: try æ
    • continue to the next test

  • TEST_IMSAMPLE_INPUT_IMW_CONT: ... contd from above

    • Type 'm' and select the 3rd ### option
    • Expected Result: try æ###

SIMPLIFIED_CHINESE:

This keyboard will display the IMX window as soon as a string the matches the pinyin for one or more characters are typed.

For all these tests install the Simplified Chinese Keyboard cs-pinyin.cmp found here

  • TEST_SIMPLIFIED_CHINESE_SINGLE: Using Text Editor or equivalent.

    • Type o - The IMX window should appear
    • Choose the 4th option
    • Expected result: 喔

  • TEST_SIMPLIFIED_CHINESE_MULTIPLE: Using Text Editor or equivalent.

    • Type h a n z i - The IMX window will appear and in the top left the letters hanzi will be present
    • Choose the 5th option
    • Expected result: 汉字变换

  • TEST_SIMPLIFIED_CHINESE_BACKSPACE_1: Using Text Editor or equivalent. This is to test the backspace occurring mid pinyin sequence before a character is output to the app

    • Type h a n z i - The IMX window will appear and in the top left the letters hanzi will be present
    • Press Backspace twice - The IMX window should now have the top left letters of han.
    • Choose the 5th option
    • Expected result: 汗

  • TEST_SIMPLIFIED_CHINESE_BACKSPACE_2: Using Text Editor or equivalent. This is to test the backspace of already output characters.

    • Type h a n z i - The IMX window will appear and in the top left the letters hanzi will be present.
    • Choose the 5th option
    • Expected result: 汉字变换
    • Press Backspace twice
    • Expected result: 汉字

SUITE_STORE_AND_CONTEXT

Test the options keyboard stores and context updating as a result of keystroke input. Introduced after #7077 and #7809 there are more tests on those PRs.

  • GROUP_WIN10
  • GROUP_WIN11

Test cases

click to expand

The key referred to as "enter" is alternatively labelled "return" on some keyboards.

Keyboards for the following test are found in the zip file store_context_kbd.zip

TEST_21_OPTION_STORE:

  1. Install 021 - options.kmx
  2. Open Text Editor
  3. Select 021-options.kmx as the current keyboard.
  4. Type a1a0a

Expected Result: "no foo.foo.no foo"

TEST_OUTPUT_KEYSTROKE

  1. Install 043 - output and keystroke.kmx
  2. Open Text Editor
  3. Select 043 - output and keystroke.kmx as the current keyboard.
  4. Type 123

Expected Result: "abd3"

TEST_OUTPUT_KEYSTROKE_INVALID_CONTEXT

  1. If not installed from the previous test - Install 043 - output and keystroke.kmx
  2. Open Text Editor
  3. Select 043 - output and keystroke.kmx as the current keyboard.
  4. Type 12Enter3

Expected Result:
"abc
3"

TEST_DEADKEY_AND_CONTEXT

  1. Install 045 - deadkey and context.kmx
  2. Open Text Editor
  3. Select 045 - deadkey and context.kmx as the current keyboard.
  4. Type yzShift+/

Expected Result: "correct"

Clone this wiki locally