Table 2.

Usability insights per round and solution adopted according to Nielsen’s usability heuristics.

Usability categories	Round 1—research staff	Round 2—patients	Solution
Visibility of system status	Login procedure was confusing owing to the complexity of password, requiring 2 symbols	Challenges adhering to password requirements for certain patients	Password requirements were spelled out for users in bold. A password visualization button was also added to the password field to allow users to ensure that passwords created matched before registering their account
	—a	Unclear whether the QuAliV number (required for creating the account) is case sensitive	Information incorporated into the presentation of the study to participants
	Validation of questionnaire unclear	—	Information buttons added to the home page of the electronic PROb module instructing users on how the questionnaires functioned and reminding them to submit their completed questionnaires. The button was also relabeled to make its functionality clearer
Match between system and the real world	Date picker was in English and began in 2018, requiring users to click to go back in time	—	The date picker was replaced with a French version. It allowed users to type in their birth dates without using the calendar
	—	Nonmutually exclusive modalities or response missing	Minor modifications made to question modalities to ensure clarity
	—	Issues stemming from the translation of questionnaire from English to French	Further cognitive debriefing with native speakers to identify the best translation of the item in question
	—	Difficulties understanding the meaning of certain questions	Less formal language substituted where possible and examples given to facilitate the comprehension of certain questions
	—	Confirmation of account on one’s smartphone (email) resulted in being locked out of one’s account on another device	Automatic connection to the site after creating one’s account deleted (temporarily) to avoid users locking themselves out of their account. Users must reenter their username and password
User control and freedom	Need for returning back to last page completed in the questionnaire	—	The user is now redirected back to the most recent page completed within each questionnaire. Scrolling from one page of a questionnaire to another automatically saves entered data
	Radio button could not be unclicked or erased	—	A refresh button was added to each item to allow users to erase their responses and therefore leave items unanswered
	Questionnaire opens in a pop-up window whose size cannot be modified	—	Double checked to ensure that text could be easily read in each window
	—	Unclear whether users had to provide first and last name	We added text indicating that typing one’s first and last name was optional
Consistency and standards	Format of certain questions was noted as being inconsistent between questionnaires. Yes/No questions appeared in a table format as soon as they used the same response thesaurus	—	Minor improvements in formatting were made where possible. Further development required to accommodate this change in the longer run
	Typos in certain questions were identified	—	—
Error prevention	Aberrant response possible for certain free text fields	—	Stricter constraints added
	The password required was complex. Instructions on password requirements were missing from the account creation page	—	Instructions on password requirements added
	Need to clarify units in free text fields		Units added in gray in each text field
	Need to indicate which questions were mandatory in the questionnaire. Need to indicate when multiple answers could be given	—	An asterisk was added to indicate which questions were mandatory. The user is sent back to mandatory questions before being allowed to progress in the questionnaire. These questions were marked in red to indicate that they were mandatory
Recognition rather than recall	Automatic logout obligations meant that users could not reconnect to their accounts for 20 min if they left the page, resulting in certain evaluators being locked out of their account	—	Error message added to the module explaining that users would be able to reaccess their accounts after 20 min
Flexibility and efficiency of use	Errors encountered with the progress bar depending on responses to questions	—	Questionnaires are programmed to open successively
	Errors on certain Web browsers	—	Further trouble shooting using full array of browsers and devices
Aesthetic and minimalist design	Methods for completing a visual analog scale unclear as definition of extreme values was missing, and a not applicable box was not included	—	An 11-point radio button scale was proposed as a temporary solution
	The IPAQc questionnaire was difficult to read on the pop-up screen	—	Alternative formatting used to improve readability
Help users with errors	Need to flag missed items	—	Progression bar for each questionnaire goes from orange to green as soon as all nonconditional questions are answered. Users are directed to unanswered obligatory questions upon attempting to go on to the next page of the questionnaire
Help and documentation	Information missing from different links (contact and preferences)	—	—
	Print button of informed nonopposition did not function correctly	—	—

^aNot applicable.

^bPRO: patient-reported outcome.

^cIPAQ: International Physical Activity Questionnaire.