Thinking Aloud Test Report

706.100 Seminar/Projekt 6SP WS 2016/17

Authors

Carina Yasmin Enzinger
Bianca Helbig

Supervisors

Keith Andrews
Ralph Wozelka

Thinking Aloud Test of the Thunderbird Add-on

Enigmail

Report of 27^th June 2017

1 Executive Summary

Enigmail is a free Thunderbird add-on for encrypting and signing emails. This thinking aloud test of Enigmail was performed in March and April 2017 with five test users (see Table 1) who had experience with Thunderbird but not with Enigmail. The software tested was Thunderbird 45.8.0 with Enigmail 2.0a1pre. The test users were first asked some background questions and were then asked to perform some typical tasks with Enigmail, first installing the add-on and then sending and receiving signed and encrypted emails.

During the tasks, the test users were asked to speak their thoughts out loud, so the test team could gain insight into their thought process. After all five test sessions, all positive and negative findings were gathered into tables and rated for their positivity and severity respectively. The list of all positive findings was sorted in decreasing order of positivity (see Section 4.4). The list of all negative findings (problems) was sorted in decreasing order of severity (see Section 4.6).

The most positive finding was that Enigmail basically does what it should. The ability to encrypt emails was described as useful and activation through buttons was considered convenient by most of the test users. Only one test user, Silvia, struggled with the buttons and could not immediately tell whether or not they were activated. Locating and installing the Enigmail add-on was quite straightforward: all of the test users were able to find and install Enigmail without difficulty.

For typical computer users (non-experts in cryptography), the largest issue appears to be lack of knowledge about how encryption actually works. One test user, Martin, even encrypted an outgoing email with his own public key. The information pop-up asked him to choose a key to encrypt with and did not warn him that the key he chose (his own public key) would not make sense.

The second most severe problem involved the automatic attachment of the user's public key to outgoing email. Every time test users sent an email, their public key was sent as an attachment by default. When explicitly asked to send their public key, emails containing two attachments were sent. A global setting to activate and deactivate the attachment of the user's public key would be desirable. It is currently only possible to deactivate public key attachment for each outgoing email separately.

Some of the findings (both positive and negative) which arose during this thinking aloud test had already been reported in the previously conducted heuristic evaluation [Enz2016]. Such findings are marked with a reference (in square brackets) to the same finding in the previous heuristic evaluation report.

2 Introduction

In 2014, Edward Snowden described the GPG (GnuPG) software [GPG2017] used to encrypt emails as "damn near unusable" [Sno2014]. Arne Padmos discussed some of the reasons for this in his talk “Why is GPG 'damn near unusable'?” [Pad2014]. The problems involved today are much the same as those specified in the 1999 paper “Why Johnny can't encrypt” [Whi1999]. Not much has changed in the past 18 years.

Enigmail [Bru2017] is a free add-on (extension) for the Thunderbird [TMF2017] open source email, news, and chat client. It can encrypt and sign emails using GPG. The Enigmail project is maintained by Patrick Brunschwig. The EnigUsab project [And2015] was initiated by Keith Andrews and Ralph Wozelka in 2015 to help improve the usability of Enigmail. In 2016, a heuristic evaluation [Enz2016] was performed and a talk was given at Grazer Linuxtage 2016 [Woz2016].

Two of the main issues from the EnigUsab Heuristic Evaluation were fixed and the changes were incorporated into Enigmail prior to this thinking aloud test. Firstly, the wording of key trust levels was changed to correspond better to the original GnuPG definitions. The labels used in the Enigmail interface now show both the trust level name and a short description. Secondly, the confusing yellow and grey buttons for encrypting and signing were replaced. The basic symbols (a padlock for encrypting and a pencil for signing) were retained, but are now marked with a green tick-mark if activated and a red cross if deactivated. In addition, forced activation is no longer indicated in the interface.

In the meantime, also prior to this thinking aloud test, other changes had been made to Enigmail by the project's developers. The Enigmail toolbar in Thunderbird's email writing window was changed: the two Enigmail buttons are now integrated into the main toolbar, and Enigmail's own toolbar with the red status messages is hidden by default. Public key attachment was also changed. Now, the user's public key is automatically attached to every outgoing email by default, and this behaviour has to be deactivated for each email separately.

3 Test Procedure

3.1 Test Methodology

A thinking aloud test is a formative test intended to find problems in a user interface and to provide recommendations for fixing them. The user is asked to think out loud while they work, so as to gain an insight into their thought processes. More details on the methods of usability testing can be found in Keith Andrews' lecture notes [And2017a]. The thinking aloud method is described in detail by Barnum [Bar2010, Chapter 7] and Krug [Kru2009].

One pilot user and four real test users were recruited. All five test users were familiar with Thunderbird, but had no experience with the Enigmail add-on. The pilot test was used to uncover any issues with the test procedure and recording, which were then fixed ahead of the four real test users. Each user was asked to perform a series of typical tasks while thinking out loud. At the end of the test, each user was interviewed to gather immediate feedback. Finally, a feedback questionnaire was administered to gather subjective rating data from each user.

After the tests, the test team analysed the session recordings and observations to compile a list of positive findings and a list of negative findings (problems), ordered by positivity and severity respectively. This report was then written, based on the template from Keith Andrews [And2017b].

3.2 User Profiles

There are two main classes of user who might want to use Enigmail: those who need secure communication for work purposes and those who want their personal communications to be secure. Employees who are obligated to use Enigmail for secure communication will often have some kind of introduction or training on how to use it and may well have access to internal technical support. Private users will mostly depend upon the official documentation and built-in features like the Setup Wizard. With this in mind, it was decided to test the usability of Enigmail for private users.

3.3 Test Users

The background and experience of the five test users is shown in Table 1. Test users are referred to by aliases rather then their real names.

Test User	TP1 (Pilot)	TP2	TP3	TP4	TP5
Date of Test	26-03-2017	27-03-2017	27-03-2017	01-04-2017	27-03-2017
Time of Test	13:00 - 16:00	11:00 - 14:00	18:00 - 20:00	12:00 - 14:00	14:30 - 16:00
General Information
Alias	"Stuart"	"Silvia"	"John"	"Martin"	"Sally"
Sex	male	female	male	male	female
Age	34	32	36	23	22
Sight Impairment	none	none	contact lenses	none	glasses
Education	secondary school	secondary school	university degree	secondary school	secondary school
Occupation	IT support	Student of Software Development and Business Management	IT support	Student of Information and Computer Engineering	Student of Software Development and Business Management
Operating System	Windows	Mac OS	Windows	Ubuntu	Windows
Thunderbird Version	45.8.0	45.8.0	45.8.0	45.8.0	45.8.0
Enigmail Version	2.0a1pre DE	2.0a1pre EN	2.0a1pre DE	2.0a1pre EN	2.0a1pre DE
Language Spoken during TA	German	German	German	English	English
Experience with Computers
Using PC since	19 years	20 years	22 years	13 years	10 years
Time (h/week)	60	25	40	30	28
Thunderbird
Time (h/week)	1	25 (in the background)	1.5	1	2.5
Services	email	email, calendar	email	email, newsgroup	email, newsgroup
Experience with Encryption
Encryption for Personal Communication	no	no	no	yes	yes
Kind of Encryption				part of university course	part of university course
Knowledge of Web of Trust	none	none	none	partial	none
Enigmail	no	no	no	no	no
Previous Usability Tests
as Test Person	no	no	yes	no	yes
in Test Team	no	no	no	no	yes
Type of Test	Thinking Aloud				Thinking Aloud, Heuristic Evaluation

Table 1: Overview of the five test users.

3.4 Test Environment

The test room and equipment is shown in Figures 1 and 2. Users brought along their own laptop computer to use for the test. The exact version of Enigmail used for the tests is documented in Figure 3.

Test room, test user in front of PC — Figure 1: The setup used in the test room (shown with members of the test team).

Screenshot of the Enigmail About Window — Figure 3: Enigmail version used for the tests.

3.5 Test Recording

The test sessions were recorded on Windows with Camtasia [TS2017a] and Morae [TS2017b], on Mac with Camtasia [TS2017a] and on Linux with vokoscreen [Koh2017]. In each case, screen video was captured with a lossless video codec, together with webcam video, microphone audio, and any system sounds. TP1 Stuart remarked that Camtasia noticably slowed down his PC.

Still frames were extracted from screen video in PNG format using VLC media player [VL2017] and user's faces were anonymised with GIMP [GIMP2017]. The videos from TP1 Stuart and TP2 Silvia were recorded with Camtasia [TS2017a] and cut with kdenlive [KDL2017]. The videos from TP3 John and TP5 Sally were recorded and cut with Morae [TS2017b]. The video from TP4 Martin was recorded with vokoscreen [Koh2017] and cut with Windows Movie Maker. All the editing of videos to anonymise users was done with kdenlive [KDL2017]. Audacity [AT2017] was used to improve the quality of the audio tracks in the videos using the effects "Normalize" and "Noise Reduction" for every video.

3.6 Training

The facilitator explained the thinking aloud technique to each test user.

3.7 Tasks

The internal task list used by the test team is shown in Table 2.

Task No.	Description	Prerequisites	Completion Criteria	Max. Time	Possible Solution Path
1	Install Please install the Enigmail Thunderbird Extension.	Thunderbird installed.	Enigmail is installed.	15 minutes.	Use a browser. Go to the Enigmail web site, download and install the app. Use a browser. Go to the Thunderbird add-on page, download and install the app. Go to "Tools" → "Add-ons" → enter "Enigmail" into the search box, and install Enigmail. Once the test user had located an official stable Enigmail release, the facilitator intervened to provide the nightly build from 26 Mar 2017 using the file `enigmail-nightly-all.xpi` on an external hard disk. Since local installation of an extension from a file involves a dialog about security risks, the facilitator then installed the add-on.
2	Generate Keys Please generate your own key pair (private key and public key).	Successful Enigmail installation.	A new key pair was generated.	15 minutes.	Go to "Enigmail" → "Key Management" → "Generate" → "New Key Pair".
3	Send your public key Please send your public key to the technical support of this evaluation: c.y.enzinger@student.tugraz.at or bhelbig@student.tugraz.at	Successful key pair generation.	The user's public key was received.	10 minutes.	Go to "Enigmail" → "Key Management" → right click on own Name → "Send Public Keys by Email". Go to "Write" → "Enigmail" → "Attach My Public Key".
4	Add a public key Please add the public key of the reply to your Key Management.		The test user added the public key.	5 minutes.	Double click on attachment of email.
5	Key server Please use the Keyserver to add the public key of "Carina Enzinger" or "Bianca Helbig".		The public key of the facilitator was added.	5 minutes.	Go to "Enigmail" → "Key Management" → "Keyserver" → "Search for Keys".
6	Send and get an encrypted e-mail Please send an encrypted email to the technical support and review the reply.		The test team received an encrypted email and the user finished looking through the reply.	10 minutes.	Go to "Write" → click on the "Encryption" button.
7	Send and get a signed e-mail Please send a signed email to the technical support and review the reply.		The test team received a signed email and the user finished looking through the reply.	5 minutes.	Go to "Write" → click on the "Sign Message" button.
8	Send and get an encrypted and signed e-mail Please send an encrypted and signed email to the technical support and review the reply.		The test team received an encrypted and signed email and the user finished looking through the reply.	5 minutes.	Go to "Write" → click on the "Encryption" and "Sign Message" buttons.
9	Send an encrypted e-mail Please send an encrypted email to the following address: enigusab@gmx.at		The test team received an encrypted email.	5 minutes.	Go to "Write" → enter recipient email → "Send" → "Download missing keys". The test user did not have the public key of `enigusab@gmx.at` already stored and hence was required to use the "Download missing keys" functionality of Enigmail.

Table 2: The internal task list used by the test team.

The task descriptions given to the test users are provided in Section A.5. A task was considered to have started when the facilitator began reading the task description and to have ended when the user satisfied the task completion criteria (or the facilitator advanced to the next task after the designated maximum time).

All test users except TP4 Martin used gmx or gmail email addresses for Enigmail setup. It took up to 30 minutes for an email from a non-tugraz email address to arrive at the test team's tugraz email address. Therefore, to speed things up in Task 3, the public keys from test users using gmx or gmail were transferred via USB stick to the laptop of the test team. TP4 Martin used his tugraz email address and his public key could be received via email. For the same reason, in order to avoid unnecessary waiting time in Tasks 6, 7, and 8, the test team did not wait for the test user's email to arrive in order to reply to it, but instead sent a pre-formulated email in lieu of a reply.

3.8 Interview Questions

The following questions were asked of each user immediately after the final task.

Opening Question
- How was it? [Wie war's?]
Standard Questions
1. Did anything strike you as particularly good?
  [Gab es etwas, dass besonders gut oder positiv aufgefallen ist?]
2. Did anything strike you as particularly bad?
  [Gab es etwas, dass besonders schlecht war?]
Individual Questions (asked of specific users)
1. Would you want Enigmail → Einstellungen → "Bequeme Verschlüsselungs-Einstellung" to work differently? [TP1]
2. Would you have pressed on the turquoise/blue panel, if the facilitator had not asked you to? [TP2]
3. Is the information given by the system itself enough for you or should there be more?
  [Sind die Informationen, die vom System gegeben werden genug für Sie oder sollten dort mehr sein?] [TP3]
4. Should the button "Search for missing keys" be somewhere else or would be a pop-up be better? [TP4]
5. Is there something you would change? [TP4]

3.9 Feedback Questionnaire

After the interview, the test users were asked to fill out the paper-based feedback questionnaire given in Section A.7.

3.10 Data Preparation

Data from the background questionnaires, task success measures, all positive and negative findings as well as the ratings from the feedback questionnaires were collected in the data spreadsheet (see Section A.8).

4 Results

All test users were able to complete all nine tasks, although sometimes some help was given by the facilitator. Some of the findings (both positive and negative) were already observed and described in the Heuristic Evaluation Report from 2016 [Enz2016]. Such findings are marked with a reference (in square brackets) to the same finding in the previous heuristic evaluation report.

4.1 Discussion and Analysis

The most positive finding is that Enigmail does what it should, i.e. signing and encrypting emails. Furthermore, every test user was able to locate the extension for downloading. The Enigmail Wiki was also very well-received by TP4 Martin, who consulted the Wiki frequently and would recommend it to anybody interested in using Enigmail.

A number of problems were also observed during the tests. The most serious problem was users using encryption in the wrong way. TP4 Martin chose his own public key to encrypt an outgoing email. There was no warning that this does not make any sense.

The second moszt serious problem is that the user's public key is automatically attached to every outgoing email. When explicitly asked to send their public key, users sent emails containing two attachments. A global setting to activate and deactivate attachment of the user's public key would be desirable. It is currently only possible to deactivate public key attachment for each outgoing email separately.

The third problem is that no "Finish" button is displayed at the end of the setup wizard. TP3 John was was uncertain as to whether setup had completed or not. This problem was already found during the heuristic evaluation.

4.2 Task Completion

A summary of how the test users completed each task is shown in Table 3. A "1" indicates successful task completion. In this study, every user completed every task. An asterisk "*" indicates that some assistance was given.

	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Task 7	Task 8	Task 9
TP1	1	1	1	1	1	1	1	1	1
TP2	1	1*	1*	1	1*	1*	1*	1	1*
TP3	1	1	1	1	1*	1	1	1	1*
TP4	1	1	1	1	1	1	1	1	1*
TP5	1	1	1	1	1	1	1	1	1

Table 3: Task completion.

4.3 Positive Findings

The five most positive findings according to their average positivity ratings are described in more detail below. The positivity rating scheme used to rank positive findings is shown in Table 4.

Positivity	Meaning
4	Extremely Positive
3	Major Positive
2	Minor Positive
1	Cosmetic Positive
0	Not a Positive
Av	Average Positivity

Table 4: Positivity Ratings.

P01. It Does What it Should

Short Title:	It Does What it Should
Description:	The ability to encrypt emails is useful [see HE P02].
Video Clip(s):	`p01-tp2-encrypt.mp4`
Timestamp(s):	TP2 00:57:00.
Positivity:	3.50

The ability to encrypt emails was described as very useful. The quote from Silvia can be heard in Figure 4.

Figure 4: The ability to encrypt emails was described as very useful by Silvia.

P02. Easy to Find and Download

Short Title:	Easy to Find and Download
Description:	It is straightforward to find the Enigmail add-on if its name is known. TP1 Stuart, TP2 Silvia and TP4 Martin used Google, TP3 John and TP5 Sally used Thunderbird's Add-ons Manager [see HE P01].
Screenshot(s):	`p02-tp1-google.png`, `p02-tp2-google.png`, `p02-tp3-thunderbird.png`, `p02-tp4-google-softwarecenter.png`, `p02-tp5-thunderbird.png`
Timestamp(s):	TP1 00:00:31, TP2 00:03:35, TP3 00:00:28, TP4 00:00:38:, TP5 00:00:39.
Location:	Google Search, Thunderbird's Add-ons Manager
Positivity:	3.50

Three test users (TP1 Stuart, TP2 Silvia, and TP4 Martin) found Enigmail through Google's search engine, while TP3 John and TP5 Sally used Thunderbird's Add-ons Manager to find it. The search results can be seen in Figures 5, 6, 7, 8 and 9.

Screenshot Google search results from Stuart — Figure 5: The Google search results from Stuart.

Screenshot Google search results from Silvia — Figure 6: The Google search results from Silvia.

Screenshot Thunderbird search results from John — Figure 7: The Thunderbird Add-ons Manager search results from John.

Screenshot Google and Ubuntu search results from Martin — Figure 8: The Google and Ubuntu software center search results from Martin.

Screenshot Thunderbird search results from Sally — Figure 9: The Thunderbird Add-ons Manager search results from Sally.

P03. Enigmail Wiki is Very Helpful

Short Title:	Enigmail Wiki is Very Helpful
Description:	The Enigmail Wiki documentation is very well done and helpful.
Video Clip(s):	`p03-tp4-enigmail-wiki.mp4`
Screenshot(s):	`p03-tp4-enigmail-wiki.png`
Timestamp(s):	TP4 00:08:21, TP4 00:14:36, TP4 00:23:51, TP4 00:27:34, TP4 00:30:59.
Location:	`http://enigmail.wiki/`
Positivity:	3.50

The documentation provided on the Enigmail Wiki web site (http://enigmail.wiki/) is well-structured and very helpful. In particular, TP4 Martin consulted the Wiki several times during various tasks and recommends it to others, as can be heard in Figure 10.

Figure 10: The Enigmail Wiki helped Martin several times and he recommends it to others.

P04. Easy to Import Another User's Public Key from Attachment

Short Title:	Easy to Import Another User's Public Key from Attachment
Description:	When another user sends their public key as an attachment to an incoming email, it can be easily imported into Enigmail's Key Management by clicking the attachment and confirming.
Video Clip(s):	`p04-tp1-import.mp4`, `p04-tp2-import.mp4`
Timestamp(s):	TP1 00:17:10, TP2 00:46:10.
Location:	Thunderbird's Email Attachment Pane
Positivity:	3.00

Importing a public key attached to an incoming email is straightforward. Figure 11 shows TP1 Stuart importing the public key attached to an email sent by the test team.

Figure 11: Importing a public key attached to an incoming email is straightforward, as can be seen here with TP1 Stuart.

P05. Encrypt and Sign Buttons are Intuitive

Short Title:	Encrypt and Sign Buttons are Intuitive
Description:	The buttons to encrypt and sign emails are considered very intuitive. Users do not have to do anything more than just press them.
Video Clip(s):	`p05-easy-buttons.mp4`
Screenshot(s):	`p05-easy-buttons.png`
Timestamp(s):	TP4 00:26:56, TP4 00:39:57, TP4 00:40:48.
Location:	Write (an email)
Positivity:	3.00

The encrypt and sign buttons were praised as a convenient way to encrypt and sign emails by TP4 Martin, as can be heard in Figure 12.

Figure 12: TP4 Martin praised the encrypt and sign buttons as "very convenient".

4.4 List of All Positive Findings Found

Table 5 shows a list of all the positive findings observed in the test, sorted in decreasing order of average positivity. The initials corresponding to the evaluators who assigned the positivity ratings are explained in Table 6

No.	Short Title	Description	Video Clip(s)	Screenshot(s)	Timestamps(s)	Location	Positivity
No.	Short Title	Description	Video Clip(s)	Screenshot(s)	Timestamps(s)	Location	CE	BH	RW	JR	Av
01	It Does What it Should	The ability to encrypt emails is useful [see HE P02].	`p01-tp1-encrypt.mp4`		TP2 00:57:00		3	3	4	4	3.50
02	Easy to Find and Download	It is straightforward to find the Enigmail add-on if its name is known. TP1 Stuart, TP2 Silvia and TP4 Martin used Google, TP3 John and TP5 Sally used Thunderbird's Add-ons Manager [see HE P01].		`p02-tp1-google.png`, `p02-tp2-google.png`, `p02-tp3-thunderbird.png`, `p02-tp4-google-softwarecenter.png`, `p02-tp5-thunderbird.png`	TP1 00:00:31, TP2 00:03:35, TP4 00:40:01	Google Search, Thunderbird's Add-ons Manager	3	3	4	4	3.50
03	Enigmail Wiki is Very Helpful	The Enigmail Wiki documentation is very well done and helpful.	`p03-tp4-enigmail-wiki.mp4`	`p03-tp4-enigmail-wiki.png`	TP4 00:08:21, TP4 00:14:36, TP4 00:23:51, TP4 00:27:34, TP4 00:30:59	`http://enigmail.wiki/`	3	4	4	3	3.50
04	Easy to Import Another User's Public Key from Attachment	When another user sends their public key as an attachment to an incoming email, it can be easily imported into Enigmail's Key Management by clicking the attachment and confirming.	`p04-tp1-import.mp4`, `p04-tp2-import.mp4`		TP1 00:17:10, TP2 00:46:10	Thunderbird's Email Attachment Pane	3	3	3	3	3
05	Encrypt and Sign Buttons are Intuitive	The buttons to encrypt and sign emails are considered to be very intuitive. Users do not have to do anything more than just press them.	`p05-easy-buttons.mp4`	`p05-easy-buttons.png`	TP4 00:26:56, TP4 00:39:57, TP4 00:40:48	Write (an email)	3	3	3	3	3
06	No Need to Re-enter the Passphrase	When looking at sent or received emails, the passphrase is remembered by Enigmail (and does no t have to be re-entered) for 5 minutes. Silvia described it as time-saving, when somebody wants to quickly read several emails.	`p06-tp2-pp-reenter.mp4`		TP2 00:58:13		3	2	3	3	2.75
07	Automatic Encryption	Enigmail checks whether the recipient's email address already has a key associated with it. If so, it automatically encrypts the outgoing message.	`p07-tp4-auto-encryption.mp4`		TP4 00:24:45		2	1	3	4	2.50
08	Enigmail Well Integrated into Thunderbird	The software blends well into the Thunderbird theme.		`p08-well-integrated-1.png`, `p08-well-integrated-2.png`	TP4 00:40:34		2	3	2	2	2.25

Table 5: Aggregated list of all positive findings, in descending order of average positivity. Timestamps refer to elapsed time from the start of the corresponding test user's session capture video (see Section C.3).

Code	Meaning
CE	Carina Enzinger
BH	Bianca Helbig
RW	Ralph Wozelka
JR	Jacqueline Rinnhofer

Table 6: The positivity ratings assigned by each evaluator are identified by the evaluator's initials.

4.5 Description of the Main Problems

The five most serious problems according to their average severity ratings are described in more detail below. The severity rating scheme used to rank problems is shown in Table 7.

Severity	Meaning
4	Catastrophic problem
3	Serious problem
2	Minor problem
1	Cosmetic problem
0	Not a problem
Av	Average severity

Table 7: Severity ratings.

N01. Easy to Choose Wrong Public Key(s) when Encrypting Outgoing Mail

Short Title:	Easy to Choose Wrong Public Key(s) when Encrypting Outgoing Mail
Description:	When encrypting an outgoing email to an email address whose public key is not yet known, the user must select a corresponding public key to use for the encryption. It is possible to choose any key (or even multiple keys) from the Key Manager, even though only the recipient's public key would make sense (since otherwise the recipient will not be able to decrypt the message). There is no warning to this effect, instead the message is simply sent encrypted with the wrong public key(s).
Video Clip(s):	`n01-tp4-wrong-key-encryption.mp4`
Timestamp(s):	TP4 00:34:45.
Location:	Write an encrypted email to an email address you do not have the public key of → send → choose a key from your list → send
Severity:	3.75

TP4 Martin sent an email encrypted with the key and did not receive any feedback from Enigmail. If the facilitator had not informed him that encryption does not work this way, Martin would not have noticed, as can be seen in Figure 13.

Figure 13: Martin had to choose a key to be able to send the email. Since he did not know any better, he used his own public key and was not given any feedback.

To solve this problem, Enigmail should provide a warning message or pop-up, before actually sending such an email.

N02. Public Key Automatically Attached to Every Outgoing Email

Short Title:	Public Key Automatically Attached to Every Outgoing Email
Description:	By default, after Enigmail has generated a key pair for the user's email address (as part of setup), the generated public key is then automatically attached to every email the user sends. If the user does not wish to attach their public key to every outgoing email, this function has to be disabled separately for every single email. There should be a general option to disable or enable this feature.
Video Clip(s):	`n02-tp1-auto-key.mp4`
Screenshot(s):	`n02-tp1-auto-key.png`
Timestamp(s):	TP1 00:13:51, TP4 00:42:42, TP4 00:43:07, TP4 00:45:18, TP5 00:17:00.
Location:	Write an email.
Severity:	3.50

The test users' public key was automatically attached to every outgoing email without them knowing about it. TP1 Stuart checked his outgoing email in the Sent folder and was confused that his public key was attached twice (the first he added himself and the second was attached automatically by Enigmail), as can be seen in Figure 14. The other four users simply did not notice that their public key was being automatically attached.

Figure 14: Stuart manually attached his public key, but Enigmail attached another one automatically too.

The public key should be attached explicitly by the user and not automatically to each outgoing email. Possibly a tutorial or guide could explain to users that their public key must be made known to recipients, in order for replies to be encrypted.

N03. Missing Finish Button for Setup Wizard

Short Title:	Missing Finish Button for Setup Wizard
Description:	The final window of the setup wizard has no "Finish" button to complete the process properly, suggesting to some users that the setup wizard had died [see HE N14].
Video Clip(s):	`n03-tp3-no-finish-button.mp4`
Screenshot(s):	`n03-tp3-no-finish-button.png`
Timestamp(s):	TP3 00:08:48
Location:	Setup Wizard → create new key pair
Severity:	3.50

TP3 John finished the key configuration process (the final part of the Setup Wizard), but hesitated because there was no "Finish" button at the end of the process, as shown in Figure 15.

Figure 15: TP3 John expected to end the configuration with a "Finish" button, but there was none.

The final step of the Setup Wizard should display a "Finish" button, to indicate to users that the process has been completed successfully.

N04. Confusing Wording for Trust

Short Title:	Confusing Wording for Trust
Description:	The wording in the Enigmail Security Info bar above an incoming signed mail is confusing. Text such as "UNTRUSTED Good signature" ("NICHT VERTAUENSWÜRDIGE Korrekte Unterschrift") is intended to convey that the signature is valid but untrusted. However, users do not understand the intricacies and terminology of the Web of Trust model. The text can sometimes lead to the conclusion that the sender of the email is untrustworthy [see HE N11].
Video Clip(s):	`n04-tp3-not-trusted-correct-signature-1.mp4`, `n04-tp5-not-trusted-correct-signature-2.mp4`
Screenshot(s):	`n04-tp1-word-trust-de.png`, `n04-tp4-word-trust-en.png`, `n04-tp3-not-trusted-correct-signature-1.png`, `n04-tp3-not-trusted-correct-signature-2.png`,
Timestamp(s):	TP1 00:30:13, TP1 00:30:55, TP3 00:25:33, TP5 00:16:27.
Location:	get and open a signed email → Details → sign/verify key open a signed email (→ Details → Enigmail security information)
Severity:	3.25

All test users except Martin were confused about the wording of the message "UNTRUSTED Good signature" ("NICHT VERTAUENSWÜRDIGE Korrekte Unterschrift") in the Enigmail Security Info bar at the top of the email panel, as can be seen in Figure 16. Sally even concluded that the sender might not be trustworthy, as can be heard in Figure 17.

Screenshot from the signature describing text. — Figure 16: The text of the signature description.

Figure 17: Sally thinks that maybe the sender of the email is not trustworthy.

The words should possibly be replaced by symbols with explanatory hover text and possibly even linked to further information about the Web of Trust system in the Enigmail Wiki. This would also be a good candidate for inclusion into an Enigmail tutorial.

N05. Missing Tutorial

Short Title:	Missing Tutorial
Description:	After installation, provision of a short tutorial for new users would make it easier to work with Enigmail. For example, it should: Describe basic functions like attaching a public key, and signing and encrypting outgoing emails. Explain that the subject of an encrypted message will be displayed as "Encrypted Message" ("Verschlüsselte Nachricht") until the recipient decrypts it. Explain that the passphrase entered by the user when viewing sent or received emails is retained by Enigmail for 5 minutes before having to be re-entered (the time period can be changed in Enigmail's Preferences). [see HE N15]
Video Clip(s):	`n05-tp1-tutorial.mp4`
Timestamp(s):	TP1 00:11:05, TP1 00:49:25, TP1 00:36:05, TP2 00:12:28, TP2 00:16:13, TP2 00:27:00, TP2 00:50:27, TP2 00:52:11, TP2 00:53:30, TP2 00:54:05, TP4 00:39:08, TP5 00:20:32.
Severity:	3.00

Figure 18: TP1 Martin would like to be shown some first steps in the course of Enigmail installation.

A short tutorial for new users would make it easier to use Enigmail. Basic functions should be explained, such as how to send someone your public key, how to sign and encrypt emails, etc.

4.6 List of All Problems Found

Table 8 shows a list of all the problems observed in the test, sorted in decreasing order of average severity. The initials corresponding to the evaluators who assigned severity ratings are explained in Table 9

No.	Short Title	Description	Video Clip(s)	Screenshot(s)	Timestamps(s)	Location (how reproducible?)	Severity
No.	Short Title	Description	Video Clip(s)	Screenshot(s)	Timestamps(s)	Location (how reproducible?)	CE	BH	RW	JR	Av
01	Easy to Choose Wrong Public Key(s) when Encrypting Outgoing Mail	When encrypting an outgoing email to an email address whose public key is not yet known, the user must select a corresponding public key to use for the encryption. It is possible to choose any key (or even multiple keys) from the Key Manager, even though only the recipient's public key would make sense (since otherwise the recipient will not be able to decrypt the message). There is no warning to this effect, instead the message is simply sent encrypted with the wrong public key(s). To solve this problem Enigmail should provide a warning message or pop-up, before actually sending such an email.	`n01-tp4-wrong-key-encryption.mp4`		TP4 00:34:45	Write an encrypted email to an email address you do not have the public key of → send → choose a key from your list → send	4	3	4	4	3.75
02	Public Key Automatically Attached to Every Outgoing Email	By default, after Enigmail has generated a key pair for the user's email address (as part of setup), the generated public key is then automatically attached to every email the user sends. If the user does not wish to attach their public key to every outgoing email, this function has to be disabled separately for every single email. There should be a general option to disable or enable this feature. The public key should be attached explicitly by the user and not automatically to each outgoing email. Possibly a tutorial or guide could explain to users that their public key must be made known to recipients, in order for replies to be encrypted.	`n02-tp1-auto-key.mp4`	`n02-tp1-auto-key.png`	TP1 00:13:51, TP4 00:42:42, TP4 00:43:07, TP4 00:45:18, TP5 00:17:00	Write an email.	4	4	3	3	3.50
03	Missing Finish Button for Setup Wizard	The final window of the setup wizard has no "Finish" button to complete the process properly, suggesting to some users that the setup wizard had died [see HE N14]. The final step of the Setup Wizard should display a "Finish" button, to indicate to users that the process has been completed successfully.	`n03-tp3-finish-button.mp4`	`n03-tp3-finish-button.png`	TP3 00:08:48	Setup Wizard → create new key pair	4	3	3	4	3.50
04	Confusing Wording for Trust	The wording in the Enigmail Security Info bar above an incoming signed mail is confusing. Text such as "UNTRUSTED Good signature" ("NICHT VERTAUENSWÜRDIGE Korrekte Unterschrift") is intended to convey that the signature is valid but untrusted. However, users do not understand the intricacies and terminology of the Web of Trust model. The text can sometimes lead to the conclusion that the sender of the email is untrustworthy [see HE N11]. The words should possibly be replaced by symbols with explanatory hover text and possibly even linked to further information about the Web of Trust system in the Enigmail Wiki. This would also be a good candidate for inclusion into an Enigmail tutorial.	`n04-tp3-not-trusted-correct-signature-1.mp4`, `n04-tp5-not-trusted-correct-signature-2.mp4`	`n04-tp1-word-trust-de.png`, `n04-tp4-word-trust-en.png`, `n04-tp3-not-trusted-correct-signature-1.png`, `n04-tp3-not-trusted-correct-signature-2.png`	TP1 00:30:13, TP1 00:30:55, TP3 00:25:33, TP5 00:16:27	get and open a signed email → Details → sign/verify key, open a signed email (→ Details → Enigmail security information)	4	3	3	3	3.25
05	Missing Tutorial	After installation, provision of a short tutorial for new users would make it easier to work with Enigmail. For example, it should: Describe basic functions like attaching a public key, and signing and encrypting outgoing emails. Explain that the subject of an encrypted message will be displayed as "Encrypted Message" ("Verschlüsselte Nachricht") until the recipient decrypts it. Explain that the passphrase entered by the user when viewing sent or received emails is retained by Enigmail for 5 minutes before having to be re-entered (the time period can be changed in Enigmail's Preferences). [see HE N15] A short tutorial for new users would make it easier to use Enigmail. Basic functions should be explained, such as how to send someone your public key, how to sign and encrypt emails, etc.	`n05-tp1-tutorial.mp4`		TP1 00:11:05, TP1 00:49:25, TP1 00:36:05, TP2 00:12:28, TP2 00:16:13, TP2 00:27:00, TP2 00:50:27, TP2 00:52:11, TP2 00:53:30, TP2 00:54:05, TP4 00:39:08, TP5 00:20:32		4	3	3	2	3
06	Key Server Connectivity	On occasion, the connection to the chosen key server was (partially) lost, resulting in confusing loss of functionality. Both TP1 Stuart and TP3 John experienced in Task 5 that old revoked keys belonging to the facilitator were not greyed out as they usually should be (TP2 00:49:25). In Task 9, the button "Download missing keys" did not work at all for TP1 Stuart. When the connection to the chosen key server is lost, the user should be notified.	`n06-tp1-search-error.mp4`	`n06-tp1-search-error-1.png`, `n06-tp1-search-error-2.png`, `n06-tp3-server-error.png`	TP1 00:21:20 and TP3 00:19:52, TP1 00:40:27	Sending an encrypted email to an email address with no matching public key stored in the key management, Enigmail → Key Management → Key Server → Search for Keys	2	3	4	3	3
07	Hidden GPG Suite Installer Window on Mac	During installation with standard configuration (Beginner's Mode) on a Mac, the GPG Suite installer window opens in the background unnoticed by the user. However, installation of GPG Suite is a prerequisite for successful installation of Enigmail. The "Enable Protection" warning symbol in the Write email panel (which replaces the Signing and Encrypting buttons when Enigmail is not configured) is easily overlooked. TP2 Silvia was unaware that Enigmail installation was incomplete (because GPG Suite had not been installed). Silvia made five unsuccessful attempts to send encrypted email and redid the Setup Wizard in beginner's mode twice, before Enigmail worked properly [see HE N1].	`n07-tp2-mac-install.mp4`	`n07-tp2-mac-install-1.png`, `n07-tp2-mac-install-2.png`, `n07-tp2-mac-install-3.png`	TP2 00:10:15, TP2 00:10:28, TP2 00:11:19, TP2 00:11:31, TP2 00:11:52, TP2 00:12:13, TP2 00:18:01, TP2 00:18:48, TP2 00:23:46, TP2 00:24:08, TP2 00:28:08, TP2 00:28:55 – 00:31:44, TP2 00:32:22, TP2 00:36:36, TP2 00:39:11, TP2 00:39:23, TP2 00:40:39, TP2 00:42:49, TP2 00:44:32	Mac Setup Wizard	3	4	4	1	3
08	"Location to Save Revocation Certificate" Message Appears Too Late	The information window that a revocation certificate should best be saved on a safe external medium such as a diskette or CD only appears after the revocation certificate has been saved [see HE N04].		`n08-tp4-revocation-certificate.png`	TP4 00:06:35, TP5 00:05:08	Setup Wizard → generate new key pair	4	4	2	2	3
09	Unclear Syntax for Search on Key Server	The key server search window does not provide any details about the possibilities to search for another person's public key. The search syntax is arcane and unclear. Commonly used wildcards such as "" do not work as expected. "Bia" only matches the exact string "Bia". Inserting a "+" character as in "Bianca+Helbig" achieves the desired result [see HE N22].	`n09-tp3-key-server-search-details.mp4`	`n09-tp1-search-name.png`, `n09-tp3-key-server-search-details.png`	TP1 00:51:00, TP2 00:47:30, TP3 00:19:13, TP4 00:18:42	Enigmail → Key Management → Key Server → Search for Keys	3	3	3	1	2.50
10	Lack of Context-Specific Help	Users would like to have access to context-specific information and help when working with Enigmail. TP1 Stuart wanted to know what "Standard configuration (beginner's mode)" would involve. A context-specific help button (say "?") or a link to the wiki entry `https://enigmail.wiki/Quick_start#Standard_configuration_.28for_beginners.29` would have helped him to understand.	`n10-tp1-help.mp4`		TP1 00:02:25, TP2 00:58:57		2	4	3	1	2.50
11	Changing Mac Menu Bar	The Key Management window has its own menu bar. On the Mac, the menu bar is integrated into the standard Mac menu bar at the top of the screen (standard OS X behaviour). However, this is sometimes not noticed by users, who then have problems finding the function to search for a another user's key on a key server [see HE N30].	`n11-tp2-mac-bar.mp4`		TP2 00:16:25	Enigmail → Key Management → Search for Key	3	3	3	1	2.50
12	Web Of Trust	Users find it difficult to understand the idea of Web of Trust. Users did not know which level of trust they should choose or how to verify that the signature really belongs to the person whose name and email they can see. Enigmail should provide hints (say, on mouse hover) or link to to wiki entry (`https://enigmail.wiki/Key_Management#Validity_of_public_keys`) about typical key usage and management strategies [see HE N12].		`n12-tp1-web-trust-1.png`, `n12-tp1-web-trust-2.png`, `n12-tp3-signing-of-key.png`, `n12-tp3-trust-level.png`	TP1 00:31:23, TP1 00:33:35	open a signed email → Details → Sign Sender's Key…	2	2	3	2	2.25
13	Misleading Key Selection Window	When a user wishes to send encrypted email, but there is no key associated with the recipient, the Enigmail Key Selection window opens. The prominent list of known keys tempts the user to choose one of these, when it is actually necessary to click the "Download missing keys" button to find and download the public key belonging to the recipient. The "Download missing keys" button should appear when the user clicks on the Encryption button in the Write window. So that the user can search for the missing key before clicking on the "Send" button. The prominent list of known keys should initially be hidden (beneath a tab or as an option), so that inexperienced users do not choose their own key.	`n13-tp4-download-missing-keys.mp4`	`n13-tp2-download.png`, `n13-tp4-download-missing-keys.png`	TP2 00:55:23, TP2 00:56:15, TP3 00:31:22, TP4 00:34:40	Write an encrypted email to an email address you do not have the public key of → send	3	2	3	1	2.25
14	Notification of Key Generation	In the Key Creation step of the Setup Wizard, the phrase "You will be alerted when key generation is completed." ["Sie werden informiert, sobald der Schlüssel fertiggestellt ist."] can lead the user to expect specific notification, particularly since they are asked to actively browse the web in the meantime. This is not the case: only the text "Your Key is Generated" ["Ihr Schlüssel ist angelegt."] is written to the (typically hidden) window. The user should be alerted via a pop-up window when the key has been generated.		`n14-tp1-fin-info-de.png`, `n14-tp1-fin-info-en.png`	TP1 00:09:39, TP1 00:49:57	End of Setup Wizard (Expert mode)	2	2	2	2	2
15	Inbox Column for Signed and Encrypted	Enigmail actually provides a column in the Inbox to indicate whether an email is signed and/or encrypted. However, it is not displayed by default and has to be manually added by the user. In the previous Heuristic Evaluation, evaluators recommended using the same symbols as on the corresponding buttons [see HE N44], positioning the two symbols consistently within the column (or using two columns) [see HE N48], and using more easily distinguishable symbols [see HE N23]. It is also recommended to make the column narrower by using the Enigmail icon (logo) instead of the word Enigmail in the column header, or by using two separate columns with the encrypted and signed symbols as headers.	`n15-tp1-symbols-1.mp4`, `n15-tp1-symbols-2.mp4`	`n15-tp1-symbols.png`	TP1 00:26:57, TP1 00:37:05, TP1 00:47:45, TP2 00:58:00	Right click on the header of the email list overview → click on "Enigmail"	2	3	2	1	2
16	Ordering of Search Results on Key Server	When searching for keys on a key server, the results are listed alphabetically, rather than by relevance. Furthermore, it is not possible to order or filter the results list. For example, searching for "Bianca Helbig" returns all Biancas and all Helbigs, with the desired match somewhere in the middle of a long list. [see HE N26]	`n16-tp2-order.mp4`	`n16-tp4-key-server-search-result.png`	TP1 00:20:27, TP1 00:20:57, TP2 00:48:39, TP4 00:21:13, TP5 00:12:14	Enigmail → Key Management → key server → search for key → OK	3	2	2	1	2
17	Different Key Wizards on Mac	On the Mac, there appear to be two different wizards for creating a new key pair: the GPG Wizard and Enigmail Key Management (in addition to the initial Setup Wizard). The GPG Wizard automatically creates a revoke certificate, but the Enigmail Key Management requests that the user generate one [see HE N09] [see HE N19]. These should be integrated into one wizard.		`n17-tp2-mac-wizards-1.png`, `n17-tp2-mac-wizards-2.png`	TP2 00:44:17	(First time) Setup Wizard Enigmail & (First time) Setup Wizard GPG & Key Manager → Generate → new key pair	2	2	2	2	2
18	Hidden Key Management Menu Bar (Ubuntu Linux)	On Ubuntu Linux, the Key Management's menu bar appears at the top of the display (like on a Mac), rather than the top of the window. TP4 Martin had to mouse to the top of the screen to even see the menu bar, although this could be due to a user-specific configuration (autohide).	`n18-tp4-hidden-menu-bar.mp4`		TP4 00:17:43	Enigmail → Key Management	2	2	3	1	2
19	Unclear Timeout to Re-Enter Passphrase	Enigmail remembers the passphrase for 5 minutes by default. After that time, the user is asked to re-enter it. It is unclear to users why they are periodically asked to re-enter the passphrase. [see P06] [see HE N29]. As mentioned in N05, Enigmail should provide a tutorial to explain some basic functions, including how to change the time period for passphrase retention. Another solution would be to show (and be able to change) the passphrase timeout setting within the Setup Wizard or Key Generation process, also providing information about where it can be changed later.	`n19-tp5-reenter-passphrase.mp4`	`n19-tp5-reenter-passphrase.png`	TP5 00:13:50, TP5 00:15:20, TP5 00:17:05	wait for 5 minutes → send an encrypted/signed email or open one	2	1	2	2	1.75
20	Key Generation Progress Bar is Confusing	The progress bar displayed during key generation jumps around confusingly. It sometimes shows nearly completed, but then jumps back to the start [see HE N34].	`n20-tp1-prog-bar.mp4`		TP1 00:08:39	Setup Wizard → create key pair	2	2	1	1	1.50
21	Standard Encryption Settings	TP1 Stuart misunderstood "Convenient encryption settings" ["Bequeme Verschlüsselungs-Einstellung"] in the Enigmail Preferences window. The word "Convenient" ["Bequeme"] should be replaced by "Standard" ["Standard"].		`n21-tp1-bequeme.png`	TP1 00:46:04	Enigmail → Einstellungen → Bequeme Verschlüsselungs-Einstellung	1	1	3	1	1.50
22	Unintentionally Disabling Encryption	Enigmail switches encryption on by default when writing an email. TP2 Silvia unintentionally disabled encryption, thinking that she was turning it on. Maybe an even larger red cross over the encryption symbol would help [see HE N32].	`n22-tp2-dis-encrypt.mp4`		TP2 00:59:11	Write email and unintentionally disable encryption.	1	1	4	0	1.50
23	Enigmail Dropdown Menu (Mac and Windows)	On Linux, the Enigmail dropdown menu has clearly visible checkboxes for the options "Encrypt Message" and "Sign Message". On Mac and Windows, it is unclear that these are clickable options.		`n23-tp2-lin-check.png`, `n23-tp2-mac-check.png`, `n23-tp2-win-check.png`	TP2 00:51:49	Write → Enigmail	2	2	1	1	1.50

Table 8: Aggregated list of all problems found, in descending order of average severity. Timestamps refer to elapsed time from the start of the corresponding test user's session capture video (see Section C.3).

Code	Meaning
CE	Carina Enzinger
BH	Bianca Helbig
RW	Ralph Wozelka
JR	Jacqueline Rinnhofer

Table 9: The severity ratings assigned by each evaluator are identified by the evaluator's initials.

4.7 Interviews

Test users seemed to appreciate the simplicity of the registration process, as four out of five test users mentioned this during their interview. The following transcripts were translated from German to English for test users TP1, TP2 and TP3. Test users TP4 and TP5 spoke English during the test.

TP1 - Stuart

F: Would you want Enigmail → Einstellungen → "Bequeme Verschlüsselungs-Einstellung" to work differently?

TP1: Because of the phrase "verschlüsselt immer ohne Rückfragen" it should encrypt email even if the user does not have yet a key for the recipient, but it is okay how Enigmail does it at the moment. It is more a wording problem.

F: Which things did you particularly like or dislike?

TP1: I would like to see the signed and encrypted signs which are shown if you open an email, also in the email overview like it is with attachments. In the course of the installation there should be information and tips, with two or three screenshots of common first steps e.g.: Only if an encrypted email is clicked on the subject will be readable. You will not be informed when the key pair generation is finished. When there is help provided it is good, but in some places there is help missing.

F: (User was using the function "Enigmail" → "Schlüssel verwalten" → "Schlüsselserver" → "Schlüssel suchen..." after the previous question.) What are you searching for?

TP1: I am interested to know how the search works and therefore typed in "*@student.tugraz.at" to see if there are any results, but there are none.

TP2 - Silvia

F: Could you name some positives or negatives?

TP2: I think that it is handy to be able to encrypt. ("Ich glaub, dass es sicher sehr praktisch ist wenn man verschlüsseln kann.") I do not like that there is no symbol in the overview to mark encrypted emails. I do not like that in Enigmail it sometimes can be hard to find specific functions and or it is confusing. The red X on the encryption button helps, but at the beginning and without help it is not clear when the email is encrypted and when it is not.

F: Since the user wonders that she does not need to re-enter the passphrase for every email, the facilitator explains that a timer is set for 5 minutes and therefore the user does not need to re-enter the passphrase in the next 5 minutes to read encrypted email.

TP2: I think it saves time when somebody wants to quickly read a lot of emails.

F: Would you press on the turquoise/blue panel if facilitator did not ask you to?

TP2: No.

TP3 - John

F: I would ask you to tell us any positives, negatives or something that comes to your mind.

TP3: Basically Cryptography etc. was new to me - sadly. I needed to get into it first. Basically the concept was somewhat clear, but this was still something new to me to do it myself. Basically I'm not sure what the signature does. Encryption is what I think it is, but what is the signature for. Maybe it is something like a digital fingerprint or digital signature like the "Bürgerkarte" which you can use to sign pdf documents.

F: Is the information given by the system itself enough for you or should there be more?

TP3: Most of the things were easy, but some were unclear and I would need to read more about the topic anyway, but outside the system.

TP4 - Martin

F: Should the button "Search for missing keys" be somewhere else or would be a pop-up be better?

TP4: I think it would be easier to use if it is better to see for the user. Because, if you use encryption and you do not do it properly, then it is useless. Maybe it would be nice to have pop-ups like "tips" for the start and they should recommend the Wiki.

F: How was it? Was it easy to use?

TP4: Yeah, I think it is very convenient. Especially since you just have these two buttons. I can install it.

F: Is there something you would change?

TP4: It blends in well into the Thunderbird theme. It is well integrated and I like it. And I really like the idea of pressing buttons. I thought it would be more complicated.

TP5 - Sally

F: How was it?

TP4: I don't like it, to be honest. I don't like Thunderbird particularly in general, and this was just an extra overhead that I would never use.

4.8 Feedback Questionnaires

At the end of their session, each test user was given a feedback questionnaire, where they rated 12 criteria (questions) each on a seven-point scale. The original blank questionnaire can be found in Section A.6. Scans of the questionnaires as completed by each user can be found in Section B.4. The seven-point scale was provided in the form of semantic differentials as 3 2 1 0 1 2 3. These were later converted to points from 0 (worst) to 6 (best) for analysis.

Table 10 shows a summary of the ratings given by users in the feedback questionnaire at the end of the test. Most test users did not use the documentation provided by Enigmail, and therefore rated Question 4 neutrally with a "0", corresponding to 3 points in the analysis. The numbers in bold indicate the (rounded) mean rating.

					TP1	TP2	TP3	TP4	TP5	Av	Std Dev
1.	Installing Enigmail.	Very easy	6 5 4 3 2 1 0	Very hard	4	5	5	6	5	5.00	0.71
2.	Quality of Key Wizard.	Very good	6 5 4 3 2 1 0	Very poor	1	3	4	5	4	3.40	1.52
3.	It is easy to find help/tutorials.	Very easy	6 5 4 3 2 1 0	Very hard	3	5	2	5	3	3.60	1.34
4	The quality of documentation.	Very good	6 5 4 3 2 1 0	Very poor	4	6	3	4	0	4.30	1.26
5	Appearance of Enigmail.	Very good	6 5 4 3 2 1 0	Very poor	4	3	5	6	2	4.00	1.58
6	Consistency of the application.	Very consistent	6 5 4 3 2 1 0	Very inconsistent	5	4	2	5	2	3.60	1.52
7	How important is it to you to understand how exactly Enigmail works?	Very important	6 5 4 3 2 1 0	Not a bit	5	2	6	4	5	4.40	1.52
8	This application cares about my satisfaction as a client.	Yes, very much	6 5 4 3 2 1 0	No, not at all	4	3	3	4	1	3.00	1.22
9	How relevant are the features Enigmail provides to you?	Very relevant	6 5 4 3 2 1 0	Not relevant at all	1	5	4	5	0	3.00	2.35
10	Overall impression of application Enigmail.	Very good	6 5 4 3 2 1 0	Very bad	4	4	5	5	2	4.00	1.22
11	How likely are you to use this application later?	Definitely	6 5 4 3 2 1 0	Never	2	5	4	5	0	3.20	2.17

Table 10: Summary of user ratings from the feedback questionnaires.

References

[And2017a]: Keith Andrews; Human-Computer Interaction: Lecture Notes; 30 May 2017. http://courses.iicm.tugraz.at/hci/hci.pdf
[And2017b]: Keith Andrews; Materials for HCI Practicals 2017; Visited 2017-05-23. http://courses.iicm.tugraz.at/hci/practicals/materials/
[And2015]: Keith Andrews and Ralph Wozelka; EnigUsab: Improving the Usability of Enigmail; Visited 2017-01-08. http://projects.iicm.tugraz.at/enigusab/
[AT2017]: Audacity Team; Audacity; Visited 2017-06-07. http://audacityteam.org/
[Bar2010]: Carol M. Barnum; Usability Testing Essentials; Morgan Kaufmann, 2010. ISBN 012375092X. http://mkp.com/testingessentials
[Bru2017]: Patrick Brunschwig; Enigmail: A simple interface for OpenPGP email security; Visited 2017-01-08. https://enigmail.net/
[Enz2016]: Carina Yasmin Enzinger and Bianca Helbig; Heuristic Evaluation of Enigmail; Visited 2017-05-23. http://projects.iicm.tugraz.at/enigusab/he/he.html
[GPG2017]: The GnuPG Project; GnuPG: The GNU Privacy Guard; Also known as GPG. Visited 2017-01-08. https://gnupg.org/
[GIMP2017]: The GIMP Team; GNU Image Manipulation Program (GIMP); Visited 2017-06-06. https://gimp.org/
[Hin2014]: Kristi Hines; How to Fix a Read Only External Hard Drive on a Mac; 14 Feb 2014. https://boxaid.com/blog/fix-read-external-hard-drive-mac/
[KDL2017]: Bushuev A., Simon A. Eugster, Jean-Baptiste Mardelle, Roger Morton, and Vincent Pinon; kdenlive; KDE Non-Linear Video Editor. Visited 2017-06-06. https://kdenlive.org/
[Koh2017]: Volker Kohaupt; vokoscreen; Visited 2017-03-23. http://linuxecke.volkoh.de/vokoscreen/vokoscreen.html
[Kru2009]: Steve Krug; Rocket Surgery Made Easy: The Do-it-yourself Guide to Finding and Fixing Usability Problems; New Riders, 2009. ISBN 0321657292. http://sensible.com/rsme.html
[Nie1990]: Jakob Nielsen and Rolf Molich; Heuristic Evaluation of User Interfaces; Proc. Conference on Human Factors in Computing Systems (CHI’90). ACM. Seattle, Washington, USA, Apr 1990, pages 249–256. doi:10.1145/97243.97281
[Nie1994]: Jakob Nielsen; Enhancing the Exploratory Power of Usability Heuristics; Proc. Conference on Human Factors in Computing Systems (CHI’94). ACM. Boston, Massachusetts, USA, Apr 1994, pages 152–158. doi:10.1145/191666.191729
[Nie1995]: Jakob Nielsen; 10 Usability Heuristics for User Interface Design; Visited 2014-04-27. https://nngroup.com/articles/ten-usability-heuristics/
[Pad2014]: Arne Padmos; Why is GPG "damn near unusable"?; Visited 2016-04-27. https://media.ccc.de/v/31c3_-_6021_-_en_-_saal_g_-_201412281130_-_why_is_gpg_damn_near_unusable_-_arne_padmos
[Sno2014]: Edward Snowden; Speaking at HOPE X in 2014 with Ellsberg; Visited 2016-04-27. https://youtu.be/onrTo1sYri8
[TS2017a]: TechSmith; Camtasia; Visited 2017-01-08. http://camtasia.com/
[TS2017b]: TechSmith; Morae; Visited 2017-01-01. http://morae.com/
[TMF2017]: The Mozilla Foundation; Thunderbird; Visited 2017-01-08. https://mozilla.org/thunderbird/
[VL2017]: VideoLAN; VLC media player; Visited 2017-03-23. http://videolan.org/vlc/
[Whi1999]: Alma Whitten and J.D. Tygar; Why Johnny can't encrypt: a usability evaluation of PGP 5.0; SSYM'99 Proceedings of the 8th conference on USENIX Security Symposium. Volume 8. USENIX Association Berkeley, CA, USA, 1999. Visited 2016-04-25. https://people.eecs.berkeley.edu/~tygar/papers/Why_Johnny_Cant_Encrypt/USENIX.pdf
[Woz2016]: Ralph Wozelka, Keith Andrews, Carina Enzinger, Bianca Helbig, and Jacqueline Rinnhofer; (Un)Usability of Enigmail; Talk at Grazer Linuxtage 2016, on Sat 30 Apr 2016. http://keithandrews.com/talks/2016/glt-2016-enigusab/

A Test Team Materials

A.1 Checklist

The checklist checklist.html was used for the tests.

A.2 Orientation Script

The orientation script orient.html was used for the English tests. For the German tests the orientation script orient-de.html was used.

A.3 Background Questionnaire

The blank background questionnaire background.html was used in the English tests. For the German tests the blank background questionnaire background-de.html was used.

A.4 Consent Form

The blank consent form consent.html was used in the tests.

A.5 Task Slips

The task slips external-tasks-ta.pdf were presented to English-speaking test users. The task slips external-tasks-ta-de.pdf were presented to German-speaking test users.

A.6 Feedback Questionnaire

The blank feedback questionnaire feedback.html was used in the tests.

A.7 Data Spreadsheet

The data from the questionnaires and the other results were collated in the following spreadsheet: data-ta.xls

B User Materials

B.1 User Aliases

The mapping of test users' real names to their aliases is documented in an internal file.

B.2 Completed Background Questionnaires

For each test user, the completed background questionnaire was scanned as PDF:

B.3 Signed Consent Forms

Each test user had to sign a consent form and those were scanned as PDF.

B.4 Completed Feedback Questionnaires

For each test user, the completed feedback questionnaire was scanned as PDF:

B.5 Data Log Forms

For each test user, the log file written by the scribe was scanned as PDF:

C Test Videos

C.1 External Videos

For each user, the entire test was captured with a tripod-mounted external video camera.

C.2 Session Capture Videos

For users TP1 and TP2, the test session on the computer was captured with Camtasia [TS2017a] and then exported in MP4 format. For users TP3 and TP5, the test session on the computer was captured with Morae [TS2017b] and then exported in MP4 format. For user TP4, the test session on the computer was captured with vokoscreen [Koh2017] and then exported in MP4 format.

C.3 Session Capture Video Transcripts

The coding scheme for session capture video transcripts is shown in Table C.1. A chronological protocol of events during the test session for each test user is given in Tables C.2 through C.6. Times are given in hours, minutes, and seconds of elapsed time on the corresponding session capture video.

Marker (Code)	Colour (r,g,b)	Event
S	(142,175,215)	Start task.
E	(172,205,245)	End task.
N	(248,180,203)	Negative observation (problem).
P	(178,208,129)	Positive observation.
Q	(255,242,230)	Quote or comment from user.
X	(238,250,184)	Error, unexpected action.
H	(230,230,230)	User helped by facilitator.
T	(217,221,251)	Timeout, exceeded maximum time for task.
I	(255,211,149)	Probe during interview (probe mark).

Table C.1: Coding scheme for session capture transcripts.

Time hh:mm:ss	Marker	Event	Description
TP1 "Stuart"
00:00:11	S	Task 1: Install
00:00:20			Stuart opens the Firefox Browser and uses Google Search with the search term "enigmail".
00:00:31			Stuart clicks on the first link which leads him to "addons.mozilla.org". See positive finding P02.
00:02:01			Stuart notices the Setup Wizard window which was opened in the background.
00:02:25	Q		"The question is what will be included in the 'Standard-Konfiguration (Anfänger)'? I'll take the 'manuelle Konfiguration (Experten)' to see what and how it will be installed." See problem N10.
00:03:01	Q		"Because you can be foist with a lot of unwanted software, I like to check if there will be something additionally installed."
00:03:53			Stuart deselects the "Startmenü Verknüpfung".
00:04:25	E	End of Task 1.	Stuart finished task 1, but stays on the installation window to look at "Einstellungen".
00:06:33	S	Task 2: Generate Keys
00:08:39	N		The loading bar begins newly to load. Stuart notices that it will take some more time and therefore follows the recommendation “im Webbrowser aktiv zu surfen". See problem N20.
00:09:39	N		Stuart thought he would be informed when it is finished, because it states "Sie werden informiert, sobald der Schlüssel fertiggestellt ist.". See problem N14.
00:10:05	E	End of Task 2.	By saving revocation certificate.
00:10:50	S	Task 3: Send your public key
00:11:05	N		"I will check where the public key has been stored. It has been generated before, but I have not seen any information where it has been stored." See problem N05.
00:11:25			Stuart opens "Schlüssel verwalten" and finds "Datei → Öffentliche Schlüssel per E-Mail senden".
00:13:40	E	End of Task 3.	By sending an email with the public key attached to it.
00:13:51	N		Stuart notices that two identical attachments were sent. He checks his process of sending the public key and notices that when sending the email it automatically attaches the public key a second time. See problem N02.
00:15:30	S	Task 4: Add a public key	Receiving the email from the test team (sent from a TU Graz email address) took some time.
00:17:09	P		Stuart imports the public key sent by the test team by double-clicking on it. See positive finding P04.
00:17:10	E	End of Task 4.
00:19:41	S	Task 5: Keyserver
00:20:13	Q	Question from user (left unanswered by the facilitator).	"On which server should I search for the key?"
00:20:27	N		Stuart searches with the search term "Bianca Helbig". See problem N09.
00:20:57	N		Stuart searches with the search term "helbig". See problem N16.
00:21:19	N		Stuart adds the key with the newest creation date. When searching for the key of "Bianca Helbig", 3 out of 4 keys should actually be greyed out because they have been revoked. See problem N06.
00:21:20	E	End of Task 5.
00:22:12	S	Task 6: Send and get an encrypted email
00:22:43	E	End of Task 6.	Stuart uses the encryption button.
00:26:57	N		Stuart would like to see in the email overview list which emails are encrypted. See problem N15.
00:27:33	S	Task 7: Send and get a signed email
00:28:10	E	End of Task 7.	Stuart uses the signing button.
00:29:13	Q		"The signing symbol is different from the encryption symbol. In the previous email which was only encrypted, a lot of options were greyed out after clicking on "Details"."
00:30:13	N		"I would be interested what would make the email 'trustworthy' ('vertrauenswürdig'), because I think it should be 'trustworthy'." See problem N04.
00:30:55	Q		I am looking at the settings of "Schlüssel unterschreiben" (Details → Schlüssel unterschreiben...), to see if it might make the email 'trustworthy' ('vertrauenswürdig'). See problem N04.
00:31:23	N		"I can't figure out 'Sie müssen das Besitzservertrauen Ihrer eigenen Schlüssel auf "Absolut" setzen, damit sie hier angezeigt werden' at the moment." See problem N12.
00:31:40	Q		"I will see what happens if I sign this." Stuart signs the key from the test team with "Ich habe es nur einfach überprüft". See problem N12.
00:33:35	N		(Details → Vertrauen zu Besitzer festlegen) Stuart would be interested to know what exactly the different options do. "Help information would be interesting. Is this a filter? Do I always have to confirm the emails of contacts with 'kein Vertrauen'"?
00:35:26	S	Task 8: Send and get an encrypted and signed email
00:35:40	E	End of Task 8.	The encryption button was automatically activated when entering the email address. Stuart presses the signing button.
00:36:50	N		Stuart expected to be asked the passphrase when sending the email. See problem N05.
00:37:05	N		Enigmail changes the subject of the email to "Verschlüsselte Nachricht", causing Stuart to believe the email had not been sent. Stuart does not see the subject in the list of sent emails. "My email was not sent." Enigmail should tell users to turn on display of the Enigmail column in the overview or should automatically add it in the installation process, but it should be narrower than at the moment. Maybe then Stuart would have noticed the sent email. See problems N05 and N15.
00:37:53	X	unexpected behaviour	Stuart is wondering why he had to enter the passphrase to send a signed email, but not when previously sending an encrypted and signed email. See problem N05.
00:39:32	S	Task 9: Send an encrypted email
00:40:27	N		Stuart uses "Fehlende Schlüssel herunterladen" button, but it did not work. See problem N06.
00:40:27	E	End of Task 9.	Stuart would have fulfilled task very fast, if "Fehlende Schlüssel herunterladen" button or Keyserver search would have worked. See problem N06.
00:46:00	S	Start of interview.
00:46:04	N		Stuart misunderstood "Enigmail" → "Einstellungen" → "Bequeme Verschlüsselungs-Einstellung". "Because of the phrase 'verschlüsselt immer ohne Rückfragen' it should encrypt an email even if the user does not have a key for the recipient, but it is okay how Enigmail does it at the moment. It is more a wording problem." See problem N21.
00:47:45	N		Stuart wants to see the signed and encrypted symbols which are shown when an email is opened, also in the email overview. See problem N15.
00:49:25	N		Stuart suggests that during installation there should be information and tips with 2 or 3 screenshots of common first steps like: Only if an encrypted email is opened will the subject be readable. See problem N05.
00:49:57	N		Stuart does not like that the user is not informed when key pair generation is finished. See problem N14.
00:51:00	N		Stuart is interested in exploring how search on the key server works. He types in "*@student.tugraz.at" to see if there are any results, but no results are shown. Stuart concludes that the search function can not search by domain. See problem N09.
	Q		"Where help is offered it is good, but help is missing in some places."
			Generally, Stuart always reads the displayed text very carefully, something other users probably would not do.
			The Feedback Questions about information and consistency are hard for Stuart to answer, because he thinks he did not give it enough attention to answer meaningfully.
00:51:00	E	End of interview.

Table C.2: Transcript of the session capture video for user TP1 "Stuart".

Time hh:mm:ss	Marker	Event	Description
TP2 "Silvia"
00:00:10	S	Task 1: Install
00:01:21	H		Silvia do not really know what an Add-on is, so the facilitator says that she use should Google "enigmail".
00:03:35	Q		Silvia finds Enigmail through the Mozilla foundation website. "Ahhh, with it I encrypt my Thunderbird." Although the facilitator said at the beginning that Enigmail is a security app, it seems that the user did not know that Enigmail is a security app. See positive finding P02.
00:06:04			Silvia chooses the configuration for beginners in the Enigmail Setup Wizard.
00:07:20 - 00:09:45			Silvia's baby begins to cry. Since the installation process is still ongoing, Silvia leaves the laptop.
00:10:15	Q		"Does it always take that long?" See problem N07.
00:10:24			The facilitator thought that the Enigmail Setup Wizard would ask the user to browse the web during the installation, as it does when generating a key pair and therefore the facilitator gives Silvia the "hint" to read the text.
00:10:28	X		The progress bar is fully loaded and there are not any clickable buttons accept "Cancel" and "Go Back". Silvia does not know what to do. Most likely because of the "hint" Silvia thought that she would have to click something and so she clicked "Go Back", but immediately returned with the "Continue" button. See problem N07.
00:11:05			Silvia returns to the main Thunderbird window.
00:11:19	X		Silvia switches to the GPG Suite window, but switches straight back to Thunderbird, maybe because she did not know that it has something to do with Enigmail. See problem N07.
00:11:31	X		Silvia is asked to think aloud and says that she is thinking how she should finish the installation process, because it was finished with loading, but she could not press the Continue button. See problem N07.
00:11:52			Silvia could not find the Enigmail installation window any more. See problem N07.
00:12:10	E	End of Task 1.
00:12:13	S	Task 2: Generate Keys	The facilitator did not see the GPG Suite window and therefore thought that Enigmail was finished with the installation process and reads the next task. See problem N07.
00:12:28			Silvia searches for help with Google Search, types in "Schlüssel generieren Thunderbird" and clicks on the first link from `verbraucher-sicher-online.de`. See problem N05.
00:13:54			Silvia thinks she has installed one of the recommended GNU Privacy Guard programs from the `verbraucher-sicher-online.de` site.
00:14:31			Silvia tries to follow the path shown in the screenshot at `verbraucher-sicher-online.de`, but the screenshot is to blurred to read properly.
00:15:00			Silvia tries to find a Mac-specific help/tutorial and uses Google Search with the search term "Schlüssel generieren Thunderbird mac".
00:15:15	H		The search phrase used by Silvia had no "enigmail" reference, so the facilitator tells the user that the key generating process is done in Enigmail. The facilitator hoped that Silvia will add "enigmail" to the search phrase.
00:16:13	H		Silvia could not find out how to generate a key pair, therefore the facilitator showed her how to open the Key Management. See problem N05.
00:16:25	H		The facilitator told Silvia to look at the menu bar. See problem N18.
00:18:01	X		Enigmail was not installed properly, therefore key generation does not work. Enigmail should tell the user why exactly the key generation step did not work. See problem N07.
00:18:48	H		Due to the problems above, the facilitator suggests a new installation of Enigmail. See problem N07.
00:23:46			After clicking the "Restart Now" button, and after the new installation of Enigmail, the GPG Suite installation window became visible and Silvia starts with the installation process from GPG Suite. See problem N07.
00:24:08	X		The "In order to finish the installation of GPG Mail we have to quit Mail" error occurred. See problem N07.
00:24:21			GPG Keychain opens "Erstellen Sie ein neues Schlüsselpaar" window. Silvia enters the requested information.
00:25:26	H		Silvia clicks in the GPG Keychain window on "Neu" to generate a new key and would have completed Task 2 if the installation process (prerequisites) had worked. Silvia already generated a key pair (TP2 00:24:21). The facilitator informs her that she already did it and does not have to do it again.
00:26:20	E	End of Task 2.
00:26:20	S	Task 3: Send your public key
00:27:00	H		Silvia does not realise she can send the public key though the normal email sending process. Since she cannot find the previous window with the key pair, the facilitator tells her again to look at the menu bar. See problem N05.
00:27:41			Silvia randomly right clicks in the Key Management window, because she remembered that a "Send Public Keys by Email" option will appear. Unfortunately, she sends the public key belonging to the GPG Tools team.
00:28:08	X		Without any noticeable problem the email with the public key from the GPG Tool team cannot be sent. The Send button is greyed out. First attempt to send an email. See problem N07.
00:28:55 - 00:31:44	X		After the break the Send button is no longer greyed out. See problem N07.
00:32:22	X		It seams that sending the public key of the GPG Tool team (0x00D026C4.asc) worked, but for some raeson the test team did not receive the email. Second attempt to send an email. See problem N07.
00:32:50 - 00:36:13		"baby sleeping break"	Silvia does not had much time left for the testing so the facilitator helps her from now on more frequently and earlier than before.
00:36:36	X		Once again, sending the public key of GPG Tool team does not work. Third attempt to send an email. See problem N07.
00:38:37	X		After receiving a special email from the test team (00:35:10) with her public key attached to it, the public key from the test team was automatically added by the Key Management. In between receiving the email and the adding public key in the Key Management, Thunderbird crashed (00:38:09). Before the crash (00:37:24), the public key was not added in the Key Management.
00:38:41	H		The facilitator points out that the sent public key was not from Silvia. Silvia sends her public key by right-clicking on her name in the Key Management.
00:39:11	X		Sending the email does not work and the "Send" button is now greyed out again. Fourth attempt to send an email. See problem N07.
00:39:23	X		Silvia switches her "from" email address and adds to the subject line "Öffentl. Schlüssel". Once again sending the email does not work. Fifth attempt to send an email. See problem N07.
00:40:39	H		Because of the problems of unsuccessfully sending emails, Silvia and the facilitator decide to deinstall Enigmail and reinstall it again. After deinstalling and clicking on the recommended "Restart Now" button, Mozilla Thunderbird crashes. See problem N07.
00:42:49	N		After installing and clicking on the recommended "Restart Now" button, it seems that the installation finally worked. See problem N07.
00:43:12	H		The facilitator also recommends to remove the already generated key pair from the Enigmail Key Management.
00:43:31			A new key pair was generated in the Enigmail Key Management.
00:44:17	Q		"This was not asked previously." Silvia notices that now Enigmail generates a certificate and asks for a passphrase. Previously when generating a key pair in GPG Suite (TP2 00:24:21), it did not ask about creating a certificate. Maybe Silvia did not notice that this was a different kind of key generating tool than before. Now she used Enigmail and before she used GPG Suite. See problem N17.
00:44:21	X		Enigmail Alert "The revocation certificate could not be created." Maybe it occurred because Silvia tried to save the certificate on a hard disk which could be written to. This could be fixed by installing an NTFS driver for MacOS from Seagate [Hin2014].
00:44:32	E	End of Task 3.	Silvia sends the public key though the option "Send Public Key by Email" from the key management. Sixth attempt to send an email, which finally worked. See problem N07.
00:45:11	S	Task 4: Add a public key
00:46:10	E	End of Task 4.	Silvia imported the public key of the test team by clicking on the email attachment. See positive finding P04.
00:47:02	S	Task 5: Keyserver
00:47:30	H		Silvia looks for an option to search for persons in the Key Management. She did not realise that she could find a key by the name of the person with the function "Search for Keys". The facilitator told her to use this search function. See problem N09.
00:48:39			The ordering of the key search results makes it hard to find the public key belonging to "Bianca Helbig". See problem N16.
00:49:25	E		Silvia clicks on the key which is not greyed out and adds the key.
00:49:36	S	Task 6: Send and get an encrypted email
00:50:27	H		The facilitator tells Silvia to think what she would normally do when she wants to send an email and then she should start to look for encryption. See problem N05.
00:51:49			Silvia looks through the Enigmail menu in the menu bar, but does not think that by clicking on "Message will not be encrypted" would encrypt the message. See problem N23.
00:52:11	H		The facilitator tells Silvia to look at the padlock symbol to the right of the Attach button and that she should choose the Gmail account as a "from" account. Silvia deselected the encryption button, because it was automatically activated when selecting the right "from" account, but maybe only because the facilitator told her to focus on that button. See problem N05.
00:53:10	E		At the second attempt and with the help of the facilitator Silvia sends an encrypted email to the test team.
00:53:30	S	Task 7: Send and get a signed email	The facilitator tells Silvia that she could sign an email with the button to the right of the encryption button. Silvia clicks on the greyed out button. See problem N05.
00:54:05	E		The facilitator tells her to select the correct "from" email. See problem N05.
00:54:13	S	Task 8: Send and get an encrypted and signed email
00:54:33	E		The facilitator forgot to ask Silvia to look at the answers from task 6, 7 and 8. The review of the incoming emails was done during the interview at 00:58:00.
00:54:42	S	Task 9: Send an encrypted email
00:55:23	H		The window "enigusab@gmx.at: No valid key" appears. Silvia chooses all keys for the encryption. The facilitator tells her to look at the window and to look for a button with which she could obtain the missing key. Silvia seems lost, so the facilitator tells her to click on the "Download missing keys" button. See problem N10.
00:56:15	E	End of Task 9.	Silvia asks "How should I know that ...?" Silvia does not know how encryption works, so for users like her it would be really useful for Enigmail to give an introduction to asymmetric encryption. See problem N10.
00:57:00	S	Start of interview.
00:57:00	P		"I think that it is really useful to be able to encrypt emails." See positive finding P01.
00:58:00	N		Silvia does not like that the encrypted emails are not marked as encrypted in the email overview. See problem N15.
00:58:13	P		Silvia wonders that she does not need to re-enter the passphrase for every email, therefore the facilitator explains that a timer for 5 minutes is set and the user does not need to re-enter the passphrase in the next 5 minutes to read encrypted email. Silvia thinks it saves effort when somebody wants to quickly read many emails. See problem N05 and positive finding P06.
00:58:57	N		Silvia does not like that it is sometimes hard to find specific functions and or it is sometimes confusing. See problem N10.
00:59:11	N		"The red X on the encryption button helps, but at the beginning and without help it is not clear when the email is encrypted and when it is not." See problem N22.
00:59:52			The baby wakes up and the facilitator decides not to ask about the information panel in encrypted or signed emails (Web of Trust). During the interview Silvia tells the facilitator that she would not press on the turquoise/blue panel on her own.
00:59:52	E	End of interview.

Table C.3: Transcript of the session capture video for user TP2 "Silvia".

Time hh:mm:ss	Marker	Event	Description
TP3 "John"
00:00:10	S	Task 1: Install
00:00:18			John goes to the Add-ons section in the Thunderbird settings and searches for an Add-on with the term "enigmail".
00:00:39			John clicks on the first search result named Enigmail in the list to install it.
00:01:34			The Enigmail Setup Wizard pops up as soon as John restarted Thunderbird. The Thunderbird program itself stays in the background.
00:01:59	Q		"Ich würd' auf alle Fälle, weil ich das meistens so mache, eine manuelle Konfiguration machen." "I would - because I most do so - do a manual configuration."
00:03:50			Enigmail Setup Wizard tells John where he can find the key manager and the expert settings. John is confused because he thinks he has to search for a key. He misunderstands a comment from the facilitator.
00:05:00			John stops expert mode, goes back to choose beginner's mode and restarts.
00:05:18	E	End of Task 1.	Enigmail installation is finished and the key generation part starts.
00:05:19	S	Task 2: Generate Keys
00:05:29			John reads the information about generating a key pair in the Setup Wizard's beginner's mode.
00:08:04			John creates the revocation certificate.
00:08:53	X		John is confused because he finished the setup but there is no button to close the wizard. See problem N03.
00:09:24	Q		"Das ich jetzt eine Art Fertigstellen-Button oder so hab, damit das dann automatisch zugeht und dann eben das Hauptfenster angezeigt bekomm von Thunderbird." "That I have some kind of a Finish button that the wizard closes automatically and then shows me the main window of Thunderbird."
00:10:04	E	End of Task 2.
00:10:12	S	Task 3: Send your public key
00:12:19	H		Facilitator explains to John that send and attach the public key means the same thing.
00:13:37	E	End of Task 3.
00:13:47	S	Task 4: Add a public key
00:14:58			John saves the public key from the test team on the hard disk by clicking on Save.
00:15:20			John chooses Enigmail in the Thunderbird menu and clicks on the Key Manager. He imports the saved key into the key management.
00:16:08	E	End of Task 4.
00:17:40	S	Task 5: Keyserver
00:18:34	N		John opens the search window for the keyserver, but dismisses it because there is just a fill out line with "Search for keys". See problem N09.
00:19:00	H		The facilitator tells John that "Search for keys" was in fact the correct way.
00:19:13	Q		"Irgendwie plausibel, aber ich hab jetzt keinen Anhaltspunkt dafür, dass ich jetzt nach Namen suchen kann." "Somehow plausible, but I did not realise that I can now search by name."
00:19:52	N		The key server did not grey out the revoked keys belonging to the facilitator. See problem N06.
00:20:20	E	End of Task 5.
00:20:33	S	Task 6: Send and get an encrypted email
00:23:05	E	End of Task 6.
00:23:15	S	Task 7: Send and get a signed email
00:25:15	N		John is irritated by the wording "NOT TRUSTED Correct Signature" ("NICHT VERTRAUENSWÜRDIGE Korreke Unterschrift") and tries to look for more details or an explanation. See problem N04.
00:25:33	Q		"Mir ist nicht ganz klar, was einerseits das nicht vertrauenswürdig aber andererseits das korrekt heißt." "I am not sure, what on the one hand not trusted but on the other hand correct means."
00:28:22	E	End of Task 7.
00:28:30	S	Task 8: Send and get an encrypted and signed email
00:30:11	E	End of Task 8.
00:30:18	S	Task 9: Send an encrypted email
00:31:22	N		John does not see the "Download missing keys" button, and tries to send the email without selecting anything. The pop-up window just tells him to choose at least one key, so John simply chooses his own public key. See problem N13.
00:31:34	H		The facilitator tells John that he needs to get the public key of the other person to be able to encrypt correctly.
00:32:50			John downloads the missing key from the keyserver like he had to do in Task 5.
00:33:31	E	End of Task 9.
00:33:35	H		The facilitator tells John about the Download button.
00:34:15	S	Start of interview
00:34:40			John has heard ot the concept of cryptography, but did not really understand how it works until now. This was completely new to him.
00:35:18			John is not sure, what the signature feature is for, but equates it to signing with the Citizen's Card ("Bürgerkarte").
00:37:06			John would like to research the topic more, but would not necessarily expect Enigmail to inform him. He would search for further information elsewhere.

Table C.4: Transcript of the session capture video for user TP3 "John".

Time hh:mm:ss	Marker	Event	Description
TP4 "Martin"
00:00:15	S	Task 1: Install
00:00:40			Martin searches "enigma thunderbird" on Google and notices that Enigmail can be obtained through the Ubuntu software center.
00:03:09	E	End of Task 1.
00:03:15	S	Task 2: Generate Keys
00:06:35	N		"Where can I save it?" Martin is unsure where to save the certificate. He would definitely give it more thought, if it were not a test. See problem N08.
00:07:21	E	End of Task 2.
00:07:30	S	Task 3: Send your public key
00:08:21	P		Martin searches for help at Google with the search term "enigmail public key" and finds the Enigmail Wiki. See positive finding P03.
00:11:42	E	End of Task 3.
00:12:05	S	Task 4: Add a public key
00:13:47	E	End of Task 4.
00:13:55	S	Task 5: Keyserver
00:14:36	P		Martin uses Google to look for more information about the keyserver with the search term "enigmail thunderbird keyserver get key" and is directed to the Enigmail Wiki again. See positive finding P03.
00:17:43	X		The menu bar belonging to Key Management is not at the top of the smaller window itself, but like on a Mac at the top of the screen. Martin has to hover over the menu bar to see the menu buttons. See problem N18.
00:18:42	N		Martin does not know which keyserver he should use and what he can search for. See problem N09.
00:21:13	X		Martin is confused because of the long list of results and cannot figure out how exactly it is sorted. See problem N16.
00:22:50			"I think it is a good way to order it. Just makes it a bit slower." If the user does not know how the other person has named their account.
00:23:00	E	End of Task 5.
00:23:40	S	Task 6: Send and get an encrypted email
00:23:51	P		Martin uses the Enigmail Wiki again to inform himself about sending encrypted email. See positive findings P03.
00:24:45	P		Enigmail encrypts automatically. Martin thinks that this is very convenient. See positive finding P07.
00:26:30	E	End of Task 6.
00:26:42	S	Task 7: Send and get a signed email
00:26:56	P		Martin says that the buttons to encrypt and sign are very convenient to use. See positive finding P05.
00:27:19	Q		"So I think it is very convenient. I don't have to do anything and I can just press the buttons and it's all done for me."
00:27:36	Q		"And apparently there is a documentation about it which is very well done for anybody."
00:27:36	P		Martin really likes the Enigmail Wiki. See positive finding P03.
00:30:22	Q		"I know a little bit about the Web of Trust. I think there are three or four stages. Any user is allowed to trust/to sign other signatures and apparently the signature of Bianca wasn't signed by anybody yet."
00:30:59	P		Martin checks the Enigmail Wiki for a documentation about the signature details and thinks it is well documented there. See positive finding P03.
00:32:01	E	End of Task 7.
00:32:07	S	Task 8: Send and get an encrypted and signed email
00:33:06	E	End of Task 8.
00:33:17	S	Task 9: Send an encrypted email
00:34:20	X		Martin is confused that a window pops up. The error message just tells him that he should choose a key. Martin chooses his own public key to encrypt the email with. See problem N13.
00:34:45	X		The wrongly encrypted email is sent without any warning that it does not make sense. See problem N01.
00:34:57	H		The facilitator explains to Martin that he just encrypted with his public key and why that would not work.
00:36:54			Martin now finds the "Download missing keys" button and can send the email correctly this time.
00:37:04	E	End of Task 9.
00:37:52	Q		"I would have thought if I press the button [send] and Enigmail didn't find it, it wouldn't find it here [keyserver] either."
00:38:10	S	Start of interview
00:39:08	N		Martin would like to have some pop-up help or similar for the start. See problem N05.
00:39:57	P		Martin says Enigmail was very convenient to use. See positive finding P05.
00:40:01	P		Martin thinks it was very easy to install the software. See positive finding P02.
00:40:29	P		Martin says the Enigmail add-on blends well into the Thunderbird theme. See positive finding P08.
00:40:34	P		Martin likes that he just has to press the buttons and nothing more complicated. See positive finding P05.
00:42:40	N		The facilitator tells Martin that his public key is automatically sent as an attachment with every email he writes. Martin thinks that it would be annoying to get an attachment with every email. See problem N02.
00:43:07			The facilitator tells Martin that he can deactivate the attachment of the public key, but has to do this every time he writes an email.
00:45:18	N		Martin thinks that no user wants to be used as an advertising tool. See problem N02.
00:45:18	E	End of Interview.

Table C.5: Transcript of the session capture video for user TP4 "Martin".

Time hh:mm:ss	Marker	Event	Description
TP5 "Sally"
00:00:15	S	Task 1: Install
00:00:24			Sally thinks that Enigmail must be a plug-in or something similar, so searches for it in the Thunderbird Add-ons section where she finds it.
00:00:54	E	End of Task 1.
00:02:15	S	Task 2: Generate Keys
00:02:37			Sally chooses the standard configuration for beginners.
00:02:56	Q		"I usually use a password manager and I hate having to think of new passwords."
00:03:57	Q		"I would need a button to see what I type 'cause I'm used to working with a password manager and it is always letting me see what my password looks like."
00:05:08	N		"Where do I save it?" See problem N08.
00:05:46	E	End of Task 2.
00:05:53	S	Task 3: Send your public key
00:08:10	E	End of Task 3.
00:08:20	S	Task 4: Add a public key
00:08:50			Sally saves her public key on the hard disk and then imports it into the Key Manager.
00:09:34	E	End of Task 4.
00:11:27	S	Task 5: Keyserver
00:12:14	N		"Okay, why did it find...? Oh my god, it found so many! Maybe I should have entered it with quotation marks." See problem N16.
00:12:54	E	End of Task 5.
00:13:00	S	Task 6: Send and get an encrypted email
00:13:54	N		"Why do I have to enter the passphrase now? Did I do something wrong? Is it 'cause of sending the message? Strange." See problem N19.
00:14:32	Q		"So Enigmail basically did the decryption for me."
00:14:48			Sally does not like that Enigmail decrypts the email automatically without asking her first.
00:14:55	E	End of Task 6.
00:15:01	S	Task 7: Send and get a signed email
00:15:20	N		"Why do I need to enter the passphrase every time? It's kind of annoying!" See problem N19.
00:16:06	Q		"Enigmail tells me that it's a not trusted... it's a correct signature from Bianca but you can't trust it. You shouldn't trust it."
00:16:27	N		"It tells me somehow to not trust Bianca when I see this." See problem N04.
00:16:33	E	End of Task 7.
00:16:40	S	Task 8: Send and get an encrypted and signed email
00:17:00	N		"Why does it always send an attachment? That's my key I guess." See problem N02.
00:17:05	N		"And now it didn't ask me for a passphrase. It's strange. What does it do? Surely random." See problem N19.
00:17:30	E	End of Task 8.
00:17:40	S	Task 9: Send an encrypted email
00:18:50	E	End of Task 9.
00:19:00	S	Start of Interview.
00:19:04			Sally does not like Thunderbird in general and would not use Enigmail.
00:20:04			Sally cannot answer the question about the documentation because she did use any.
00:20:32	N		Sally missed an introductory tutorial like some other software has. See problem N05.
00:20:42			Sally thinks it would be better if she would know more about how encryption works. She thinks if you understand how it works, it would make more sense and would be more useful.
00:20:42	E	End of Interview.

Table C.6: Transcript of the session capture video for user TP5 "Sally".

	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Task 7	Task 8	Task 9
TP1	1	1	1	1	1	1	1	1	1
TP2	1	1*	1*	1	1*	1*	1*	1	1*
TP3	1	1	1	1	1*	1	1	1	1*
TP4	1	1	1	1	1	1	1	1	1*
TP5	1	1	1	1	1	1	1	1	1

	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Task 7	Task 8	Task 9
TP1	1	1	1	1	1	1	1	1	1
TP2	1	1*	1*	1	1*	1*	1*	1	1*
TP3	1	1	1	1	1*	1	1	1	1*
TP4	1	1	1	1	1	1	1	1	1*
TP5	1	1	1	1	1	1	1	1	1

	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Task 7	Task 8	Task 9
TP1	1	1	1	1	1	1	1	1	1
TP2	1	1*	1*	1	1*	1*	1*	1	1*
TP3	1	1	1	1	1*	1	1	1	1*
TP4	1	1	1	1	1	1	1	1	1*
TP5	1	1	1	1	1	1	1	1	1