Cerina User Study
The main purposes were to investigate the changes in anxiety, depression, and general functioning scores over time among employees who used Cerina for six weeks.
The study was designed as a single-group feasibility open trial where all eligible participants have access to the Cerina application for six weeks (See table 1). The main purposes were to investigate the changes in anxiety, depression, and general functioning scores over time among employees who used Cerina for six weeks. Other purposes of the pilot study were: 1) to explore the usability (i.e. strengths and weaknesses (e.g. user errors, glitches)of the User Interface (UI) of the app, usefulness, and acceptability) of Cerina and 2) to learn about users` views on the clinical content and potential suggestions for improvement. Participants were asked to fill in online self-report measures of anxiety, depression, and general functioning baseline (t0) and at 6 weeks (t1) (see appendix 1 for study materials). Additionally, they filled out the SUS questionnaire and semi-structured interviews were conducted with the participants at 6 weeks (t1) to explore which elements of the intervention facilitated or hindered participants` progress during their participation and how the intervention can be optimised to increase its acceptability, relevance, and user-friendliness in real-life settings (see appendix 2 for the topic guide).
Table 1. Measures collected and the time point of data collection
Outcome measures Baseline (t0) 6 weeks (t1)
GAD7 X X
PHQ-9 X X
WSAS X X
SUS X
Semi-structured interviews X
Participants were employees of Tag Worldwide (a big corporate specialised in marketing production) and National Care Group (NCG) (one of the UK`s leading providers of care and support services to vulnerable adults).
Key results
The users have been recruited from the employee networks of Tag Worldwide and National Care Group (NCG). The recruitment started in April 2022 and up until now, 118 employees registered their
interest in using Cerina, 59 (Tag N=24, NCG N=35) filled out their pre-assessment questionnaire, 43 downloaded Cerina, and 20 (Tag N=11, NCG N=9) completed their post-assessment questionnaire. Of all these 20 people, the majority (89%) were female, aged between 30 and 45 (37%) and 18 and 30 (37%), and had an undergraduate degree (42%). The majority of them (67%) were not receiving psychological support at the time of the study and identified friends as their main support network. Additionally, about half of them were on medication for the past 6 months for a physical and/or psychological problem.
Overall, users had mild anxiety (M=10.15, SD=5.75), moderate depression (M=13.52, SD=8.22), and significant functional impairment (M=19.10, SD=11.00) based on their scores on the pre-assessment questionnaire. The initial comparisons of pre and post-assessment questionnaires indicated that there is an overall, but non-significant reduction in anxiety [t(18)=0.76, p=0.46>0.05] and depression [t(16)=0.72, p=0.48>0.05] and significant improvement in general functioning scores [t(180=0.03<0.05] over 6 weeks (see Appendix B). Due to the small sample size, bootstrapping, using 1000 bootstrapped samples, was used for the analyses as implemented in the SPSS version 28.
The usability of Cerina was acceptable (M=70, SD=17.37) based on the users` scores on the System Usability Scale. We conducted interviews to obtain more in-depth information on the specific aspects of usability (see appendix 3 for the main themes). The initial feedback from the interviews with the users pointed out that feeling validated was one of the main potential therapeutic benefits. One user talked about Cerina as if Cerina was a person. She mentioned that she identified herself with a case example about an employee who overprepared him/herself for a presentation. Finding the content relevant helped her to continue the sessions. Another user, who was seeing a therapist for her anxiety problems, said that using Cerina was re-assuring, and Cerina gave her the flexibility to practice anxiety management techniques in her own time and space (see appendix 4 for user testimonials):
“Cerina was very hands-on and it gave me the flexibility to work on my own anxiety, whereas therapy gives me a space to talk about it… It was very convenient that things we were talking about during therapy such as breathing techniques were available on Cerina and I was able to practice them between the sessions” User X, female, Tag
In terms of the usability of Cerina, convenience and flexibility were the main personal facilitators, whereas the simple UI and friendly language were highlighted as the main app-related facilitators whilst using Cerina. There were also barriers restricting the use of Cerina in daily life. The main personal barrier was forgetting to use the app due to workload or not feeling particularly anxious during that week. Also, a few technical problems were named such as not receiving notifications about an action plan for a particular exercise, issues with the navigation (i.e. frozen screen), and the high volume whilst listening to the audio of some of the anxiety management techniques.
There were also some suggestions to improve the usability of Cerina. These were mainly about making Cerina more interactive by incorporating pop-up reminders for using it more frequently and personalising the pop-up notifications. For instance, one user said that she would like it if Cerina realises when she feels bad and sends her a notification directing her to a specific session and/or an exercise momentarily:
“When I feel bad, Cerina could advise me to go to a session or practice a specific technique…I would like it if the notifications are personalised depending on how I feel” User Y, female, Tag
Other suggestions included adding more sessions and exercises representative of the holistic perspective (e.g. mindfulness exercises) and being able to use Cerina on different platforms (i.e. desktop version, mobile version) in future versions.
Key takeaways
It is important to note that the first employees who completed their assessments had mild clinical symptoms, but their perceived functional impairment in their day-to-day life was more severe. It is known that the impact of having anxiety and depression symptoms (irrespective of the work environment) accounted for almost 18 million lost working days among employees (PHE, 2019). The recent meta-analytic studies also indicated that COVID-19 pandemic and its implications in work life such as decreased employment security, and the imposed demands of working irregular hours resulted in increased perceived impairment and psychological burn-out as well as a higher prevalence of common mental health conditions among employees (Schelhorn et al., 2021; Global Burden of Disease, 2022). Therefore, psychological interventions implemented in the workplace should not only target observable symptoms but also employees` perspectives on the impairment of daily functioning. In this context, if incorporated into an employee well-being programme, Cerina could be beneficial, especially for employees with mild to moderate symptoms of generalised anxiety and other clinical symptoms. Also, it could potentially help improve the manner in which observable symptoms impair the ability to function day to day.
To summarise, these initial results highlight Cerina`s usability and its potential therapeutic benefits. They should be considered in light of the limitations. The small sample size and underrepresentation of males restrict the conclusions drawn from the study. We also did not control for the life events which might have affected the questionnaire scores. Nonetheless, the initial feedback highlighting the usability of Cerina promises its potential to engage with the target users and strengthens its likely clinical effectiveness in future trials.
Key References:
Bangor, A., Kortum, P. T. & Miller, J. T. (2008) An Empirical Evaluation of the System Usability Scale, International Journal of Human–Computer Interaction, 24:6, 574-594, https://doi.org/10.1080/10447310802205776.
Mental, G. B. D., & Collaborators, D. (2022). Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. The Lancet Psychiatry, 9(2), 137–150. https://doi.org/10.1016/s2215-0366(21)00395-3
Mundt, J., Marks, I., Shear, M., & Greist, J. (2002). The Work and Social Adjustment Scale: A simple measure of impairment in functioning. British Journal of Psychiatry, 180(5), 461-464. doi:10.1192/bjp.180.5.461
PHE document - Commissioning_effective_mental_health_prevention_report.pdf. (n.d.). Work-related stress, anxiety or depression statistics in Great Britain. (2019). March. https://www.hse.gov.uk/statistics/causdis/stress.pdf
Schelhorn, I., Ecker, A., Lüdtke, M. N., Rehm, S., Tran, T., Bereznai, J. L., Meyer, M. L., Sütterlin, S., Kinateder, M., Lugo, R. G., & Shiban, Y. (2021). Psychological Burden During the COVID-19 Pandemic in Germany. In Frontiers in Psychology (Vol. 12). https://doi.org/10.3389/fpsyg.2021.640518
Spitzer RL, Kroenke K, Williams JBW, Löwe B. (2006). A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7. Arch Intern Med.166(10):1092–1097. doi:10.1001/archinte.166.10.1092.
Spitzer, R. L., Kroenke, K., & Williams, J. B. (1999). Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA, 282(18), 1737–1744.
Appendix 1. Study Materials
Materials
Primary outcome measure
1. The Generalised Anxiety Disorder Scale-7 (GAD-7) (Spitzer et al., 2006) is a 7-item self-report scale that identifies and measures the severity of GAD. Scores range from 0 to 21, with a cut-off score of 5 distinguishing between clinical and non-clinical populations. The scale has good psychometric properties (Spitzer et al., 2006; Lovell et al., 2017).
Secondary outcome measures
2. The Patient Health Questionnaire (PHQ-9) (Spitzer, Kroenke & Williams, 1999) is a 9-item self-report scale that measures depression symptoms. Scores range from 0 to 27, with a score of 10 and above considered to be a clinically significant level of depression. The PHQ-9 has good reliability and validity Spitzer, Kroenke & Williams, 1999).
3. The Work and Social Adjustment Scale (WSAS) (Mundt et al., 2002) is a 5 item self-report measure that assesses functional impairment. Scores range from 0 to 40. The scale assesses the impact on work, home, social and private activities, and personal and family relationships. A score 20 and above is considered to indicate severe functional impairment, scores between 10 and 20 suggest severe but functional impairment, and scores of 10 and less are considered subclinical. The scale has good reliability and validity (Mundt et al., 2002).
4. The usability of Cerina will be measured with the System Usability Scale (SUS) and with semi-structured interviews. The SUS is composed of 10 statements that are scored on a 5-point scale of the extent of agreement (score 0 to 100). The reliability is good (Cronbach's alpha 0.91) (Bangor, Kortum & Miller, 2008). Interventions with scores of 70 and above are accepted as highly usable (Bangor et al., 2008) and scores between 50 and 70 indicate acceptable usability of an intervention. Interventions with scores of 50 and below are subject to concerns about their usability by the target population and should be investigated further (Bangor et al., 2008).
Appendix 2. Topic Guide for the Semi-Structured Interviews
1. What did you think of the device overall? Usability
a. Did you find any parts complex?
b. On a scale of 1-10 how easy did you find the app to navigate?
c. How often did you encounter bugs and glitches whilst navigating through the app?
d. On a scale of 1-10 how would you rate the look and feel of the app?
e. Do you have any suggestions for what could we do differently to make it easier to use?
f. Do you have any suggestions for what could we do to make it look better?
g. Since your first time using the app, did you notice any changes, can you give any examples?
2. What words or feelings come to mind when you think about Cerina? Branding/ customer experience
a. In terms of information provided by the app did you feel confident in using the app effectively?
b. Do you have any suggestions when it comes to content for the app?
c. Would you recommend this app to a friend who suffers with anxiety problems?
2. Clinical Questions
1. How much time did you spend on the app per day or week? Did you complete all the sessions and how long did each session take you? If not, how far did you get? Why did you not complete?
2. Does the app help you in a way that you had expected? If not, what was missing or what exceeded your expectations?
3. Do you feel Cerina understands individuals with anxiety problems?
4. Is there anything that Cerina got wrong, did you feel misunderstood or any parts of the app felt irrelevant for your anxiety problems? (Please describe)
5. Do you think Cerina helped you in managing your anxiety problems?
If ‘yes’, please describe which aspects you found helpful
If ‘no’, please describe why you think Cerina was not helpful and share your ideas about what might have been more helpful
6. How did you find the anxiety management techniques?
7. Were you able to easily perform the tasks, if not what further information do you need, what would have helped you, do you have any suggestions?
8. Was there anything in the app that triggered you to feel upset or more anxious?
9. Closing General Question: Do you have any other feedback for us regarding the development of this app, or in general about the use of mental health apps?