A novel general practice registrar to supervisor feedback system for distance education in rural areas

Abstract

Background

Quality supervision in general practice (GP) is critical for the progress and satisfaction of GP registrars and for attracting future rural GPs. However, there is limited research to inform the implementation of feedback systems for enhancing supervision by rural supervisors, and no published evidence specific to distance education where a remote supervisor may be in a different practice and supervising from afar. This study aimed to develop and explore the outcomes of an easy-to-administer, safe and constructive, registrar-to-supervisor feedback system for a distance (or remote) supervision model.

Methods

Participatory action research involved the design of a standardised short-form questionnaire and an administration, data analysis and feedback process between registrars and supervisors. The questionnaire was administered each year between 16–20 weeks of the first year of registrar training within a 3–4-year rural and remote GP training program—the Remote Vocational Training Scheme (RVTS) (2020–2022). Participation in the project was voluntary. Registrars were asked 12 standardised questions about supervision over three domains: bond strength, task agreement and goal setting. Responses were summed by domain and evaluated using set criteria of high (> 80%), medium (51% to 79%) or low (50% or lower). High- and medium-level narrative feedback reports were provided to supervisors. Low domain scores were followed up by relevant internal staff to negotiate and resolve issues.

Results

All 106 commencing registrars completed the questionnaire, of which n = 99/106 (93%) reported high performance related to the bond with their supervisor, n = 94/106 (89%) reported high performance on training tasks, and n = 53/106 (50%) reported medium or low performance for supervisor’s understanding the registrar goals. The majority of supervisors found the feedback useful. Ten registrars identified to be in need (9% of 106) were offered additional support.

Conclusions

The system was found to be feasible, safe, and constructive for reviewing the quality of a distance supervision model for rural and remote registrars. It enabled prompt resolution of issues that would have otherwise been difficult to address and facilitated more open discussions about the quality of supervision. This process has been standardised within the RVTS.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12909-025-06820-9.

Background

Several articles describe quality supervision in general practice (GP) training and the benefits of doctor learning in rural and remote areas outside of larger regional centres [1, 2]. However, ongoing quality improvement of supervision is critical for general practitioners to learn effectively on the job, progress towards Fellowships (in Australia, this occurs with one of two colleges, the Australian College of Rural and Remote Medicine [ACRRM]) and/or the Royal Australian College of General Practitioners [RACGP]) and feel satisfied and interested in future rural work. Continuous feedback systems play an important role by providing timely responses to relevant issues [2]. However, there is minimal evidence to guide the implementation of feedback systems for rural and remote supervisors concerning the quality of supervision in general practice, and no published evidence specific to distance education involving remote supervision where the supervisor may be overseeing registrars from a different practice location.

The Remote Vocational Training Scheme (RVTS) is an Australian Government-funded vocational training and medical workforce program that provides comprehensive general practice/rural generalist fellowship training. Each year since 2000, the RVTS recruits up to 32 registrars working in rural, remote and First Nations communities [3].

RVTS supervisors, who often work in different practices or towns, support registrars remotely through a combination of modalities and face-to-face visits. Supervisors are experienced GPs who have extensive knowledge of rural, remote, and First Nations communities and the dynamics of more rural locations (modified Monash model (MMM) 4–7), where most RVTS registrars are situated [4]. However, being mostly offsite from their registrars, they rely on systems to obtain specific, timely and relevant feedback from registrars to inform supervisor improvement. The feedback system should safeguard and strengthen the long-term supervisory relationship, as registrars in the RVTS are connected to the same supervisor throughout the 3–4-year training period [5]. Establishing standardised, respectful feedback systems for supervision is also helpful for the accreditation and ongoing quality improvement of GP training organisations.

There is extensive literature on various methods that supervisors can use to provide feedback to their registrars [6–8]. While GP registrar to supervisor feedback has always been encouraged, there is limited evidence about two-way feedback systems, and existing research suggests that such systems can be difficult to implement [9, 10]. Multiple barriers, such as the power imbalance between registrars and supervisors and time pressures, may inhibit the collection and use of quality improvement information between supervisors and registrars in the context of GP training [11–13].

In the Australian context the unique structure of the RVTS, where the supervisor works remotely from the practice employing the registrar and is not involved in the employment contract, and where registrar training progression to a different practice does not require a supervisor training report, addresses some of the more important factors involved in power imbalance between registrar and supervisor.

This study aimed to develop and explore the outcomes of an easy-to-administer, safe, and constructive registrar to supervisor feedback system, which overcomes any additional power imbalances and is tailored to distance general practice education for registrars based in rural, remote and First Nations settings.

Method

This project used the participatory action approach for designing, implementing and reflecting on the feedback system through shared learning within the RVTS teaching and learning community (including medical educators, supervisors and registrar training coordinators) [14]. The project was deliberately pragmatic, as it was undertaken in the context of the limited resources of rural and remote medicine within a complex, highly distributed national GP training program.

The system design principles were first agreed upon by the project team (RVTS management staff and key supervisor roles). These were to ensure the feedback system was easy to administer which was defined as feasible to implement for timely feedback year by year within the RVTS program operational context; safe which was defined as inviting reflection without causing objection; and constructive which was defined as specific, relevant and timely.

Codesigning a suitable questionnaire

An easy-to-administer system including a short, standardised data collection instrument was codesigned to gather authentic registrar feedback regarding supervisors. As there are no specific supervisory feedback tools that are routinely applied in general practice training in Australia, particularly regarding distance education with remote supervision, it was necessary to explore applicable tools from other disciplines.

A key tool for gathering information was the Working Alliance Inventory (WAI), initially developed as a 36-item questionnaire to assess the therapeutic relationship between psychologists and their clients [15]. The 12-question short-form of the inventory was found to be equally valid [16]. The theoretical basis for the WAI conceptualised the relationship between psychologists and patients as having three dimensions [17, 18]. It was suggested that the model is applicable to the “educational alliance” as summarised in Table 1.

Table 1.

Dimensions of the educational alliance

Goals: Mutual agreement regarding the aims of the change process Tasks: Agreed learning activities designed to address registrar and training program aims Bond: The emotional strength of the working relationship

With the aim of achieving the collection of specific, timely and relevant feedback from the registrar the project team discussed twelve questions based on the short WAI, exploring goal, task, and bond issues, with the RVTS Registrar Training Coordinators (RTCs) and selected supervisors over several meetings. A modified RVTS Supervisor Educational Alliance Questionnaire (SEAQ) was agreed upon (Table 2).

Table 2.

Supervisor educational alliance questionnaire

(Registrar response: Never; rarely; occasionally; sometimes; often; very often and always) 1. My supervisor and I agree on how to improve my clinical practice 2. What I am discussing with my supervisor gives me new ways of approaching patient problems 3. I believe my supervisor respects me 4. My supervisor does not understand my training needs 5. I am confident in my supervisor's ability to help me 6. My supervisor and I are working towards mutually agreed-upon goals 7. I feel that my supervisor appreciates my contribution 8. We agree on what is important for me to work on 9. My supervisor and I trust one another 10. My supervisor and I have different ideas on my knowledge gaps 11. We have established a good understanding of the kind of changes that would improve my practice 12. I believe the way we are working with my training is correct The sum of the scores for each domain is as follows: goal dimension questions 4, 6, 10 and 11; task dimension questions 1, 2, 8 and 12; and bond dimension questions 3, 5, 7 and 9 Scoring and benchmarking Positively phrased questions (Never = 1 to Always = 7) Goal questions 4 and 10, negatively phrased (Never = -1 to Always = -7) High – 80% + summed score for the domain, medium 51–79% and low 50% or less

Informing participants and collecting the data

A practical process for administering and managing specific and relevant feedback was developed by the lead author and agreed upon by the project team. The process sought to ensure that the data were included within the wider RVTS administrative dataset which is used to manage the program. Over the three calendar training years from 2020–2022, at approximately 16–20 weeks of the first year of training, both registrars and supervisors were sent an information sheet outlining the purpose and voluntary nature of the SEAQ (Supplementary material – Part A). Sixteen weeks into the first term of the 3–4-year RVTS training program was considered the point at which supervisory relationships could be appropriately assessed and supported with early intervention resources. The information sheet included the registrar questions, the reflective activity for supervisors and who to contact if either the registrar or the supervisor wanted more information. It informed the registrar and supervisor of the nature of the questionnaire and that the results of the SEAQ would be analysed and used to develop high-level feedback for the supervisor (not specific results) for the purpose of improving RVTS training as well as possibly being summarised for reporting in research and evaluation. One week later, the SEAQ was sent to the registrars via email containing links to an online survey tool. Registrars choosing to participate responded to 12 questions via a Likert-like verbal scale with seven possible responses (Table 2). There were no implications for nonparticipation, but a reminder was sent out to nonparticipants if they had missed information because of their busy work environment. The responses were stored in a password-protected, secure area of RVTS SharePoint Drive, which was accessible only to authorised RVTS staff.

Scoring and giving feedback

The registrar responses were analysed by the lead author, an independent moderator who was not directly involved in the supervision of the RVTS registrars and was responsible for supervisor quality improvement within the RVTS. The sum of the scores in each domain was calculated (Table 2) and expressed as a percentage. The benchmark levels of high, medium or low (cut-off scores) were related to the percentage score. There is no guidance in the rural medical literature regarding performance standards for WAI indicators, nor is there guidance specific to a rural and remote training context (involving distance education and remote supervision). To demonstrate high performance, the project team determined that 80% + was a reasonable high-performance benchmark. In support of this, the Australian General Practitioner Training (AGPT) program survey in 2020 suggested that 74.5% of GP registrars were satisfied or very satisfied with their supervisor support [19]. Furthermore, other RVTS administrative data reinforces > 90% registrars are satisfied with key supervision measures in de-identified annual surveys [20] The project team considered the 80% cut-off point would support the goal of the feedback system which was to build reflection and learning into supervision practice. The project team then discussed arbitrary cut-offs for demonstrating medium performance of between 51 and 79% to indicate that supervision quality could be improved in some way and low performance of 50% or less to suggest that supervision quality might be compromised, and further investigation would be required. The team agreed to reflect on whether the benchmarks were practical and relevant over the course of the pilot project.

To provide clear standardised feedback, the lead author responsible for supervision quality improvement, guided the process of developing a simple narrative proforma and reflective activity based on benchmarking of the scores for each of the three domains (Supplementary material – Parts B and C). This included reference to specific guided activities to prompt quality improvement activities for individual supervisors. For example, the feedback for a score of 80% + in the Bond domain advised, “You have an excellent educational alliance with your registrar”. Supervisors were still asked to reflect on their role to identify further improvements. A score between 51 and 79% in the bond dimension offered the remark “You have a good educational alliance. You should reflect on whether there are any barriers to regular meetings, whether you have prepared adequately for the educational sessions and whether you give nonjudgmental feedback. In addition, you should reflect on the feedback in relation to goal setting and task development. The registrar’s medical educator or the medical educator supervisor support can assist with guidance on further improvement in your educational alliance”. A score of 50% or less in any domain received the same narrative feedback as the previous example but also triggered a medical educator-led discussion described below.

Applying feedback and improving practice

Supervisors with scores above 50% in all domains were asked to undertake personal reflection, and no further action was required (Supplementary material – Part B). When a registrar’s SEAQ evaluation gave a score of 50% or less in any domain, the alliance was considered to require action. In these cases, the lead author contacted the registrar, as part of the RVTS role regarding managing supervision quality, advised about the confidential nature of the call and, with the registrar’s permission, discussed the reasons for the low SEAQ score. If no consent was given, the discussion ended, and registrars were assured that any information they provided would be confidentially handled and not reported. If they agreed to continue, the registrars proceeded to disclose their concerns. Once the nature of the difficulty was understood, permission was sought to contact the supervisor and discuss the responses at an aggregate level. With permission, discussions with the supervisor included suggestions for improving the educational alliance only and not for any other reason. Registrar psychological safety was always prioritised, and no conversations occurred without the registrar’s verbal consent. If the registrar misread the negative question the score was adjusted to reflect the registrar’s true intention.

Throughout and at the project’s conclusion, the RVTS team reflected on key learnings to guide useful aspects to embed in ongoing practice into program policies and procedures. This project received ethical approval and the need for consent to participate was waived by the University of Queensland 2023/HE001926.

Results

The design of the SEAQ aligned with the operational delivery of the RVTS and was administered with ease to each of the RVTS cohorts commencing the 2020–2022 period. In total, 113 registrars with RVTS were included during that period. Of these, 106 were available and had a confirmed supervisor at the 16-week point in their first year of training when the information sheet followed by the SEAQ was sent out. All 106 (100%) registrars returned a fully completed SEAQ by approximately five to six weeks. Seven registrars were in the middle of the program or supervision changes and did not participate. Overall, 88 supervisors provided support for the responding registrars, and 18 of these supervisors supervised two registrars from different year groups. The registrar and supervisor demographics are described in Table 3.

Table 3.

Characteristics of registrars and supervisors in the Remote Vocational Training Scheme cohorts 2020–2022

Characteristic	Registrar cohort (n = 106)	Percent	Supervisor cohort (n = 84)	Percent
Age group
< = 30	4	3.8%	0	0.0%
31–35	24	22.6%	1	1.2%
36–40	31	29.2%	5	6.0%
41–45	28	26.4%	13	15.5%
46–50	7	6.6%	9	10.7%
51–55	8	7.5%	17	20.2%
56–60	4	3.8%	16	19.0%
61–65	0	0.0%	10	11.9%
66–70	0	0.0%	8	9.5%
> 70	0	0.0%	5	6%
Total	106	99.9%	84	100%
Sex
Female	50	47.2%	37	44.0%
Male	56	52.8%	47	56.0%
Total	106	100%	84	100%
Ruralitya
MMM 1–3	9	8.5%	42	50.0%
MMM 4–7	97	91.5%	42	50.0%
Total	106	100%	84	100%
Supervisor experience (years)
1–10	Na		43	51.2%
11–20	Na		26	30.9%
= > 21	Na		15	17.9%
Total	Na		84	100%

The number may not add up to 100% because of rounding

Administrative data were missing for seven registrars and four supervisors

Na Not applicable

^aMMM Modified Monash Model, https://www.health.gov.au/topics/rural-health-workforce/classifications/mmm [4]

Analysis based on the registrar’s perspective across the three domains of the educational alliance revealed that n = 99/106 (93%) had a high educational alliance bond score (80% +) and that n = 94/106 (89%) had a high task score (80% +) and that n = 53/106 (50%) had a high goal score (80% +) For visualisation, the goal, task and bond score distribution are displayed as bar charts (Fig. 1).

Fig. 1.

Goal, task and bond scores (total 106 registrars)

Twenty-one (19%) of the 106 registrars gave a low score of 50% or less in any of the three domains of the SEAQ. The lead author, in the role of overseeing RVTS supervision quality, acted as the moderator and contacted these registrars, confidentially seeking feedback to understand the reasons for the scores and to discuss ways to safely improve the training relationship. Eleven (10%) of the 106 registrars had misread the negative goal questions and, in all cases, the adjusted score was > 50%. (The adjusted data was used in Fig. 1.)

In total, following adjustment for the misread question, 96 (91% of 106) registrars gave a score of > 50% in any of the three domains of the SEAQ. In these cases, the feedback was selected from the appropriate benchmark level and sent to the supervisor for reflection and action as they deemed necessary.

The remaining ten (9%) of the 106 registrars gave permission to discuss the feedback with their supervisors. In seven cases, the discussion process with those supervisors led to quality improvement and resolution by highlighting issue(s) that supervisors were not aware of and provided clear actions that they were happy to implement. When the discussions disclosed that the registrar and/or supervisor had specific needs, the RVTS Registrar in Difficulty Policy was followed, thereby providing access to additional supervisor and/or cohort medical educator support as needed [21].

In the three remaining cases, there were irreconcilable differences related to supervision quality, and they were discussed confidentially at an internal RVTS Quality Assurance meeting with the registrar’s and supervisor’s consent and in liaison with the RVTS chief executive officer (CEO). Following further consultation with the registrar and their supervisor, alternative arrangements were made for supervision (a new supervisor) in all cases. There were no complaints regarding the processes followed.

Discussion

This project describes a safe and constructive technique to collect registrar evaluations of the supervisory relationship and, with permission, discuss and share the drivers of any negative feedback with individual supervisors, which promoted supervisor reflective activity on the nature and content of their supervision. All 106 registrars were prepared to complete the 12-question, SEAQ, suggesting that using a short questionnaire is an effective way to obtain a good registrar response rate. Routinely applying the SEAQ to cohorts enrolled in training was considered to assist with normalising feedback and improving supervision quality. The narrative, bench-marked feedback enabled systematic, constructive and respectful feedback that was found to be acceptable and useful for the supervisor. The benefit to registrars is demonstrated by the 78.3% training-program completion rate coupled with the retention of RVTS registrars in challenging rural and remote practices [22].

The power imbalance between registrar and supervisor is reduced by the fact that the supervisor is remotely located from the registrar and is not in a contractual arrangement. Additionally, this novel method used short-form, broad questions designed to act as indicators in the three domain areas of the educational alliance. Further, registrars were advised that the survey responses would be reported to an independent person who would check with the registrar and get their consent before discussing matters with the supervisor. Finally, the registrars were informed that their feedback would be summarised in a high-level narrative report for their supervisor. This approach reassured registrars and promoted honest feedback regarding any issues that could then be discussed in a safe environment.

In seven of the ten registrars who gave the supervisor a goal score < 50% the problem related to communication issues regarding the registrar’s needs or aims. The issues were readily addressed and potentially prevented on-going registrar dissatisfaction. In the last three cases both the registrar and supervisor were experiencing difficulties. The independent mediator enabled a safe resolution for both registrar and supervisor. Further description is limited due to privacy and safety concerns.

The RVTS program experience over a 25-year timeframe suggests that a registrar in difficulty often relates to a supervisor in difficulty and vice versa. However, disclosure opportunities are essential because good registrars may draw on self-reliance and other resources to manage poorly performing supervisors and quality supervisors manage registrars in difficulty without involving outside support mechanisms.

As a sign of reflective practice and continuous improvement, the RVTS has continued to use the SEAQ for more than 18 months since the completion of this project, suggesting that the codesigned survey, process and feedback system is feasible and safe.

Despite all care being taken some registrars may still have not felt safe to disclose issues regarding their supervisor. The RVTS uses a range of other feedback methods to explore supervisor and registrar satisfaction, including using internal reports from RTC’s, monitoring by medical educators, multisource feedback questionnaires, self-disclosed clinical incident reports, external clinical teaching visits, webinar attendance (over 80% attendance is required) and workshop performance outcomes. Triangulation of these measures with the SEAQ enables registrar and supervisor issues to be identified. No registrars were identified as having performance concerns at the time of administering the SEAQ, however the administration of the SEAQ at 16 weeks upon commencement of the first year of training enabled early detection and troubleshooting of any issues within the registrar and supervisor relationship.

This project expands the evidence of the wider medical literature where there are no examples of systems using easy-to-administer, safe, constructive feedback for supervisors overseeing registrars in remote medical training sites. Moreover, this is the first study describing a simple feedback system for high workload rural and remote settings where registrar and supervisor relationships can be time-pressured, and issues can easily remain unresolved unless explicitly explored. Around four in five supervisors found the feedback helpful and one in two supervisors anecdotally reported that this process stimulated them to appreciate their registrars’ goals more comprehensively. The SEAQ was also found to promote more open communication between supervisors and registrars to identify obstacles and meet the registrar’s agenda. Further, it informed the content of supervisor training. In most cases, issues were easily and appropriately resolved, and more intractable concerns were further investigated through clear escalation and remediation pathways. In a small proportion of cases, appropriate referrals for deeper investigation led to new supervisors being appointed.

The study had some limitations. That negative questions were misread by eleven (10%) of the 106 registrars was a concern. However, they were easy to identify as the registrars had scored the supervisor highly in their other responses and this was confirmed on discussion with the registrars. All registrars were advised to read the questions thoroughly and complete the survey in a quiet room and the negative questions provided a useful check and balance as to whether registrars were giving valid responses. More evaluation is planned to test the impact of the SEAQ and further research is needed to explore the reasons why registrars feel that their supervisor has a limited understanding of their goals; to assess supervisor attitudes more formally to their registrar feedback; to evaluate the registrars’ perspective of supervisor behavioural change subsequent to receiving feedback; and to explore how valid and reliable the tool is for the assessment of a supervisory relationship in difficulty.

Conclusions

This article presents a novel, standardised, and easy-to-administer process for gathering and summarising registrar feedback for supervisor reflection. The process is particularly suited to distance education models of rural general practice training, including remote supervision, and for doctors in busy work settings. The data collected is valuable for informing quality improvement actions and is suitable for managing sensitive information while protecting supervisory relationships. It provides important insights into key domains of the educational alliance. Applying these findings to improve supervision quality for registrars based in small communities, including those in distance education models, should enhance the recruitment and retention of GP registrars.

Abbreviations

ACRRM

Australian College of Rural and Remote Medicine

AGPT

Australian General Practice Training

CEO

Chief executive officer

EFFECT

Effective Clinical Teaching

GP

General Practice or General Practitioner

GP-SRMS

General Practice Supervisory Relationship Measure

MMM

Modified Monash Model

RTC

Registrar Training Coordinator

RACGP

Royal Australian College of General Practitioners

RVTS

Remote Vocational Training Scheme

SEAQ

Supervisor Educational Alliance Questionnaire

SRM

Supervisory relationship measure

WAI

Working Alliance Inventory

Authors’ contributions

T.M. developed the original concept and collected the data. T.M., P.G. and B.O.S. analysed and interpreted the results. T.M. wrote the original draft and prepared Table 3 and Fig. 1. T.M., R.G., P.G. and B.O.S. wrote and edited the main manuscript text and Tables 1 and 2. All authors reviewed the manuscript.

Funding

The RVTS is funded by the Australian Government Department of Health and Aged Care. The RVTS staff and researchers worked independently.

Data availability

To protect the privacy of study participants, data access is coordinated with the main author and is subject to ethical approval.

Declarations

Ethics approval and consent to participate

This study received ethical approval from The University of Queensland 2023/HE001926 as part of evaluating the process and outcomes of the RVTS supervision and support model.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.