A “high-stakes test”.
It’s a neat description of a pivotal but controversial test of Irish medical school applicants: the Health Professions Admission Test (HPAT).
In the coming days, at six locations in Ireland, students will be tested on their ‘general skills’. What transpires could be very important — about 30 per cent of available points for undergraduate medicine come via HPAT.
HPAT Ireland is devised and operated by the Australian Council for Educational Research (ACER), which has much expertise in its sphere.
Nevertheless, in Ireland, the HPAT continues to elicit debate. Questions have surrounded its purpose, scheduling, impact on students, and the proliferation of independent preparatory courses. Meanwhile, the Language Commissioner is concerned about the absence of an Irish-language version (see panel).
The HPAT was introduced in 2009, on foot of the <em>Fottrell Report</em>, as an adjunct to the ‘academic’ assessment provided by the Leaving Certificate. The multiple choice test assesses logical reasoning, problem-solving skills, interpersonal understanding and non-verbal reasoning.
Over 2,500 candidates took HPAT-Ireland in both 2014 and 2015. The registration fee for 2016 was €120.
There are divergent views on HPAT. In 2012, when he was Minister for Health, Dr James Reilly said anyone who attained 600 points in their Leaving Certificate had “bloody well” earned the right to study medicine, if that’s what they wanted.
The test had merely led to “the development of another industry and another course”, Minister Reilly told the Seanad.
More recently, Prof Arnie Hill, Head of the RCSI School of Medicine, described the test as having increased “diversity of access” but added that it was an extra burden on students. Moreover, he said section 3 (non-verbal reasoning) was “complete and utter nonsense” in respect of becoming a doctor.
Via Twitter, the <strong><em>Medical Independent</em></strong> (<strong><em>MI</em></strong>) sought feedback from medical students. University College Cork’s Medical Society (@UCCMedSoc) responded: “In the misquoted words of F Gump Snr, ‘The HPAT is like a box of chocolates, you never know what you’re going to get’ and that’s true for both its results and its selection of future medical professionals!”
The exam body, ACER, offers an alternative perspective. Its spokesperson says the combination of HPAT and the Leaving Certificate “allows for the selection of an able, but more diversified, candidature than if just Leaving Certificate scores are used to select students”.
<h3><strong>Preparation</strong></h3>
The issue of HPAT preparation is especially contentious. According to ACER literature, neither it nor the Irish universities recommend or endorse any tests or preparation courses. Indeed, ACER also seeks to discourage test-takers from sharing information with third parties on the test, labelling it as student misconduct.
Wide and critical reading “may provide helpful preparation” for sections 1 and 2, it says. “There is little benefit to be gained from last-minute ‘cramming’. However, some familiarity with the question types typically presented in HPAT-Ireland is beneficial.”
Four preparation materials are currently produced by ACER: Three full-length tests and additional practice questions (one practice test is covered by the registration fee).
However, a 2012 study reported that HPAT Ireland candidates who undertook commercial preparatory courses appeared to “slightly outperform” those who did not. The finding was published in an interim report of the National Research Group Evaluating Revised Entry and Selection to Medicine, convened under the auspices of the Council of Deans of Faculties with Medical Schools in Ireland (CDFMSI).
ACER developed further materials for candidates, arising from this review.
Yet conflicting standpoints on preparation have continued to fuel Internet message boards and anxious minds. There are many independent preparation courses available. One of the best-known providers is Australian company MedEntry. With a long history in the field, it vigorously contests ACER’s stance.
“Individuals succeeding without HPAT Prep simply don’t prove that everyone else can do the same, any more than Madonna’s success proves that everyone can become a star,” advises MedEntry’s online marketing literature, perhaps conscious of a large teenage audience.
Founder of MedEntry Dr Ed Boyapati tells <strong><em>MI</em></strong> that ACER’s position on courses “is to be expected”. He says setting exam papers is difficult, and the more students know about an exam, the harder the job becomes.
<img src=”../attachments/9ade2d64-6302-47ab-858c-ab37e5a40a8c.JPG” alt=”” /><br /><strong>Dr Ed Boyapti, Founder of MedEntry</strong>
Dr Boyapati says there are two aspects to “training”: one is becoming ‘test-wise’, and the other is developing the skills tested by HPAT.
He maintains that Irish students can gain greater improvements than counterparts in Australia and New Zealand. They may not be as accustomed to these kinds of thinking skills being taught at second level, and also may not be as ‘test-wise’, he suggests.
Asked for data, Dr Boyapati says it is on its website. He says the skills assessed through HPAT are important in any career.
“When people say HPAT is an additional burden and so on, I wouldn’t quite agree with it… Ireland in any case must be among the very few countries where there was no [such] test until seven years ago for medical entrance,” he says.
The ‘standard price’ for MedEntry’s ‘most popular’ package, comprising a two-day workshop and online learning resources, is €595.
The Institute of Education, Dublin, also provides a preparation course. Every year, around 500 students attend the two-day programmes at a cost of €350, according to Director Mr Peter Kearns and Business Development Manager Mr David Ball.
The programme outlines the exam’s three components and includes a test in an exam environment, followed by a review of solutions.
<h3><strong>Psychometric testing</strong></h3>
The Institute commissions a psychometric testing agency to produce a new sample exam each year. Mr Kearns says feedback from students is positive. “We ask them simple questions — ‘do you feel you would have been worse off if you hadn’t done the course?’ and they say ‘yes’; and ‘do you find that it improved your grade?’ and they say ‘yes’. I suppose that is at odds with what the examination body — ACER — are saying [on preparation courses], but it really comes down to common sense.
“It is an exam with a very, very tight time limit structure to it. It is an exam probably like no other exam that the Leaving Cert student has seen before; it is effectively an aptitude test… ”
Students taking HPAT may be unused to feeling uncomfortable in exams, he notes. “In the HPAT, some of the questions are very, very hard; one might say they are intentionally hard to throw them. And we coach them — it is almost psychological — in how to deal with that sort of situation, where you are in an exam, this is their crunch exam, where possibly it is going to be career-defining and they get confronted with a question where they have not a clue how to answer and this could be a very new experience for them.”
Mr Ball says there is a plateau after which practice “becomes comforting rather than just effective”. However, he reiterates Mr Kearns’s position that such preparation “makes sense”.
It is acknowledged that there is no ‘control group’ to allow for statistical evidence to be provided.
HPAT is scheduled at a stressful time for students and would be more suitably held early in the academic year, they add.
<h3><strong>Investigation</strong></h3>
Following the 2014 HPAT-Ireland exam, ACER launched an investigation amid allegations that students who attended a preparation course run by MedEntry had seen, and been coached in, how to answer certain questions in the exam.
An ACER spokesperson comments: “Following HPAT-Ireland 2014, comments were made by some students about the exposure of the content of three units (10 questions) in section 1. The alleged exposure of the questions came from the coaching college MedEntry. ACER thoroughly investigated the claims. In the case of one unit, ACER had access to the material provided by MedEntry. While there were similarities between this material and the HPAT-Ireland 2014 questions, there was surface resemblance only. More limited information was available for the two additional units. While detailed psychometric analysis refuted any evidence of item ‘exposure’, standard test scoring processes were adopted and the two units were removed from scoring.”
This disquietened some students and their families. According to documentation obtained from the Department of Education and Higher Education Authority (HEA) under Freedom of Information legislation, a complainant informed the Department that their relative had, that year, undertaken the HPAT for a second time when, as they put it, the “ground rules changed”.
The complainant wrote: “From a legal point of view, I think that a person would have a strong case for ‘reasonable expectation’ of the conduct of the exam…” The complainant also queried the issue of ‘ownership’ of the exam (ACER are contracted to deliver HPAT by each of the five medical schools that require HPAT).
<blockquote> <div> <p class=”QUOTEtextalignedrightMIstyles”>‘In the HPAT, some of the questions are very, very hard; one might say they are intentionally hard to throw them’
</div> </blockquote>
In August 2014, a Department official forwarded the complaint to a colleague in the HEA for their observations. The same summer, the Department also sought “clarity” on issues regarding HPAT following representations made to the Minister.
The Department queried the HEA on whether there was a HPAT office within Ireland (there is not), who ACER was accountable to, and who verified and audited the HPAT results.
From the correspondence obtained by <strong><em>MI</em></strong>, it appeared the HEA routed these queries to the CDFMSI, or a group under its auspices.
According to ACER’s spokesperson, there were no challenges from candidates following the exam.
Meanwhile, in March 2014, an <em>Irish Times</em> report on the aforementioned ACER investigation (which found no impropriety by any party), stated that “MedEntry sent an email to students this year encouraging them to give feedback about the exam. Among other things the email states: ‘Emails that result in changes to the MedEntry resources (to make them better simulate the HPAT) will attract a bursary’.”
When <strong><em>MI</em></strong> put it to Dr Boyapati that this could be interpreted as seeking specific information on the test questions in order to improve MedEntry materials, he said every business or educator asks how they can improve services or courses.
“Universities do it, hospitals ask it of their patients, airlines of their customers; it is exactly the same thing we do. We ask our students how can we improve our services, how can we improve our questions, and they provide feedback and we act on those…
“As part of our commitment to improvement of services delivered, we constantly seek feedback from all our stakeholders, including students.”
MedEntry had not heard of any students being accused of misconduct for “providing feedback on our services”, he says. “Further, students these days talk about any test/exam on various forums on the internet. How are ACER or universities going to police it or track them down? It seems an impossible exercise.”
On the development of MedEntry materials, he states: “What we do is, we go through the types of questions — not the exact questions but the types of questions — that are likely to appear in the HPAT. By attending various international conferences, [analysing] the current trends and so on, we know not only what happened in the past but what is likely to happen in the future, the types of questions that are likely to come up, the ‘21st Century skills’, as I call it, problem-solving skills and so on… If ACER tries to use those questions, or those types of questions, it is good. It is a happy coincidence, I would say.”
<h3><strong>Impact</strong></h3>
Some observers have expressed concern that students from disadvantaged backgrounds could be further excluded by HPAT.
However, the aforementioned report from 2012 found no evidence of this.
Yet this report also found that students who repeated HPAT were likely to improve their score. Scores improved in all three sections, but were most evident in section 3 (non-verbal reasoning). Since 2010, approximately one-third of medical school entrants had repeated the HPAT, it reported.
Subsequently, the weighting of section 3 was reduced to 20 per cent of overall marks (it had accounted for one-third).
Meanwhile, research has not determined evidence of a “definite gender bias” associated with the introduction of HPAT or that it strongly predicts clinical and communication skills in the first two years of undergraduate medicine.
The CDFMSI’s research group is continuing to study selection methods, including outcomes of the HPAT.
Prof Michael Larvin, Head of the University of Limerick’s Graduate Entry Medical School, points out that medical school entry is not purely the province of the CDFMSI. Rather, there is a national context and the Council is currently engaging with the Task Group on Reform of University Selection and Entry (TGRUSE).
“The remit of the Council of Deans really is to manage the interests and collaborations between the six medical schools in this country; we don’t take final decisions on entry mechanisms, it would always be done in concert with the IUA [Irish Universities Association] but ultimately with the HEA.”
In the same interview, Prof Seán Dinneen, Head of the School of Medicine, NUI Galway, notes that the first takers of HPAT-Ireland are now only interns. “The idea is that we select well, to form really good doctors, really good graduates, but it is very difficult to assess the impact of the test because we have very little information to go on.”
Large number of participants are required from a research perspective, they outline.
<h3><strong>Research</strong></h3>
“The individual numbers per year are not huge and the outcome variables we would look at — for example, the success of the graduates — can take many years after qualification,” says Prof Larvin. “So the research group would be the first to say there are no strong indicators coming out of the research we have — in other words, there isn’t strong enough statistical evidence to either indicate a change in the way that we assess students for entry, or indeed to stop the HPAT. There isn’t strong enough evidence either way, is what we are reading at the moment, and a little bit more time is needed.”
Prof Dinneen says that ACER come over from Australia to meet medical school representatives, when HPAT Ireland and the GAMSAT are through the cycle.
He says ACER is “a very professional outfit” and expert in psychometrics. Prof Larvin adds that ACER has been in this area “for quite some time in these isles”.
“We have a relationship with ACER through HPAT, and [Prof Dinneen] attended that meeting in June; I attended a meeting with ACER the day before in Dublin, concerning the GAMSAT, so we have a regular relationship with ACER… ” He adds that ACER presents data, which is scrutinised.
“In a sense, they are actually the external party to Ireland and we are, in effect, the internal scrutinisers I suppose.”
On the issue of preparation courses, Prof Larvin says that, in relation to GAMSAT (which applies to UL GEMS entry), students “almost with one voice” advise to buy and practice from ACER materials.
“That is something we hear very strongly about the GAMSAT here…our students tell us the preparation courses, in their view, do not give a significant edge. It is a difficult test to prepare for, in some ways, but by looking at past papers they tell me here in UL that has always been the best preparation… ”
Prof Dinneen supports the comments of Prof Larvin in respect <br /> of HPAT.
<div style=”background: #e8edf0; padding: 10px 15px; margin-bottom: 15px;”> <h3><span style=”font-size: 1.17em;”>Testing times — the language of learning</span></h3>
The absence of an Irish language version of the HPAT is “inherently unfair” on Irish speakers, the Language Commissioner has told <strong><em>MI</em></strong>.
Mr Rónán Ó Domhnaill says that, while there is no statutory right to take the test in Irish, the situation is “unfair” on candidates schooled solely in Irish at primary and secondary level.
<strong><em>MI</em></strong> has learned that a complaint was recently made to the office of the Language Commissioner on the matter. However, when queried by <strong><em>MI</em></strong>, Mr Ó Domhnaill said he could not discuss individual complaints.
Mr Ó Domhnaill’s predecessor as Language Commissioner, Mr Seán Ó Cuirreáin, raised the matter after the establishment of the HPAT in 2009.
<img src=”../attachments/417ca7d6-e39f-4067-9ed6-7987f581108a.JPG” alt=”” /><br /><strong>Language Commissioner Mr Rónán Ó Domhnaill</strong>
A letter to Mr Ó Cuirreáin in April 2009, from the Irish Universities Association (IUA), was obtained from the Higher Education Authority (HEA) following a Freedom of Information (FOI) request. The IUA was responding on behalf of the universities.
In the letter, Mr Lewis Purser, Director of Academic Affairs, IUA, said the issue of providing the test through Irish was “seriously considered”. However, due to three factors, the test was only made available in English.
Firstly, the timeline requested by the Minister for Education in May 2008 meant that the new route needed to be in place for students admitted in autumn 2009; as such, there were only a few weeks to undertake the tender process and prepare an information brochure for the medical schools.
Secondly, there were difficulties from a testing perspective in translating a psychometric test into a second language; and thirdly was the need to keep costs for students as low as possible.
Mr Ó Domhnaill told <strong><em>MI</em></strong> his office had yet to see evidence that the test could be compromised by offering it in more than one language. He said Foras na Gaeilge had a database of “highly -trained” translators.
In July 2015, the issue was raised with the HEA by the parents of a student. The partially-redacted correspondence was obtained under FOI legislation.
According to the parents, the student was “at a disadvantage” as he/she was not given the opportunity to sit HPAT in their first and native language.
The HPAT was “such an important aspect for gaining entry to the undergraduate degree in medicine” that they felt an Irish version should be available.
The letter referred to “extensive international research” that students who were not native speakers had difficulty in timed exams because they needed more time to interpret the English language.
It said McGill University, Montreal, Canada, dropped the Medical College Admission Test (MCAT) because there was no French version and this was “causing a barrier for French speakers”.
“We would appreciate it if special consideration could be extended to Irish speakers, as I believe our family is part of the ‘minority’ that you mention in your equality statement… ”
The correspondence trail shows that it was sent by the HEA to the Council of Deans.
Speaking generally on the Irish language issue, Head of the School of Medicine, NUI Galway, Prof Seán Dinneen, said the Council has asked NUI Galway, which has a particular affiliation with the language, to advise on the issue of availability in Irish.
To his knowledge, the matter had seldom been raised by test-takers.
Prof Michael Larvin, Head of University of Limerick’s Graduate Entry Medical School, said the matter needed to be examined, but financial ramifications for test-takers would be a key concern.
Schools were already concerned that HPAT was borne at a cost to applicants and were keen to maintain “broad access”.
</div>
