Quality Performance in the Immigration Appeal Division 2017-18

Ta​ble of contents

• Context
• Performance results
• Timely and complete pre-proceeding readiness
• Respectful proceedings
• Focused proceedings
• Clear, complete, concise and timely decisions
• Summary
  
• Appendix - checklist
• Notes

Context

This study measures the quality of decision-making in the Immigration Appeal Division (IAD). Quality measurement assesses the extent to which the division achieves the four expected results of quality established by the Immigration and Refugee Board of Canada (IRB):

1. Timely and complete pre-proceeding readiness
2. Respectful proceedings
3. Focused proceedings
4. Clear, complete, concise and timely decisions

This study’s objective is to identify trends, patterns and issues concerning quality to inform the management of quality at the divisional level. Results complement other available data sources—decisions by the Federal Court, evaluations, key indicator reports, and quality measures from previous years—to understand performance in the IAD in both qualitative and quantitative terms.

The study reviewed 47 appeals randomly selected from a total of 298 sponsorship appeals finalized between November 2017 and February 2018 that were decided on their merits after an oral hearing before a single-member panel. Twelve out of 20 experienced IAD members who finalized at least one appeal during this period are represented in the sample.

The file review was performed by Maryanne a, at the time a former Assistant Deputy Chairperson in the Western Region. The file reviewer, also referred to in this report as the evaluator, examined all evidentiary and administrative materials on file, listened to the complete audio recordings and assessed these against qualitative measures in a checklist developed by the Planning, Evaluation and Performance Measurement Directorate (PEPM) and approved by the Chairperson in 2017 (see Appendix). Each measure is assessed along either a 1-to-3 rating scale or a categorical yes-no scale.

Two different targets set by the IRB are applied.

1. “Meets expectations” is represented as “2” on the 1-to-3 scale.
2. Results are also expressed as a percentage of cases that scored at least “2”. For example, 70% is the target for indicator #11 (The member ensures the parties focus testimony and documentation on relevant issues), meaning that 70% of the files are expected to score 2 or higher against this indicator.

This study acknowledges the inherent limitations to qualitative research. Qualitative measures do not generate precise data as do quantitative metrics. Moreover, a small sample size limits the inferences that may be made about the broader caseload. Also of note, the consultant was unable to evaluate all cases due to a public appointment while the study was in progress. As such, the sample under-represents the Eastern Region, and general conclusions are based principally on the results of Western and Central Region cases.

This study’s aggregate findings are accurate within a margin of error of ±11%, 18 times out of 20. This margin increases when data is broken down to the regional level or case type. Therefore, the study does not advance any conclusions where the sample size is insufficient to do so.

The findings in this report are solely those of the evaluation team.^{Footnote 1} Their observations are necessarily subjective in nature and do not lend themselves to firm conclusions on legal matters such as the correct application of the law, the weighing of the evidence, or the fairness of the proceedings from a natural justice perspective. Only a court reviewing the case can arrive at such conclusions. This report aims to provide a perspective to improve the division’s performance overall.

The sample

The evaluator analyzed a sample of 47 appeals drawn from a broader population of 298 eligible appeals finalized between November 2017 and February 2018.The following charts illustrate the makeup of the sample.

Performance results

Timely and complete pre-proceeding readiness

Why measure th​is:
The groundwork for quality is set pre-hearing when the Registry prepares a timely, organized and complete case docket and the member assimilates the facts and key issues of the case.

What was measured:

Targets: Respectively 80%, 90% 100% and 100% of cases achieving at least 2.0 out of 3.

What the numbers say:

• As a national aggregate, registry and member average scores were universally above the IRB standard in the vast majority of appeals examined.
• Nearly 94% of files were properly organized or complete with all the required information. These results fall within the margin of error.

Strengths

• Docket preparation. Overall, the Registry delivered a docket to the member that was timely, organized and complete with all the required documents in 98% (n=47) of the cases.
• Member preparedness. Members clearly benefited from being handed complete and timely dockets. Members consistently showed that they were ready at the start of the hearing with demonstrated knowledge of the facts and key issues of the appeal. On only one occasion, the evaluator noted that the member’s introduction was “somewhat confused and unfocused”.

Opportunities

• Docket organization. In 6% of cases examined, all in the Central Region, documents in the file docket were not organized in a logical and standardized manner. Some examples include administrative records or evidence not arranged in reverse chronological order and a hearing disposition record that was left blank. Properly organized dockets allow the member to quickly retrieve evidence and information, which contributes to the efficient management of proceedings, and a completed hearing disposition record allows registry staff to input the results of the hearing into the NOVA system.

Respectful proceedings

Why measure this:
Individuals appearing before the IRB expect that they will be treated with sensitivity and respect. Any shortcoming in this regard potentially undermines tribunal integrity and public confidence. Indicators #5 and #6 are considered universal—they apply to all cases.

What was measured:

Target: 100% of cases achieving at least 2.0 out of 3.​

What the numbers say:

• Results show that respectful proceedings consistently score at or near 100% nationally and in each region.
• The number of cases found to be applicable against question #9 were too few to cite.

Strengths

• Sensitivity and respect. All the parties were treated with sensitivity and respect with some minor areas for improvement noted below.
• Presenting evidence. In all 47 hearings examined, the member ensured that each party had an opportunity to present and respond to evidence. In some instances, members granted additional time to unrepresented appellants to prepare submissions on legal issues or to produce relevant documents. On two occasions, written submissions by the Minister’s counsel were presented for the record despite the fact that they were unable to attend the hearing in person.
• Members also address and rectify technical problems that might have a bearing on the disclosure of evidence. In one example, the recording of the hearing did not start due to oversight. The member proceeded to summarize the unrecorded evidence and verified that the parties were in agreement with the statement of facts before proceeding with the hearing.

Opportunities

• Unrepresented appellants. Study results show a marked increase in the proportion of unrepresented appellants which reached 40% in the study sample (n=19). This represents the largest proportion in any year’s study sample to date and a 9% increase since the last study in 2016-17. In addition to finding more unrepresented appellants, results found that unrepresented appellants have significantly lower success rates (21%) than appellants represented by counsel (54%). When looking at all appeals finalized during the fiscal year, the percentage of appeals allowed among unrepresented appellants drops to 12% versus 43% for represented appellants. Possible causative factors were outside the scope of the study.
• Sensitivity and respect: Study results indicate that while members scored 2.7 out of 3 in this area, there is room for improvement. For example, the evaluator noted that for hearings by teleconference, “member[s] should [identify themselves] and [other] participants for the party on the phone”. In another instance, it was noted that a member sought to clarify an appellant’s testimony “in an argumentative way”. On another occasion, the member did not provide Minister’s counsel the “opportunity to address the issue that was decided”. The examples represent the exception while in the vast majority of cases, members demonstrate a high level of consideration for hearing participants.
  
• Ex parte communication. Members must ensure that all parties are privy to statements expressed during the hearing. On one occasion, a member initiated a discussion with Minister’s counsel on a matter pertinent to the hearing process, but did not disclose it or provide a summary to the appellant who only joined the hearing afterwards by teleconference.

Text format

This is a bar chart including 2 bars describing the percentage of Appeals that were allowed from 2017 to 2018. Forty-three percent of appeals were allowed for represented appellants compared to only twelve percent for unrepresented appellants.

Focused proceedings

Why measure this:
Proceedings that are efficient and well managed create conditions for quality outcomes to emerge and support the IRB’s efforts to make the most effective use of its resources. Criteria #10 to #12 are assessed against all cases while #13 to #20 assessed on an as applicable basis.

What was measured:

Target: 70% of cases achieving at least 2.0 out of 3.

What the numbers say:

• Focused proceedings scored at or near 100% nationally, exceeding the target of 70%.
• Figures are not reported for Question 18 which applied to very few cases.

Strengths

• Focus on issues. Members are adept at clearly defining the agenda for the parties at the beginning of the proceeding with 100% of cases meeting or exceeding the target. In one example, a member clearly explained at the outset the calculation formula applied when assessing if an appellant fulfills their residency obligation and how the issue would be examined during the proceeding. Members also ensured that parties presented testimony and documentation that was relevant to the issues being considered. In one case, the member identified the specific considerations applicable to claims made on humanitarian and compassionate grounds. In another instance, the member discussed the nature of the issues to be examined during the hearing and allowed counsel supplemental time to refocus their presentation. Additionally, members generally led questioning that was relevant, with some members demonstrating in-depth knowledge of case details. The evaluator did observe some performance issues in this area which are explored further in the section below.
• Challenging situations. The evaluator found that members are tactful when managing challenging situations. For example, in one instance, a member who was having “difficulties eliciting [an] appellant’s testimony” showed considerable sensitivity when attempting to draw out additional information. Members also made efforts to resolve problems affecting the quality of the hearing. In one example, a member periodically adjusted the phone speaker volume so that the appellant could more clearly hear the discussion. In instances where parties were connected by videoconference or teleconference, members had to manage “multiple people speaking” and rectify connection issues that interrupted proceedings.
• Plain language. Appellants may have difficulty comprehending either official language. Members recognize when appellants are experiencing difficulties understanding the hearing proceedings and will normally adopt plain language to improve communications with an appellant. For example, one member often rephrased statements to ensure that the appellant could understand.

Opportunities

• Narrowing issues. Although members generally define the issues to be examined at the outset of the hearing, the evaluator noted that members rarely narrow the issues after the hearing has started. Rather, members tend to focus on the relevance of testimony and ensure the timely progression of the hearing.
• Focus on issues. The evaluator observed that on two occasions members did not address the key issues raised during the hearing. In another case, the member altogether disregarded the key issue in the final decision; according to the evaluator, while the hearing statements mainly focused on “H&C” (humanitarian and compassionate grounds), the member cited “legal validity” as the key issue in the final decision. Examples where members diverge from the key issues make it difficult for the parties to follow the member’s reasoning path and provide grounds for potential judicial review.
• Relevant questions. The data reveals that five members did not consistently articulate focused, organized and relevant questions. In these instances, the evaluator described member questions as “argumentative”, “unclear” or “vague”. In one case, a member often used leading questions. For example, a member asked a witness: “Why should I allow your husband’s appeal?” In another case, the member’s questions did not attempt to elicit evidence directly from the second appellant, who was married to the principal appellant. Furthermore, the evaluator noted that, in one case, a member negatively characterized the appellant’s testimony by making statements such as “that is smoke, which doesn’t help… not impressed by that…”

Recommendation. In a subset of cases, questioning by members was found to be argumentative, unclear, vague or leading, departing from the IRB’s expectations for relevant and focused questioning. This was not a systemic issue affecting the division as a whole. As such, the division should consider a proportionate response to ensure consistently high quality hearings, such as a professional development refresher or a communiqué to members on effective and relevant questioning.

• Communication issues. Communication issues can also represent a challenge for members. In one example, the appellant participating by teleconference spoke with a strong accent that made it difficult for the member to understand the testimony. Rather than have the appellant testify in their first language using an interpreter provided by the IRB, the member allowed counsel, who claimed to have understood the testimony, to restate it for the record. The member accepted this practice and obtained “the evidence as restated by the appellant’s counsel”.
• Ineffective counsel. Members on occasion intervened when an appellant was not adequately represented. In one case, a member spent considerable time instructing the lay person and guiding counsel. In the opinion of the evaluator, the member could have directed the process and questions rather than engage in a lengthy instruction to the lay representative.

Clear, complete, concise and timely decisions

Reasons state conclusions on all the determinative issues

Why measure this:
The Supreme Court of Canada set the requirement for justifiability, intelligibility and transparency in a decision of an administrative tribunal. Through indicators #21 to #30 on the following three pages, this study applies the Court’s requirement in the context of IRB decision-making. Indicators #21 and #22 and applied universally.

What was measured:

Target: 95% of cases achieving at least 2.0 out of 3.

What the numbers say:

• 46 of 47 appeals met or exceeded the target of 2 out of 3 and compare well to scores in previous reports

• Determinative Issues. In the large majority of appeals, reasons and decisions dealt with the issues addressed during the hearing and the evidence adduced, with rare exceptions where the member’s reasons parted from the key issues cited at the hearing.
• In one hearing, the Minister’s counsel objected to considerations made under section 130 of the Regulations. The member sustained the objection but in the written reasons reversed that ruling and rendered a decision that ultimately favoured the appellant.

Decisions provide findings and analysis necessary to justify conclusions

What was measured:

The target is 95% of cases achieving at least 2.0 out of 3.

What the numbers say:

• Members scored at or near 100% nationally.
• Although all the targets were met for these indicators, scores surpass the target marginally and are notably lower compared to other scores in this report.

• Contrary evidence. One indicator against which the quality of decisions is measured is the level of consideration the member demonstrates towards contrary evidence. The evaluator noted that occasionally, member reasons were not sufficiently comprehensive in addressing why some evidence was preferred. Also, in one instance, in a residency obligation appeal the member did not address the Minister’s main argument about skilled worker expectations in becoming a permanent resident in their reasons.

Reasons are transparent and intelligible

What was measured:

Target: 100% of cases achieving at least 2.0 out of 3.

What the numbers say:

• All scores in all regions met the target or fell within the sampling error.
• Scores averaged near 2.9, which is consistent with results from previous years.

• Oral decisions. Among the 13% of decisions in our sample delivered orally, the average score for questions 29 to 31 examining transparency and intelligibility was 2.9 out of 3. The small sub-sample sizes do not permit direct comparisons but the quality of oral reasons is broadly equal to that of written decisions in every measurement. The results also indicate that members were consistent and proficient in delivering reasons using plain language. Members also surpassed targets and demonstrated improvements in presenting clear and concise reasons that are easily understood and logically sequenced.

Decisions are rendered as soon as possible

What was measured:
Decisions rendered as soon as practicable is defined as the oral reasons rate each for allowed appeals and dismissed appeals. For this measure, the study broadened its data source from the sample to all principal decisions (n=1693) rendered during the period of November 2017 to February 2018.

What the numbers say:

• Among all allowed decisions (n=989), 23% were rendered orally.
  
• Among all dismissed decisions (n=704), 8% were rendered orally.
• Altogether, 17% of all principal decisions were delivered orally.
• The oral reasons rate stood at 21% for C​entral Region and 11% for both the Eastern and Western Regions during the period examined.

Summary

Following a review of 47 randomly selected appeals finalized during 2017-18, this study found that the members and Registry combined to achieve an aggregate score of 2.8 out of 3, exceeding the target of 2.0. A summary of the main areas of success follows:

• Dockets were organized, complete and timely 98% of the time.
• Members were prepared for their hearing 100% the time.
• Members were always courteous and respectful in nearly 100% of the hearings.
• An issue agenda was used in 100% of hearings.
• Vulnerable persons were designated and afforded procedural accommodation.
• Members’ questioning was relevant and organized nearly 100% of the time.
• In 98% of cases, reasons met expectations; key issues were addressed, findings of fact were clear, and analyses explained the member’s decision path.

Opportunities to focus attention are:

• File dockets were organized in a logical and standardized manner 94% of the time, falling short of the 100% target in the Central Region.
• In a few cases, member questions were “argumentative”, “unclear” or “vague”.
• There is a wide variance in appeals allowed between unrepresented appellants and represented appellants.
• A few reasons appeared to part from the issues cited or rulings made in the hearing.
• Ex parte communication was not disclosed and summarized to the other party in one case.
• Decisions delivered orally fell short of the target for allowed and dismissed appeals.

One recommendation is made: In a subset of cases, questioning by members was found to be argumentative, unclear, vague or leading, departing from the IRB’s expectations for relevant and focused questioning. This was not a systemic issue affecting the division as a whole. As such, the division should consider a proportionate response to ensure consistently high quality hearings, such as a professional development refresher or a communiqué to members on effective and relevant questioning.

Appendix - Checklist