Independent review of Quality Assurance Framework

​​​ ​Printable version (PDF, 326​ KB)

​​​Dear Richard:

Thank you for asking me to review the Immigration and Refugee Board's Quality Assurance Framework and underlying artefacts. In summary, the Framework is robust when measured against national and international standards and, indeed, could provide a model for similarly situated administrative tribunals in Canada and abroad. I have no hesitation in stating that the Framework is optimally designed to ensure quality decision-making and high performance at the Board going forward. My qualifications are set out in Annex A.

Background

The Board is Canada's largest administrative tribunal. It is organized into four divisions: the Immigration Division and Immigration Appeal Division deal with immigration matters, especially sponsorship applications and inadmissibility; the Refugee Protection Division and the Refugee Appeal Division deal principally with questions relating to refugee status. These divisions contain, in turn, geographic regions from coast to coast to coast. Board members render around 70,000 decisions annually.

This snapshot provides a modest picture of the complexity with which the Board must grapple. Some further details help to illustrate the challenging environment in which the Board operates:

• Members are not recruited in the same way: appointments to the Refugee Protection Division and Immigration Division are made by the Board; but by the Governor in Council to the Immigration Appeal Division and Refugee Appeal Division. This has implications for recruitment strategies.
• The Board must set and meet quantitative and qualitative standards: it must render a large number of decisions each year and these decisions must be of an acceptable quality, as defined by the expectations of the individuals who appear before the Board and the federal courts sitting on judicial review of the Board's decisions. Moreover, this large quantity of decisions must be produced through a process which is qualitatively fair.
• The Board must be responsive to a variety of external stakeholders: the Treasury Board; Parliament; the federal courts; and the public. These stakeholders have a variety of interests – economy, efficiency, effectiveness, legality, fairness and accuracy – which are not always readily reconcilable.
• The Board's work is often in the public eye: immigration and refugee policy is regularly a source of political and public debate; and the quantity and quality of the Board's decisions, and the fairness of the Board's processes often come under scrutiny. In this regard, it is worth noting that the Board is required to give effect to Canadian values: on the one hand, ensuring that individuals appearing before the Board are treated with concern and respect for their dignity; on the other hand, ensuring that the Board's decisions accurately accord appropriate status to those individuals.
• The Board often deals with individuals who are vulnerable and who do not have access to legal advice. Board members conduct inquisitorial hearings which are quite different from the adversarial processes which characterize judicial decision-making and much administrative tribunal adjudication. As they cannot necessarily rely on an individual's representative or lawyer to make the individual's case, Board members often have to delve deeply into highly sensitive material with only their training and experience to guide them.
• The underlying subject-matter of the Board's work is, by any measure, inherently complex. Contemporary immigration and refugee law contains interwoven strands of international treaties, domestic statutes and regulations, ‘soft' law promulgated by international and domestic bodies, judicial decisions and past Board practice. Furthermore, the terrain on which the Board makes factual findings is constantly shifting, especially in refugee matters as conditions in individuals' countries of origin change regularly. Making accurate decisions which treat similarly situated individuals similarly is an ongoing challenge.
• Lastly, members enjoy a high degree of adjudicative independence and cannot be micromanaged by Board leadership.

In general, the guiding principles supporting the Framework – compliance, consistency, equity, independence, collaboration and ongoing process of improvement – are well designed to respond to this inherently challenging environment. The Framework is organized around the stages of a continuous improvement cycle: Plan, Do, Monitor and Measure, and Adjust.

At this point in time the Framework and its underlying artefacts speak mostly to the Refugee Protection Division and Refugee Appeal Division. The Immigration Division and Immigration Appeal Division feature less prominently though they also have similar practices. In terms of assessing the robustness and optimality of the Framework, I have assumed that the practices on the refugee side will gradually be implemented on the immigration side to the extent they are not already, with due attention to any relevant differences in the operational context.

The 'Virtuous Circle'

There have been relatively few sustained attempts to establish publicly available criteria of excellence in administrative decision-making and regulatory performance: these are listed in Annex B. One notable example is the Alberta Energy Regulator's Model for Regulatory Excellence – the Regulator, however, is quite different from the Board. The Regulator is a policy-making body with some adjudicative functions which it performs in a quasi-judicial manner. The Board is essentially an adjudicative body which functions much more like a court of law.

In the United Kingdom's Administrative Justice and Tribunals Council Landscape Paper, the administrative justice system was envisaged as a virtuous circle: see Annex C. In essence, quality administrative decision-making is about getting it right, setting it right and putting it right. I address these in turn by reference to the Framework.

Getting it Right

Getting it right is about making accurate decisions through an appropriately fair process. The Framework is well designed to ensure that Board members get decisions right in terms of substance and procedure.

As noted in section 3.6, “Adjudicative and Support Tools”, the Board has developed a variety of tools “primarily concerned with improving information access and horizontal member engagement, which subsequently enhances decision quality”. Procedurally, these include the Chairperson's guidelines, designed to set out guiding principles for adjudicating and managing cases, for example those relating to child refugees or vulnerable persons. Substantively, these include jurisprudential guides developed to generate consistent decision-making across claimants and across regions: the legitimately wide scope for such guides was recently affirmed by the Federal Court of Appeal in Canadian Association of Refugee Lawyers v. Canada (Immigration, Refugees and Citizenship), 2020 FCA 196. Adjudicative strategies of this type are subject to ongoing review and revision on the basis of objective, metrics-driven analysis (section 2.3).

Relatedly, the Refugee Protection Division has the capacity to put in place specialized teams whose Members “develop advanced subject matter expertise in areas related to country conditions, claim types, questioning techniques, and dealing with sensitive claims and vulnerable claimants” (section 3.7). Again, this enhances the substantive accuracy and procedural fairness of decisions.

Lastly, reasons review by legal services (section 3.5) allows the Board to identify decisions which have substantive weakness. Ideally, draft reasons would be reviewed in every case, but this is more realistic in low-volume Divisions than in high-volume Divisions where decisions have to be rendered in a timely manner.

Overall, the Framework sets out appropriate means for getting it right.

Setting it Right

Setting it right is about empowering decision-makers to make substantively accurate and procedurally fair decisions; put another way, it involves calibrating institutional structures to ensure that decision-makers are in a position to get it right. Much of the Framework is oriented to putting processes in place which give Members the skills and knowledge they need to reach substantively accurate and procedurally fair decisions.

The Framework takes a holistic approach to setting it right, beginning with merit-based recruitment where feasible (section 3.1), training and mentoring for new Members (section 2.1) and ongoing professional development to provide Members with appropriate substantive and procedural updates including in the writing of reasons (section 3.3). The presence of a Member Learning Secretariat, which draws on adult education principles in designing programming, is especially notable and praiseworthy. The Board also has adjudicative consistency groups (section 3.7) designed to achieve consistency across claimants and regions, in the interests of promoting accuracy and equal treatment of similarly situated individuals.

Furthermore, these processes are subject to dynamic revision as part of the Adjust component of the Framework. When prompted by audits and QMIs (section 6.1), training (section 6.2), adjudicative strategies (section 6.3) and individual remedial programs (section 6.4) are adjusted. As such, ongoing monitoring of performance feeds holistically into the recalibration of key components of the Framework. This is an innovative means of responding to the challenging environment in which the Board operates.

A key institutional innovation here is the development of Quality Centres (section 4.5). These will have a wide remit and will no doubt become a cornerstone of the Framework.

Overall, the Framework sets out appropriate means for setting it right.

Putting it Right

Putting it right is about remedying decisions which are substantively inaccurate or procedurally unfair. The most common means of putting it right in administrative agencies are internal appeal and judicial review. The Board has such mechanisms at its disposal: the Refugee Appeal Division and Immigration Appeal Division are capable of putting it right, as are the federal courts.

Commendably, the Board has been proactive in developing alternative means of putting it right.

Quantitatively, the Refugee Protection Division operates with a set of performance indicators as part of a “nationally agreed upon approach for performance expectations” (section 4.1). Board-wide, internal and external reporting also ensures strong performance (section 4.2).

Qualitatively, the Board engages third parties to conduct reviews, for example of reasons for decision in order to measure decisions against a “set of standards for quality decision-making” (section 5​.1). Mentors assist members to ensure the quality of their decisions and decision-making processes (section 3.4); where necessary remedial programs can be put in place (section 6.4).

Lastly, there is a robust process for dealing with complaints about Member conduct which falls short of substantive inaccuracy or procedural unfairness (section 5.4). Complaints and responses are a means of putting it right which also may feed back into the revision of processes for setting it right​.

Overall, the Framework sets out appropriate means for putting it right.

Conclusion

The Framework is an extremely impressive document, well supported by the underlying artefacts and which represents best practice in getting it right, setting it right and putting it right. Across all these components of the ‘virtuous circle’ of administrative decision-making, the Framework is clear, comprehensive and innovative, a thorough response to the Board’s challenging environment.

Yours faithfully,

Signed: Paul Daly

Annex A: Short-form CV

Select professional experience

Current (2019-): University Research Chair in Administrative Law & Governance, Faculty of Law (Common Law Section), University of Ottawa; Review Officer (Part-time), Environmental Protection Tribunal of Canada

Previous: Senior Lecturer in Public Law, Faculty of Law, University of Cambridge (2016-2019 and 2017-2019: Deputy Director of Graduate Research); Professeur invité, Université Paris II Panthéon-Assas (2017-2018); Professeur adjoint, professeur agrégé & vice-doyen, Faculté de droit, Université de Montréal (2012-2016); Student-at-Law,Lerners LLP (2011-2012); Replacement Professor, Faculty of Law, University of Ottawa (2010-2011)

Education

2007-2010       Ph.D.   University of Cambridge, Faculty of Law & Queens' College

2006-2007       LL.M.  University of Pennsylvania Law School

2005-2006       LL.M.  University College Cork

2002-2005       B.C.L. University College Cork

Select professional associattions

2017- Canadian Journal of Administrative Law & Practice (Editorial Board)

2012- Law Society of Ontario

2009- Attorney at Law (New York)

Expertise

Administrative Law: active researcher and teacher across multiple jurisdictions since 2005; described as Canada's leading academic administrative law academic; award-winning author of dozens of scholarly books, articles and book chapters and the internationally read blog Administrative Law Matters; sought-after bilingual public speaker; sustained involvement in training programmes for the judiciary and legal profession; scholarship cited more than 50 times by courts and administrative tribunals in Canada, Ireland and Australia

Consulting and Public Service: produced a report on the Copyright Board's decision-making procedures; wrote an expert report on administrative justice for a s. 96 constitutional challenge to the British Columbia Civil Resolution Tribunal (2020); conducted an independent, third-party review of decision-writing and record-generation at the Canadian Nuclear Safety Commission; acted as consultant and counsel in multiple files before the Supreme Court of Canada; currently on retainer to the Ontario Financial Services Regulatory Authority and the Canadian Securities Administrators;  serving part-time member of federal administrative tribunal.

Annex B: List of publications on regulatory and tribunal excellence

Alberta Energy Regulator, The Alberta Model for Regulatory Excellence (AER, Calgary, 2016)

Coglianese, Cary, Listening, Learning, Leading: A Framework for Regulatory Excellence (Penn Program on Regulation, Philadelphia, 2015)

Daly, Paul, Independent review of structure and organization of Commission's records of decision (Canadian Nuclear Safety Commission, Ottawa, 2020)

Daly, Paul, Best Practices in Administrative Decision-Making: Viewing the Copyright Board of Canada in a Comparative Light (Heritage Canada, Ottawa, 2016)

Tomlinson, Joe and Robert Thomas, “Administrative justice – A primer for policymakers and those working in the system” (UK Administrative Justice Institute, Colchester 2016)

United Kingdom Administrative Justice Institute, A Research Roadmap for Administrative Justice (Nuffield Foundation, London, 2018)

Annex C: The virtuous circle

Version accessible here