PACP Committee Meeting

    Hello, everyone. I call this meeting to order.

     Welcome to meeting number 58 of the House of Commons Standing Committee on Public Accounts.

    Pursuant to Standing Order 108(3)(g), the committee is meeting today to study Report 4, entitled "International Assistance in Support of Gender Equality—Global Affairs Canada," which is one of the 2023 Reports 1 to 4 of the Auditor General of Canada.

     I'd now like to welcome our witnesses.

    From the Office of the Auditor General, we have Karen Hogan, Auditor General. Thank you for being here today with your team. We have Martin Dompierre, assistant auditor general; and Susan Robertson, director.

    From the Department of Foreign Affairs, Trade and Development, we have Christopher MacLennan, deputy minister, international development; Patricia Peña, assistant deputy minister, partnerships for development innovation; and Natalie Lalonde, chief audit executive, office of the chief audit executive.

    Ms. Hogan, welcome back again. You have the floor, for five minutes, please.

    Mr. Chair, thank you for this opportunity to discuss our report on international assistance in support of gender equality, which was tabled in the House of Commons on March 27.

    I would like to acknowledge that this hearing is taking place on the traditional unceded territory of the Algonquin Anishinabe people.

    Gender equality and the empowerment of women and girls is a priority concern across the world, one that is captured in the United Nations sustainable development goals. In 2017, Canada launched its feminist international assistance policy, which applies to all of Global Affairs Canada's spending for international assistance.

    When women and girls have equal opportunities to succeed, they can be powerful agents of change, driving economic growth and improving the quality of life for their families and communities. Under the feminist international assistance policy, Global Affairs Canada committed to funding projects to improve gender equality and the conditions of the world's poorest and most marginalized populations.

    Our audit focused on whether the department supported gender equality in low- and middle-income countries, and reported on the results for the approximately $3.5 billion it spent on bilateral assistance projects each year since the launch of the policy. We found that the department was unable to demonstrate that its spending improved the lives of women and girls. Significant weaknesses in the department's information management practices resulted in missing or incomplete project files that could not support decision-making or demonstrate due diligence. As a result, the department is unable to track overall outcomes against policy goals.

    This is a serious issue. We recommended that the department take immediate action so that it can fully report on the outcomes of the projects it funded. This is crucial to demonstrating the value of these investments and to ensuring that women and girls are benefitting from the programs that Canada is funding.

    Global Affairs Canada also needs to address how it measures longer-term outcomes. We found that the department focused on short-term outputs, with only 2 of 26 policy indicators measuring outcomes. An example of this was a project to build washrooms and handwashing stations in schools to make them more welcoming for girls who might otherwise miss class when they are menstruating. The department did not track this project’s outcomes, such as whether school attendance and completion rates improved for girls over the long term.

    We found that although Global Affairs Canada implemented a thorough process to assess the extent to which gender is integrated in its projects, the department could do more to incorporate different identify factors. For example, while it regularly included age, it did not routinely consider other factors, such as ethnicity, family size and structure, or rural or urban location. The department should improve the integration of intersecting identity factors in its project development so that it can deliver more inclusive programming.

    This concludes my opening remarks. We would be pleased to answer any questions the committee may have.

    Thank you for your remarks.

     Turning now to Global Affairs Canada, Mr. MacLennan, you and your team have the floor for five minutes.

     Thank you very much, Chair.

    Let me begin by thanking this committee for the invitation to appear today to speak about this report.

    I am pleased to be joined by Patricia Peña, assistant deputy minister for partnerships for development innovation, who will be leading the department's response to the report, and Natalie Lalonde, our chief audit executive.

    First, I would like to acknowledge the work carried out by the Office of the Auditor General and thank them for this report. The report highlighted the importance of Canada's feminist international assistance policy, but also noted the urgent need for Global Affairs to improve its monitoring and reporting on the results of our work.

    I would like to assure the committee of how seriously Global Affairs takes the findings in this report. Canada's development assistance programming is making a tangible contribution to improving the lives of women and girls, from reducing maternal and child mortality to supporting the completion of primary education, reducing gender-based violence and supporting women's economic empowerment. These efforts matter.

    It is for this reason that we have to redouble our efforts to ensure that we have the effective information management and outcome measurement systems in place to demonstrate to Canadians the value of their investments.

    Let me begin by touching on how we are already addressing the recommendations made in this report.

    On the first recommendation, regarding GAC's information management system, I have tasked the department with immediately putting in place interim measures for a single project documents database to store and manage all project documents. At the same time, the department will integrate the findings from this report into our larger, ongoing overhaul of our grants and contributions management system. This major transformation project, launched one year ago, will allow GAC to automate and streamline its business processes and systems to give us better information for decision-making and improve the impact of Canada's international assistance.

    In response to the second recommendation, to measure and report on results of our programming, we recognize that we must find a better way to aggregate and communicate our results. With more than 1,500 international assistance projects underway at any given time, we have a large amount of data and information connected to the overall objectives of the Feminist International Assistance Policy.

    Under Ms. Peña's leadership, the department will undertake a complete review of our current approach to gathering and disseminating the outcomes of our work and will oversee the strengthening of our corporate tools, updating governance mechanisms and development of options to better report on the impacts of our development assistance programming.

    The third recommendation looks at identity factors beyond gender and age in project-level equality assessments. Global Affairs Canada is committed to improving how we ensure that every development project receives a full gender-based analysis, including questions of intersectionality. The department will immediately launch a review of the changes made in 2020 to identify where further adjustments are needed in terms of management guidance, tools and training.

    In our international assistance work, Canada has been a leader in advancing gender equality and the empowerment of women and girls for several decades. Canada works with partners across the globe to support the development of more inclusive communities and to achieve poverty reduction.

    I welcome the work of this committee and look forward to continued collaboration with the Auditor General on implementing the recommendations of the report.

    Thank you.

    Thank you.

    We will begin our first round. The first four members will have six minutes each.

    Mr. Genuis, you have the floor for six minutes, please.

     Thank you, Chair.

    Thank you to the Auditor General for her important work on this subject.

    My questions will be directed at the officials.

    Obviously this report is profoundly disappointing. This government has, with great fanfare, tried to rebrand some of the efforts that actually started under the previous government related to improving the lives of women and girls around the world. A government that is actually serious about making the lives of women and girls around the world better would be interested in bothering to measure outcomes as part of their policy indicators. That should be a pretty basic thing. If ministers, staff and the government are actually saying this isn't just about politics and they're actually trying to make people's lives better, then they would insist on the measurement and tracking of results.

    Unfortunately, the government has clearly not done that. After eight years of lots of talking about feminism and a feminist international assistance policy, we find out that the government has loved to talk about it in a domestic context, but has not in fact been asking the right questions or putting in place the mechanisms to link their actions with outcomes.

    To the officials, before us today you're even saying contradictory things, as far as I understand it. On the one hand, you are telling us that you take the work of the Auditor General seriously, but on the other hand, you are asserting that the work you have done has been making a tangible impact. The Auditor General's work says that you do not know if you are making a tangible impact because your policy indicators are not actually linked to outcomes.

    How can you sustain the position, sir, that you take seriously the findings of the Auditor General while continuing to say that you know things that in fact, based on the data, you don't know?

    I agree one hundred per cent with your point that gathering the results of our activities is critically important. For that reason, for a number of years the department has been collecting results at the outcome level for all of our projects. At any one given time, we have 1,500 projects active across all of our development assistance spending. Each one of those projects is required to have a results framework that includes ultimate outcomes, intermediate outcomes, immediate outcomes and a series of activities that support those outcomes. Each and every one of them does.

    Now, the challenge—which the Auditor General has shone a light on—is a challenge that we've known for many years. It goes well back, particularly to.... You made note of the great work that we've done, going back to Muskoka in maternal, newborn and child health. We've been learning these lessons over a number of years. The difficulty that we have is rolling up results across 1,500 individual projects in a manner that rolls it up into a corporate-level reporting against a single policy.

    Let me put this to the Auditor General then, because I don't think your explanation is entirely consistent with the report.

    The report says that, yes, you have trouble aggregating those impacts and in various cases you didn't have data, data wasn't being accessed or in some cases data wasn't available at all. More fundamentally, the policy indicators you have identified are not outcome-related. They're policy indicators, but they're not outcome indicators, so you have a problem not just with aggregating the data but with tracking the wrong things.

    Can I just ask the Auditor General if I am correct in my reading of the report?

    You are correct in that, of the 26 indicators, only two of them are outcome-based and those are at the policy level.

    I guess I would provide that, at the project level, of the ones that we looked at, we saw that project officers were doing a good job managing, monitoring and paying out when milestones were met, but I didn't see those as outcomes. Those are really more outputs.

     I acknowledge that you need both, however. You need outputs to measure medium term and short term, but you need outcomes for the long-term objectives.

     Right, and just to demystify this for those who are watching, the difference between an output and an outcome is something like this. An output is how many doctors' visits were funded for women needing a particular kind of service. An outcome is what the health outcomes are across a population. You can say that we spent a lot of money and we facilitated a certain number of doctor visits, but if that didn't actually change the health status of people overall, then you haven't achieved your outcome. If you're measuring outputs and not outcomes, then you're not actually measuring whether or not you're making a difference.

    Just to go back to the officials in the time I have left, I think we've clearly established the point that you don't actually know if your policy is making a difference in terms of outcomes, and that you cannot in fact assert that we're making a tangible impact, because you're not measuring outcomes. Do you acknowledge that?

    With respect, I would have to disagree.

    Are you disagreeing with me or with the Auditor General?

    I'm disagreeing with you—your characterization of what we're doing.

    Okay.

    Could the Auditor General clarify? Is my characterization correct?

    I can speak to only the files we looked at. Out of the 1,500 files my colleague mentioned, we looked into 619 of them and pulled a representative sample from that. I can speak only to those groups, and the ones we looked at did not have outcomes associated with them.

    You're not just disagreeing with me; you're disputing the finding of the Auditor General.

    Mr. Genuis, that is your time. We'll be back to you, I'm sure.

    Mr. Sidhu, before I turn to Ms. Bradford, who is online, I've been advised that the headset you have is outdated. I just want to flag that to you if you're planning on asking some questions. You seem to indicate it won't be an issue. Perfect—I just wanted to double-check. Thank you. I wouldn't want to overlook you for any reason.

    Ms. Bradford, you have six minutes, please.

    Thank you, Mr. Chair.

    Thank you to the witnesses for appearing before our committee today. We really appreciate hearing from you directly.

    I'm going to start with my questions for you, Mr. MacLennan.

    I was wondering if you could just give us an overview and speak to how Canada's feminist international assistance policy is supporting the most vulnerable.

    Thank you very much.

    When this policy was instituted in 2017, it was quite honestly a radical shift for the department. Not only is it about what we focus our projects on but also about the way we undertake the complete undertaking of the international development assistance responsibilities that we have.

    Since 2017, we have, over the course of the first five years of the policy...and I should note that this is the first audit of the policy we've had. The projects that are associated with this policy are only now starting to produce the results after five years. As everybody is aware, producing results in international development does not take place in two years. It has very long-term investments, which are required to achieve lasting sustainable results.

     What the department has done is that it has adjusted completely its approach, with a clear focus on gender equality and a clear focus on helping the most vulnerable in the countries in which we are active. I can give you one example of what that means. It means two things.

    The first one is that we specifically address some areas where we have inequalities between men and women, and we attack those problems directly. As people would have seen in the report, that's the 15% target we were trying to achieve, which means projects whose purpose ultimately is to address inequalities. That's a very sharp tool to go after only those elements in international development assistance.

    With the other 80% of our programming, it means that in all types of programming we undertake we ensure there are gender equality elements to everything we do, in everything from the distribution of food aid, which takes into consideration the needs of women and girls.... A really simple example is that, if you are distributing food in a food aid situation and the sacks of millet or the sacks of grain you're distributing are in 50-pound bags, it's more likely that women will not be able to carry them. That is the type of analysis we apply to ensure we distribute that food in an effective way to women, who are the leaders of households and the ones who make sure the families eat.

     That goes beyond the dollars we would spend. It goes to our efforts to work with organizations—with the WFP, the World Food Programme, and with other UN agencies such as the UNFPA, the fund for population, and other partners—to ensure that in everything they do, even if it's not specifically focused on gender equality, all of their development programming is delivering in a feminist way across all gender equality results.

     Building on that, how does GAC work with its partners—you've referred to some—including those in civil society organizations to systematically track its investments and capture FIAP's impact, assess its progress and share it? How do you work with your partners to capture that?

    There are a couple of ways. It depends on the context within which we're working. I'll use an example of working in a partner country with a local partner, let's say.

    I can talk to you about a fantastic project we have in Tanzania, looking to help girls stay in school. We work with local partners to build out their logic model of what their efforts are going to accomplish.

    That includes, as one of the elements, as I mentioned, the project performance measurement framework. It includes ultimate outcomes, intermediate and immediate outcomes, and the host of activities that are designed to support that. It should be noted that, when we're working with partners in that way, it's the partners themselves who build those frameworks. We work with them closely. We encourage them to choose the proper indicators, whether they be performance indicators on outputs or performance indicators on outcomes, to ensure they line up to the best of their ability with things such as the SDG indicators, but it is the partner who builds those frameworks.

    We then work with the partners to ensure the proper execution of the project. One of the core elements that we do is a gender-based analysis. We do a gender-based analysis—now called gender-based analysis plus—across every single one of the projects. For all the 1,500 projects I've mentioned, we do that analysis.

    That analysis is used for two reasons, basically. The first one is to enable us to determine exactly what type of a project it is. That is the gender coding you would have seen in the report. The other thing it does is that it enables us to help the partners to better integrate the gender learnings we have on the ground in their work. That is just one example of how we go about doing that.

    The AG indicated that one of the issues was GAC's ability to manage information.

    Could you elaborate on what improvements will be made with respect to that? Is there any update on the progress, and when do you anticipate these improvements will be completed?

    This is actually a very important question. It was unfortunate in the way this transpired.

    The truth is that, at Global Affairs Canada, we're aware that we have some antiquated and very slow systems for the creation of documents, the storing of documents and the retrieval of documents. This is something that we were aware of.

    It's the reason why we launched last year, or about a year ago, a transformation exercise for our entire approach to grants and contributions. We needed to do a transformation in any event, because we have to migrate to a new IM/IT platform. The platform we are currently using is expiring. We're using that as an opportunity to rethink the entire process of our grants and contributions management. That will start with a better understanding of the creation, retrieval and use of documents on an ongoing basis.

    In the interim...and this is one of the things I've learned and the department has learned from this audit: We can't wait for that to take place. This will be a multi-year process to overhaul our entire grants and contributions process. In the interim, we are going to immediately install a new database, which will be a single repository for our documents. It will enable us, in a more efficient and efficacious way, to build the document base for all of our projects, to use the document base for all of our projects and also to be able to share them when required.

    Thank you very much. That is the time.

    Ms. Gaudreau, you have the floor for six minutes.

    Thank you, Mr. Chair.

    Thanks to the witnesses for being here.

    This is an important moment. People are watching us. Personally, I think it is surprising to see some things that are going on right now when it comes to aid. I need to offer some background. I have been here since 2019, but before that I worked for several years in the community sector. For every dollar we were granted, in addition to having an accountability process that I might even call extreme, we were able to determine the impact of the money on the ground, not just in a riding, but also in a town or in a family. We know that every dollar counts. So I find it hard to understand what is going on today, but also to accept it.

    My first question is for the Auditor General.

    You requested access to certain documents in order to do your job properly, but you could not obtain them. What happened, exactly? What answer were you given? Had the documents disappeared?

    I would first like to say that I never had the impression that the department did not want to provide us with the information. In fact, it was that it was having trouble locating the information. We were given a lot of explanations. I understand that the department has many projects all over the world and the information may be difficult to access, but that is no excuse. It is crucial to have an information management system to which all senior managers have access.

    Once we received the information, we observed that the matter had been well managed. The problem was that no one knew where to find the information. That may be because individuals were saving information on their office computer and it was wiped when they left. They simply did not know where the information had been saved. The thing that was missing in the department was therefore a directory.

    This explains why it took us about four months to get access to the information. It was not because good will was lacking. It was simply attributable to mismanaged information.

    In my next questions, you will see that I am looking for solutions, but first there is something I would really like to understand.

    The organizations struggle to have enough staff to do their own accounting. We know that the department often butchers their grant applications. How is it, then, that the same department does not have all the performance measurement tools that would enable the Auditor General to reassure the public by telling them how each dollar was spent? This is a matter of considerable concern to me.

    Can you tell me more about that?

    We did have a number of gaps, in fact. It wasn't only a matter of the system that the Auditor General was just talking about. We certainly had challenges in terms of information management.

    In some cases, there were good reasons. First, we have program analysts who come from all around the world, and sometimes they use different software, because it's what is available where they are.

    As well, our projects under development often last three, four, five, six or even seven years. That means that in a department like ours, where many of the staff move around, there are changes over the life of a project. These are not excuses, but that is an explanation.

    We absolutely have to have a centralized system to which everyone has access, whether they are working on site or in Kiev or Tanzania, for example.

    However, it must be said that this was a well-known problem. We knew about it in advance. Once we received the request from the Office of the Auditor General, we knew that we would not be able to assemble all those documents in five days, the deadline given.

    So it was simply a matter of time. If you had been given a bit more time, you would have been able to provide more information.

    That is not what I had understood. I had understood that documents had been lost or records of them had not been kept.

    We were able to find all the documents, except for one project where there was a very specific situation. The problem was the deadline we were given for doing it. As I said, it was a known problem.

    In most cases, the reason was that our systems were a bit inadequate, but there were also gaps relating to our work in the department, our communication with the Office of the Auditor General, and the process of collecting documents within the department.

    What do you say to organizations whose accounting is unbelievable? How can it be that in 2023 a meeting has to be held about orchestrating it all? I know people who work in international aid. What do you tell those people? Do you tell them that you are going to sort it out in six months and then see what the result is? We know you have good intentions, but what about the process?

    I don't know what I am going to tell my constituents when I go back to my riding. Do you understand? What can I tell them?

    You are talking about another problem that goes back a very long time. It is a problem that relates to our processes within the department, which are often governed by federal laws and policies, it must be noted.

    Many of our partners tell us flat out that we are slow and it is difficult to work with Global Affairs Canada. One of the reasons is that we need to ask for a lot of information, repeatedly. For example, each year, we ask for information from each of our partners for preparing a report about the results or an update on the project. We are well aware of the situation.

    As well, as part of the initiative we undertook last year for transforming our grants and contributions, we took this issue very seriously and we have completely integrated the idea that we need to manage our relations with our partners better. We need to find a better way of inspiring more confidence on our partners' part while at the same time, of course, asking them to account for the money spent.

    Thank you.

    Mr. Desjarlais, you have the floor for six minutes.

     Thank you very much, Mr. Chair.

    I actually want to thank the Auditor General and her team for this report. I also want to thank the witnesses who are present with us today.

    It's no secret, I'm sure, in hearing the testimony from my colleagues that it is a disappointment and a terrible shame, I think, to Canada's international reputation but also to our domestic reputation...in order to actually deal with the information management systems in a fair and consistent way.

    You made mention in the previous testimony...and it's part of my benefit of going last in the round to have heard many of my colleagues' questions. They focused specifically on information management. I want to thank you for your candid response to the questions in a direct way.

    I do want to get more specific. I know it's a difficult topic, and you've had to repeat yourself many times. The information management system.... We're dealing with $3.5 billion in international support. It's arguable that a program that large would have of course contemplated what a strong information management system would look like prior to deployment or prior to looking at partnerships.

    At the very beginning and onset of these programs, why was this not ever an issue that was contemplated in terms of the weaknesses that are obvious to the Auditor General, but must also be obvious to you, Deputy Minister?

    Thank you very much.

    That's exactly right. It has been obvious to us. As I said, we have launched our process to fix the problem.

    One of the challenges—and this is a challenge that I'm sure exists across other ministries as well—is that technology is changing at rates that are difficult for us to keep up with. One of the things we've noticed is that we've ended up using multiple systems to share, create and build document bases. In our system there are some systems that are simply built for storing documents. You don't create the document there. You just store the document there. In other places a different type of system is used.

    In some of our missions abroad, they don't have the same bandwidth, for example, so they don't access in the same way. They choose to find workarounds that are more efficient—a different type of software, for example.

    This is not acceptable, I agree. That's one reason why we have launched the process to completely overhaul the way we do this, but we do have an interim solution. It won't solve the fact that we have multiple systems that are creating documents, but for the purposes of being able to ensure that we always have access to those documents in a quick and efficient manner.... I had no concerns, to be honest, that we wouldn't be able to find the documents. I just knew that it would be labourious to find the documents. In the end, we did find all of the documents.

    As you mentioned, the size of the budget that we're managing is something that we have to address immediately.

    This leads me to trying to understand how this problem was able to manifest itself so clearly.

    When were you aware of these deficits within the ministry?

    We had had an internal audit in 2021 that noted there were difficulties with our information management systems.

    I should note that one of the things that's interesting is that, in Global Affairs Canada, for the international development mandate, we actually operate at the project level, first and foremost. These types of issues, in terms of document retrieval across a series of projects all at the same time, don't manifest themselves on a daily basis. Individual projects and their project managers are on top of all of their things, and they're working on them.

     When you ask to have all of the projects and to pull up all of the following six documents, say, for each of these projects, and you pick them randomly across numerous responsibility centres, that's where you really see the breakdown in the systems.

     That's interesting.

    To try to put it in perspective.... Auditor General, it would be great if you can comment on this. Given the fact that this internal audit took place in 2021, it's probably more obvious that the ministry would have been aware of this prior to 2021. This is considering the fact that there would have been numerous other kinds of reviews.

    Do you think the position that the deputy minister's outlining today is reasonable in terms of the short-term solution? Is that really enough to provide transparency for Canadians?

    Whether this existed before 2021, I would have to argue that, yes, it would have. That's why the 2021 internal audit identified it. How far back it goes is something that would be up to the department to assess.

    I think the conversation that I had with the deputy, when we were going through this, was that the fact that they have an action plan to deal with it is great, but an IM/IT solution is a very long-term project, so they need something in the meantime to be able to demonstrate due diligence and to be able to have that global aggregate picture across the organization. That's why I was pleased to see in his commitment and opening statement today that they're going to do that.

    That, to me, is the right interim measure. You don't want to keep operating this way until a permanent solution is found, because you won't be able to demonstrate it any better next year or the year after. You'll be waiting for this IT solution, and it shouldn't just be an IT solution.

     There's training and education for individuals to understand the importance of properly gathering information, putting a file together and bringing it up the ranks. It's important for senior management to be asking about outcomes and progress. None of that was happening for this to go on as long as it has, in my perspective.

    Thank you.

    You have time for a very brief question, or I can come back to you.

    I have a question. I'm not sure if it's quick, though.

     I'll try to put it on the record, and maybe you can think about it a bit afterwards.

    It's in relation to the Auditor General. You asked for 65 projects for review, and you reviewed 35. I understand that was because of time constraints and the ministry being unable to produce those documents.

    Is it likely that those 35 projects you reviewed were, in fact, cherry-picked—I can try to find better language—in order to demonstrate a better condition for the ministry, rather than taking a total amount of or taking into account the whole file? Of course, there would be the remaining, which would be the most difficult—

    Would you stop right there? I'll allow an answer if you stop there.

    Go ahead, please.

    I'm going to correct some of the numbers. We actually selected a sample of 60 files. We received 50 that we deemed to be complete. There were nine that were incomplete and one for which we received no information whatsoever. From a statistical perspective, that's enough to allow you to infer the population of the 619 projects.

    I don't believe there was an intention to pick out certain documents or to leave an empty file. The one file missing is actually a file in Ukraine. I believe it was understandable that they didn't go and get that document, but that speaks to the need for an IM/IT solution.

    Thank you.

    Going to our second round now, Mr. Genuis, you have the floor for five minutes.

    Thank you, Chair.

    What we just heard from the Auditor General is, essentially, don't attribute to malice what can be explained by incompetence.

    The government didn't have complete information on multiple projects, but we're going to charitably assume that they didn't exclude information because it was inconvenient. They didn't provide it because they didn't have it.

    Following up on Mr. Desjarlais' testimony, Mr. MacLennan, I find the explanation that technology has changed as a basis for saying you don't have and haven't kept proper information for measuring outcomes somewhat incredible. Databasing has existed as a concept for thousands of years. It's just a matter of making sure that the new technology is able to receive information from the previous technology. However, if you never collected the information, if you were looking at the wrong indicators or if you weren't measuring outcomes-based indicators, that's not a problem of technology, surely. Is it?

     That's a problem of policy and an administrative failure on the part of the government to say, “We need this information, we need it stored somewhere and we need to ensure that, as we roll out new technology, we are incorporating the information from the previous systems into the new systems.”

    Thank you for the question.

    There are two things in there that got a little jumbled up in terms of whether or not we're identifying the outcomes we want to achieve in our projects and whether we're collecting information to do that. There were also questions around our ability to effectively manage documentation—no matter what that documentation might be.

    My point about technological changes is that, over time, new technologies come online, they get used by staff and they add another type of technology with another type of database that it's being saved in. We didn't have a single repository for documentation, no matter what the purpose of the documentation was—whether it be the original approval note that went up, which is a simple Word document, or whether it be an Excel spreadsheet or something along those lines.

     I'm sorry to jump in, but I think all of these things are problems. You didn't have a single repository, but you also weren't collecting the right information. None of those problems would have been solved if there hadn't been technological change. It seems to me that if, after months of requesting information, the Auditor General still received incomplete information.... In many cases, for the 50 projects for which information existed, the information was there but it took a long time to get it. Clearly, it wasn't being accessed on a regular basis. However, for the other projects it just didn't exist.

    I want to come back to my previous question. We ran out of time on it.

    You have continued to assert that you know the work of the government in this area has made a difference in terms of outcomes. The Auditor General has said you have no data to support that claim.

    How do you sustain that claim in light of the fact that the Auditor General says you don't have the data that would justify making that claim?

    Thank you for that question. That's actually very good.

    I will not speak for the Auditor General. What the Auditor General looked at was something that we've called the FIAP—the feminist international assistance policy—KPIs or key performance indicators. This was a set of 26 indicators that were created at the program or corporate level, i.e., they were not associated with a specific project, instead, they were a set of indicators designed to help us with the implementation of the policy.

    As noted about the $3.5 billion in annual spending, when the new policy was put in place, the government had already been working under a completely different policy in terms of where Canada's development assistance was being focused. The 26 indicators, as the Auditor General rightly points out, were fundamentally about performance or outputs—what it is that the project money is doing as opposed to outcomes. That's exactly right at this corporate level of reporting.

    The purpose of those indicators was to demonstrate to the government and to Canadians that the department was following the policy. It was readjusting the allocations that we were making—

    I'm sorry. I am almost out of time. We will have to pick this up.

    Why was the policy designed to measure outputs instead of outcomes? How was that supposed to be satisfactory to policy-makers or Canadians?

    The policy is a different document from the placemat of indicators that was developed after the policy. The indicators that were developed were about ensuring that we were turning the boat. As everybody can understand, when you have—

    I'm sorry. You're telling me you have two different sets of indicators. You have the outcome and the output indicators. The Auditor General was looking at the output indicators. You're now suggesting that you have outcome indicators somewhere, but that information wasn't provided to the Auditor General.

    Where is this outcome-based data if it wasn't referred to in the Auditor General's report?

    The outcome-based data is included in each and every one of the individual projects that the department undertakes on an annual basis. There are, on average—it varies from one year to the next—1,500 individual projects at any one time. Every single one of those projects is required to have a results framework that includes both outcomes and outputs.

    If you want I can explain where the difficulties are, and we do have real difficulties.

    Actually, gentlemen, we're going to have to come back to this. The time is up. There will be time for the opposition bench.

    I will turn now to Ms. Yip.

    You have the floor for five minutes, please.

    Thank you for coming.

    I'll start off with Mr. MacLennan.

    How much of Canada's aid or assistance is focused on gender equality?

     I would argue that all of the aid we undertake is driven by a feminist approach to delivering international assistance.

    That means multiple things. One thing it means—and this is one of the targets in the policy—is that we will specifically work with partners to design types of programming that directly address gender inequality, so that it is the purpose of the project. It informs all of our spending.

    We have very long-term relationships with some multilateral organizations, such as the Global Fund to Fight AIDS, Tuberculosis and Malaria. We have been a member of that for 20 years. Since the feminist international assistance policy, we continue to support that organization, but we have taken a far more aggressive role on the board—we have a seat on the board—to encourage and ensure that the organization also adopts a much more focused approach to gender equality. That's one way.

    Across our other development assistance projects, we integrate gender outcomes across all of our projects. They vary according to what the type of project is. There are some projects where you can have gender equality results that are built into the program that you can understand in an immediate sense. Then there are others where quite honestly it's more difficult and it's less possible to have a gender equality result per se. Every single project receives the analysis to tell us what is possible, what is needed and what we should do within each and every project.

     We work with partners. Often, these partners are women's rights organizations on the ground in the countries in which they are operating. They tell us what their needs are. We work with them to build out what makes sense from the perspective of supporting that women's rights organization to do its job. Then we build the program around that.

    I noted that the OECD ranked our programs as fourth in the world. Is that right?

    I think it was first, actually, in terms of overall contributions to gender equality.

    The OECD has something called the development assistance committee. That development assistance committee provides a uniform way for all donors to account for what they are spending all of their development assistance on.

    Over the past five years there has been a rather significant increase in the focus of Canada's international development assistance on gender equality objectives.

    You also mentioned that the “plus” is being applied to all the programs.

    Can you expand on that? Is it truly inclusive?

    That's a very good point. It does speak to one of the recommendations in this report.

    Global Affairs Canada has for a long time been one of the leaders among other donors in applying a gender lens to all of our programming. This began back in the late 1990s with one of our first policies, called women in development, and has evolved significantly since that time.

    When the feminist international assistance policy was originally launched, we were well placed. We had a dedicated gender equality team that had been basically undertaking these gender-based analyses across all of our programming for years. When the policy was launched, it became pretty clear that we needed to expand it and it needed to be the “plus”. It needed to look at other types of intersecting factors that also contribute to the vulnerability of some of the clients—some of the people we're most looking to serve.

    In 2020, we adjusted quite significantly our approach to our gender-based analysis to add that “plus” to ensure that we had a more inclusive approach to our gender-based analysis to include things such as ethnicity, religion and age, as an example. These are these intersecting criteria or issues that may lead to increased vulnerability.

    Thank you very much.

    I again yield the floor to Ms. Gaudreau, but this time for two and a half minutes.

    Okay. So I'm going to move on, although I am a bit gobsmacked.

    If you have found your documents, is it possible to provide them to the auditor general? In fact, I'm talking about the ones that were missing for responding to the request. I think it is important to finalize everything. Can you undertake to do that?

    We have continued to give her all the documents as we collect them. We have collected everything, but if there are some missing for the Auditor General, it will be possible to send them to her, certainly.

    Excellent.

    You have talked about a lot of things, be it the constraints associated with policies, the efforts you have made, the information technologies, the fact that you are a huge organization, or the structure specific to each project. I don't think those are good reasons that can be hidden behind.

    I would like to know whether you received a memorandum or a recommendation saying what it really was important and urgent to do, to demonstrate political will. Was that political will, to demonstrate transparency to people and make sure they would be able to know what results this $3.5 billion produced, tangible at Global Affairs Canada?

    I can absolutely assure you that the question of results is central to everything we do. However, it doesn't make things easier to do, particularly organization-wide. That is where our challenge lies. The real fly in the ointment is in our method and our capacity to work on 1,500 projects, to collect the indicators that are similar, and to produce a report that encompasses all those indicators. We are not the only ones to have this problem.

    What are you lacking? What can we do to help you ensure that this never happens again and that you immediately have the figures at hand if you are ever asked the question?

    You know, I am an entrepreneur. I have worked in big organizations, and that could never have been acceptable. Either it would have gone bankrupt or it would not have received any more grants.

    For example, we are already working within the Organization for Economic Cooperation and Development, the OECD. That is one of the organizations trying to help all the donor countries. The OECD itself understands the issues and problems well. For example, the Nordic countries calculate only the results at the project level. They do not even try to report on all their programming. They simply accept that in this area, the issue relates to the projects. So if you want to know what the results are, you have to look at each project to see the results properly.

    Other countries are trying other things. In our case, we are trying new things. I can give you an example, for health...

    I'm sorry. You have no time left right now, but we will be able to come back to it.

     Mr. Desjarlais, you have the floor for two and a half minutes as well.

    Thank you very much, Mr. Chair.

    Considering just how short the time is, I'll leave my last question. Perhaps we can follow up with some supplemental questions later on if we can find a better explanation for the difference between some of the outcomes on the 60 projects and the 30 projects. We just don't have enough time for it today, unfortunately. If you can provide any more information related to those projects, or the issues in relaying that information to the Auditor General and why that information was delayed, perhaps somewhat expanding some of your points in this discussion.... Perhaps you can provide in writing why the delay exists in such a significant way.

    I'll move on to the Auditor General. We're trying to sort out how we actually deal with this in a fair way so that Canadians can expect that this issue can be concluded. The 2021 internal audit was evidence of this failure.

    Auditor General, you made mention previously that it obviously would have been an accumulated problem that required addressing. Your office, of course, took the liberty of ensuring that a more in-depth audit took place, but how common is it, for example, to have an internal audit, have your audit and then monitor the action and implementation? How long does it often take, particularly for this ministry, which is such a large spending ministry and has these really unique problems of being unable to second this information in a quick fashion? It leaves me with the perspective that this may never be fixed, in some ways, just because of how complicated this is.

    Have you seen audits like this in the past? Have they reconciled themselves with the need for better transparency?

     I think this is the first time that we've audited Global Affairs Canada in a very long time, which is exactly why we wanted to go there and see how they were doing. It's a lot of money, and hopefully it really drives an impactful and important change for women and girls across the world.

    The issue is one of information management. You don't need a big, complex tool to be able to have an ability to consolidate information. I hear clear goodwill and desire from the deputy to put something in place in the interim. They had already started a project that was long term. I did not know that when we went in and looked at it, but after we found the internal audit and we saw the outcomes, I would have expected that an interim measure would have been put in place, because it is important to be able to show globally what you can do.

    I thought about all the numbers when you talked about 35. We do mention 35 somewhere in the report. It's that we found that only 35 of the projects we looked at were contained in the external reporting. It's only a little more than 50%, which speaks to the case that, when you don't have a central repository of everything, you can't speak globally to what you're accomplishing. That's why having an IM/IT tool is really critical when you have so many projects and such a volume of transactions going on.

    Thank you very much.

    Turning again to Mr. Genuis, you have the floor for five minutes.

    Thank you, Chair.

    I will summarize where I think we've been with the questions I've asked and then try to get more clarity on this.

    A central objective of our development assistance is to make life better for women and girls around the world. The Auditor General has said, in my reading of the report, that we don't know whether the government policy is having the intended outcome because we're not tracking outcomes at a corporate level. Our deputy minister has said that we have output and outcome measurements. We're aggregating the output measurements, yes, but we're not aggregating the outcome measurements, but we do look at outcomes at the individual project level. That's what I understood you to say.

    Further, we know that the information on projects is not being regularly accessed. We know this because it took months for the Auditor General to get data—in the end, incomplete data—about the various projects. Integrating this concern about what is being measured and how that information is accessed, we can see that, if outcomes are being looked at in individual, local projects, they are not being measured systematically across the board, and information is not being accessed and used by policy-makers. I come back to the point that I think the Auditor General made, which is that we don't know. We're not tracking or testing to see if our policy is having the intended effect of making life better for women and girls around the world.

    Mr. McLennan, is my summary of things correct?

    I would argue that what we are doing is—

    Can you answer my direct question and then offer an explanation rather than skipping over the answer to the explanation?

     I agree with you in part.

    Which parts do I have wrong?

    At the very end, the notion that because we cannot and have difficulty in translating all of the individual outcomes at the project level into a higher-order corporate roll-up of outcomes, that we're not getting outcomes for the policy, that's—

    I didn't say you're not getting outcomes. I'm saying you don't know if you're getting outcomes—

    But I do know I'm getting outcomes. The problem is—

    There's information about outcomes that exists in these individual projects. That information was not made available to the Auditor General. It is not rolled up or looked at in a systematic way, and that information, even the output information, is not accessed on a regular basis, but you nonetheless are sure that, in a general sense, outcomes are being achieved. I don't know how you know that.

    Maybe let me take one step back, if you'll permit me.

    There were significant challenges within the department to deliver the documents to the Auditor General. In part, that was because of our systems, but it was only in part because of that. There were also some unfortunate gaps that took place over the course of the two to three months of providing documents, and miscommunication between my department and the Office of the Auditor General—

     I'm sorry to jump in again, but that's not the issue. Yes, maybe there was miscommunication between offices, but the bottom line is that, if the senior offices at Global Affairs had regular access to those documents and were using the information in those documents in an ongoing way, if that were the case, it would have been easy to provide.

    It seems, in the Auditor General's conclusion, which she had referred to previously, that because those documents were not readily available to senior managers, it suggested that those documents were not being used and accessed on a regular basis by senior managers.

    The documents are used at the project level by the project officers. The problem took place at a corporate level in terms of effectively requesting, tracking and understanding which documents we had actually provided to the Auditor General's office, and understanding which documents of everything that we provided were actually core to ensuring that each individual project file was complete, according to what the Auditor General required.

    Right, but the fact that you're not accessing those documents at a corporate senior level makes it impossible for me to understand on what basis you assert that, at a corporate global level, you're making a difference.

    Maybe I'm just misunderstanding your connection between the fact—

    The assertion that you're making a difference for women and girls depends on the ability to corporately measure outcomes. If you're not corporately measuring outcomes, how can you say that you're making a difference for women and girls around the world?

    Because we measure outcomes at the project level, first and foremost. That is the place where you know whether or not you're getting outcomes for the actual project in which you are investing money.

    The difficulty, and it's pointed out by the Auditor General, is in being able to roll that up, 1,500 individual projects, all with individual project indicators.

    But you don't have the outcome measured if you're not rolling it up.

    We'll have to come back to this.

    Mr. Fragiskatos, you have the floor for five minutes.

    Thank you, Chair.

    Thank you to all of you for being here.

    Auditor General, are you aware of and does your report delve into other democracies that have had this challenge? Presumably, if Canada, when it comes to overseas development assistance, has had this issue, particularly relating to information management and tracking, do we know of other democracies that have experienced this?

    Mr. Chair, I am going to ask Mr. Dompierre to respond.

     I have a quick answer. We did not specifically look at other democracies in terms of their systems or their challenges. We looked at how Canada ranked in terms of the feminist policy against other countries, so we didn't go down and look specifically at other challenges that exist abroad or in other countries.

    I'm trying to understand how unique this is. Is it something that's unique to our country?

    I can go to the deputy minister, perhaps, Mr. MacLennan.

     I can add, and then I'll allow the deputy minister to speak to it.

    Everyone can acknowledge that in this area of assistance, it's difficult to demonstrate outcomes, but just because it's difficult, it doesn't mean you shouldn't attempt to do it.

    In our perspective, we saw that the department hadn't set itself up from the get-go to measure and monitor those outcomes globally. We saw that individual programs are well tracked, but it stopped there. That information didn't flow up, and that's what the deputy has been explaining.

    Deputy, in this vein, in engagements with G7 partners, which I expect have been regular for you since you've held the role, what have you heard? Is this something that other countries are experiencing, whether in the G7 or outside of the G7?

    The short answer is yes. This is a difficulty that extends across all donors, and it includes organizations such as the United Nations and the World Bank as well.

    Part of the problem is that, when you do international assistance, you are spread across every possible sector of public policy. You are working in the health care sector, the social development sector, the governance sector and the economic sectors. We have grants and contributions. We actually have repayable contributions of various types.

    All of this means that we are spread across every possible thing, for example, that all the government departments of the Government of Canada are spread across, plus the provinces. The ability, on an annual basis, to take the results from individual projects...because that is the fundamental unit of account that my team and I hold ourselves to. We are making an individual decision to finance an organization to do something. That is the fundamental unit of account.

    The point that's been made and the finding in this audit is a very valid one: We absolutely still need to attempt to find ways to tell that story at a more corporate level.

     I'm going to again situate this within a broader context, looking at democracies that are struggling with this. There must be successes as well.

    Are you looking at other countries that have had greater success in terms of tracking outputs and outcomes?

    Where is that? Which countries would you say are doing it well, and is there room for modelling their approach into the Canadian experience?

    That's a very good point. The truth is that, for most of our partner countries, each one is doing it in a slightly different way. Sometimes that's because they're governed by different legislation within their own frameworks, and they're required to do things differently, but they are all facing the same challenge. We all function in more or less the same way, which is the fundamental project is the unit of account. They all have that issue.

    For some countries, such as the Nordic ones, for the most part they've decided to not try. They've decided that the best approach is to focus on an effective discussion and reporting on results at the project level. Other countries, I believe the Netherlands—I don't have my notes in front of me—also use country-level reports. For example, you not only have country programs, but you have investments in multilateral organizations that operate in countries and you roll up the results at the country level. What does your contribution contribute to?

    Even there, that's where it starts to fall apart. The reason it starts to fall apart is this: How do you determine exactly what your contribution is vis-à-vis other donors who are acting in the same space vis-à-vis the national government itself? Canada works in Tanzania, for example. We provide direct support to the health sector in that country. We're not the only country that provides support to the health sector. Also, we're working to support local organizations that are working in the health sector, with all of them looking to achieve a KPI result on something like reducing child mortality and reducing maternal mortality.

    You can sometimes get into games of methodologies where you try to account for what the Canadian contribution could be. Given that we provided 17% of the total program funding for this project, does that mean we get 17% of the results? It's there where you start trying to figure out whether it makes a lot of sense to do that.

    It's a real conundrum, how to do it at strategic levels—

    Thank you. I hate to do that, but I have to at some point. I try to allow for answers, and I think we had an answer there. I'm sure we can come back to that line of questioning as well from the government side.

    Mr. McCauley, you have the floor for five minutes, please.

    Thanks. I'll be passing my time to Mr. Genuis—no, I'm just kidding.

     AG Hogan, thanks again as usual for the report.

    Mr. MacLennan, you spoke about KPIs. Who is developing those KPIs? Who's setting them?

    There are numerous types of KPIs in the department—

    Yes, more specifically—

    Is it specifically the ones that the Auditor General referred to?

    It's specifically the international aid and the issue at hand, which is the lack of recorded outcomes.

    One set of KPIs, the one that the Auditor General looked at specifically, was a set of KPIs that exists at the corporate level in our department. They were designed immediately within about one year of the launch of the feminist international assistance policy. Their primary purpose was to ensure that the department was refocusing its development assistance programming to align with the new policy.

    Is it fair to say they're very high-level ones and not...?

    They're very high level. They are, to the Auditor General's point, focused more on outputs than on outcomes.

    Okay. You mentioned 1,500 projects on the go. Who is developing those? Who is approving those? Is anyone setting targets and outcomes for those?

    That varies quite significantly. For example, a significant portion of our development assistance spending goes to large multilateral organizations that we are supporting. When we work with something, and I'll use an example, the Global Fund to Fight Malaria, Tuberculosis and AIDS, they actually set their own results framework and we contribute to that framework. We know going in; they set it.

    Do we follow up on it?

    Absolutely, we have a seat on the board, as a matter of fact. We sit on the board—

    We're able to record and measure....

    Absolutely—and much more. We also use that seat on the board to hold the Global Fund to account on the effective spending of the Global Fund money at strategic levels and in the countries in which they work.

     How long will it take us to get to a point where the AG and Parliament will be satisfied with what we're actually achieving with the money that taxpayers are contributing?

    To that question, I really do hope to work very closely with the AG.

    I don't want to be difficult here, but this is a project that Global Affairs has to develop and implement. I'm not sure if conversing with the AG is going to speed up the effort. It's pretty clear what we need to do.

    You have a very large budget. We're not talking about a system that needs to do quantum physics. It's just tracking simple outcomes from a relatively small number of projects. It's 1,500 projects. We're not talking about a million different outcomes.

    What is it going to take to get to simply tracking the outcomes?

    I would like to agree with you that it's not a lot of projects and that it's not actually very difficult to do this. Nothing would make me happier than being able to say to the committee that this was a simple oversight and can be corrected really quickly and easily.

    The reason I mentioned working with the Auditor General is that this is the type of thing around knowing whether you have it right or not.

    What's the difficult part?

    I'm looking at your departmental plans and you have here that 167,318,000 people got a double dosage of vitamin A. You have that exact amount. How difficult is it to measure the outcomes of 1,500 projects so that you can report to Parliament or the AG? If you have the detail of how many vitamin A pills you've given out, how difficult is it to measure an outcome like that?

    Tim Hortons can have a franchise where they press a button and can see.... They have sales similar to the amount of money you're spending per year. They can measure exactly how much they're achieving.

    Perhaps I'm oversimplifying, but why is it so difficult to track the outcomes of 1,500?

    Absolutely. The example you've used is actually a very good example. What you're referring to is something called Nutrition International, which is a Canadian organization. I'll use data from 2020. In 2020, it provided 98.9 million children with two doses of vitamin A. That's the output. In the Auditor General's language, that would be an output. That led to 78,000 child deaths averted in 61 countries. Now, that's the outcome.

    The problem and the challenge that we have is that at the project level.... We have a project with Nutrition International with that framework that I mentioned. That's why I know that result immediately. The problem is that with 1,500 projects with multiple outcomes, you're looking at potentially thousands of outcomes. Some of those outcomes can be combined perhaps at higher orders. For example, reducing child mortality or reducing maternal mortality are relatively simple ones. We know how to measure those.

    Improving governance, for example, is one of the other things we do. I think, as parliamentarians, you would understand this really well. We actually do parliamentarian training in countries. The outcome of that is much more difficult to nail down exactly, but we could figure it out at a project level. What it rolls up to and what it combines with, unless we're simply reproducing potentially 3,000 indicators in a report, is where it becomes more difficult.

    It's not insurmountable. It's going to require a lot of hard work to get there, though. One thing that I am very confident about—for example, one of the more significant commitments that the government has made on global health and sexual reproductive health and rights is a commitment of $1.4 billion in annual spending—is that we've learned a great deal from Muskoka, Muskoka 2.0 and the work we did on SRHR from 2017 to 2020 about finding better ways to create a report that will speak to Canadians at the outcome level.

    Now that's within one portion—

    Thank you. We'll have to come back after this next portion.

    Mrs. Shanahan, you have the floor for five minutes please.

     Thank you, Chair.

    I've been listening with great interest to Mr. MacLennan.

    I always appreciate and thank you, Auditor General, for the work you do, because I really feel that this is an evolving area of practice, measurement, output and outcomes. We've had previous audits regarding organizations here in Canada. During the pandemic, for example, that was one of the issues that was brought up. This is really an executive MBA or public administration course, sitting here, to understand better how we can...and I get it.

    This is my first question. If the outcomes are measured on an individual project basis, are you able to show us that? I believe one of the comments the Auditor General made was that GAC was just not able to show those outcomes at the aggregate level. Are you able to show those outcomes to us at a project level?

    The short answer to that is yes. Where there were challenges...and I think the Auditor General pointed to one of those. In the projects that were looked at—all projects approved basically between 2017 and 2020—very few of those projects were actually completed yet at the time the audit was done. Some projects can demonstrate interim results, but results at the project level usually require the full completion of the project before you get the final report with the results rolled up according to the results framework developed for each and every project. Even then....

    Maybe you'd like to hear from Patricia, if you're getting tired of my voice. She has a great example of a project that is completed and the types of results we're seeing. Normally, you don't see results even sometimes after the project is completed—for example, sometimes when you're dealing with issues that require cultural change. In the gender equality feminist approaches, a key thing that we're trying to accomplish is an actual change in culture. That doesn't happen in three years. You don't change a culture in three years, but you can start to turn the ship on the culture there, as an example.

    I would like to hear further examples. You mentioned Nutrition International. I had a chance to see one of their projects when I was in Senegal recently with the Canada-Africa Parliamentary Association. I know that all my colleagues have travelled as well, and we've had a chance to visit projects. You realize that you're in a very different world as well when you're visiting. It's so moving and impactful to see what people are doing on the ground.

    Yes, I would like to hear some more examples. I think that will help us in our work.

    Thank you.

    Maybe I'll refer to one that's not health-related, because we've had a few examples of that. In the area of economic empowerment, one area that we've been working on in Bolivia is aquaculture, helping women fishers to feed their communities and to build some capabilities in small business by essentially growing fish. This has a lot of different dimensions to it.

    First of all, we've established specific people who are going to benefit, targeting women but benefiting all their communities. We know that the indirect benefit is huge. They've increased their food security. We give them training. We give them some skills. We give them resources to build this capacity, and then that multiplies. They have better nutrition, because they're eating more fish. They're starting small businesses. They are hiring other people, because as businesses grow they create businesses along the value chain. A restaurant now sells some of those products. They participate in local value chains. They sell their fish to markets. They're better educated. They're then able to keep their children in schools, keep them healthier and use that income.

    It creates that long-term tail. What's really exciting about projects like this is that our involvement has a particular time frame, but we're actually able to know that three, four and five years later the benefits of those early interventions are really delivering for those communities long after we're gone.

     Would you say that the output indicator, then, would be the number of children staying in school or the income levels? Is that the kind of thing that we're looking at?

    I can give you some specifics. Productivity has increased by a factor of 6% in these communities. There are 254 loans, for instance, in this project that would never have existed before. The diets of targeted communities toward fish have increased by 20%, to 30% to 40%, depending on the community.

    We're seeing it in real results in the various communities in these areas, and they're healthier.

    So that's all rolling out, on an aggregate level.

    Thank you very much.

    Ms. Gaudreau, you have the floor for two and a half minutes.

    Thank you, Mr. Chair.

    We have been told a lot of things. However, I have to be honest: what I have been told to justify the failure to disclose certain projects does not satisfy me. It has not persuaded me.

    Based on what I have heard from Ms. Peña, I am wondering, technically, how it can be that the tools are not available to measure the effect and consequences of all the mechanisms for helping these women become autonomous and start up businesses. I'm not trying to find out the value of a dollar invested, because even if I asked the United Way, they would tell me that $1 is equivalent to $20 in food aid for a family.

    Ms. Hogan, what are the procedures when you ask for something, but you don't receive everything you need to do your audit properly? How does that work? Could you give me an explanation? I'm not an expert in this field.

    It is a process of continuous dialogue between our audit teams and the department. When the request is made, we ordinarily set a deadline. Sometimes we negotiate it, but there always has to be a deadline set; otherwise, we would never get the information needed for doing our work.

    In this case, we made a request again and did continuous follow‑up over several months, during which we received a very small volume of documents, little by little. We drafted a version of our report that showed the amount of information we were missing. I believe that it was sent to the deputy minister, who said that it was not acceptable and the documents would have to be found. The process took its course, and ultimately we received a lot of documents.

    However, even though we have now reached the end of the audit, there are still ten files that are incomplete or do not contain any information.

    I would like to know whether you have concerns about the management of the institution. I would like to know what you think about that.

    It concerns me because in the files we have received, it is clear that it was the manager piloting the project at the outset, and it was senior management and supervisors who took over after that.

    Why did no one ask for a summary of the information on all the projects? A comprehensive picture of everything the department does could have been provided. That shows that there is a lack of training about the importance of processing information properly and about how to interpret data in order to make properly informed decisions.

    Thank you, Ms. Gaudreau.

    Mr. Desjarlais, you have the floor for two and a half minutes. The clock is ticking.

    Thank you very much, Mr. Chair.

    I want to turn to one of the recommendations by the Auditor General, 4.34. For the convenience of the members, I'll read it: “Global Affairs Canada should consider identity factors beyond [gender and] age in its project-level gender equality assessments to support more inclusive programming.”

    It was noted by the Auditor General that it assessed aspects of the gender inequality work within your department. However, it found that it didn't necessarily expand that review to the other aspects of equality and equity that are now commonly seen as important factors in human rights in this country and hopefully beyond—so that “plus” portion of it.

    In speaking directly to that, how are you going to compensate for deficiencies going forward?

     Of the three recommendations, this is one that's quite interesting.

    As I mentioned, the policy was launched in 2017. Between 2017 and 2020, we absolutely recognized exactly the issue that the Auditor General has pointed out here.

    In 2020, significant changes were made to our approach to doing gender-based analysis, and that's when the plus was added. We added those criteria of, as I've said, ethnicity, race, religion and age, these intersecting things that may contribute to increased vulnerability and that should be integrated into the way that projects are designed and rolled out.

    The 10 projects that the Auditor General looked at were all approved prior to 2020, so they were all approved prior to the change that was made in the department in 2020.

    What we're going to do, though, is launch immediately a review to verify that the changes we made in 2020 respond to the findings that the Auditor General had of projects that existed prior to 2020, and, if changes are necessary, if we need to change procedures, if we need to change training, for example, or if we need to change the guidance documents that our project officers use, then we'll undertake that.

    Who would be responsible for that, would you say, in your ministry? Is it some of the colleagues you have present with you?

    Patricia, thankfully, is going to oversee all of the response—

    Oh, wow.

    —but we have a dedicated team of gender equality experts.

    In our department, we have specialists across a variety of sectors such as agriculture and food, climate and health, but we also have a dedicated group of gender equality specialists. We probably, in the OECD DAC, have one of the larger teams who does this work. We are recognized as being some of the more advanced. For example, at Global Affairs Canada, we have a gender-based approach to evaluations now that we do in the department.

    This team is the one who supports it—

    Thank you.

    Mr. Desjarlais, I'm going to try to get one more round in, so if you'll allow me, I'll keep moving here.

    Mr. McCauley, you have the floor for five minutes, please.

    Mr. Chair, thanks.

    AG Hogan, looking at your report 4.3, you mention that the “audit only looked at bilateral development assistance.”

    Is there a specific reason for that and not others, or what other ones are there?

     I would point to some testimony that the deputy minister gave.

    In international assistance, there are multilateral investments, where a pile of money from many countries gets accumulated, so it's really hard to identify and target what Canada's contribution might have done to advance the outcomes of a multilateral organization, which is why we focused in on the bilateral. It also makes up $3.5 billion out of the $5.2 billion. It's a majority of the spending.

    It almost sounds like you were giving him a break by trying to look at the ones that are easier to track, and we still failed.

     I was trying to focus in on where we can truly assess the value of Canada's investment.

    Right.

    It was not at all to give them a break but to be able to target and show the value to Canadians.

    Great.

    Jumping over to 4.15 and, Mr. MacLennan, this may be more for you.

    The AG writes that, to conduct their work, they needed Global Affairs to provide documents. The department didn't provide all the documents. They said they were too difficult to locate and not available to audit, they didn't keep data repositories or they were left on computers that they couldn't access. I'm flummoxed how we could have billions of dollars being spent that we contract....

    Have every one of these what I think are relatively simple issues been immediately addressed? How did it get to this?

    In part, it's that there are multiple systems that teams are working on. Global Affairs is a large, amalgamated department and has multiple IM/IT systems that are being used.

    I understand that, but I mean, “Bill's no longer with us. We can't access the data on his computer.” It's 2023.

    Exactly. Those types of problems will be immediately resolved.

    Was the data lost, or did we just not know where to look for it?

    No. The only data that was lost was for the one project that was in Ukraine. That data had to be destroyed when the team left Kyiv last year. All of the other documentation has been retrieved.

    The problem is exactly as you pointed out. It was long and labourious to do it because it required the spoke activities to find all of the documents, which is not at all.... It's a huge gap in our system. We have a long-term solution, but we can't wait for the long-term solution. That, I think, is the real benefit of the audit.

     When you say “long term”, what's long term to you? How many years is that?

    The overall project, if it's.... Patricia can speak in greater detail—

    I'm just looking for a ballpark figure of how many years it is when you say “long-term solution”.

    Up until the end of 2026 is the entire project. The interim solution will be immediate, which is the ability to have a single database, where all project information is stored and is easily accessible.

    When you're looking to do something far more advanced in terms of things like machinery and being able to use AI to help with some of the problems of how to get to aggregation, for example, that's to come in this larger project of the transformation initiative.

    How much is your department looking for in the main estimates and budget for funding for this fiscal year?

    I'm sorry. I'd have to get back to you on that specifically.

    Mr. McCauley, do you want that answer?

    No. I can look it up, I'm sure. Thank you.

    I'll read something to you from section 4.40. This is the most disturbing line:

    Have you fixed that?

    Ms. Hogan, are you satisfied, with the response to this specific issue or to your report, that we can get to that?

    I'll speak first, and then we'll ask the deputy to answer on how he plans to address it.

    This speaks to the overall policy. The policy had just spending commitments. Spend 15% on direct gender integration. Spend 50%—

    That's why I said it's the most worrying.

    That doesn't really show the outcome and the whole objective of the feminist policy. That shows where to spend money. In my view, that's likely a reason why the majority of indicators were output-based. They were in line with spending commitments versus outcomes.

    Perhaps the deputy would like to talk about solutions.

    Yes. We'll have to come back to that, but I am hoping for another round.

    I'll turn now to Ms. Bradford. You have the floor for five minutes, please.

    Thank you, Mr. Chair.

    I want to change the line of questioning a bit.

    Mr. MacLennan, the OAG found that the department did not meet its commitments to spend 15% of its assistance on gender-targeted projects—

    Ms. Bradford, pardon me. Can you raise your boom mike a bit?

    I'll double-check with translation. I've stopped the clock.

     I'm getting a thumbs-up. Go ahead, please. I'll restart the clock.

    Okay.

    I'm changing up the line of questioning a bit, Mr. MacLennan.

    The OAG found that the department did not meet its commitments to spend 15% of its assistance on gender-targeted projects or to spend 50% of bilateral funding for sub-Saharan Africa, at least until 2021-22. Are you able to tell us whether the commitments were met in 2022-23?

    No. At this point, I don't have that data yet.

    What is the department doing to ensure that the three spending commitments under the feminist international assistance policy are being tracked?

     I can assure you that those three commitments are tracked very regularly. We've been tracking them since the moment the policy was launched.

    The degree of change from 2017, quite honestly, is quite significant. As an example, the commitment with respect to gender equality-focused programming moved from our baseline of 3% in 2015-16 to a total of 12% in 2021. It had actually gone to 14% in 2019-20.

    We track it on a regular basis throughout the year and the change over time was quite significant for that one. On sub-Saharan Africa, we reached a high of 49%, which is just shy of actually achieving the target that was made.

    I can let the committee know that the department regularly manages its budget against a long number of programming commitments and spending commitments. That's normal. It's gone on for as long as I've worked in international assistance, which is numerous years now and through numerous governments. The difficulty with a percentage target, however, is that we don't always control the denominator. What took place with COVID-19, followed by the invasion of Ukraine, fundamentally changed the denominator.

    Ukraine and other crises are not in sub-Saharan Africa, so Canada ramping up its efforts to respond to those crises obviously had an negative impact on the percentage total going to sub-Saharan Africa. For COVID-19, the nature of the investments we made were fundamentally about immediately procuring and distributing vaccines. That doesn't lend itself to being a GE3, as it's called. As a result, our ability to hit that target was diminished.

    Okay.

    The OAG “reviewed documents from relevant senior management committees from June 2017 to September 2022, and...did not find evidence that senior managers regularly reviewed gender equality outcomes or progress on policy goals.”

    Do you agree with this finding? If so, what's the explanation for it?

    To my knowledge, the committees the Auditor General looked at were the Global Affairs corporate committees, which are a series of committees that are spread across the full mandate of all Global Affairs Canada. My assumption is that the report is accurate.

    Since I was named deputy minister, which was a little over a year ago, in January 2022, I've created a dedicated international assistance operations committee, whose very purpose is to look at the breadth and depth of operations challenges we have in the department. That includes performance management and the tracking of results. That's now been integrated fully into this new committee that I've struck, which includes all of the assistant deputy ministers in the department and is dedicated to international development assistance.

    Okay, that took care of my next question.

    I don't actually have any other questions at the moment.

    Thank you.

    Thank you very much.

    I'm going to endeavour to get through a last round. I want to just cut the time from five minutes to four minutes for the government and official opposition members, and two minutes for the two other opposition parties.

    Mr. Genuis, you have the floor for four minutes, please.

    I had asked some questions earlier about outputs versus outcomes. I want to bring the Auditor General in on it.

    A few minutes have elapsed since my previous rounds, but I was referencing you a few times, Madam Auditor General. At a broad level, do you think the government has the evidence to support the claim they're making when they say that this is making a difference for women and girls around the world?

    I think at a broad level that's exactly what we said in the report. It's that they couldn't show us at a broad level.

    I do acknowledge that, at a project level, they can track. I do think we should acknowledge that. For the example I gave of building washrooms and putting washrooms and handwashing stations in schools, that is of value to the students in that school. However, when the goal is to increase attendance at school, that's what you should also measure. There is some value, but the ultimate goal is really the one that needs to be measured.

     I think that's an important point. If you're spending large amounts of money and governments are saying something was done with that money, and it seems to, in particular circumstances, be making someone's life better, then that is important in a certain sense, but the larger point is this: Are we getting value for money and having the broadest, greatest impact we can? Are we tracking to make sure the measures are working to achieve the outcomes we've established we want to have?

    Also, for the Auditor General, the government says that they have outcome-based data, but they're just not consistent in the indicators they're using. Did you see outcome-based data at the project level, or is your impression that they don't have that data at the project level?

    I would say that at times we did see outcome-based data. My bathroom project is one for which we wouldn't have. Ultimately, when you're spending $3.5 billion, accounting for it in million-dollar tranches is difficult. You need to have a global picture to even decide whether or not you should reallocate funds to a certain area of the world.

    That information's needed in order to have well-informed decision-making about investments.

    In terms of the existence of outcome-based data at the local level, you said there were cases in which it was there. It sounds as though the implication is that there were also cases in which there was not outcome-based data. Of the 50 projects you looked at, in how many was there identifiable outcome-based data at the local level?

    I'm happy for others from your office to chime in.

    I'm going to have to turn to someone else to see if they have that level of detail, or we will have to get back to you.

    We'd have to get back to you on that.

    We can say that around half of the projects were reporting against the corporate indicators. As the department's been saying, all of the projects are required to do logic models and identify ultimate outcomes. As has also been pointed out, because we were fairly early on in the projects' implementation, many of them hadn't completed their work. It wasn't time to report on outcomes.

    We didn't see serious problems with the way those tools that were being implemented at the project level were created. Where we were seeing the disconnect was that what the projects were trying to do wasn't translating into corporate indicators.

    Right.

    If there aren't local outcomes in about half of the cases, then I think the best the government can say is that some of the projects are doing some good, and for some of the projects we don't know. I would appreciate a written follow-up if there are things you haven't had a chance to say, certainly with a breakdown showing the projects for which outcome-based data was available in some form and the projects for which it was not.

    Thank you.

    Are you able to provide that?

    We'll look into it. If we don't have anything further to add, then we'll let you know either way.

    That would be appreciated. Thank you.

    Ms. Yip, you have the floor for four minutes. Go ahead, please.

    Mr. MacLennan, in your opening statement you talked about strengthening your corporate tools and updating governance mechanisms. Could you elaborate more on that, especially on the governance?

    Sure. Why don't I begin at the top? I should indicate to the committee that I've worked in this space and at the former CIDA for many years. I came back to the department—nominated as the deputy minister—in January 2021 with a deep knowledge. Following the amalgamation of the department, one of my immediate observations was that we really did require a senior table that looked at development operations—not policy but operations. That is first and foremost the biggest tool that we've put in place to manage governance. It's not quite a year old yet. We're still running it.

    I can assure the committee that the findings of the Auditor General's report have caused quite a significant reflection on even how much more important this committee is than we thought previously. The question of results—I've heard the members around the table—clearly is something that all members hold close and expect the department to deliver on. I expect that from myself, and I expect that from the committee and from the department as well.

    The team is up to the challenge of doing this. I do believe, actually, that the teams themselves are the ones that hold most closely the importance of delivering actual results. The teams that are in place are fully capable of doing the job. They want to see results more than anybody else does, to be perfectly frank. They're the ones on the ground working directly with partners to do this.

    Maybe I can turn to Patricia, who's going to be leading the....

     As an example, the deputy mentioned that I'll be following up on the action items for the recommendations in this report. I'll be leading a team that looks at those, and we'll be tracking progress. When we reach particular milestones, we will bring that to this international assistance operations committee so that the committee is apprised, so that we can get feedback and so that we can seek direction and then be able to do everything we've committed to do.

    Thank you.

    Did the pandemic hinder your programs and impact outputs and outcomes?

    Yes, obviously the pandemic had a significant impact, and quite a varied impact, depending on what part of the world you're talking about. In some parts of the world, it required us to slow down projects, as teams on the ground were forced to go off-line. They went through the exact same experience that we went through. It was figuring out how to do this online. How do you monitor and evaluate the work you're supporting and funding? It did lead to a slowdown.

     In some cases, we needed to extend the projects for time lost during COVID to allow them to catch up on the results they were trying to accomplish. In other circumstances, within a project that was, for example, largely health or education—two really good examples—we worked with the partners to adjust what they were doing to respond more clearly to the challenges that COVID was causing. That's a really good example. In some of our education programs, we allowed and worked with partners to figure out ways to support online learning for students in various developing countries where we were doing those types of projects.

    That is the time.

    Thank you.

    Ms. Gaudreau, you have only two minutes, I'm sorry.

    Thank you, Mr. Chair.

    Mr. MacLennan, I have three requests. First, I would like to know how funds are allocated among the various countries. Can that information be submitted to the committee?

    Absolutely.

    Great.

    Second, when all the projects have been provided to the Auditor General, can we be informed, so we can note in our report that all the information requested was found, compiled and sent?

    Third, you have one minute to tell us about the concrete action plan and the measures to be taken to make sure that all efforts are made, starting now, and that things are back on track when it comes to gender equality. When we meet again, things will have to have been resolved.

    I can assure you that this is central to everything we are going to do to give specific responses to the recommendations, which is only a start. We are going to continue our momentum in this matter and completely overhaul the way we show Canadians everything we are doing in relation to development.

    Even though we may invite you back to the committee, I suggest that you let us know what you have done, because we need to be reassured. I said at the outset that I was very concerned, but I know there will soon be results.

    Thank you.

    Thank you.

    Mr. Desjarlais, you have the floor for two minutes now as well.

    Thank you very much, Mr. Chair.

    I want to turn directly to some comments that were made earlier in the last round of questions in regard to the deadline of 2016. Mr. McCauley was able to have you admit to a deadline in terms of some of the major commitments for change. You mentioned, as well, that AI and some other aspects of technology would also be implemented. I believe you provided evidence to suggest why it would be so long.

    In terms of some of that work that will be undertaken for the robustness of this new system, can you provide details as to what this system is in terms of its difference from the existing one?

     Certainly. To clarify, on the database we've committed to implementing, we're working on that immediately to make sure that we can find all the documents and they can be tracked. That's our interim solution.

    In terms of our longer-term approach, immediately, we've already started working on essentially taking all of these disparate legacy systems that don't talk to one another, and we're going to be building a new way of managing grants and contributions. On one hand, it's an IM/IT solution, so it's a technological solution, but it's really looking at how we operate.

     It's going to look at how we manage data—so that's data collection—but then also the analytics around that. It's going to link to our finances. Currently, we don't have a way to get the systems to speak to one another. It's then going to track our results. It's taking bits of information and bits of ways to work that we're doing in a variety of different ways and that we can now cobble together. However, as the report has shown, it takes us time and it's not straightforward.

    Very importantly, we are working very closely with our partners, including our Canadian partners. They are involved with us in this process to make sure that it's a system that works not just for us, as Global Affairs Canada and as the Government of Canada, but also for them, because they are the front line in delivering these projects on our behalf.

    Thank you. That was perfect.

    Turning back to the official opposition, is it Mr. Genuis or Mr. McCauley? Do you have any more questions? You have four minutes.

    If you're satisfied, that's okay too.

    Mr. Chair, I'm far from satisfied with my time or with the situation at the department.

    Can I pick up where I left off? I didn't know I would have more time, but maybe if the Auditor General....

    You were answering some of the previous question. I wonder if you can pick up on this issue of what information is available around outcomes on an individual project level and the challenges you had in accessing that information.

    I'm not sure I can add too much more. I can say it in different words.

    I would repeat that it was there sometimes, but it's really tracked and monitored just at the project level. We didn't see senior management asking for it, but there also wasn't a way for project managers to roll it up and push it. Sometimes it happens, because about 50% of the projects are rolled up into external reporting, so it's about consistency across all of the areas of the organization.

    Thank you for that.

    Mr. MacLennan, do you accept as true what was just said in that outcomes-based measurements are only available or only exist in the case of about half of the projects? What's going on with the other half of the projects?

    No, I can't agree with that, actually.

    One of the things I can say is that only 20% of the projects the Auditor General reviewed were actually completed. The other 80% were still, as we understood.... Because they were begun basically over the first three years of the policy, 80% of the projects are still actually active in the field, so we haven't gotten to the point of a completed project to collect the results yet.

     Every project has not only the outcomes that are expected, but they also have, for example, the indicators to tell those outcome stories.

    Going back to the Auditor General, is that fair from your perspective, or is something being missed?

    When we picked the projects, we made sure that the projects had at least one year of reporting, because we didn't want to select projects and say there was really nothing to look at. I acknowledge that outcomes sometimes take more than one year to get there, but you need to set yourself up to monitor and gather that information. I would say that's a nuance I would add to the response.

    For the projects that were not complete, did you see evidence of the collection of outcomes-based data that was simply ongoing? Did you see an indication that outcomes-based data was not being collected, or that there was no plan to collect outcomes-based data?

    Again, I'm going to turn to Susan's comments that, at times, there was outcomes-based data, and at times, there was not. We saw project managers doing a good job using the tools, but you need to set yourself up to monitor the right things.

     In the absence of having more.... I don't think we have more at our fingertips to give you more specifics than what you—

     What I got was that in 50% of the projects there was outcome-based data collected and in 50% of the projects there was not outcome-based data collected, or is it that, of those 50%, there might have been outcome-based collection going on but the data was just not available yet?

    Mr. MacLennan is kind of painting the picture that they were trying to collect outcome-based data in every case. It was just that they hadn't gotten to it in some cases yet, but what you're saying is—

    I'm at a place where I'd prefer us to go back and really look through our files to give you a more accurate answer, instead of misspeaking.

    I will turn to Susan to see if she wants to add something.

    I can clarify where some of the disconnect is here. The projects are generating their own outcomes and objectives, and then Global Affairs is approving the projects with these pre-existing set of objectives in them. The problem is that those objectives don't necessarily map onto the KPIs that the department is tracking and collecting.

    We can tell you—and I don't remember off the top of my head—how many projects we looked at that were performing according to expectations and how many were below expectations. Those kinds of things are being tracked at the department level. We know whether or not the projects are performing as expected, which I think is partly what you're trying to get at here.

    The disconnect is that what those projects are actually tracking as outcomes are not the same as the KPIs that the department is trying to track. For example, if you have a project that doesn't happen to have anything to do with vitamin A but is doing something related to nutrition, it won't contribute to the vitamin A statistic that's being reported. It's not that it's not producing a result. It's that it's not producing that result, so it can't contribute to the reports that are being generated at the departmental level.

    I hope that helps clarify the disconnect I think I'm hearing.

    Thank you. We are out of time. I appreciate the answer. Can we expect some information, or do we think that covers it?

    We've committed to go look, and if we want to expand or provide more, we will.

    Thank you.

    Mr. Fragiskatos, you have the last four minutes. You're welcome to use as much or as little of that as you'd like.

    It's over to you, sir.

    Thank you very much, Chair.

    With the last question, let me say at the outset that I very much believe and have always believed that Canada needs a very strong overseas development assistance policy.

    The challenge here is with the problems that have been identified, and they are serious in nature. I'm glad to hear that there have been reflections and that there will be efforts to rectify, based on the Auditor General's report. The worry that I have is that, when things like this do happen, they fuel an argument that exists in Canada and that persists about development assistance in general: Is it worth it?

    It's a question of getting back to basics. Deputy Minister, tell us why it's still important for us to have a strong overseas development assistance policy. What's in it for Canada, and what's in it for Canadians?

    Obviously, it's difficult for me to not agree with that statement.

    I think our official development assistance is one of the primary tools in Canada's tool kit when it comes to our overseas interests—our national interests that are overseas.

    The best example of this was the pandemic. I can tell you that, in the world of international assistance, we've been talking about the value of investing in health care systems in places like Tanzania and Mozambique, to enable those countries to not only treat their own citizens but to also contribute to global health. That's an actual thing. The control of infectious diseases requires all countries to work in co-operation. We need to be there to help our partners build those systems because, in the end, they also contribute to protecting Canadians.

    On climate change, we are working with our developing country partners to help them respond to climate change, to adapt to the impacts of climate change but also to ensure that they're not contributing and that they don't take the pathway of high emissions in their development, so that we are actually contributing to the reduction of global and greenhouse gas emissions for all of the planet. Canada can do everything it wants, but we know that the rest of the world also has to act. We need to be there to help developing countries do that as well.

     I take your point. I obviously agree, but I'm going to push you a bit more.

    Tell it to my constituents, who won't be watching this because they're busy at work trying to put together their next paycheque so they can feed their family. Why is it important that we continue to invest large sums? Granted, it's not what Pearson called for, 0.7% of GDP, but my understanding is that it is around 0.3% of GDP, which is not insignificant at all.

    What's in it for the everyday Londoner? What's in it for the everyday Canadian? You did touch on this when you talked about the pandemic and global health systems, and how ultimately a focus on the international does help to ensure the security of Canadians, but can you go into that a bit more, Deputy Minister?

    South Korea used to be one of the poorest countries on the planet. It is one of the prime development success stories. South Korea is now an important trading partner with Canada. In the Indo-Pacific, South Korea is increasing as an important ally for Canada. It is a thriving democracy in a world in which democracy is in retreat. We need more South Koreas for Canada.

    Thank you very much everyone. We ran a few minutes late.

    You have my apologies, Mr. Zimmer. I didn't start right on the money, so we're two minutes late.

    However, I appreciate everyone coming today. Have a good weekend for those of you who are getting away tonight.

    I will adjourn the meeting.