Disclosure of internal audits 2006-2007


VALUE FOR MONEY AUDIT REPORT AIR FRANCE ACCIDENT INVESTIGATION

TABLE OF CONTENTS

  1. EXECUTIVE SUMMARY
  2. I. INTRODUCTION
  3. II. BACKGROUND
    1. 2.1 Costing of the Investigation
    2. 2.2 Management of Organization for the Conduct of Investigations
    3. 2.3 Accountability for Investigation
  4. III. AUDIT OBJECTIVES AND SCOPE
  5. IV. METHODOLOGY
  6. V. AUDIT OBSERVATIONS
    1. 5.1 Compliance to the legislation, policies and procedures
    2. 5.1.1 Policies, standards and procedures guiding the
    3. implementation of the CTAISB Act
    4. 5.1.2 Conferring the powers of investigators to new employees
    5. 5.1.3 Complying to the Financial Administration Act
    6. 5.2 Probity and prudence is exercised
    7. 5.2.1 Development of a Major Occurrence Investigation's Budget
    8. 5.2.2 Exercising due diligence in managing costs during
    9. an investigation
    10. 5.2.3 Management of overtime
    11. 5.2.4 Management of other significant elements during the
    12. field phase
    13. 5.2.5 Compliance to Occupational Safety and Health requirements
    14. 5.3 Appropriateness of management practices and controls
    15. 5.3.1 The participation of Administration Officers in the deployment
    16. and field phases
    17. 5.3.2 Tools and resources available to investigators
    18. 5.3.3 Use of the MOI Team List in ensuring investigators are
    19. prepared for the next major investigation
    20. 5.3.4 TSB constantly improving its investigation process
    21. 5.3.5 Management of priorities after the field phase
    22. 5.3.6 Carrying out a feedback process at the end of an investigation
    23. 5.3.7 Horizontal communications during the investigation
    24. 5.3.8 Functional support services during the investigation
  7. VI. SUMMARY OF RECOMMENDATIONS
  8. VII. MANAGEMENT RESPONSE
  9. Appendix A: Audit Objectives & Criteria

EXECUTIVE SUMMARY

The Transportation Safety Board (TSB) undertook a value for money audit of the Air France accident investigation, with emphasis on the deployment and field phases of the investigation to obtain assurance that the process to conduct major investigation is sound, that it complies with relevant legislation governing its operations and gives due regard to economy while being efficient and effective in advancing transportation safety. Auditors have used the Air France accident investigation to assess the process. This accident occurred on August 2, 2005. TSB deployed approximately 35 staff, investigators, administrative support, information technology experts and media relation specialist. Approximately 45 observers coming from Air France, Airbus, BEA, FAA, NTSB, NavCan and other organizations assisted the investigations.

The audit found that the Air France accident investigation was carried out in compliance with the Canadian Transportation Accident Investigation and Safety Board Act, the Financial Administration Act, and internal policies and procedures in place within TSB. Nonetheless, auditors believe that internal procedures could be improved to strengthen controls with respect to how investigators are assigned duties involving the exercise of the powers of investigators and with respect the financial control of certain expenditures against the investigation's budget.

The audit also established that the investigation was managed with probity and prudence. The Investigator in Charge (IIC) and Deputy IIC demonstrated the capacity to manage the activities of the investigation and have controlled appropriately the costs of the investigation. Even so, the auditors have made a number of recommendations on how the controls over certain activities such as budget preparation and costs, such as overtime could be better exercised by TSB.

Finally, auditors have recognized that TSB is constantly improving its air investigation process working for ways to accomplish its functions and discharge its responsibilities in an effective and efficient manner. Nevertheless, some areas were cited to be in need of improvement and included the use of the Major Occurrence Investigation Team List, the participation of Administration Officers during the field phase of an investigation, the management of priorities between the field phase until the completion of all groups' reports and the formalization of a feedback process for less significant investigations.

Overall, when evaluating how the investigation of the Air France Accident was conducted, auditors believe that TSB has obtained value for money in the way resources were managed. Furthermore, TSB has demonstrated its professionalism in being able to assemble an investigation team and be operational at the Operations Centre the morning following the accident, and all this happening in the peak season for summer vacation. The investigation proceeded rapidly with the deployment and field phases completed in 10 days and the initial draft investigation report tabled in July of 2006 with an expected final report to be released to the public in early 2007.

I. INTRODUCTION

The Transportation Safety Board (TSB) undertook a value for money audit of the Air France accident investigation, with emphasis on the deployment and field phases of the investigation after it was suggested by TSB management to the Treasury Board Secretariat that such an exercise would bring assurance about the process to conduct major investigation. Treasury Board approved the proposal and funded the conduct of this audit.

This report summarizes the conclusion related to an Audit Engagement Plan presented and approved by TSB management in the spring of 2006.

II. BACKGROUND

The Transportation Safety Board of Canada (TSB) is an independent agency that investigates marine, pipeline, railway and aviation transportation occurrences. Its aim is the advancement of transportation safety. It does not assign fault or determine civil or criminal liability.

Approximately 4000 occurrences are reported to the TSB every year. On average, 100 of these occurrences are investigated during any fiscal year using the TSB's base budget. Whenever an investigation is anticipated to exceed $50,000 the Transportation Safety Board can present a submission to Treasury Board seeking additional funding to cover the costs of such investigations. This was the situation with the Air France accident investigation, which the Transportation Safety Board categorized as a class 2 major investigation. Treasury Board approved $553,933 in incremental funds to the Transportation Safety Board to conduct this investigation.

The Transportation Safety Board has developed an investigation process which is documented in Manuals of Investigation (MOIs); a specific Air Investigation Branch's Major Occurrence Investigation Checklist (MOIC); and a Major Occurrence Investigation Team List. Investigators are appointed in accordance with section 10 (4) of the CTAISB Act. A TSB employee acquires the powers of an investigator specified in Section 19 of the CTAISB Act when he/she is appointed to a position which requires him/her to perform the duties of an investigator. Prior to being assigned duties that will involve the exercise of these powers, the employee must meet the following requirements:

  • completion of a recognized accident investigation course (adopted by mode) and in-house training on TSB investigation methodology, standards, and practices;
  • acquisition of sufficient experience, under supervision, in the conduct of investigations and demonstration of the knowledge and ability to perform effectively investigative tasks independently; and,
  • completion of all other mandatory training courses required by the applicable branch.

The Director of Investigation (DOI) shall ensure that the employee will not be assigned duties which involve the exercise of the powers of an investigator until the above-noted training and experience requirements have been met.1 The Air Safety Information System (ASIS) and the new TSB Investigation and Information Management System (TIIMS) are two systems that are used to manage investigation information, thus support the conduct of investigations.

All investigations are assigned a unique project code and for some major investigations a distinct responsibility centre is also assigned. The delegation of authority chart provides the Investigator in Charge (IIC) with specific authority for spending the major investigation approved budget.

Together, the investigation process, MOIs, and the Air Branch MOIC constitute the cornerstone of the management control framework for the conduct of investigations.

2.1 Costing of the Investigation

The following table provides information as to the final budget for the conduct of the Air France Accident Investigation and the actual costs at this point into the conduct of the investigation.

Table 1. 

Financial Summary based on final budget and actual costs of the Air France Accident Investigation

Detailed Costs in dollars Final Budget Actual Costs as of
Oct 17, 20062
Personnel (Overtime) $240,100 $230,284
Transportation and Communication 135,700 143,738
Information 9,000 5,690
Professional Services 60,000 49,178
Rentals 15,400 15,360
Utilities, Materials and Supplies 14,500 7,342
Gross TB Submission Amount 474,700 451,592
EBP (20%) 48,020 46,057
TSB Appropriations portion of TB Submission 522,720 497,649
Accommodation (13%) 0 0
TOTAL TB Submission Amount $522,720 $497,649

A group of approximately 35 investigators and supporting staff participated in the deployment and field work phases of this investigation. Another group of approximately 45 observers were deployed as representatives of Air France, Airbus and from other safety organizations world wide (BEA, FAA, NTSB, NavCan, and others) were part of the investigation team.

2.2 Management of Organization for the Conduct of Investigations

The Transportation Safety Board is organized with an Executive Director and two directorates: Investigation Operations and Corporate Services. Within Investigation Operations, there are three modal branches (marine, rail / pipeline, and air) and four supporting units (Engineering Branch, and Macro Analysis, Human Performance, and Publishing and Linguistic Services sections).

The Investigation Process

The regional office is the first responding TSB unit when an occurrence happens. Regional offices are organized to constantly have an officer on duty (24 hours per day / 7 days a week). In accordance with the TSB Manual of Investigation Volume 2, Part 4, the regional office must notify Head Office (HO) of any significant occurrence3. Based on the circumstances of the occurrence, the Director of Investigation (DOI), in consultation with the Chief of Investigation Operations (Chief of InvOps), may decide to establish a major occurrence investigation team. In such a situation, the DOI, in consultation with the Chief of InvOps appoints an Investigator-in-charge (IIC) and determines the initial composition and organization of the investigation team. As soon as possible after the occurrence and usually by the next day, an Operations Centre is organized at the occurrence site. The number of investigators sent to an occurrence site varies from one or two, for a relatively straightforward investigation, to 10 or more for a major investigation.

The deployment phase represents the period of time from when the decision to form a major investigation team is made until its arrival at the accident site; the field phase represents the period of time that the investigation team members arrive at the occurrence site until the majority of them return back at their normal place of work. The field phase can last from one day to several months. During this period, team members generally:

  1. secure and examine the occurrence site;
  2. examine the equipment, vehicle or wreckage;
  3. interview witnesses and company and government personnel; collect pertinent information; select and remove specific wreckage items for further examination;
  4. review documentation; and,
  5. identify potential unsafe acts and unsafe conditions.

Significant investigation activity takes place after the TSB team departs the occurrence site. The post-field phase can take many months, depending on the investigation size and complexity. At the end of the post-field phase, the IIC produces an initial draft investigation report.

The tabling of the initial draft investigation report to the TSB Board initiates the report production phase of the investigation. The Board reviews the initial draft investigation report, which may be approved, amended or returned for further staff work. Once the draft report is approved, a confidential draft report is sent to persons and corporations whose interest may be affected by the report and who are most qualified to comment on its accuracy. They then have the opportunity to dispute, correct or contradict information that they believe is incorrect or unfairly prejudicial to their interests.

This process is intended to ensure procedural fairness and the accuracy of the Board's final report. The Board considers all representations (comments) and will amend the report if required. Once the Board approves the final report, it is prepared for release to the public.

TSB Investigation and Information Management System (TIIMS)

TSB embarked in this project with the goal to develop an integrated information technology (IT) and an information management (IM) platform to support the data collection, archiving, retrieval and assessments required by investigation teams. Once the development of this system is completed, it will consist of an integrated set of documents, contents, records, cases, workflows, forms, and project management tools. TSB is currently transitioning to this information management system and some working tools. The system was piloted as part of the Air France accident investigation.

2.3 Accountability for Investigation

The Director of Investigation has exclusive authority to direct the conduct of investigations on behalf of the Board4. The designated Investigator-in-charge is responsible for the day-to-day conduct of the investigation and must report to the DOI on a regular basis. The procedures in place call for regular feedback to the DOI during the field phase on the progress of an investigation followed by a wrap-up meeting at the end of the field phase and a 60-day report during the post-field phase. At the end of an investigation, the team must do a wash-up (team back session) to assess what worked well and what needs to be improved in terms of investigation and management practices.

The Transportation Safety Board has established a series of mechanisms that introduces good management practices surrounding the conduct of investigations.

Amongst these practices, the following are highlighted:

  • Issuance and upkeep of Manuals of Investigation and an Air Investigation Branch MOIC to enable investigators and other team members to conduct safe and efficient investigations.
  • Maintenance of an adequate number of trained and available investigators.
  • Maintenance of protocols and procedures for the acquisition and use of non-TSB personnel and expertise to augment TSB resources.
  • Provision of a basic training program and on-going training for investigators.
  • Provision of advice, as required by the Director of Investigation or the Chief of InvOps to IIC and Deputy IIC or other group chairpersons.
  • The organization of the investigation team into sub-group of experts led by a group chairperson.
  • During the field phase, the conduct of daily opening and closing meetings with the IIC, Deputy IIC, group chairpersons and team members.
  • The conduct of a wrap-up session at the end of the field phase and the conduct of a wash-up at the end of the investigation.
  • Following the completion of the investigation, a review of lessons learned and the adequacy of policies, standards, procedures, checklists and equipment.

III. AUDIT OBJECTIVES AND SCOPE

The TSB has undertaken a value for money audit of the deployment and field phases of the Air France accident investigation to provide Transportation Safety Board's Executive Committee, and Treasury Board officials, with the assurance that the process in place is sound, that it complies with the Act and regulations, and gives due regard to economy while being efficient and effective in advancing transportation safety.

More specifically the scope of the audit includes an examination of:

  • Compliance: to determine if the investigation team has conformed to all laws and regulations that govern its operations, including all relevant legislation and spending authorities;
  • Financial Management: to examine the financial results of operations against plans and validate the reliability of financial data; and,
  • Performance: (value for money) to examine management practices, goals and objectives, controls and monitoring and reporting systems to assess the economy, efficiency and effectiveness of operations during the deployment and field phases of the investigation.

IV. METHODOLOGY

The methodology used consisted in the review of key documents to identify and familiarize with the policies and directives that guide the conduct of major air investigations. The auditors also met with management of the Air Investigation Branch, notably the Director of Investigation (DOI), the Investigator in charge (IIC), and the Manager, Regional Operations for the Richmond Hill regional office, several investigators who participated in the Air France accident investigation, financial management officers as well as administrative support staff who also participated in the investigation. One auditor travelled to Toronto to meet with regional staff who were directly involved with the initial response to the investigation. The auditor visited the accident site and the hotel where the Operations Centre was organized to accommodate the deployment and field phases of the investigation.

Auditors also conducted a sample review of financial transactions to ensure compliance with financial policies and regulations. They have carried out an analysis of the overtime component, as this represented a major cost in this investigation.

The audit team presented in August its initial observations to the IIC and to the Director General, Corporate Services.

This report summarizes the audit observations and provides a conclusion in relation to the three audit objectives and audit criteria established at the outset of this audit engagement. Finally, the audit report proposes recommendations to strengthen the Air Branch investigation process where necessary.

V. AUDIT OBSERVATIONS

The following section provides a conclusion with respect to each of the three objectives established at the outset of this audit engagement as well as audit observations and recommendations when appropriate.

5.1 Compliance to the legislation, policies and procedures

The audit has found that the Air France accident investigation was carried out in compliance with the Canadian Transportation Accident Investigation and Safety Board Act, the Financial Administration Act, and internal policies and procedures in place within TSB. Nonetheless, auditors believe that internal procedures could be improved to strengthen controls with respect to how investigators are assigned duties involving the exercise of the powers of investigators, with respect to the recording of commitments for the purpose of financial control and of certain expenditures against the investigation's budget.

5.1.1 Policies, standards and procedures guiding the implementation of the CTAISB Act

The TSB, as an Agency, has developed Manuals of Investigation and the Air Investigations Branch has developed a comprehensive checklist entitled the Major Occurrence Investigation Checklist (MOIC). This checklist is revised periodically when participants to an investigation identify needs for additional guidance or when efficiencies in the investigation process are identified.

An experienced Investigator-in-charge was appointed to lead the Air France accident investigation. The IIC had previously led six investigations and participated in a number of other investigations in various functions. Although the Air France accident happened during the peak of the summer holiday season, the Agency was able to respond rapidly with staff on site within a few hours of the accident. The Agency was also able to assemble a team of investigators the same day of the occurrence and an Operations Centre was established the next day ready to deal with more than 30 investigators and over 45 observers from the Air France, Airbus and other investigation agencies who participated in the investigation.

The investigation proceeded rapidly with a deployment and field phases that lasted 10 days and a subsequent post field phase managed tightly with an initial draft investigation report tabled in July of 2006 and with an expected final report to be issued to the public in early 2007.

5.1.2 Conferring the powers of investigators to new employees

The CTAISB Act specifies the process for designating investigators under the Act. Furthermore, TSB has developed procedures defining the requirements for when an investigator can be assigned duties that involve the exercise of the power of an investigator. These procedures have not been followed exposing the Agency to having unauthorized investigators exercising the powers of an investigator.

The MOI states that a TSB employee acquires the powers of an investigator when he/she is appointed to a position which requires him/her to perform the duties of an investigator. It also states that prior to being assigned duties that will involve the exercise of the investigator's powers, that the employee must meet a number of requirements pertaining to training and acquisition of sufficient experience. A process is described as to how the Agency will document that the investigator has achieved the requirements and is entitled to exercise the powers of investigators.

The audit revealed that the process described in the MOI, leading to the documentation and confirmation that all requirements have been satisfied by the investigator has not been followed. All investigators interviewed who participated in the Air France accident investigation had indeed met the requirements pertaining to training and experience as all had completed a recognized accident investigation course and had acquired several years of experience working as an investigator. However, few were aware that they also had to complete a number of investigations before being assigned duties that involve the exercise of the powers of an investigator as per MOI Vol 1, Section 3.8.3, 4th paragraph. Most have also expressed having participated in an accident investigation early upon being hired and did not realize that they were not entitled to exercise the power of an investigator until formally approved by the Director of Investigation.

RECOMMENDATION:

  1. The Director General Investigation Operations should:
    1. put a process in place to ensure that new investigators, who receive their certificate of appointment on their first day of employment, are informed that the certificate does not entitle them to exercise the powers of an investigator until they have been so authorized, as specified in MOI Volume 1; and
    2. review the validity of the process described in MOI Volumes and ensure its implementation once validated or updated.

5.1.3 Complying to the Financial Administration Act

The Agency has managed the investigation in accordance with the Financial Administration Act (FAA) with the exception of a few transactions who were approved without proper financial authorization. Auditors have completed a review of a sample of transactions of the Air France accident investigation and have found that goods and services procured at the deployment site were approved by the IIC in accordance with section 34 of the FAA. An exception to this practice relates to costs incurred in Head Office which were not always authorized by the IIC or the DOI. For instance, overtime carried out by TSB employees in Head Office and temporary help provided to the IT department were charged to the investigation budget without the IIC or DOI approval. These costs were charged to the investigation budget based on the rationale that they resulted in the replacement of staff participating in the investigation. Indirect costs should not be charged to the investigation budget unless a direct link can be established to the investigation. This can only be done by having those officers (IIC or DOI), specifically delegated with the authority to approve financial transactions under section 34 of the FAA for a distinct major investigation responsibility centre, authorize such transactions.

In the same manner, payments were approved by the Financial Officer in accordance with section 33 of the FAA who did not notice that approval of such transactions under section 34 was done by officers who were not delegated with the authority for the distinct major investigation responsibility centre.

For their part, commitments of major expenditures coincided with the establishment of the first budget approximately nine days after the beginning of the investigation. All major expenditures including overtime and accommodation are committed.

RECOMMENDATION:

  • 2. The Director of Investigation with the assistance from the Director General Corporate Services should:

    1. Issue instructions to all divisions of TSB to the effect that all costs incurred and to be charged to the investigation budget must be pre-authorized by the IIC or DOI who have delegated financial authority under section 34 of the FAA for a distinct major investigation responsibility centre budget;
    2. Ensure that Financial Officers exercising payment authority under section 33 are knowledgeable of the name of officers who have been given financial authority under section 34 for major investigations assigned with a distinct responsibility centre; and,>
    3. establish clear procedures on what type of indirect costs can be charged to an investigation's budget and disseminate such procedures to those who need to know.

5.2 Probity and prudence is exercised

In the context of this audit, probity and prudence was articulated around the development of an investigation's budget, the control of key costs during the investigation (i.e. overtime, transportation, accommodation), and the compliance with occupational safety and health requirements.

The audit found that the investigation was managed with probity and prudence and that the IIC and Deputy IIC demonstrated the capacity to manage the activities of the investigation and have controlled appropriately the costs of the investigation. Even so, the auditors have made some recommendations on how the controls over certain costs could be better exercised by TSB.

5.2.1 Development of a Major Occurrence Investigation's Budget

An investigation budget for the Air France accident investigation was developed rapidly upon initiating the accident investigation which facilitated the Treasury Board submission process. However, areas for improvements have been noticed which, when corrected could improve the efficiency of both the budget preparation exercise as well as the efficiency of the investigation.

Auditors believe that the preparation of an investigation budget took disproportionate attention having to be produced within the first few days of an investigation. The morning of the deployment, before the IIC had reached the Operations Centre, he received an e-mail reminding him to prepare the budget spreadsheet showing costs, commitments and forecast for all investigation areas. During the first week of the field phase, the IIC and Deputy IIC spent the equivalent of a week's time (over 35 hours) to prepare the budget not counting the time spent by one Administration Officer who devoted the major portion of her first week to the preparation of the budget. The audit found that the team lacked the proper tools, instructions and training, and aimed at an unnecessary level of precision in the estimation of costs. For instance, the calculation of overtime was done using each investigator precise hourly rate rather than use an average salary based on classification and level. The auditors were also surprised to hear from the IIC that the measure of success of the investigation was the fact that the costs of the deployment and field phases of the investigation did not exceed the planned budget. Although important, this measure should not be the most important measure of the success of an investigation. A budget process exists to ensure proper management of costs and expenditures.

Auditors believe that the Corporate Services Directorate and other Head Office personnel could play a more active role in the development of the investigation budget. This would streamline the process and relieve the investigation team from spending time which could otherwise be spent toward the investigation itself. For instance, they could develop and formalized an investigation budget template and make it part of the Administrative Support Go Kit. They could provide training to IIC and Deputy IIC and Administrative Support staff that might be called to participate in such exercises. They could also assist the team in determining certain cost estimates such as overtime using readily available information they have, etc... They could also provide data entry capacity into the GX system if the information was provided by the Administrative Group deployed to the accident site.

RECOMMENDATIONS:

  • 3. The Director of Investigation and the Director General of Corporate Services should agree on the supporting role to be played by the Corporate Services Directorate as part of a major investigation in order to relieve the investigation team from spending too much time on budgeting and recording of transactions, allowing the IIC and Deputy IIC to concentrate on managing efficiently the investigation.
  • 4. The Director General of Corporate Services should develop an investigation budget template that would be included in the MOIC and become part of the Go Kit of the Administrative Support Group and provide training to those expected to play a role in the development of major investigation's budget.

5.2.2 Exercising due diligence in managing costs during an investigation

The DOI, IIC and Administration Group Chairperson demonstrated having a good appreciation for the eligibility of expenditures and they have complied with the acquisition process.

The audit found that the investigation team is very concerned about costs during an investigation. In fact, during interviews, all have indicated the level of efforts they devote to negotiate with other stakeholders the sharing of costs of various services required when responding to an air accident. For instance during the Air France accident investigation the following services were received without costs:

  • security services received from the Peel Regional Police Force;
  • site survey and services received from the Greater Toronto Airport Authority;
  • safety services (ambulance services) received from the City of Toronto;
  • wreckage removal received from the Air Claims Insurance Company; and,
  • expert advice received from various external transportation safety organizations.

Each region Administrative Support Officer has developed a list of potential specialized suppliers to expedite the acquisition of specialized services during an investigation. A procurement specialist provides remote assistance (from HQ) to the investigation team to ensure that procurement activities are carried out in accordance with the contracting policies and regulations.

5.2.3 Management of overtime

The management of overtime during the investigation and most importantly during the field phase could be improved through formal guidance and procedure from the Agency and increased monitoring and reporting.

TSB recognizes that the overtime costs can be one of the most significant expenditures during the deployment and field phases of an investigation. For instance, the Treasury Board presentation indicated that overtime cost was estimated at $240,100 representing 50% of the overall estimated cost of the Air France accident investigation. Although overtime represents an important cost of an investigation, it must be realized that overtime cannot be avoided during an investigation and that pressure is on the investigation team to complete their field work rapidly to ensure that evidences do not naturally disappear and to permit the clean-up of the wreckage and re-instate the use of airport runways rapidly when an accident occurs at an airport site.

Auditors noted that investigators typically worked from 7:00 in the morning until 10:00 at night during the field phase of the investigation with some investigators working until midnight or later. The IIC responsible for the management of on-site investigation activities expressed having controlled the overtime during the field phase by approving all overtime worked and by conveying to the team members the schedule for a day of work during the investigation and setting up the time for the daily meeting. Although commendable, this approach to managing overtime did not prove sufficient to avoid charges of overtime beyond these periods. All but five on-site participants have charged overtime in excess of the IIC guidelines working as much as 18 hours per day for several consecutive days. In fact, we noticed that the MOIC neither provide guidance on the number of consecutive hours that can be worked by an investigator, nor on the number of consecutives days that can be worked before a rest day is made mandatory.

TSB as an organization needs to strike a balance between completing the investigation within a reasonable time and ensuring health and safety of its workers. In formulating direction to its staff, TSB should consider whether the same administrative support staff needs to be on site or whether shift work could not be accommodated to relieve administrative staff from doing as much overtime as investigators.

Auditors also observed that TSB does not produce an overtime report for management and therefore management cannot easily appreciate the details of overtime for a particular investigation, the number of overtime hours worked per day, per investigator and for the various phases of the investigation. Without such information, it is difficult to appreciate the full picture of overtime, assess reasonableness of overtime costs and develop guidelines and/or policy for the management of overtime.

RECOMMENDATIONS:

  • 5. The Director General of Investigation should document a policy that would provide guidance with respect to overtime worked by the various participants during the field phase of an investigation as well as during subsequent phases. These guidelines should aim at striking a balance between the need to work rapidly to collect the necessary evidences before they naturally disappear while also exercise a greater control of overtime and limit the risk to the health and safety of participants during the field phase of the investigation.
  • 6. The Director General of Corporate Services should develop an overtime report that would be used by management to assess and oversee the use of overtime generally as well as during investigations.

5.2.4 Management of other significant elements during the field phase

One of the most important aspects of the investigation is locating the Operations Centre closed to the accident site, yet away from media distraction. The accommodation must respect certain criteria that have been included in the MOIC. Some of these criteria were not met in the location of the Operations Centre for the Air France accident investigation.

The Air France accident investigation team established its Operations Centre in a hotel facility on Dixie Road in Mississauga that did not have a telephone line in the Operating room nor offered a high speed internet access. Furthermore, the Investigation Team completed only a cursory evaluation of the security of the Operations Centre which led to the loss of four lap top computers that were stolen on the second night of the field phase.

The auditors learned during the audit that telephone lines and high speed internet access could not easily be installed at the hotel as TSB had to rely on a third party, Bell Canada for the installation. The installation of telephone lines and high speed internet was done without delay because of the initiative of the Deputy IIC, who through a personal contact at Bell Canada, was able to have the phone lines and the high speed internet access installed. However without such privileged relation, the set up of the telephone lines and high speed internet access would have taken several more days and would have severally jeopardized the efficient conduct of the investigation. Such a requirement to have high speed internet access is not reflected in the current version of the MOIC.

The hotel had no security feature other than keys to lock the main door to the room serving as the Operations Centre. However a door at the back of the room could not be secured, compromising the security of the Operations Centre which led to the loss of four lap top computers. The four stolen computer were new and did not contain any investigation data. However, this incident brought into question the security of investigation data and this aspect must also be addressed in securing a safe place to establish an Operations Centre. Following this incident, the investigation team arranged for additional security to be provided to the Operations Centre and defrayed approximately $500 for such services.

RECOMMENDATION:

  • 7. The Director of Investigation should review the minimum criteria for the selection of accommodation that will house a temporary Operations Centre during the field phase of an investigation. Such criteria should now include the availability of high speed internet and proper security features for the Operations room. Once documented, the criteria should be disseminated to those tasked with selecting accommodation and those tasked with ensuring the security of the Operations Centre during the field phase.

5.2.5 Compliance to Occupational Safety and Health requirements

The Air France accident investigation revealed some weaknesses with respect to Occupation Safety and Health which brought the Agency to revise some of its practices.

The MOIC identifies that the Site Coordinator/Safety Officer is the investigator responsible to ensure that all activities at the occurrence site are properly coordinated with specific emphasis on site security and site safety. This responsibility comprises ensuring that personnel on the site are adequately equipped with personal protection equipment.

The audit revealed that the investigator charged with site coordination and safety was not formerly appointed until later during the investigation. This was confirmed with the investigator and IIC. This may have led to some safety and health issues not being managed as it should have been.

A number of issues with occupational safety and health were recognized during the investigation including the unavailability of equipment to TSB regional investigators conducting initial site hazard assessment and unavailability of protective personnel equipment (PPE) to investigators during the field work, the wearing of golf shirts which are not recognized as PPE as well as the lack of awareness and training of site safety procedures by investigators. All together, nine issues were identified at a special joint meeting of the TSB OSH Committees. Corrective measures have been undertaken with respect to all nine issues identified with some issues referred for further study to the National OSH Policy Committee and the PPE sub-committee.

Following the field phase of the Air France investigation, TSB has decided to separate the function of Site Coordinator/Safety and is now recognizing two separate functions for "Site Manager" and "Site Safety". The new version of the MOIC covers each function as a separate task.

RECOMMENDATIONS:

  • 8. The Co-Chairs of the TSB National Occupational Safety and Health Policy Committee should ensure completion of the studies undertaken as part of the follow-up actions to the nine issues identified at the special joint committee meeting held subsequent to the Air France accident investigation.
  • 9. The Director of Investigation should ensure that the corrective measures recommended by the National OSH Policy Committee are reflected in the MOIC and implemented during investigations.

5.3 Appropriateness of management practices and controls

Auditors have established that TSB is constantly improving its air investigation process looking for ways to accomplish its functions and discharge its responsibilities in an effective and efficient manner. Nonetheless, some areas were cited in need of improvement and included the use of the MOI Team List, the participation of Administration Officers during the field phase of an investigation and the management of priorities after the field phase and until the completion of all groups' reports.

5.3.1 The participation of Administration Officers in the deployment and field phases

The Administration Officers participating in the field phase of the investigation were not well prepared for their role and TSB is still exposed to this difficulty should another major accident occurs.

The auditors noted that the Administration Group checklist does not provide for this group to have a group chairperson responsible for other members of the group. This is however the practice for other investigation groups participating in an investigation. Rather, the checklist identifies a role for the Head Office Administration Member and a Regional Administration Member. At the occurrence site, the checklist identifies a role for the Investigation Team Administration Officer. The checklist does not identify the criteria to consider in deciding how many administration officers are required for a major investigation. When asked this question during interviews, most have indicated that three administration officers with one of those officers being responsible for document control would be the appropriate number of support staff to conduct a major investigation.

In the Air France accident investigation, three administration officers were initially dispatched to the occurrence site. Although all had several years of experience working for the TSB, none had experience acting as administrative member on a major investigation team. A fourth member was called in during the second week to assist the initial three Administration Officers at the site. This fourth Administration Officer had previous experience through the MK Airline investigation as well as the Swiss Air investigation. She was also the only one having previous experience in using the TIIMS document control software, which was being piloted to manage documents for the Air France accident.

During the Air France field phase, confusion was evident early as to who was in charge of the administration group. Personality conflicts and difficulties were sustained in the group until the end of the investigation.

Although TSB prepares its investigators through formal training, no training is offered to administrative staff on their role and responsibilities during a major investigation, a role that is very different from their regular duties. Although TIIMS has been piloted for two major investigations since October 2004, Administration Officers have not yet received training on how to use the software package for recording documents during a major investigation. Some training to better prepare the Administration Officer to its role during a deployment and field phases of an investigation is planned for the next Administration Officer meeting to be held in the fall of 2006.

The auditors noted that all Administration Officers arrived at the occurrence site with their own go-kit and none of these go-kits were similar. There is no official go-kit for the administration group. When arriving at the site, the Administration Officers were presented with lap top computers that included Microsoft Word software when all of their forms and diskette were for Corel WordPerfect software. The computers provided were new and did not have Corel WordPerfect software loaded, which was described to the auditors as an oversight. One Administration Officer working out of the Toronto Office has indicated that a year after the accident, forms required during a major investigation have still not been distributed to them. Steps are currently being taken to correct the situation with the Air Branch currently distributing a common basic kit to all its Administration Officers and with training being organized for later this fall.

RECOMMENDATIONS:

  • 10. The three Directors of Investigation working with Human Resources should:

    1. identify the skill sets required of Administration Officer who participate in a Major Accident investigation deployment and ensure that new Administration Officers recruited possess the required skills;
    2. formalize the training for Administration Officers to include the tasks required from an Administration Officer during a major investigation including how to cope with the pressure of the work during these periods;
    3. develop a list of Administration Officers having the required skills and training required during a major investigation deployment and field phases; and,
  • 11. The Director of Investigation should review the MOIC to identify that one Administration Officer is the Group Chairperson.

5.3.2 Tools and resources available to investigators

Investigators generally have the information, tools and essential resources to discharge their responsibilities. This is the conclusion reached by the auditors following a series of interviews conducted with investigators participating in the Air France accident investigation. Investigators have stated having the information and tools to carry their job efficiently. Investigators are provided with go-kits which they maintain ready at all times. They are also provided with lap top computers and the Air Branch investigators benefit from automatic update to the policies and checklists which they required when conducting an investigation. Other non Air Branch investigators do not have automatic update of the policies and checklists.

RECOMMENDATION:

  • 12. The Director of Investigation should review practices for sending the automated update to the policies and checklists to include investigators from other supporting groups such as engineers; human performance; macro analysis and publishing and languages services.

5.3.3 Use of the MOI Team List in ensuring investigators are well prepared for the next major investigation

TSB's has launched a process to ensure that investigators expected to participate in the next major occurrence know about their roles and responsibilities. The Major Occurrence Investigation (MOI) Team List is part of that process. Improvements to the use of the list would facilitate the investigation process.

The Air Branch maintains a MOI Team List which identifies who may be called to participate in the next major investigation. The list is reviewed twice yearly in January and June of each year. Although not set in stone, the purpose of the list is to allow TSB staff to know they could be called upon should a major accident happen. The staff named on the list are then expected to review the checklist associated with their expected role and be ready should an accident occur.

The value of the list is that it shows a good level of planning and preparation. However, the list is not complete as the name of managers may be used rather than the name of the proposed investigators. This was the case for the list in place prior to the Air France Investigation. The list dated April 20, 2005 had the name of the engineering manager for three functions identified in the MOIC. The same was also true for manager responsible for human performance. The use of managers' name prevents the list from acting as a planning and preparation tool.

The list prepared in the spring of 2006 was not completed at the end of August when this report was being developed. Difficulties to confirm certain administrative support functions were holding the completion of the list. The dissemination of the list was also noted has not transpiring as it should. Several investigators interviewed had not seen the list where their name appeared. Although investigators realize that they may be called on an investigation at any time, if the MOI Team List is not properly disseminated, investigators can not prepare as easily in anticipation of the next major accident.

RECOMMENDATION:

  • 13. The Director of Investigation and the Chief of Investigation Operations should aim at completing and disseminating the MOI Team List on a timely basis so that investigators can always be ready should a major accident occur.

5.3.4 TSB constantly improving its investigation process

TSB is constantly working to improve the leadership of its investigation. An audit conducted by the International Civil Aviation Organization (ICAO) of the civil aviation system in Canada concluded that TSB was an appropriately established, properly organized and adequately funded organization capable of accomplishing its functions and responsibilities effectively and efficiently. It commented positively on the qualification criteria for the selection of investigators and the specialized training offered to acquire and maintain the level of expertise required. It established that all the necessary facilities, equipment and documentation were provided to TSB staff. Certain recommendations were formulated which are being addressed by TSB staff.

Furthermore, the auditors noted that in the recent four to five years, several major accidents have occurred allowing TSB to test its processes and constantly improve what does not work so well. Amongst some of these recent changes to the process is the reduction of the IIC's span of control by introducing two lead investigators in the process, one for the technical side and the other one for the operations side.

5.3.5 Management of priorities after the field phase

Increased management of priorities following the field phase would ensure that all reports are collected in time to support the initial draft investigation report.

The IIC has explained that maintaining the team integrity once the field phase is completed is a challenge as several investigators return to normal assignments with only some members of the investigation team returning to the Ottawa Engineering Laboratory for the completion of data gathering work and preliminary analysis. Management of priorities amongst a dismantled team remains of strategic importance at that time to ensure that investigators who have participated in the accident investigation complete their work (data gathering, analysis and reporting) on time.

Auditors have found that for the Air France investigation, this aspect proved a challenge with the initial draft investigation report being released when groups' reports were not all completed. In fact the following situation5 existed at the time of tabling the Initial draft investigation report to the Board:

. Operations Group 100% completed
. Aircraft Performance 85% completed
. Cabin Safety 80% completed
. Weather 75% completed
. Airports 50% completed
. ATS 100% completed
. Structures 100% completed
. Power plants 100% completed
. Survey Group 100% completed
. Systems 70% completed
. Recorders 70% completed

Prior to issuing its initial draft investigation report, TSB should have obtained all the groups' specialized reports in order for its conclusion to be sufficiently and appropriately supported and substantiated. TSB remains exposed when such group's reports are not completed.

The auditors believe that the major cause for delays in groups preparing their report on time is the lack of priority management between the IIC and involved regional managers after the field phase.

RECOMMENDATION:

  • 14. Based on data collection requirements defined in TSB MOI Volume 2-4, Chapter 1, the Director of Investigation together with experienced IICs and Managers of Regional Operations should review the established procedures to ensure that groups' reports are produced on time and prior to the initial draft investigation report being completed. These procedures should then be included in the MOIC.

5.3.6 Carrying out a feedback process at the end of an investigation

Although a feedback process occurs as part of a major investigation process, a similar process rarely takes place when less significant investigations are carried out.

The MOIC identifies that the IIC should conduct a "wash up" session at the end of the investigation and some more informal feedback process at the end of the field phase. Since the investigation process for the Air France accident is not yet complete, a formal wash-up can not have happened. Auditors have confirmed that a wash-up has recently occurred for the MK Airline accident investigation and that a similar activity is planned when the Air France accident investigation is completed. Investigators interviewed believe that this process should also occur when less significant investigations are carried out.

Performance evaluation of each group chairperson has also not been developed as part of the Air France investigation process. Such mechanism is required as part of the post field phase activities as per the MOIC, section 2-3. The preparation of performance evaluation following the field phase would provide useful information on the strengths of the group chairperson as well as on his/her potential improvements.

RECOMMENDATIONS:

  • 15. The Director of Investigation together with experienced IICs should:

    1. review if a feedback process should not be carried out when less significant investigations are carried out; and,
    2. ensure that performance evaluation of each group chairperson takes place during the post field phase to be in compliance with the MOIC.
  • 16. The DGIO and DOI should review all observations and recommendations to determine their applicability to other modes and take appropriate action as required.

5.3.7 Horizontal communications during the investigation

Horizontal communications are maintained throughout the investigation. Per MOIC, the DOI is responsible to keep senior management informed of investigation issues. Per interview with the DOI such responsibility is discharged in a formal manner at management and Board meetings and in a casual manner to the Chair between meetings.

5.3.8 Functional support services during the investigation

The audit has demonstrated that support services were provided by the Communications and Corporate Services groups whenever the DOI, Air Branch requested such services as part of the Air France accident investigation. The audit has demonstrated a need for more collaboration on specific aspects of the investigation so that Corporate Services and Communication may better support the DOI during investigation.

VI. SUMMARY OF RECOMMENDATIONS

  1. The Director General Investigation Operations should:
    1. put a process in place to ensure that new investigators, who receive their certificate of appointment on their first day of employment, are informed that the certificate does not entitle them to exercise the powers of an investigator until they have been so authorized, as specified in MOI Volume 1; and<
    2. review the validity of the process described in MOI Volumes and ensure its implementation once validated or updated.
  2. The Director of Investigation with the assistance from the Director General Corporate Services should:
    1. issue instructions to all divisions of TSB to the effect that all costs incurred to be charged to the investigation budget must be pre-authorized by the IIC or DOI who have delegated financial authority under section 34 of the FAA for a distinct major investigation responsibility centre budget;
    2. ensure that Financial Officers exercising payment authority under section 33 are knowledgeable of the name of officers who have been given financial authority under section 34 for major investigations assigned with a distinct responsibility centre; and,
    3. establish clear procedures on what type of indirect costs can be charged to an investigation's budget and disseminate such procedures to those who need to know.
  3. The Director of Investigation and the Director General of Corporate Services should agree on the supporting role to be played by the Corporate Services Directorate as part of a major investigation in order to relieve the investigation team from spending too much time on budgeting and recording of transactions, allowing the IIC and Deputy IIC to concentrate on managing efficiently the investigation.
  4. The Director General of Corporate Services should develop an investigation budget template that would be included in the MOIC and become part of the Go-Kit of the Administrative Support Group and provide training to those expected to play a role in the development of major investigation's budget.
  5. The Director General of Investigation should document a policy that would provide guidance with respect to overtime worked by the various participants during the field phase of an investigation as well as during subsequent phases. These guidelines should aim at striking a balance between the need to work rapidly to collect the necessary evidences before they naturally disappear while also exercise a greater control of overtime and limit the risk to the health and safety of participants during the field phase of the investigation.
  6. The Director General of Corporate Services should develop an overtime report that would be used by management to assess and oversee the use of overtime generally as well as during investigations.
  7. The Director of Investigation should review the minimum criteria for the selection of accommodation that will house a temporary Operations Centre during the field phase of an investigation. Such criteria should now include the availability of high speed internet and proper security features for the Operations room. Once documented, the criteria should be disseminated to those tasked with selecting accommodation and those tasked with ensuring the security of the Operations Centre during the field phase.
  8. The Co-Chairs of the TSB National Occupational Safety and Health Policy Committee should ensure completion of the studies undertaken as part of the follow-up actions to the nine issues identified at the special joint committee meeting held subsequent to the Air France accident investigation.
  9. The Director of Investigation should ensure that the corrective measures recommended by the National OSH Policy Committee are reflected in the MOIC and implemented during investigations.
  10. The three Directors of Investigation working with Human Resources should:
    1. identify the skill sets required of Administration Officer who participate in a Major Accident investigation deployment and ensure that new Administration Officers recruited possess the required skills;
    2. formalize the training for Administration Officers to include the tasks required from an Administration Officer during a major investigation including how to cope with the pressure of the work during these periods;
    3. develop a list of Administration Officers having the required skills and training required during a major investigation deployment and field phases; and,
  11. The Director of Investigation should review the MOIC to identify that one Administration Officer is the Group Chairperson.
  12. The Director of Investigation should review practices for sending the automated update to the policies and checklists to include investigators from other supporting groups such as engineers; human performance; macro-analysis and publishing and languages services.
  13. The Director of Investigation and the Chief of Investigation Operations should aim at completing and disseminating the MOI Team List on a timely basis so that investigators can always be ready should a major accident occur.
  14. Based on data collection requirements defined in TSB MOI Volume 2-4, Chapter 1, the Director of Investigation together with experienced IICs and Managers of Regional Operations should review the established procedures to ensure that groups' reports are produced on time and prior to the initial draft investigation report being completed. These procedures should then be included in the MOIC.
  15. The Director of Investigation together with experienced IICs should:
    1. review if a feedback process should not be carried out when less significant investigations are carried out; and,
    2. ensure that performance evaluation of each group chairperson takes place during the post field phase to be in compliance with the MOIC.
  16. The DGIO and DOI should review all observations and recommendations to determine their applicability to other modes and take appropriate action as required.

VII. MANAGEMENT RESPONSE

  • TSB management fully accepts the observations of the auditors. Management will consider the specific recommendations and will develop a detailed management response and action plan.

Appendix A: Audit Objectives & Criteria

The audit will provide assurance on the following key aspects of a good framework for conducting investigations. Based on preliminary results, the audit team has defined the following specific objectives and audit criteria for which it will provide assurance:

OBJECTIVE 1

Provide assurance that investigations are carried out in compliance to the Canadian Transportation Accident Investigation and Safety Board Act and regulations as well as the Financial Administration Act.

Audit Criteria

  • Only authorized investigators and observers participated in the Air France accident investigation; All observers have signed the appropriate sign off sheet defining their role and responsibilities.
  • Investigators have respected the powers conferred to them as part of the Canadian Transportation Accident Investigation and Safety Board Act and regulations;
  • The DOI and IIC managed the Air France accident investigation in accordance with the investigation policies, standards, process, procedures and the checklists in place;
  • Financial transactions approved during the investigation were adequately committed (section 32) and duly approved by an officer(s) delegated with the spending authority (section 34) and payment authority (section 33);
  • The DOI and IIC authorized only expenditures that are eligible under the approved additional funding decision from TB; and

OBJECTIVE 2

Provide assurance to the effect that probity and prudence (due diligence) is exercised when managing resources during the conduct of an investigation.

Audit Criteria

  • An investigation budget was developed in due time to facilitate the TB submission process;
  • Adequate levels of support from the Corporate Services Directorate were provided to facilitate the preparation of the budget, subsequent changes to the budget, the production of financial information, the acquisition process, and the overall financial and administrative management during the conduct of an investigation (budget template; facilitate access to GX and other systems to collect information; development of macros and formulas);
  • Options are being explored (both internally and externally) to facilitate the authorization of additional funding in cases of major occurrences;
  • The DOI, IIC, and the Administration Group chairperson understand what eligible as expenditures are during an investigation, know the acquisition process, facilitate the management of financial transactions and ensure proper monitoring of activities and costs during the deployment phase (i.e. overtime is managed and groups that have completed their work depart the occurrence site in due time);
  • A series of criteria facilitated the decision-making process with respect to key expenditures (transportation means; choice of accommodation; overtime); and
  • Only expenses related to the Air France accident investigation were captured under the special responsibility centre and project codes. All expenses related to the Air France accident investigation were captured under the special coding.
  • The investigation was carried out in compliance with the Occupational Health and Safety requirements.

OBJECTIVE 3

Assess the appropriateness of management practices and controls to assure efficient and effective investigations.

Audit Criteria

  • Available resources (investigators and Administration Officers; functional support personnel, equipment and go-kit) are there when an occurrence happens;
  • Investigations team members have the knowledge, the training and the expertise to conduct assigned duties in an investigation;
  • Access to external expertise is readily available when such expertise is not available in-house;
  • Investigations team members have the information, tools and essential resources to do their job well (policies/procedures);
  • All participants to an investigation are clear about their roles and responsibilities (DOI; IIC; Deputy IIC; group chairpersons; team members; observers; and functional support personnel);
  • The Air Investigations MOIs, and checklists are maintained up-to-date and are adequately disseminated to potential team members;
  • Effective horizontal communications are maintained across the organization throughout the investigation (sit reps, etc...);
  • Appropriate supervision is in place within each group (a group chairperson is appointed) and general supervision of technical, expert and administration group is appropriately exercised;
  • Progress reporting is done and corrective measures to the conduct of the investigations are made promptly;
  • Sufficient attention is dedicated to the maintenance of a document control log from the outset of the investigation; and
  • A feed back process examining investigation activities and processes (wash up) takes place and provides team members with an opportunity to improve the process.
  • Effective and efficient functional support services (i.e. Communications, Corporate Services) are provided on a timely basis throughout the investigation.

1.   TSB Manual of Investigation Operations, Volume 1, Operations General, Section 3.8 Designation of Investigators.

2.   As this is an ongoing investigation, auditors have audited the costs incurred as of the May of 2006. Costs recorded after May 2006 were only subject to a cursory review.

3.   TSB Manual of Investigation Operations, Volume 2, Part 4, Investigations Standards and Procedures, Page 37.

4.   Canadian Transportation Accident Investigation and Safety Board Act, Section 10.(2).

5.   The percentages were adjusted to mid-value when different % were presented by the IIC and the Operations Lead.