Continuous Monitoring Report (CMR) - 2017 Quarter 4

European Aviation Safety Agency Continuous Monitoring Report Estonia 2017 - Quarter 4 Limited distribution(*) This quarterly report is disseminated to National Standardisation Coordinators. It provides: - the data provided by Member States in accordance with Article 6(1)(a) of Regulation (EU) No 628/2013, - the status of open findings in each domain, and a summary of the open key findings, and - the output of the Model for Standardisation inspections' planning, in the form of a Country dashboard. Guidance on the Model and the related Dashboard is provided in the EASA Information Paper The Model for Standardisation inspections' planning. Should you notice any incomplete or erroneous entry, please report it to [email protected]. Key acronyms used in the Country Dashboard Acronym Description CAT Commercial Air Transport CPI Corruption Perception Index FTEs Full-Time Equivalent (for the purpose of calculating staff figures) GDP Gross Domestic Product, at constant prices ISC Immediate Safety Concern LEI Lack of Effective Implementation ROs Regional/Remote Authority Offices performing oversight SSC Significant Safety Concern SSP State Safety Plan Y / Y2Y Year / Comparison between current year versus previous year Exposure indicators used in the Country Dashboard Domain Exposure 1 Exposure 2 AIR Licenses Organisations OPS Aircraft operated in Commercial Air Transport Air Operator Certificates FCL Licenses Approved Training Organisations MED Certificates Aeromedical Examiners FSTD Not available Devices RAMP Number of ramp inspections in the database Total number of findings per participating state Air Traffic Control Officers licensed with at least Air Traffic Service Providers and Air Traffic ANS one valid endorsement Controllers’ Training Providers (*) This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Continuous Monitoring Report - 1 Estonia 1. Key Activity Figures / General Safety Indicators Submitted: 03/11/2017 1.1 Authority Technical staff AIR OPS FCL MED FSTD ANS RAMP ADR ROs 0 Number of FTEs 3.75 3 1.84 0.8 0.16 5 0.1 1 Y2Y change % -12.8 -31.8 -8 -20 0 -16.7 -83.3 0 Budget (Y2Y change %) 4.72 Turnover rate % 0 0 0 0 0 0 50 0 Major changes (last 2 Y) N Sharing/Outsourcing N Y N N Y N N N 1.2 Undertakings AIR OPS MED FSTD ANS ADR CAMO 13 AOC (A) 6 AeMCs 2 FFS (A) 0 ATSPs 2 ADR certified 4 AMO 145 9 AOC (H) 1 AMEs 4 FFS (H) 0 AISPs 1 ADR exempted 0 AMO F 0 NCC (A+H) 1 GMPs 0 FTD (A) 0 CNSPs 3 ADR > 5 MPax 0 MTO 147 2 SPO (A+H) 3 MC C.1 237 FTD (H) 0 METPs 3 HEL in scope 0 POA 21G 1 High Risk 1 MC C.2 206 FNPT (A) 1 TO (I) 1 Mov (ADR certified) 48676 MC C.3 117 FNPT (H) 1 TO (U) 2 Mov (ADR exempted) 0 AIR Lic FCL OHMPs 0 BITD 0 EASA 176 ATO (A) 5 JAR 0 ATO (H) 3 National 0 ATO (S) 0 ATO (B) 0 ATO (As) 0 FCL Lic MPL ATPL(A) ATPL(H) CPL (A) CPL (H) PPL (A) PPL (H) Other ANS Lic ATCO ATSEP FISO EASA 0 100 11 53 23 66 9 4 EASA 92 JAR 0 0 0 0 0 0 0 0 ICAO 0 National 0 0 0 1 0 3 0 28 National 0 0 8 1.3 Safety / Activity / General / RAMP Aircraft EU CofA Annex II Valid ARC CAT ops Documented Implemented SSP Aeroplanes 120 58 87 38 Y N Helicopters 13 0 11 2 Major Extraordinary Interim RAMP events elections Government (last 2Y) N Y Ramp Inspectors 1 New Ramp Inspectors 1 RITO 0 Annual Inspection Points (in October only) 87 This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Continuous Monitoring Report - 2 Estonia 2. Current findings' follow-up status Updated: 07/11/2017 2.1 Status of open findings Class C findings Class D findings Domain Open On track Not agreed Overdue Open On track Not agreed Overdue AIR 0 0 0 0 0 0 0 0 OPS 1 1 0 0 1 1 0 0 FCL 0 0 0 0 0 0 0 0 MED 0 0 0 0 1 1 0 0 FSTD 0 0 0 0 0 0 0 0 ANS 1 1 0 0 1 1 0 0 RAMP 0 0 0 0 2 2 0 0 Total 2 2 0 0 5 5 0 0 Not agreed = Corrective actions not agreed within the legal timeframe / Overdue = Findings with agreed corrective actions out of which at least one is overdue / Grey: never inspected in that domain / This report takes into account only findings for which a Final Report has been issued. Cross-domain findings are counted for as one finding in each domain to which they are applicable. 2.2 Open findings - Highlights Highlights include: - class D findings - class C findings deserving special attention (repetitive, long overdue, under Supplementary report, etc.) - class C findings having a substantial impact with regard to the 'even playing field' (free movement, fair competition, etc.) - findings applicable to more than one domain Domain No. Summary CE Dated Class Status SR Lack of qualified inspecting personnel for CAT aeroplane 19734 4 06/2016 C On track operators. OPS Inadequate verification of compliance with ORO.FTL 19736 6 06/2016 D On track requirements prior to the issue of approval. Appointment of the medical assessor without meeting relevant MED 19295 4 10/2015 D On track requirements. The CA/NSA could not provide evidence that they had verified an ANS 19726 ANSP’s compliance with all the common requirements before 6 06/2016 D On track issuing / renewing its certificate. due to other tasks, insufficient time to perform ramp inspection 19469 7 01/2016 D On track tasks (oversight, planning, performance) RAMP ECAA ramp inspectors do not have sufficient time available to 19473 7 01/2016 D On track perform an adequate ramp inspection preparation. CE= ICAO Critical element / SR = Supplementary Report This report takes into account only findings for which a Final Report has been issued. This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Country Dashboard Estonia Code: EE Time period: Dec-17 Updated: 06/11/2017 Authority & Follow Up indicators AIR OPS FCL MED FSTD ANS RAMP Standardisation (STD) rating 96 91 98 96 95 95 84 Months since last comprehensive insp. 33 18 26 26 21 38 21 Exposure 1 indicators 176 40 298 560 92 41 Exposure 2 indicators 25 7 8 4 2 12 15 ARI (Average Reactivity Index) 76.1 82.1 94.4 99.7 98.1 91.5 STD rating scale in quartiles: Highest Lowest Data not available SAFA Ratio (SR) Operators with SR ≥ 2 ISC Months since last comprehensive insp ANS 0.42 0 OPS AIR Safety Accidents Ratio (vs. total) Accidents last 5 years FCL MED Indicators A H A H FSTD CAT 0 0 0.00% 0.00% OPS Other 1 0 3.03% 0.00% CAT: number of CAT accidents by State of Operator / Other: number of non-CAT accidents by State of Registry Movements Movements GDP % chg CPI ICAO LEI ICAO SSC General Y2Y % chg Standardisation rating Indicators - Exposure 1 - Exposure 2 16040 -5.89 4.00 70 36.56 No Additional information Operators with SAFA Ratio ≥2 Output levels D D D D D D C AIR OPS FCL MED FSTD ANS RAMP D C D D D D D Output levels scale: Low priority A B C D High priority Level not available n/a The Country Dashboard is based on SIS data as submitted during the last data collection campaign and all changes of indicator values greater than 25% notified to [email protected]. This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Country Indicators Estonia Code EE Domain AIR Reactivity Index) 100 91.45 85.62 84.61 84.61 84.61 84.61 84.61 76.07 ARI (Average 76.07 80 60 40 20 0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 ARI (Average Reactivity Index) Average ARI Standardisation 100 96 96 96 96 97 96 96 95 93 Rating 90 86 85 80 75 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Standardisation Rating Average Standardisation Rating Licenses Exposure 1 indicators 190 60 180 40 170 20 160 0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 1 indicators Exposure 1 indicators/TS FTEs Organisations Exposure 2 indicators 40 10.00 20 5.00 0 0.00 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 2 indicators Exposure 2 indicators/TS FTEs Notes This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Country Indicators Estonia Code EE Domain OPS Reactivity Index) 100 80.6 82.3 83.9 84.5 84.5 85.5 82.5 81.5 82.1 ARI (Average 80 60 40 20 0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 ARI (Average Reactivity Index) Average ARI 91 Standardisation 100 84 90 84 90 81 77 77 79 80 60 Rating 40 20 0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Standardisation Rating Average Standardisation Rating CATops Exposure 1 indicators 60 20.0 40 10.0 20 0 0.0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 1 indicators Exposure 1 indicators/TS FTEs AOCs Exposure 2 indicators 10 4.0 5 2.0 0 0.0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 2 indicators Exposure 2 indicators/TS FTEs Notes This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Country Indicators Estonia Code EE Domain FCL 100.0 100.0 100.0 ARI (Average 100 Reactivity 93.8 93.8 94.4 94.4 94.4 94.4 95 Index) 90 85 80 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 ARI (Average Reactivity Index) Average ARI 100 98 98 98 Standardisation 97 97 97 97 95 93 93 Rating 90 85 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Standardisation Rating Average Standardisation Rating Licenses Exposure 1 indicators 400 200 200 100 0 0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 1 indicators Exposure 1 indicators/TS FTEs ATOs Exposure 2 indicators 10.0 6.00 4.00 5.0 2.00 0.0 0.00 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 2 indicators Exposure 2 indicators/TS FTEs Notes This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Country Indicators Estonia Code EE Domain MED 100.0 100.0 100.0 99.4 99.4 99.7 99.7 99.7 99.7 Reactivity Index) 100 ARI (Average 95 90 85 80 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 ARI (Average Reactivity Index) Average ARI Standardisation 100 96 96 96 94 93 95 95 95 95 Rating 89 90 85 80 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Standardisation Rating Average Standardisation Rating Certificates Exposure 1 indicators 600.0 1000.0 400.0 500.0 200.0 0.0 0.0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 1 indicators Exposure 1 indicators/TS FTEs AMEs Exposure 2 indicators 6.0 6.0 4.0 4.0 2.0 2.0 0.0 0.0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 2 indicators Exposure 2 indicators/TS FTEs Notes This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Country Indicators Estonia Code EE Domain FSTD 98.1 98.1 98.1 98.1 Reactivity Index) 93.2 89.3 100 ARI (Average 80 68.2 68.2 68.2 60 40 20 0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 ARI (Average Reactivity Index) Average ARI 100 Standardisation 94 95 95 95 95 91 91 91 93 91 Rating 90 85 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Standardisation Rating Average Standardisation Rating Exposure 1 indicators 1.5 1.0 0.5 0.0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 1 indicators Exposure 1 indicators/TS FTEs Devices Exposure 2 indicators 4.0 20.0 2.0 10.0 0.0 0.0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 2 indicators Exposure 2 indicators/TS FTEs Notes This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties.  European Aviation Safety Agency Country Indicators Estonia Code EE Domain ANS Reactivity Index) 100 97.4 97.5 97.5 97.6 97.6 ARI (Average 95 94.2 92.2 92.0 91.5 90 85 80 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 ARI (Average Reactivity Index) Average ARI Standardisation 100 95 91 91 90 93 95 Rating 90 86 87 90 88 85 80 75 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Standardisation Rating Average Standardisation Rating Certified ATCOs Exposure 1 indicators 100.0 20.0 50.0 10.0 0.0 0.0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 1 indicators Exposure 1 indicators/TS FTEs Organisations Exposure 2 indicators 13.0 3.0 12.0 2.0 11.0 1.0 10.0 0.0 Dec-15 Mar-16 Jun-16 Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Exposure 2 indicators Exposure 2 indicators/TS FTEs Notes This Report may contain confidential information in accordance with the provisions of Article 15 of Regulation (EC) No 216/2008. It is destined exclusively for the Services to which it has been expressly addressed and its contents shall not be communicated to other parties. European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning European Aviation Safety Agency Flight Standards Directorate INFORMATION PAPER The Model for Standardisation inspections’ planning October 2016 The information contained in this report is intended for internal communication and documentation of work in progress. Such information is not necessarily definitive and consequently the Agency cannot be held responsible for any damage resulting from unauthorised use of this information. If you rely on this information for purposes other than its intended use, you assume all risk associated with such use. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 1 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning EXECUTIVE SUMMARY The European Commission and the Agency identified the need to review the current working methods for conducting Standardisation inspections, to ensure they become more system-oriented, risk-based, and provide for a more efficient use of resources. Regulation (EU) No 628/2013 introduces the concept of Continuous Monitoring Approach or CMA. The main CMA objective is to transition from a system of compliance audits, which are planned and performed at regular intervals regardless of the identified issues, to a more risk-based and performance-based approach that prioritises the conduction of inspections when and where required. This will allow the Agency to allocate its resources in a targeted and proportionate manner to address the identified risks, and to avoid generating any undue burden on States’ Competent Authorities. The implementation of CMA requires a Model for collecting and analysing compliance, safety and activity indicators on a continuous basis. The Model is designed to be a decision support tool, assisting the Agency’s management in prioritising EASA’s Standardisation activities and in determining the scope and the frequency of comprehensive and focused inspections. The current Model has been developed on a continuous engagement and consultation with the Agency’s Standardisation experts, and relies on a robust statistical methodology to analyse the available data. Nevertheless, it is a “living” system, which shall be reviewed and updated regularly to ensure it remains valid and current. This paper presents (a) the CMA elements of the Standardisation Regulation, (b) the structure, the components and the outputs of the Model and (c) the methodology employed to build the Model. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 2 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning TABLE OF CONTENTS 1 THE CONTINUOUS MONITORING APPROACH 4 2 THE MODEL 6 2.1 Purpose and development 6 2.2 Model components, indicators and sources 7 2.3 Model operation 8 2.4 Trend monitoring 13 3 FURTHER STEPS 14 ANNEX I – LIST OF MODEL INDICATORS 16 ANNEX II – METHODOLOGY 19 ANNEX III – SCORING POINTS AND EXPOSURE RANGES 25 ANNEX IV - ACRONYMS 27 IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 3 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning 1 THE CONTINUOUS MONITORING APPROACH The Commission Implementing Regulation (EU) No 628/2013 (hereinafter - Standardisation Regulation), in comparison to the preceding European aviation standardisation requirements, has introduced a fundamental change in the way the Agency monitors the application by Competent Authorities of the common rules in the field of civil aviation. As stipulated in Article 3, the monitoring shall be continuous and risk-oriented, on the basis of the information available to the Agency. The main objective of this new approach to Standardisation is to transition from a system of compliance audits, which are planned and performed at regular intervals, to a more risk- and performance-oriented approach that prioritises the conduction of inspections when and where required. This would allow the Agency to allocate its available resources and technical capacities in a targeted and proportionate manner to address and manage the identified risks, thus ensuring maximum efficiency gains on a continuous basis, and to avoid generating any undue burden on States’ Competent Authorities. Such approach is in line with the evolution of the ICAO Universal Safety Oversight Audit Programme (USOAP), which has now adopted a continuous monitoring approach as described in ICAO Doc 9735*. Furthermore, the Memorandum of Cooperation signed in 2010 between the EU and ICAO creates the framework for a structured cooperation between parties. A Working arrangement was subsequentely signed between EASA and ICAO in February 2015. As a consequence, the Agency's Standardisation inspection programme and the ICAO USOAP are expected to become gradually more inter-related. Article 7 of the Standardisation Regulation defines the key continuous monitoring components as follows: (a) the collection and analysis of data and information provided by the Competent Authorities of Member States, ICAO, the European Commission or other relevant sources; (b) the assessment of the Competent Authority’s ability to discharge its safety oversight responsibilities; (c) depending on the assessment referred to in point (b), the prioritisation, planning and determination of the scope of inspections; (d) the conduct of such inspections, including the related reporting; (e) the follow-up and closure of findings of non-conformity stemming from the inspections. In essence, the Agency’s Continuous Monitoring Approach to Standardisation (CMA) is a cyclical approach that entails the following components: • Collect information: Collection of data and information from States’ Competent Authorities, ICAO, the European Commission and other sources; • Analyse and prioritise: Assessment of the Competent Authority’s ability to discharge its safety oversight responsibilities. Depending on this assessment, the prioritisation, planning and the scope of inspection is determined; * ICAO Doc 9735 AN/960 - USOAP Continuous Monitoring Manual - Third edition, 2011. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 4 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning • Act as appropriate: Conduction of such inspections; • Follow Up: Follow-up and closure of findings stemming from these inspections. Such components are illustrated in Figure 1 below: Figure 1: CMA components In the CMA logic, the maximum interval between two comprehensive (full-scope) Standardisation inspections is larger than former baseline frequency of inspections (2-years). Today the Agency performs its Standardisation inspections with a 4-years reference cycle; such interval will be further adjusted on the basis of the experience built. In between two comprehensive inspections, focused Standardisation inspections, aimed at inspecting specific areas within one or more technical domains, and/or at assessing the implementation status of agreed corrective actions, can be planned and performed on the basis of the identified priorities. Figure 2: CMA planning IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 5 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning This allows an optimal utilisation of available resources, both for the Agency and for Competent Authorities. Such concept is illustrated in Figure 2 above. 2 THE MODEL 2.1 Purpose and development The Standardisation Regulation tasks the Agency to collect and analyse relevant data and information and to assess the Competent Authority’s ability to discharge its safety oversight responsibilities. The Agency shall then prioritise, plan and determine the scope of its inspections on the basis of the outcome of such assessment. To ensure an objective and consistent assessment, the Regulation requires the Agency to establish, develop and maintain a single model (from now on referred to as “the Model”). Such Model shall take into account at least the following elements†: (a) the size and complexity of the aviation industry; (b) serious incidents, accidents, fatal accidents and related fatalities; (c) the results of Ramp inspections; (d) the results of previous inspections; (e) the ability of the Competent Authorities to implement effectively corrections and corrective actions; (f) the result of audits carried out under international conventions or State safety assessment programmes; (g) the existence of measures pursuant to Article 11(2) of Regulation (EC) No 216/2008 or to Article 258 of the Treaty. In essence, the Model represents a decision support tool aimed at enabling the Agency to organise and prioritise its inspections in the most effective and efficient way. The Model therefore shall compile, process and combine the available information in order to obtain a comprehensive picture of a Competent Authority’s safety oversight capability as well as its exposure to risk, in a consistent manner across all countries. The Agency’s Flight Standards Directorate initiated the development of the Model in agreement with the European Commission back in 2011, as soon as consensus was reached within the EASA Committee upon the move towards a risk-based CMA to Standardisation. A dedicated Working Group (CMA WG), involving subject matter experts from each Standardisation domain, was established in April 2011. The WG developed the overarching “logic” of the Model, defined its main components, and identified the relevant indicators as well as the associated data sources. Furthermore, it identified three key principles in order to maximise the Model’s effectiveness and to ensure it will produce consistently reliable output: † Article 7(2) of the Standardisation Regulation. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 6 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning • The Model is a “living” system which shall be kept constantly up-to-date with fresh data; • The Model indicators shall be reviewed regularly to ensure their validity and predictive power; • The Model output shall be subject to regular “sense-checking” and calibration by subject matter experts. 2.2 Model components, indicators and sources The first stage of the Model development was the identification by subject matter experts of specific indicators and elements which characterise Competent Authority, its performance, and its level of exposure to safety risks. The WG agreed on a combination of indicators of leading and lagging nature which reflect the Agency’s confidence in an authority’s ability to discharge its safety oversight responsibilities. The WG also ensured that all retained indicators could be objectively measured, in order to avoid subjective assessments. Indicators of a similar nature were grouped into six main components, as shown in Figure 3 below; the detailed list of all indicators and their related data sources is provided in Annex I. Authority Indicators Follow-up Indicators Activity Indicators •No. of regional offices •Progress of corrective •Aircraft registered •Staffing and budget levels actions •Organisations approved •SSP implementation •Reactivity index •Licenses issued •Standardisation findings •Supplementary reports •Movements •... •... •... Safety Indicators Time elapsed General Indicators •Immediate safety concerns •Time elapsed since last •Political stability •SAFA ratio comprehensive inspection •GDP variations •Accidents •ICAO LEI •... •... Figure 3: CMA Model components The values for the indicators described above are collected from three main sources: • EASA: Standardisation findings, follow-up status, safety indicators, SAFA ratios, etc; • Authorities: key information (size, complexity), activity data (approvals, licences, etc); • Other: ICAO (Aircraft movements and traffic trends), International Monetary Fund (macro-economic data), etc. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 7 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning In order to enable an efficient data collection process, EASA has developed a web-based interface, called Standardisation Information System (SIS). This platform aims to streamline the process and to facilitate the data submission task for Competent Authorities at regular intervals. A regular update of the SIS takes place every 6 months, unless the value of an indicator changes significantly (e.g. by more than 25%), thus requiring an ad-hoc update. As required by the Standardisation Regulation, each country has to assign a National Standardisation Coordinator (NSC) who shall be responsible for data submission via SIS (for both, regular and ad-hoc data updates). The SIS data within the Competent Authority could also be collected with an involvement of responsible persons from different domains, however the overall responsibility for data submissions remains with NSCs. A user guide for SIS has been made available to all NSCs and EASA organises a series of recurrent NSCs meetings to enable the effective exchange of information on SIS, improve the data quality, provide feedback and exchange experiences. 2.3 Model operation The Model processes all information collected in three subsequent steps. The first step processes the indicators related to the Authority and Follow-up components, in order to calculate the resulting “Standardisation rating”. The second step includes the Activity, Safety and Time components. Finally, the third step relies upon expert judgement in order to determine the final result for each specific technical domain (“output level”), while taking into account the General indicators and any other relevant data as background information. An overview of the Model’s operational sequence is portrayed in Figure 4 below: Figure 4: Model overview The following paragraphs provide a short description of the actions performed at each step. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 8 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning Step 1: Standardisation rating calculation In this step the indicators belonging to the Authority and the Follow-up components are evaluated, weighted and combined (averaged) into a single score - the “Standardisation rating”. This rating aims to emulate the expert’s confidence in the Competent Authority’s ability to discharge its safety oversight capabilities, taking into account the status and relevance of Standardisation findings as well as the outcomes from inspections, once the final report has been sent. The rating is calculated by means of a scorecard system, where the indicators are individually weighted and then combined to produce a single score ranging from 100 (best) to 0 (worst). The relative weights associated to each indicator have been determined using a sound statistical methodology, as described in Annex II. The resulting weights were then evaluated and fine-tuned through validation meetings involving all Agency’s Standardisation Team Leaders as subject matter experts, in order to ensure that the resulting Standardisation rating would “match” the expert judgement to the highest possible extent. The weights have then been converted into scoring points, which decrease the starting score (100) when a given condition is met. A list of all indicators, conditions and scoring points which are used to calculate the Standardisation rating is provided in Annex III. Step 2: Overlay indicators The Standardisation rating resulting from the calculation described above is then complemented by further “overlay” indicators which include the following components: • Activity: the size, scope and complexity of the aviation industry in each technical domain, on a relative scale (low / medium / medium high / high), as a way to measure exposure to risk, • Time: the time elapsed since the last comprehensive inspection in each domain, in order to take into account the inevitable decay with time in the confidence that can be put in the “Standardisation rating” (especially if all findings have been closed in the meantime), and • Safety: the available safety indicators, such as SAFA Country Ratio, accidents, immediate safety concerns raised in recent inspections, etc. Such components are kept separate from the Standardisation rating, as the related indicators are of a different nature; therefore, blending them in a single result would not be meaningful. Step 3: Validation of output In the last step, the Standardisation rating and the overlay indicators are portrayed, together with the General indicators and any relevant additional information, in a Country Dashboard aimed at supporting the decision- making process. The Dashboard concept is illustrated below: IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 9 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning Figure 5: CMA Dashboard The Dashboard displays, for each Country, aggregated information for each technical domain (Airworthiness, Air operations, Aircrew, Medical, Flight Simulator Training Devices, and Air Traffic Management / Air Navigation Services) and for each Model component, as previously described. Note: The technical domains on the country dashboard also include the RAMP domain for which the Standardisation rating is the output of an in-house Model assessment, based on similar principles to the present Model. The RAMP Model is developed with objective to derive the performance of a state. To achieve this, several indicators have been aggregated to one index as follows: IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 10 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning - Compliance status assessed during the Standardisation inspection - Performance during the follow-up phase (e.g. reactivity index, Supplementary Reports) - Performance observed analysing the data in the central SAFA Database: • Results of the Quality Reviews; • Average Findings/Reports rate (too low or too high); • Significant under completion of the SAFA checklist (average). Similarly to the Model for Standardisation inspections’ planning, other factors are also considered in order to complement the Standardisation rating result and make informed decisions on the planning of the SAFA Standardisation inspections. These factors include: a. the time since last comprehensive inspection to account for the “erosion” of the relevance of the performance indicator, with time and b. the exposure measures to capture the impact of Standardisation. In order to facilitate the analysis, the Standardisation rating stemming from the Authority and Follow-up indicators is presented in different shades of blue highlighting to which quartile the rating belongs to (1st quartile: white, 2nd: pale blue, 3rd: light blue, 4th: blue, where the 1st quartile is the highest 25% and the 4th is the lowest 25% of all resulting ratings of other Member States in particular domain). Where data is not provided by the States, the calculation engine is unable to calculate a numerical value at aggregated level, therefore a greyed out indication appears as data is not available. The time elapsed since the last comprehensive inspection is calculated in number of months. The icon theme follows this convention: 0 to 12 months – one square, 13 to 24 – two squares, 25 to 36 – three squares and 37 and over – four squares. For comprehensive inspections where the final report has not been finalised, the date of the previous comprehensive inspection applies in the Model calculations. Similarly, the Activity component is presented as an overall indication of the level of aviation activity in each technical domain; The activity indicators consist of two exposure measures representative for each technical and they are shown in Table 1. Activity Exposure 1 Exposure 2 AIR Airworthiness Licenses Organisations (Maintenance, Training, Production) OPS Aircraft operated in Commercial Air Air Operator Certificates Transport FSTD N/A Flight Simulator Devices FCL Air Crew Licenses Approved Training Organisations MED Medical Certificates Aeromedical Examiners RAMP Number of SAFA ramp inspections in the Total number of findings per participating state database ANS Air Traffic Control Officers licensed with at Air Traffic Service Providers and Air Traffic least one valid endorsement Controllers’ Training Providers Table 1: Activity indicators in each technical domain IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 11 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning The level of activity is a relative measure which is obtained by clustering the data submitted by all Authorities in fours tiers for each domain, i.e. low / medium / medium high and high. These are displayed in one-bar, two- bar, three-bar and four-bar symbols respectively. The resulting exposure ranges for the above clusters are shown in ANNEX III – Scoring points and exposure ranges. The Dashboard also encloses an indication of an Average Reactivity Index (ARI) which caracterises countries’ reactivity in addressing standardisation findings’ closure . A reactivity index is calculated per corrective action (depending on its status: open, open overdue, closed on time, closed late) and then aggregated at finding level. The final reactivity index is calculated by domain by aggregating all findings issued in the last 4 years and the ones older than 4 years and still open. The higher the Average Reactivity Index the higher the country’s reactivity in addressing findings’ closure. The icon theme follows this convention: no square if ARI between 0 and 30, one square if 30<ARI≤47,5, two squares if 47,5<ARI≤77,5, three squares if 77,5<ARI≤90, and four squares if ARI≥91. Each domain is plotted on a Country Heat map with the Standardisation rating on the x-axis and the months elapsed since the last comprehensive inspection on the y-axis, in order to capture the overall country picture across all technical domains. Each domain is represented with a circle, where the size and the colour of the circle indicate the exposure level i.e. the bigger the circle the larger the exposure and the exposure measure i.e. black circle indicates Exposure 1 indicators and grey circle indicates Exposure 2 indicators, respectively. The Safety indicators include the Country SAFA country ratio, the technical domain where Immediate Safety Concerns have occurred and the number of accidents for aircraft above 2250 kg. The latter is split in two categories: CAT (Commercial Air Transport), where the concerned State is State of Operations and Other, where the concerned State is State of Registry. The ratio of the accidents versus the total number of accidents for all EASA member states is also available to show the contribution of the state to the total number of accidents. The Dashboard also portrays the General indicators, which provide further background information with regard to the overall situation at Country and Authority level. These namely include traffic figures (movements and their year-to-year change), ICAO inspection outcomes and other useful information at state level. For the sake of consistency, selected symbols characterise the General indicators depending on their values. For example, green, yellow and red flag, indicate low, medium and high rating, respectively. The Additional information field allows subject matter experts to capture any further information considered relevant in the frame of the Agency’s continuous monitoring activities. For example SAFA operators with a SAFA ratio greater than 2. At this stage the Model enables the Agency’s Flight Standards Directorate to validate the situation of each technical domain within each Country, as well as to compare the situation in each given domain across all Countries, on the basis of the aggregated information provided in the Dashboard. The outcome of such assessment is used for planning purposes at various levels: • Strategic level: considering a 4-years baseline cycle for comprehensive inspections, the overall results could be clustered in 4 output levels – e.g. “A” (low priority) to “D” (high priority). Such levels would then provide the basis for developing the multi-annual plan of comprehensive inspections foreseen by the Standardisation Regulation; IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 12 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning • Tactical level: trends and variations in the values of individual Model components in one or several domains can be used to determine the timing and scope of focused inspections, as well as to revise the multi-annual inspection plan in order to address any emerging risk; • Operational level: major variations affecting the values of one or more Model indicators may trigger requests for clarification and/or specifically focused inspections as appropriate. Such assessment is carried out as a regular management review activity, requiring the active participation and involvement of Standardisation experts from each technical domain, in order to agree on the identified priorities and needs and to take informed decisions. It is worth noting that the Model has been developed and validated using data from the so-called “EASA Countries‡” and has then been extended and applied to all Countries subject to Standardisation inspections. As a consequence, the quartile and exposure ranges are defined by the Model results of the EASA Countries, whereas the Model results of the remaining Countries are compared against the formed baseline. 2.4 Trend monitoring EASA has been now collecting the data for more than two years. Therefore, one of the future steps anticipated is the Model’s key indicator trend monitoring and analysis. Starting with the CMR for 2015 quarter 4, EASA Standardisation Sections, National Standardisation Coordinators and the European Commission are provided with the CMRs, which are complemented with key indicator trend monitoring. The CMR key indicator trend monitoring report depicts key indicator quarterly changes in the following technical domains - Airworthiness, Air operations, Aircrew, Medical, Flight Simulator Training Devices and Air Traffic Management / Air Navigation Services. CMRs include trend monitoring for the following indicators: 1. Average Reactivity Index (ARI); 2. Standardisation Rating; 3. Exposure 1 (in accordance with Table 1: Activity indicators in each technical domain); 4. Exposure 2 (in accordance with Table 1: Activity indicators in each technical domain); 5. Ratio Exposure 1/Technical Staff Full Time Equivalents (TS FTEs); and 6. Ratio Exposure 2/Technical Staff Full Time Equivalents(TS FTEs). Avearage Reactivity Index and Standardisation Rating trend monitoring also includes a trend of the EASA Member State average values of the respective indicator. As agreed at the 8th National Standardisation Coordinator’s Meeting (online) on 19th April 2016, the trend monitoring will only be published twice a year with quarter 2 and quarter 4 CMRs, following the SIS data collection campaigns. Figure 6 depicts the layout of CMR’s trend monitoring graphs in the OPS domain as an example. ‡ i.e. all EU Member States plus Switzerland, Iceland, Norway and Liechtenstein IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 13 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning Figure 6: Trend graphs 3 FURTHER STEPS The Model is built on a robust statistical methodology, and is the result of various iterations and workshops involving all Standardisation experts. It has been improved and refined by: • applying additional inputs and feedback provided by Standardisation experts throughout the development process, IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 14 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning • validating the Model outputs against expert judgment on a continuous basis, in order to ensure that the Model is robust and “credible”, and • taking into consideration the comments received from States’ NSCs and Authorities in the frame of EASA committee meetings and of a dedicated workshop (February 2013). The Model has been developed as a living system, which is kept constantly up-to-date with fresh data. To ensure that the Model is kept current and validated against expert judgement, the Agency continuously performs the following activities: - Input data improvement The data used to develop the Model were provided by the Competent Authorities in Q4/2012. At that time, the Authorities’ response rate had been around 80%. Furthermore, the data received for a few indicators (e.g. turn- over rate of technical staff) appeared to be not fully accurate in all cases. Therefore, the Agency entered into contact with several authorities in order to enrich the data set, and some improvements have been made. Since the first data collection campaign, EASA reviewed several times the SIS User guide in order to provide better guidance on the indicators to be collected, and the issue of data quality was addressed in all the regular meetings conducted with the National Standardisation Coordinators. Further improvements have been noticed, however, some inconsistencies can still be spotted in the reported data. Therefore, in 2015 Standardisation team leaders have started to perform consistency checks during the on-site phase of standardisation inspections. - Validation of the Model Validation checks on the Model output are performed every year in June in the frame of the development of the Standardisation Inspections’ Annual Programme (SIAP) for the upcoming year. - Continuous improvement The Agency reviews its indicators regularly to ensure their continued validity and predictive power (regular “sense-checking”); These actions will result in a continuous improvement of the Model as key enabler for an effective and efficient EASA CMA. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 15 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning ANNEX I – LIST OF MODEL INDICATORS Authority and follow-up indicators Source Indicator Description Authority Number of ROs Number of Regional / remote offices covering at least 1 technical domain (besides Headquarters) Authority Budget Y2Y % chg Percent change in the overall Authority budget (current year vs. previous year) Authority Major changes Major Authority organisational / structural changes occurred in the last 2 years (YES/NO indicator) Number of Technical Staff (not counting administrative staff) performing certification/surveillance Authority TS FTEs activities in each technical domain Percent change in staffing figures between the current year and the previous year, for each Authority TS Y2Y % chg technical domain Ratio of staff leaving in the last 12 months compared to total number of staff in each technical Authority TS turn-over rate domain (%) Sharing: more than 1 Competent Authority designated in a given domain; Outsourcing: use of Authority TS sharing/outs. qualified entities in a given domain (YES/NO) Authority SSP status State Safety Programme status (documented / implemented) (YES/NO) Open class D findings related to ICAO Critical Element 1 (Primary aviation legislation) in a given EASA D findings - CE 1 domain Open class D findings related to ICAO Critical Element 2 (Specific operating regulations) in a given EASA D findings - CE 2 domain Open class D findings related to ICAO Critical Element 3 (Authority structure and staffing) in a given EASA D findings - CE 3 domain Open class D findings related to ICAO CE 4 (Technical staff qualification / competence) in a given EASA D findings - CE 4 domain Open class D findings related to ICAO CE 5 (Internal and external procedures / guidance) in a given EASA D findings - CE 5 domain Open class D findings related to ICAO CE 6 (Licensing, certification, approval obligations) in a given EASA D findings - CE 6 domain EASA D findings - CE 7 Open class D findings related to ICAO CE 7 (Surveillance obligations) in a given domain EASA D findings - CE 8 Open class D findings related to ICAO CE 8 (Resolution of safety concerns) in a given domain EASA Total D findings Total number of all open class D findings in a given domain EASA Total C findings Total number of all open class C findings in a given domain Average Reactivity Index; it captures the Authority’s ability to implement corrective actions properly EASA ARI and on time, in each domain EASA CAP not agreed Failure to agree on a corrective action plan (CAP) in a given domain EASA D findings in SR Number of class D findings addressed by a Supplementary Report (SR) in each domain IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 16 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning Activity indicators Source Indicator Description Authority Aircraft Number of aircraft registered/operated, as follows: Aircraft EU CofA Annex II Valid ARC CAT ops A 0 0 0 0 H 0 0 0 0 Authority Organisations Number of approvals issued in each domain, as follows: AIR OPS MED CAMO 0 AOC (A) 0 AeMCs 0 AMO 145 0 AOC (H) 0 AMEs 0 AMO F 0 MTO 147 0 FCL GMPs 0 POA 21G 0 ATO (A) 0 OHMPs 0 ATO (H) 0 AIR Lic Total ATO (S) 0 MC C.1 0 EASA 0 ATO (B) 0 MC C.2 0 JAR 0 ATO (As) 0 MC C.3 0 National 0 FSTD ANS FFS (A) 0 ATSPs 0 FFS (H) 0 AISPs 0 CNSPs 0 FTD (A) 0 METPs 0 FTD (H) 0 TO (I) 0 FNPT (A) 0 TO (U) 0 FNPT (H) 0 BITD 0 Authority Licences Number of licences issued in each domain, as follows: AIR Lic Total ANS Lic ATCO ATSEP FISO EASA 0 EASA 0 JAR 0 ICAO 0 National 0 National 0 0 0 FCL Lic MPL ATPL(A) ATPL(H) CPL(A CPL(H) PPL(A) PPL(H) Other ) EASA 0 0 0 0 0 0 0 0 JAR 0 0 0 0 0 0 0 0 National 0 0 0 0 0 0 0 0 Other Movements Aircraft movements (scheduled flights, CAT) (source: ICAO) Other Movements Y2Y % chg Percent change in movements between the current year and the previous (source: ICAO) IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 17 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning Safety indicators Source Indicator Description EASA ISC Domain(s) with a raised Immediate Safety Concern (current year + previous year) EASA Accidents Number of accidents (last 5 years) in CAT (by State of Operator) and Other (by State of Registry) involving aircraft with mass greater than 2250kg EASA SAFA Ratio Current overall SAFA Country Ratio Other ATM/ANS indicators Runway incursions, Separation Minima infringements, ATC technical events (last 2 years) (source: Eurocontrol / EASA) Time indicators Source Indicator Description EASA Time elapsed Time elapsed since the last comprehensive Standardisation inspection General indicators Source Indicator Description Authority Extraordinary Extraordinary Elections occurred in the last 12 months (YES/NO) elections Authority Interim government Interim Government occurred in the last 12 months (YES/NO) Other LEI % Lack of Effective Implementation in each domain (source: ICAO) Other GDP Y2Y % chg Percent change of GDP at constant prices, calculated as average of last two years (source: IMF, World Economic Outlook Database) Other CPI Current corruption perception index (source: Transparency International, perceived levels of public sector corruption) IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 18 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning ANNEX II – METHODOLOGY The weights (or ultimately the scoring points) of the indicators are the output of a statistical regression analysis which used the EASA Standardisation expert judgment, the data provided by the Member States and the Standardisation inspection findings. The methodology steps followed in order to determine the scoring points were: 1. Expert rank ordering: obtain expert rating of the Competent Authorities based on Standardisation visits 2. Single indicator analysis: examine how the values of indicators vary with expert rating 3. Indicator transformation: transform indicators to a common scale 4. Indicator normalisation: standardise to a common mean and standard deviation to allow for sensible comparison between the indicators 5. Regression analysis: solve the system of normalised indicators to acquire indicator weights and ultimately the scoring points Throughout the development and the validation of the Model, the EASA Standardisation experts were kept in close communication and were continuously consulted in every step of the process. Full details of the statistical approach are provided below. Expert rank ordering In order to determine the relative importance of each indicator i.e. its weight, the expert judgment of the inspectors was obtained in each technical domain. In the form of a workshop or a series of workshops, the Standardisation team leaders were requested to position each Competent Authority on a scale from 1 to 5. The word pictures for 1 and 5 are as follows: 1: Highest confidence in the Country’s ability to discharge its oversight capabilities and adherence to the Standardisation regulations basing their judgment on outcomes from regular or follow up Standardisation inspections and tracking the progress of action plans. 5: Least confidence in the Country’s ability to discharge its oversight capabilities and adherence to the Standardisation regulations basing their judgment on outcomes from regular or follow up Standardisation inspections and tracking the progress of action plans. The intent was that the expert should consider each indicator, before proving an overall rating from 1 to 5 for the Competent Authority as a whole. The output of this exercise was a rating table for all inspected Countries and in each technical domain. Single indicator analysis Each indicator was then examined independently of the other indicators on how it varies, or in statistical terms how it correlates with the expert ranking. To assess each indicator’s relation to the dependent variable i.e. that IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 19 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning expert ranking and its ability to predict and explain the dependent variable, the following assessment criteria were considered: • Distribution of the indicator values (spread in buckets) • Correlation or statistical relationship with expert ranking (trend in the intuitive direction) • Differentiation of data points in terms of expert rating (steepness of curve) An illustrative example of a predictive indicator (Number of findings) is shown below: Figure 6: a. A scatter plot of the expert rating vs. Number of findings b. Average expert rating vs. Number of findings There are several correlation coefficients, measuring the degree of correlation between two ranked variables or two sets of data (i.e. the indicator and the country rank order provided by the experts). In this case Spearman’s correlation coefficient is selected as the appropriate rank order performance measure and in addition the following two data characteristics justify the selection: • the available data are ordinal in a discrete (e.g. number of findings) format representing a rank order. • there is a monotonic relationship between the variables i.e. as one variable increases, the other variable tends to increase or decrease, without requiring that increase to be represented by a linear relationship. The Spearman’s correlation coefficient is defined as follows with values ranging from -1 to 1. 1 or -1 meaning perfect correlation. Negative correlation, is correlation in opposite direction i.e. if one variable increases other decreases and vice versa. 6∑ 1 1 where: !" # $ % " IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 20 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning " The performance of the indicators was discussed and reviewed by the experts and in the cases where poor performance was identified it was mainly attributed to: • Data quality: poor or missing of data • Outlier expert rating i.e. an expert rating that was numerically distant from the rest of the ratings To smooth out such anomalies and prevent inaccurate input parameters from leading to imprecise outputs, appropriate action was taken to clean/remove/alter the “corrupted” values. In particular, the missing values (null responses from some Competent Authorities) were omitted during the analysis phase to minimise data “contamination” effects of invalid values. Significance testing was also performed at the critical values of 5% and 1% to ensure that the correlations were significant and not random. Bucketing of indicators Single indicator analysis facilitates the allocation of the expert ratings into buckets which were determined by the concentration pattern of the values. In the present case, all the indicators are categorical in nature meaning that the values lie in well-defined buckets of observations (e.g. number of findings). For example, for the discrete indicator, Number of findings the expert rating values are distributed across 4 evident buckets (A, B, C & D) as shown below: with indicator values 1, 2, 3 & 4 respectively. This bucketing exercise is repeated for all indicators of the Model. Figure 7: Bucket allocation for Number of findings Indicator transformation Once the list of indicators had been identified, and the data quality checks had been completed, the indicators were transformed to a common scale and standardised to a common mean and standard deviation in order to IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 21 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning allow for sensible comparison between the indicators. The transformation was linear, i.e. a linear relationship exists between the ordinal and the transformed data. For a given discrete value of an indicator, the transformed value is the average of the expert rating for that particular indicator value. An illustrative example of a discrete indicator transformation (Number of findings) is shown in Table 1 (see also Figure 6.b) : Bucket name Indicator value Transformed value (Number of findings) (Average rating) A 0 1.53 B 1 2.46 C 2 3.70 D 3 4.75 Table 1: Indicator value transformation Indicator normalisation In order to normalise the indicators the mean is subtracted from each transformed indicator to standardise it to a zero mean. The indicator is then divided by its own standard deviation. Standardisation of transformed indicator values to a common mean and standard as follows: ' " ( & " ) This transformation and Standardisation process ensures that indicator weights are comparable across indicators and ready to be used in the next step, the regression analysis. Regression analysis A linear regression analysis is performed to test the effects of the independent (indicators) predictor variables on a single dependent (expert judgment) variable and determine the indicator weights. Regression analysis is to solve a system of equations that maximises correlation with expert ranking to acquire the optimum weight for each indicator. The system of equations to be solved is: -,- ,- 4,- 5- D- , -, 2 , , 2 , 4, 2 ,5 2 <=>>;?7@ =4 ,D 2 678 9 :; . . + .. 1 $- 3 + .. 1 $ 3 . . 3 + .. 1 $4 + 1 ABBBBBBC + 1 . . * -, 0 * , 0 * 4, 0 5 * 0 D * 0 IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 22 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning where: !" # 5 D E F $ $ F " " Different combinations of indicators were tested to understand the dynamics and the inter-dependencies between indicators and evaluate their impact on the outcome of the Model. The process involves running series of regressions of different indicator combinations versus the dependent variable, the expert rank order. The findings of this exercise were presented to the experts in order to provide logical explanations on the correlations between indicators and ultimately advise on the final selection of weights. To use the Model output as a scoring matrix (scorecard), a translation must be made from the "weight domain" to the "score domain". in doing this two effects must be captured: A. The weight associated with the indicator B. the "distance" between buckets of the indicator (i.e. the change in the Model output in moving from bucket 1 to bucket 2 or similar. To assess the appropriate score points the weight is multiplied by the distance between two buckets. The example below shows this in action. Referring back to the bucketing example in Table 2 and assuming that the indicator weight is 0.15 then the resulted impact in the scoring domain is shown under Incremental change in Table 2. Indicator bucket Transformed value Bucket output Incremental change (Number of findings) (Average rating) (Weight x Average) (Output difference) A 1.53 0.22 - B 2.46 0.36 0.13 C 3.70 0.55 0.18 D 4.75 0.71 0.15 Table 2: Conversion to scoring domain This allows to measure the variation in the Model result for all the indicator buckets in order to derive the scoring points corresponding to each bucket or condition. This exercise was performed for all indicators and their buckets and the resulting table values were converted to a scale with range from 100 to 0. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 23 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning The produced scoring points of each indicator were defined such that they optimise predictive power of the Model and maximise correlation with the expert input. Although they were statistically tested, they were further adjusted by the experts to match their professional opinion. In other words, by means of continuous iteration with the experts these were further challenged and revised to take into account empirical input and be subject to reality checks. This way the generated scoring points were not only statistically significant but also expert verified. Standardisation of scoring points across all domains The regression analysis was performed in each domain leading to an optimal set of scoring points for each domain and in accordance with technical experts guidance. It was noted that for the general authority indicators the scoring points were similar across the majority of the domains, while for the Standardisation related indicators, the variation was slightly larger but within the acceptable limits to allow for an approximate best value to be used to fit all domains. It is important to mention that for the domains undergoing transition to new regulations the data were insufficient to draw statistically significant conclusions and therefore their scoring points were automatically allocated in accordance with the other domains. A series of scoring point adjustment tests were conducted to determine a standard score for each indicator whilst maintaining a satisfactory correlation (greater than 70%) with expert judgment in all domains. The optimisation process led to adjusted scoring points that fit all technical domains at a minimum expense (less than 2%) of the correlation with expert judgment in each domain. IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 24 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning ANNEX III – SCORING POINTS AND EXPOSURE RANGES The indicators, conditions and scoring points used to calculate the Standardisation rating are: Point Indicator Condition s Number of ROs More than 2 ROs 2.5 GDP Y2Y % chg If (GDP % chg > 0) Movements Y2Y % chg and (Movements % chg > 0) 3.5 and (budget Y2Y % chg < 0) Budget Y2Y % chg Major changes If applicable 6 TS Y2Y % chg If (TS Y2Y% chg < 0) Aircraft and (Aircraft / Organisations / Licences) > 0 6 Organisations (depending on the relevant technical domain) Licences If technical staff turn-over rate ≥ 20% 6 TS turn-over rate If 10% < technical staff turn-over rate < 20% 2.5 TS sharing/outs. If applicable 6 SSP Documented but not Implemented 3.5 AUTHORITY SSP status SSP neither Implemented nor Documented 6 1 or more open class D findings related to ICAO Critical Element 1 (Primary aviation D findings - CE 1 3 legislation) in a given domain 1 or more open class D findings related to ICAO Critical Element 2 (Specific operating D findings - CE 2 3 regulations) in a given domain 1 or more open class D findings related to ICAO Critical Element 3 (Authority structure D findings - CE 3 3.7 and staffing) in a given domain 1 or more open class D findings related to ICAO Critical Element 4 (Technical staff D findings - CE 4 5 qualification / competence) in a given domain 1 or more open class D findings related to ICAO Critical Element 5 (Internal and external D findings - CE 5 3.7 procedures / guidance) in a given domain 1 or more open class D findings related to ICAO Critical Element 6 (Licensing, D findings - CE 6 6 certification, approval obligations) in a given domain 1 or more open class D findings related to ICAO Critical Element 7 (Surveillance D findings - CE 7 6 obligations) in a given domain 1 or more open class D findings related to ICAO Critical Element 8 (Resolution of safety D findings - CE 8 5 concerns) in a given domain Total D findings 5 or more open class D findings in a given domain 2.5 Between 2 and 5 open class C findings in a given domain 2.5 Total C findings 6 or more class C findings in a given domain 5 77,5 < ARI ≤ 90 2.5 ARI 47,5 < ARI ≤ 77,5 5 (Average FOLLOW-UP Reactivity 30 < ARI ≤ 47,5 7.5 Index) ARI ≤ 30 10 Corrections and corrective actions for Class D findings not agreed within (or agreed after) CAP not agreed 5 the regulatory timeframe in a given domain D findings in SR 1 or more open Class D findings addressed by a SR in a given domain 6 IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 25 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning The ranges for the Exposure 1 indicators in each technical domain are as follows: Level AIR OPS FCL MED FSTD ANS RAMP Low 0 – 600 0 – 50 0 – 1000 0 – 500 N/A 0 – 500 0 – 100 Medium 601 – 2500 51 – 200 1001 – 5000 501 – 2000 N/A 501 – 1000 101 – 200 Medium high 2501 – 6000 201 – 600 5001 – 20000 2001 – 10000 N/A 1001 – 2000 201 – 500 High 6001 - 601 - 20001 - 10001 - N/A 2001 - 501 - The ranges for the Exposure 2 indicators in each technical domain are as follows: Level AIR OPS FCL MED FSTD ANS RAMP Low 0 – 30 0 – 10 0 – 10 0 – 10 0 – 10 0 – 10 0 – 50 Medium 31 – 150 11 – 40 11 – 40 11 – 60 11 – 30 11 – 40 51 – 200 Medium high 151 – 500 41 – 100 41 – 100 61 – 200 31 – 100 41 – 80 201 – 500 High 501 - 101 - 101 - 201 - 101 - 81 - 501 - IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 26 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning ANNEX IV - ACRONYMS Acronym Description 145 Part-145 147 Part-147 (A) Aeroplanes (As) Airships (B) Balloons (H) Helicopters (S) Sailplanes 21G Part-21 Subpart G AeMC Aeromedical Centers AIR Airworthiness AIR Lic Maintenance Licences AIS Aeronautical Information Service AMEs Aeromedical Examiners AMO Approved Maintenance Organisation ANS Air Traffic Mgmt / Air Navigation Services ANS Lic ATCO, FISO, ATSEP Licences / Certificates ANS TO Air Traffic Controllers' Training Providers (Initial / Unit) AOC Air Operator Certificate ARC Airworthiness Review Certificate ARI Average Reactivity Index ATC Air Traffic Control ATCO Air Traffic Control Officer ATO Approved Training Organisation ATPL Airline Transport Pilot Licence ATS Air Traffic Service ATSEP Air Traffic Safety Electronics Personnel BITD Basic Instrument Training Device CAMO Continuing Airworthiness Management Organisation CAT Commercial Air Transport CNS Communication, Navigation and Surveillance CofA Certificate of Airworthiness CPL Commercial Pilot Licence F Part-M Subpart F Maintenance Organisation FCL Aircrew (former Flight Crew Licences) IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 27 of 28 An agency of the European Union  European Aviation Safety Agency Indirectly Controlled Document Information Paper - The Model for Standardisation inspections’ planning Acronym Description FCL Lic Aircrew Licences FFS Full Flight Simulator FISO Flight Information Service Officer FNPT Flight and Navigation Procedures Trainer FSTD Flight Simulation Training Devices FTD Flight Training Device FTE Full-Time Equivalent (for the purpose of calculating staff figures) GDP Gross Domestic Product GMP General Medical Practitioner LABs Language Assessment Bodies MC Medical Certificate (C. 1 = Class 1; C. 2 = Class 2) MED Aircrew (Medical) MET Meteorological Service MPL Multi-Crew Pilot Licence MTO Maintenance Training Organisation OHMP Occupational Health Medical Practitioner OPS Air Operations POA Approved Production Organisation PPL Private Pilot Licence RAMP Inspections of aircraft of operators under the safety oversigh of another Member State or of a third country RI Reactivity Index RITO Ramp Inspection Training Organisation ROs Regional/remote Authority offices performing oversight SAFA Safety assessment of foreign aircraft SIAP Standardisation Inspection Annual Programme SP Service Provider SSP State Safety Programme TS Technical Staff Y Year Y2Y Comparison between current year vs. previous year IC.PD.STD.CMA.Model.Paper.4.2 © European Aviation Safety Agency. All rights reserved. ISO9001 Certified. Proprietary document. Copies are not controlled. Confirm revision status Page 28 of 28 An agency of the European Union Dear Ms Adamson, In accordance with Art. 7(3) of Regulation (EU) No 628/2013(1), we are pleased to submit to your attention the quarterly Continuous Monitoring Report (CMR) � 2017 Quarter 4 for your country. The information provided is accurate, to the best of our knowledge, with regard to the following reference dates: data submitted in SIS: see date stated on the top right corner in section 1 status of open findings and key findings, section 2: 18/09/2017; output of the Model: 21/09/2017 (establishment of output levels: 01/12/2017;) trend analysis for domains AIR, OPS, FCL, MED, FSTD &amp; ANS. Please note that each CMR is sent exclusively to the National Standardisation Coordinator of the State concerned and to the European Commission. Additional guidance on the Model and the Country Dashboard is provided in the attached updated Information Paper. We would like to ask you to include the guidance when sharing the CMR with other staff in your organisation. The document will also be available in the NSC SINAPSE community. We remain at your disposal for any clarification or question related to the content of the CMR. Should you notice any incomplete or erroneous entry, we would kindly ask you to bring it to our attention at [email&#160;protected]. (1) �The outcome of the model [for inspection planning] and the input data and results of the assessment shall be made available to the national standardisation coordinator (NSC) of the State concerned.� With kind regards Filippo Sellara Data Coordination � Policy &amp; Planning European Aviation Safety Agency Tel.: &#43;49 221 89990-2021 Postal: Postfach 10 12 53, 50452 Cologne, Germany An agency of the European Union