A Business Case For Human Factors Investment 1k5b4

EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION

EUROCONTROL

This document is issued as EATMP Reference Material. The contents are not mandatory. They provide information and explanation or may indicate best practice.

Human Factors Module A Business Case for Human Factors Investment

HUM.ET1.ST13.4000-REP-02

Edition Edition Date Status Class

: : : :

1.0 13.12.1999 Released Issue EATMP

EUROPEAN AIR TRAFFIC MANAGEMENT PROGRAMME

DOCUMENT IDENTIFICATION SHEET

DOCUMENT DESCRIPTION Document Title A Business Case for Human Factors Investment EWP REFERENCE: HUM.ET1.ST13.4000 PROGRAMME REFERENCE INDEX:

EDITION:

1.0

EDITION DATE:


13.12.1999

Abstract This document is, within the Human Resources Domain (HUM), one of the human factors modules dealing with human performance. This module suggests that human factors should be integrated throughout the Air Traffic Management (ATM) system life-cycle in order to reduce cost and make ATM systems safer and more effective. The module will provide the reader with an awareness of the benefits from integrating human factors into the ATM system life cycle and the cost associated herewith. The module will accommodate an understanding of the effects of both taking specific human factors initiatives and the way in which cost develops over the life-time of the system. Cost Benefit Analysis System Life-Cycle Human Factors Usability Engineering

Keywords Quality Management Human Factors Plan Risk Management Stakeholders Change Management Life-Cycle Cost Project Management Business Case

PERSON:

J. KJÆR-HANSEN

TEL: 4773

AUTHOR:

Johan KJÆR-HANSEN

Human Factors Integration Cost Effectiveness Analysis Human Factors Investment DIVISION:

DIS/HUM

DOCUMENT STATUS AND TYPE STATUS Working Draft Draft Proposed Issue Released Issue

o o o þ

CATEGORY Executive Task Specialist Task Lower Layer Task

o o þ

CLASSIFICATION General Public EATMP Restricted

o þ o

ELECTRONIC BACKUP INTERNAL REFERENCE NAME: HOST SYSTEM Microsoft Windows

g:\Own_use\Delvrabl\Released\Hf_modul\Business.doc MEDIA

SOFTWARE

Type:

MicroSoft Office 97 (MS97)

Media Identification:

Visio 4.0

A Business Case for Human Factors Investment


DOCUMENT APPROVAL The following table identifies all management authorities who have successively approved the present issue of this document.

AUTHORITY

NAME AND SIGNATURE

Human Factors Expert ATM Human Resources Unit (DIS/HUM)

DATE

09.12.1999 J. KJÆR-HANSEN

Manager Human Factors Sub-Programme ATM Human Resources Unit (DIS/HUM) Manager Human Resources Programme (HRS) ATM Human Resources Unit (DIS/HUM)

09.12.1999 M. WOLDRING

09.12.1999 M. BARBARINO

Chairman Human Resources Team (HRT)

09.12.1999 A. SKONIEZKI

Senior Director EATMP (SDE)

13.12.1999 W. PHILIPP

Edition: 1.0

Released Issue

Page iii



DOCUMENT CHANGE RECORD The following table records the complete history of the successive editions of the present document.

DATE

0.1

15.11.1998

Working Draft

All

0.2

15.01.1999

Draft

All

0.3

05.03.1999

Proposed Issue HRT11

All

0.4

16.08.1999

Proposed Issue HRT12 (editorial changes)

Where applicable

1.0

13.12.1999

Released Issue (editorial changes)

Where applicable

Page iv

REASON FOR CHANGE

SECTIONS PAGES AFFECTED

EDITION

Released Issue

Edition: 1.0



TABLE OF CONTENTS

DOCUMENT IDENTIFICATION SHEET................................................................................ ii DOCUMENT APPROVAL .................................................................................................... iii DOCUMENT CHANGE RECORD ........................................................................................ iv EXECUTIVE SUMMARY ....................................................................................................... 1 1.

INTRODUCTION ........................................................................................................ 3 1.1 Scope and Purpose of this Document ......................................................................... 5 1.2 Outline of the Document ............................................................................................. 6

2. 2.1 2.2 2.3 2.4

THE ATM SYSTEM LIFE-CYCLE AND HUMAN FACTORS INTEGRATION............. 7 System Life-Cycle Phases .......................................................................................... 7 Managing Risk in the System Life-Cycle ................................................................... 10 Human Factors ......................................................................................................... 11 Integrating Human Factors in the System Life-Cycle................................................. 12

3.

THE BENEFITS OF HUMAN FACTORS INTEGRATION......................................... 13 3.1 Benefits for Whom?................................................................................................... 13 3.2 Investing to Gain ....................................................................................................... 14 3.3 Adding it all up .......................................................................................................... 20

4.

THE COST OF HUMAN FACTORS INTEGRATION ................................................ 21 4.1 Life-Cycle Investments.............................................................................................. 21 4.2 Life-Cycle Cost Development.................................................................................... 24

ANNEX:

MANPOWER ESTIMATIONS............................................................................ 29

REFERENCES .................................................................................................................... 31 GLOSSARY ........................................................................................................................ 33 ABBREVIATIONS AND ACRONYMS................................................................................. 35 CONTRIBUTORS................................................................................................................ 37

Edition: 1.0

Released Issue

Page v



Page intentionally left blank

Page vi

Released Issue

Edition: 1.0



EXECUTIVE SUMMARY The module is intended to promote the integration of human factors throughout the Air Traffic Management (ATM) system life-cycle in order to reduce cost and to make ATM systems safer and more effective. It will provide the reader with an awareness of the cost and benefits of integrating human factors in the process of developing and operating ATM systems. Chapter 1, ‘Introduction’, presents the issues addressed by the module and outlines the scope, objectives and target audience. Chapter 2, ‘The System Life-Cycle and Human Factors Integration’, proposes some definitions and fundamentals for understanding and managing risk in the ATM system lifecycle and outlines the integration of human factors in this process. Chapter 3, ‘The Benefits of Human Factors Integration’, analyses the benefits of integrating human factors in the system life-cycle. Chapter 4, ‘The Cost of Human Factors Integration’, identifies some of the costs of integrating human factors in the system life-cycle. The Annex provides a sample cost estimation of utilising some selected human factors methods. Further annexes consist of a list of the ‘References’ made in this document, a ‘Glossary’, a list of the ‘Abbreviations and Acronyms’ used in this publication and finally a list of the Contributors.

Edition: 1.0

Released Issue

Page 1




Page 2

Released Issue

Edition: 1.0


1.


INTRODUCTION

“One of the lessons we learned early on in our modernisation program, was that ignoring human factors in our major acquisitions can cost us dearly, both in the expense of re-engineering and in schedule delays. We’ve (the FAA) made it a requirement that human factors must be systematically integrated at each critical step in the design, testing, and acquisition of any new technology introduced into the air traffic control system.” (Del Balzo, 1993, p. 3)

There is an increasing appreciation that taking appropriate of human factors in the ATM system life-cycle will not only make that system safer to operate, but will in fact save a lot of money. In traditional thinking engineers will provide the Air Traffic Control Officers / Air Traffic Controllers (ATCOs) with the latest technology, who will in turn have to perform so that the entire system will cater for the traffic flow. While this arrangement on the surface seems satisfactory it assumes a number of things. It assumes that the system designers know what is reasonable to expect in of controller performance and limitations; it assumes that the system designers can anticipate how the controller and the system will actually work; it assumes that the controllers know what the system designers intended with specific system functions, etc. However, we may deceive ourselves if we rely on these assumptions - they usually do not hold. Human factors are traditionally not considered as part of the core issues in an industrial or commercial environment. The reasons are many: ATM systems are developed by engineers, and human factors do not normally form part of an engineer’s curriculum. Psychologists who make up a large proportion of human factors specialists are traditionally more interested in analysis than in design, and the two communities have different cultures; consequently, communication can be awkward. Finally, human factors may be seen as something being ‘politically correct’ to address but, due to their ‘soft’ nature, they are not conceived to produce tangible and relevant results. To the extent that human factors are considered, it is typically too little and too late. Figure 1 illustrates a typical development scenario:

Edition: 1.0

♦

If the requirements analysis does not address human factors, the subsequent inadequacies will leave those issues to be resolved in the Design phase.

♦

The problems that remain after the system has been designed will be left for the implementation phase to iron out.

Released Issue

Page 3



♦

Once the system has been implemented human factors experts may occasionally be brought in to solve specific and typically grave usability problems.

♦

As it is very difficult to change a design due to the technical problems implied and the financial cost related herewith, issues that cannot be dealt with by the human factors experts will be addressed in the system documentation.

♦

Certain problems may be dealt with through training of the s of the system.

♦

The outstanding issues will have to be dealt with by maintenance and personnel.

♦

At the end of the ‘food chain’ the s will have to work around the problems which hasn’t been dealt with in the previous.

This scenario is not an exception; it should rather be considered as the norm.

Requirements Analysis

Documentation what cannot be dealt with will be put into the ...

inadequacies will be dealt with in ...

Human Factors Experts

Design

Training

what the training cannot fix will be left for ...

problems not solved will be left for ...

Implementation

what is not covered will be left for ...

usability problems will be given to ...

Maintenance and

Figure 1: ing the buck ... However, there is an increased understanding of the importance of human factors in relation to aviation safety; statistics show that a vast majority of aircraft incidents and accidents are caused by human factors. The relation between human factors and safety has been identified, and measures such as Team Resource Management (TRM) are beginning to emerge. In addition, there is an emerging awareness that a proper integration of human factors can save vast amounts of money and risk in the system life-cycle of an ATM system.

Page 4

Released Issue

Edition: 1.0



This report argues that, in order to increase cost-effectiveness and safety, human factors should be introduced as early as possible in the ATM system life-cycle and integrated in all phases. The return on investments of human factors is long-term and should be seen over the life span of a system rather than as ‘quick-and-dirty’ measures to provide corrective action and an apparent swift return. When promoting the idea of integrating human factors into the system lifecycle many questions will be raised: ♦

How much will we benefit from this?

♦

How will we know if it helped us?

♦

What will be ‘enough integration’?

♦

How much will it cost?

♦

How much will we gain in of revenue or safety?

♦

How to persuade my management to do it?

Although this document will not answer all of these questions it is intended to give the reader a feel for how to approach the answers and where to look.

1.1

Scope and Purpose of this Document The objectives of this document are: ♦

To introduce some basic concepts in addressing the ATM system life-cycle and human factors;

♦

To identify the benefits and beneficiaries of integrating human factors;

♦

To identify relevant cost factors in integrating human factors and elaborating on cost development;

♦

To outline some basic strategies in approaching the system life-cycle.

The target audience for this module may have one of the following profiles:

Edition: 1.0

♦

manager of Air Traffic Services (ATS), responsible for system development or procurement,

♦

human factors specialist involved in system development,

♦

system developer,

♦

operations and maintenance officer.

Released Issue

Page 5



A previous module entitled ‘Human Factors Module - Human Factors in the Development of Air Traffic Management Systems’ (EATCHIP, 1998a) gives an introduction to the integration of human factors in the ATM system life-cycle.

1.2

Outline of the Document This document should answer the following questions:

Question

Page 6

Chapter

♦

What is included in the ATM system life-cycle, what are human factors - and how do the two things interact?

2

♦

What are the benefits of integrating human factors in the ATM system life-cycle?

3

♦

What kind of costs are associated with integrating human factors in the ATM system life-cycle?

4

♦

How much effort will it take?

Released Issue

Annex

Edition: 1.0


2.


THE ATM SYSTEM LIFE-CYCLE AND HUMAN FACTORS INTEGRATION “That, in order to maximise safety and cost effectiveness of CNS/ATM systems, the proactive management of human factors issues be a normal component of the processes followed by designers, providers and s of the systems” (Conclusion 6/2 from ICAO Rio Conference, 1998)

The increasing air traffic and the demand for effective air transportation have made it necessary to invest in ATM systems with larger capacity. Development and operation of ATM systems are characterised by employment of vast resources, which makes potential failures costly and the investments risky. In this chapter a model of the system life-cycle is introduced and an outline of system development methodologies is given. Important human factors issues are related with the life cycle; a brief description of how to integrate them will be given.

2.1

System Life-Cycle Phases In order to describe and analyse the entire life cycle of a system we may subdivide it in a number of phases, each dedicated to particular aspects in the life-cycle process and calling for different concerns.

Initiation

Planning

Feasibillity

DevelopPrement operational

ImplemenLocal tation Implemen- Operations Planning tation

Figure 2: The EATCHIP project life-cycle (EATCHIP, 1998b) The life-cycle phases can, in a general form, be described through the following phases (EATCHIP, 1998b):

Edition: 1.0

♦

Initiation: Approval to start the project and to invest associated time, funds and effort.

♦

Planning: Obtain approval for the detailed structure of the tasks, staffing and funding.

Released Issue

Page 7



♦

Feasibility: To test the technical, operational and financial feasibility of different development options and seeking approval for the chosen option.

♦

Development: To develop specifications for the operational performance and to produce specifications for equipment, interfaces and procedures.

♦

Pre-operational: To have an extra reality check on the proposed solution by building a pre-operational prototype and/or undertaking a real-time simulation to the specifications.

♦

Implementation Planning: To obtain a detailed understanding of all the practicalities of a successful implementation of the proposed solution and to co-ordinate it with affected airspace s and stakeholders.

♦

Local Implementation: To implement the system; performing detailed design, development of standard product, integrating the system components, testing, education and training, procedures, phase-in and phase-out.

♦

Operations: This phase covers several stages in the system life-cycle: Ø Operations and Maintenance: To start operations of the final project output as part of the ATM system. To assess and record performance. Ø System modification: To modify the system as necessitated by technological, economic and/or operational experience. Ø Recycling: To dismantle and dispose of the system, and to re-use the human resources.

2.1.1

System Development The development phases are typically conducted using a ‘waterfall’ model (Royce, 1970), as shown in Figure 3. The waterfall model recognises that development has to take place in stages and has to be given between the stages.

Page 8

Released Issue

Edition: 1.0



Development

Operations

Concept Development Preliminary Design System Modifications

Detailed Design Construction and Comissioning

Operation and Maintenance

Recycling

Figure 3: Waterfall model of system life-cycle While the waterfall model gained popularity in the seventies and has been used extensively, recent developments suggest an iterative spiral approach (Boehm, 1988), as shown in Figure 4. The spiral model suggests that system development should be carried out in an iterative succession of phases gradually expanding the scope. The model suggests the early use of prototypes and risk modelling. Determine objectives, alternatives and constraints

Cumulative cost

Evaluate alternatives, identify and resolve risks

Progress through steps

Risk analysis

Risk analysis

Risk analysis

Commitment partition Review

Risk analysis Requirements plan life-cycle plan

Development plan

Integration and test plan

Prototype 1

Concept of operation

Prototype 2

Simulations, models, benchmarks Software requirements

Software product design

Requirements validation Integration and test

Design validation and verification

Implementation

Operational prototype

Prototype 3

Acceptance test

Plan next phases

Detailed design Code

Integration and test

Develop and next level product

Figure 4: The iterative spiral model of system development (Boehm, 1988)

Edition: 1.0

Released Issue

Page 9


2.2


Managing Risk in the System Life-Cycle Risk and uncertainty are at a maximum in the early stages of the ATM system life-cycle but opportunities for adding value are also greatest at this point. The risk remains relatively high during the first phases of the life cycle. It only starts to decrease once the development has turned into implementation and the system starts to prove itself. This development is shown in Figure 5. On the other hand, the amount at stake, i.e. the money spent in the life cycle, increases moderately in the initial phases of the project, while it increases significantly once the design has to been turned into hard evidence. These two basic risk characteristics of the life cycle, i.e. the risk/opportunity and the amount at stake, define two time intervals; while the early phases of the life cycle contain the period in which the highest risks are incurred, failures can be repaired relatively easy. In the latter phases, when the system has left the drawing board and has been implemented, the risk of failure falls. However, the impact of potential failures is rather severe as the system or parts of it will have to be reconsidered at great expense.

RISK

Development

Opportunity and risk

Operation

Period of highest risk impact

Period when highest uncertainty is incurred Amount at stake

LIFE-CYCLE

Figure 5: Risk in the system life cycle 2.2.1

System Assessment Once the system has been developed and brought into operation, the performance of the total system will determine the level of success reached due to the efforts involved. The capacity provided by the system and its optimal use are key in determining its success. While the system may provide an increase in its capacity for handling aircraft whether measured by the quantity and quality of services - the ultimate factor that will determine whether the capacity provided will translate into an overall

Page 10

Released Issue

Edition: 1.0



benefit - is the extent to which it is accepted by its s and the extent to which it is usable. The acceptance of the system means whether the humans coming into with the system accept its existence and the way in which it operates. The usability of the system signifies whether a system is easy to use and to learn, and efficient for the human to apply in performing certain tasks. Integrating human factors in the system life-cycle will amplify the acceptability of the system and considerably improve its usability.

10

10 Usability

SYSTEM

Acceptability

5

5

Human Factors Integration 0

0

Figure 6: Deg for acceptability and usability

2.3

Human Factors Human factors can be defined as a multi-disciplinary effort to compile and generate knowledge about people at work, and apply that knowledge to the functional relationships between people, tasks, technologies and environment, in order to produce safe and efficient human performance. As a brief introduction to some of the concerns and issues within human factors, the SHEL concept (Edwards, 1972) offers some insight (ICAO, 1989). The SHEL model, as shown in Figure 7, identifies four components: Liveware (i.e. the human element), Software (procedures, symbology, etc.), Hardware (machine) and the Environment (within which the S-H-L system must function). The model introduces a concern for the interfaces and relation between the different blocks. For example, the Liveware-Liveware relation reflects the teamwork aspects, while the Liveware-Software interface is concerned with the interaction between the human and functions provided by software in the broader sense.

Edition: 1.0

Released Issue

Page 11



H S

L

E

L

S = Software (procedures, symbology, etc.) H = Hardware (machine) E = Environment L = Liveware (human)

Figure 7: The SHEL model (ICAO, 1989)

2.4

Integrating Human Factors in the System Life-Cycle Introducing and integrating human factors in the ATM system life-cycle requires some changes to the existing processes, and introduction of new elements in the concern for system effectiveness. Some of the changes will be straight forward and fall naturally in the development and operational phases, others will require changes to the philosophy upon which the ATM system is traditionally operated. Figure 8 summarises the life-cycle phases as well as some of the questions to answer in the process of integrating human factors. A detailed description of the integration of human factors into the system lifecycle is provided in (EATCHIP, 1998a).

Operations

Initiation

Planning

Feasibility

Development

PreOperational

Implementation Planning

Local Implementation Operations and Maintenance

· What should the humans do? · What should the machines do? · What should be done in collaboration between the humans and the machines

· Which working positions are wanted? · How many staff are needed? · Are the system and procedures workable?

· Do the working procedures and HMI correspond? · Does the test plan cover all the requirements?

· Does the implementation comply with requirements and expectations? · Have all ATM staff received adequate training and are they ready to operate the new system?

· Are the controllers satisfied with the system? · Are there inadequacies in the system?

System Modification

· Do the proposed modifications comply with the philosophy? · Will the modification increase human performance?

Re-cycling

·How can we retain and apply the human expertise from the old system? · What kind of training is needed to re-use the workforce?

Figure 8: Human factors questions to be answered in the system life-cycle

Page 12

Released Issue

Edition: 1.0


3.


THE BENEFITS OF HUMAN FACTORS INTEGRATION

“There is no free lunch - but sometimes, if you eat a good breakfast, you won’t need to spend as much money on lunch.”

(Hayne, 1996)

This chapter outlines some of the potential benefits induced through integrating human factors in the system life-cycle. The system will typically benefit on three levels: the working level, the development process and the overall safety level.

3.1

Benefits for Whom? Integration of human factors in the ATM system life-cycle produces different benefits for different groups of people and stakeholders. Table 1 summarises some of the end s and stakeholders, as well as the benefits that human factors integration poses to them. Table 1: Beneficiaries of human factors integration

Co st

sa vin gs Inc rea se ds afe Im ty pro ve dw ork Pr oje ing ct co on nd tim itio Inc ns e an rea db se ud ds ge yst Inc t em re a ac se c ds ep tan yst Co em ce st/ be us ne a b fit i l i t e Ris y vi d km en ce itig ati on

TYPE OF BENEFIT

BENEFICIARIES Air Traffic Services Providers Civil Aviation Authorities Air Traffic Control Officers Project Managers ATM System Developers Airspace

Edition: 1.0

ü ü ü

ü ü

ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü

Released Issue

Page 13


3.2


Investing to Gain The existing processes and ways of doing things will have to be adapted to facilitate human factors methods and approaches. Investments in the life-cycle process have to be made as early as possible. Tasks will have to be adapted and some new ones will have to be added.

Existing processes and systems

+ investments + adaptation of processes + addition of new tasks

New processes and systems

Figure 9: The change process The system life-cycle phases will be altered through new or adapted tasks, paying primary attention to how best to take into human strengths and limitations. Each life-cycle phase will need particular initiatives to facilitate specific needs. Figure 10 illustrates how different initiatives will result in benefits in the system life-cycle, and their overall impact. A more in-depth description of how to integrate human factors in the system life-cycle can be found in EATCHIP (1998a).

Page 14

Released Issue

Edition: 1.0



Initiatives

Result

Impact

better human factors planning

improved human factors integration

(early) ends involvement

increased productivity

improved job design

increased acceptability

improved humanmachine interface

better working environment

improved selection and training

increased safety

more effective product testing

improved usability

more usable system documentation

increased system reliability

Life-cycle Phases Initiation & Planning Person responsible for human factors s' group establishment Plan for human factors integration

Feasibility Definition of s profiles and analysis of tasks Preliminary human-machine interface design Performance measurements in simulators Approved specifications and prototypes

Development & Pre-operational Application of human-centered design Integration of working procedures In-depth design of human-machine interfaces Preparation for recruitments Development of transition training Usability testing

Implementation Final approval from s' group Final training of controllers for new system

Operations and Maintenance Establish improvement mechanism

System Modification Specification of modifications Task analysis and task change documentation Re-assessment of human and system reliability

Re-cycling

increased maintainability

Plan for transition Preparation of transition of human resources to new functions

Figure 10: Human factors integration initiatives, their results and impact 3.2.1

Better Human Factors Planning A primary prerequisite for a successful human factors integration is the nomination of a person responsible for human factors. This person should institute a human factors plan and organise end s to provide in the design process. Secondly, the haphazard inclusion of human factors as described in Section 1 needs to be avoided by means of a plan of how to take into human factors in the development and life-cycle process. A similar plan needs to be compiled when changes are to be made to the system or when a new system is to be brought into use.

Edition: 1.0

Released Issue

Page 15



Table 2: Initiatives for better human factors planning Initiative

3.2.2

Rationale

Person responsible for human factors

A person should be assigned as responsible for human factors. This person should oversee the integration of human factors through participation to the system development from the earliest point possible and should participate in the planning and management throughout the life cycle of the system.

Plan for human factors integration

An early planning of the process will help the end s to understand the intentions better, just as it will allow the planning team to get a better overview.

Plan for transition

Transition to a new system or modifications to an existing system requires an early plan to accomplish communication and overview.

End Involvement Involving the people that will have to use the system in the end (the end s) and representatives from parties that have a stake in the process or the final system (the stakeholders) is mandatory in the design process. Its importance cannot be overstated. The earlier this happens, the better. A group representing the end s of the system (such as ATCOs, supervisors and maintenance engineers) needs to be set up to review and guide the development process. The approval of the preliminary design by the s’ group is an important step in retrieving valuable on the design and increasing the acceptability of the end s. Table 3: Initiatives for end s involvement Initiative

Page 16

Rationale

s' group formation

The formation of a s’ group will significantly increase the acceptability of the system to the ATM staff population.

Approval of specifications and prototypes

Approval of a preliminary design from s' group and ATM staff is essential in the early phases of development.

Preparations for transition of human resources to new functions

The frustration among personnel during a transition phase can be greatly reduced through proper change management measures, notably extensive communication and frequent interaction.

Released Issue

Edition: 1.0


3.2.3


Improved Job Design In order to maximise the use of human strengths and to minimise the effects of human weaknesses it is vital that the job and tasks to be carried out are carefully designed. Definition and analysis of the tasks to be undertaken by the human and application of human-centered design guidelines are among the many aspects to consider. Rigorous usability testing will take out all unpleasant surprises and will cater for a system that bridges the end s capabilities and the tasks to be solved. Table 4: Initiatives for improved job design

3.2.4

Initiative

Rationale

Definition of s profiles and analysis of tasks

To ensure optimum job design it is important that job and task profiles are developed for each of the positions needed and that a task analysis is performed to the job design.

Human performance measurements

Measurements of human performance in semioperational environments such as simulators are necessary to investigate whether the requirements imposed by the system can be adequately handled by the humans.

Application of human-centered design

Application of human-centered design principles will significantly improve the system design.

Integration of working procedures

The design of working procedures as an integral component in the human-machine design process is a requirement for an optimum design.

Usability testing

Usability testing will ensure that the system is easy to learn, pleasant to use, error-free, error-forgiving, easy to and efficient.

Task analysis and change documentation

It is important to analyse the effect on job design when changing the system, and to provide adequate documentation of the changes.

Improved Human-Machine Interface Clarity in communication between the system and the human is imperative for the human to be effective in his or her control of the system and for the system to provide the necessary information on time and in an adequate format. It requires that both the human and software understand each other’s intentions and have realistic expectations from each other. The development of Human-Machine Interfaces (HMIs) requires that end s are presented with early versions for and that rigorous usability testing is applied.

Edition: 1.0

Released Issue

Page 17



Table 5: Initiatives for improved Human-Machine Interface (HMI) Initiative

3.2.5

Rationale

Preliminary HMI design

The end s need to provide on early HMIs to enhance design decisions and to gain their acceptance of the system

Application of human-centered design

Human-centered design principles are vital in the development of HMIs.

Usability testing

The usability testing of the HMI is an important component in the overall usability testing of the system

Improved Selection and Training Assuring effective human performance is something that starts long before the controller takes the seat at the working position. The recruitment, selection and training for the job are important to bring the best human resources to the scene. With the introduction of automated functions, the role of the ATCO will change, and selection and training need to take such changes into . Table 6: Initiatives for improved selection and training Initiative

3.2.6

Rationale

Preparation for recruitments

Early concern for recruitment of personnel for the system and concern for their profiles will improve the selection of staff for the system.

Training of controllers for new system

In the transition to a new system staff need adequate training before taking up their new responsibilities.

More Effective Product Testing The involvement and increased reliance upon the humans in the system development will provide for a more effective product testing. The end s of the system will, if included in the process early enough for them to have a real saying in the development, provide invaluable to the system developers.

Page 18

Released Issue

Edition: 1.0



Table 7: Initiatives for more effective product testing

3.2.7

Initiative

Rationale

Establish improvement mechanism

While the s’ group represents the end s in the life cycle, mechanisms need to be established enabling the end s themselves to provide continuous on the system.

Re-assessment of humans and system reliability

The re-assessment of humans and system reliability when making changes to the system will ensure a better testing of the t human-machine systems performance ability.

More Usable System Documentation Concerns for the end s will be reflected in the system documentation, which will become more understandable and -friendly, and hence will increase the effectiveness of the s and their acceptance of the system. Table 8: Initiatives for more usable system documentation Initiative Specification of modifications

3.2.8

Rationale More -friendly documentation will increase the understanding of the system among the end s and enable good performance.

Types of Benefits Some of the benefits resulting from integrating human factors in the system life-cycle are directly measurable in money , while others are less tangible. Some have a direct impact on the effectiveness of the system, while others merely enable the improvement of additional factors in the processes.

· Cost savings · Increased productivity · Increased capacity · Decreased delays · Project delivered on time

· Reduced life-cycle risk · Increased acceptability · Improved usability · Increased safety · Increased system reliability · Increased maintainability

· Reduced number of human errors · Reduced maintenance costs

· Early end- involvement · Improved job design · Improved human-machine interface · Improved selection and training · More effective product testing · More usable system documentation · Better working environment

Qualitative benefits

Quantitative benefits

Delivered benefits

Enabling benefits

Figure 11: Types of benefits

Edition: 1.0

Released Issue

Page 19


3.3


Adding it all up Integrating human factors in the system design will result in benefits on three levels: 1. The development cycle of the system will benefit, from its initial concept to its installation, from issues such as more effective product testing, improved training and selection, better system documentation and a product which is delivered on time and within budget. The emphasis on usability and acceptability will also ensure that the system will be up and running quickly, getting it right the first time. 2. At the working position level human factors integration will contribute to reduce fatigue and stress, monotony and boredom, increase job satisfaction and motivation, improve comfort and working environment, and decrease the number of human errors. 3. The overall system will be safer through a better utilisation of the human resources and improved system reliability.

Improved selection and training

Reduced fatigue and physical stress

More effective testing Improved system design

Reduced impact of errors Fewer errors

· Reduced overall cost of system development, implementation, maintenance and operation. · Increased human efficiency and productivity through adequate design and acceptance.

Reduced time for training Easier system installation More usable system documentation

Reduced monotony and boredom

· Increased safety through better integration of humans and improved system reliability

Increased human comfort Improved working environments

Figure 12: Some benefits of human factors integration

Page 20

Released Issue

Edition: 1.0


4.


THE COST OF HUMAN FACTORS INTEGRATION

“There is an ‘iron’ law that should never be ignored. To consider Human Factors properly at the design and certification stage is costly, but the cost is paid only once. If the operator must compensate for incorrect design in his training program, the price must be paid every day. And what is worse, we can never be sure that when the chips are down, the correct response will be made.” (Wiener, 1988)

It is generally believed that the decisions made during the first stages of a system design determine the main body (more than 70%) of a system's lifecycle cost. In an increasingly competitive world of privatised Air Traffic Services (ATS) providers and national istrations under tight budget control, it is therefore important to pay extra attention to these first steps when initiating the acquisition of a new system or when modifying an existing one. This section will investigate some of the costs related with human factors integration. The cost items involved will not be quantified as the specific amounts depend on many factors.

4.1

Life-Cycle Investments The integration of human factors means investments into people, processes and material. An outline of some of the investments to make is shown in Figure 13. The cost items may vary between systems.

Operations

Initiation

Planning

Feasibility

Development

Preoperational

Implementation Planning

Local Implementation Operations and Maintenance

· Person responsible for human factors · s' group establishment

· Task analysis · Rapid prototyping · Human performance assessment · Usability engineering

· Human-machine interface · Transition training development · Usability testing

· Final training of controllers

· Task analylsis · Usability testing

System Modification

Re-cycling

· Re-training of ATM staff · Establish plan for transition

Figure 13: Life-Cycle Cost items

Edition: 1.0

Released Issue

Page 21


4.1.1


Initiation & Planning As the infrastructure to the integration processes needs to be set up early, two important investments need to be undertaken. The assignment of a person responsible for human factors is a significant first step in the process. This person needs to be a human factors specialist with good qualifications and experience in applying human factors principles and methods. The establishment of a s’ group is another important step. Both of these constructs need to follow the system in its entire life cycle, with the development and adaptation phases being the most intense. Table 9: Cost items in the Initiation & Planning phase Cost Item

4.1.2

Detail

Person responsible for human factors

The person responsible for human factors will have to be assigned for the entirety of the system lifecycle.

s’ group

The s’ group will have to exist for the entirety of the system life-cycle.

Feasibility The Feasibility will see a lot of activity as it is a very crucial stage at which a solid foundation has to be laid for the integration of human factors. Some of the important cost items are shown in Table 10. Table 10: Cost items in the Feasibility phase Cost Item

Page 22

Detail

Task analysis

To understand the tasks to perform and their specific cognitive requirements a task analysis is needed. Fast-time simulations can be used to augment steps in the task analysis with, for instance, workload indices.

Rapid prototyping

Rapid prototyping provides a flexible tool to quickly present a design and get from involved ATM staff.

Human performance assessment

In order to get a first feel of the benefits of a particular design, assessment of the human performance can be obtained in small-scale realtime simulations.

Usability engineering

Early considerations need to address the usability of the system.

Released Issue

Edition: 1.0


4.1.3


Development The Feasibility phase and the Development phase may blend into one using an iterative system development strategy. The human factors cost items in the Development phase may therefore closely resemble the ones in the Feasibility phase, as shown in Table 11. Table 11: Cost items in the Development phase

4.1.4

Cost Item

Detail

Human performance assessment

A detailed of the human performance can be obtained using, for instance, small-scale real-time simulations.

Transition training development

The development of transition training for the ATM staff needs to be aligned with the requirements and characteristics posed by the system.

Usability testing

Thorough usability testing should iron out most of the usability problems.

Implementation The Implementation phase covering the implementation of the system should ensure that the system is implemented as per the guidelines developed during the previous phases, and that the ATM staff concerned receive their final training for the system. The staff assigned to operate the system need to receive the necessary training early enough for them to be fully qualified and up to speed when they switch to the new or modified system. Table 12: Cost items in the Implementation phase

4.1.5

Cost Item

Detail

Final training of ATM staff

The ATM staff should receive the necessary training enabling them to take up their duties.

Operations Modifications to a running system are inevitable. The effects on tasks and working environment need to be analysed and usability aspects need testing. In the final phase of the system life-cycle two major tasks need to be undertaken; first of all, the transition of ATM staff needs to be carefully planned and the plan needs to be communicated; secondly, the ATM staff need to receive the proper re-training to take up their new functions. This phase is the major link to a new incarnation of the system life-cycle.

Edition: 1.0

Released Issue

Page 23



Table 13: Cost items in the Operations phase Cost Item

4.2

Detail

Task analysis

The impact of changes that are made to the system on the tasks of ATM staff needs to be analysed carefully.

Usability testing

The usability of system changes needs to be tested.

Transition plan

Concern needs to be addressed to a well-planned transition to a new system. The transition plan needs to be communicated to all relevant parties to maximise the effect and acceptance.

Re-training of ATM staff

The ATM staff need proper training to take up their new duties.

Life-Cycle Cost Development Cost within the system life-cycle is typically assessed on a short-term basis, focusing on whether specific tasks should be carried out or not. The rationale is understandably to minimise the cost and to adapt to the available resources. Focusing the attention narrowly on the cost of single tasks may however unintentionally jeopardise the overall cost of developing and operating a system. Figure 14 illustrates this aspect. The dashed curve shows how cost is generated over the life-cycle. The curve represents an instant on the ‘current cost’ at any given point in time within the life-cycle. From this it is quite obvious that the main bulk of the system life-cycle cost is obtained during the implementation and operation phase. On the contrary, the curve represented by the solid curve shows how cost is being decided in preceding phases and spent on beforehand - the so-called locked-in cost. The locked-in cost represents points in time at which decisions have been made about future spending. The locked-in cost is less obvious and largely neglected in the management of system development and operation despite its obvious importance. Looking at the locked-in cost it becomes obvious that a major part of the system life-cycle cost is determined in the very early phases. It is interesting to note that 70% of the locked-in cost is generated before the Detailed Design phase which only occupies about 10% of the total system life-cycle time. This aspect only amplifies the importance of doing it right from the beginning and of making the right decisions from the early phases.

Page 24

Released Issue

Edition: 1.0



100

95 % 85 % Locked-in Cost (%)

75 70 %

50 50 % Actual Cost (%) 25 20 %

18 %

7%

2%

7%

50%

100%

Op era Su tions pp ort and

Pro du ctio n

F De ull-s ve cal lop e me nt

De sig n De tai led

De sig n

Pre lim ina ry

Co nc ep t.

De sig n

1%

% of Lapse Time

Figure 14:Life-cycle cost during the Conceptual and Preliminary Design phases (Gawron et al., 1996) While Figure 14 illustrates the general development of cost in the life cycle, Figure 15 illustrates the effects of not getting it right the first time. The figure illustrates how the cost of changing a system design amplifies depending on the point in time when the change is introduced. Compared to the cost of changing the design during the definition phase of the life cycle, the cost of the changes made during the development will be increased by 1,5 to 6 times. The cost of the changes made to the system, after it has been finalised and delivered to the end s, will be amplified by 60 to 100 times (Pressman, 1992).

Edition: 1.0

Released Issue

Page 25



Cost of change 60 - 100 x 100

50

1.5 - 6 x 10

1x Definition

Development

System released

Life-cycle

Figure 15: The amplification of design change cost 4.2.1

Three Life-Cycle Strategies As a way of dealing with human factors in the life cycle three different strategies can be found: a) ‘Do nothing’ approach: No initiatives are taken to counter human factors problems; only when problems arise will they be addressed. b) ‘Reactive’ approach:

Concern for human factors is left to the last stages of the development process.

c) ‘Proactive’ approach: Problems are fixed before they occur. The cost scenarios of the three different strategies are illustrated in Figure 16. The first (‘Do nothing’) approach illustrates how cost related with human performance issues will increase rapidly over the life-cycle of the system. If some concern for human performance issues is dealt with in the final stages of the development process (as shown in Figure 16 as the ‘reactive’ approach), the cost scenario will develop in a less aggressive yet increasing manner. However, if an early awareness to the human factors and human performance issues is introduced in a proactive manner (shown in Figure 16 as strategy c), the cost will develop in a rather different manner. The figure does not only illustrate how cost is higher compared to the other approaches due to the investments made early in the process, but it also shows how the early anticipation of problems takes the air out of later and more expensive problems.

Page 26

Released Issue

Edition: 1.0

Cost of Detecting and Resolving Human Performance Issues



a) No human factors integration

100

75

50 b) Re-active human factors integration 25 c) Pro-active human factors integration

Design

Implementation

Operation

Life-cycle

Figure 16: Cost scenarios of three different life-cycle strategies The reluctance to provide the necessary resources to embark on a proactive approach is probably based on the notion that it is better to wait and see where the problems occur and then intervene. While this strategy may, apparently, save some money, especially when the system is being developed, experience shows that the bill will have to be paid later ... with interests. Unless human factors are emphasised as an important part of the requirements for a new or adapted system, contractors bidding for the contract are likely to leave them out to save cost (and therefore increase the likelihood of winning the contract). Therefore, the requirements from any system development or from making changes to an existing system need to address human factors specifically. In summary, we can make the following observations: ♦

70% of cost is determined in the first 10% of the project;

♦

It is more cost-effective (60 to 100 times) to change the design of a system at the initial phases of development than to do it once the system has been built and is in operation.

To put it briefly, it is a matter of either paying up front for detecting and resolving the problems - or paying more later, which by all s, will be significantly more costly.

Edition: 1.0

Released Issue

Page 27




Page 28

Released Issue

Edition: 1.0


ANNEX:


MANPOWER ESTIMATIONS

It is difficult to provide cost estimates as no two projects are exactly alike. Some experience has been obtained through usability engineering projects, where software products are brought through the same integration and validation process as advocated in this report. Table 14 gives an overview of cost/benefit aspects of the methods. It gives a rough estimate of the duration of the s validation process for three different s validation scenarios: ♦

Small s validation process: detecting deficiencies in a single design,

♦

Medium s validation process: comparing three designs,

♦

Extensive s validation process: repeated validation at different stages in the development process.

The table describes whether additional cost is incurred by subjects or material needed for performing the s validation. Finally, the reliability of measurements is shown in the far-right columns. The table refers to methods that are not covered in this module. Details can be found in Melchior et al. (1996).

Edition: 1.0

Released Issue

Page 29



Table 14: Estimation of the cost/benefit of applying a method for s validation (Melchior et al., 1996) Cost of Evaluation (man-days) for Project Size

Cluster

10 TAFEI 1 Software checker 2 Ravden & Johnson 3 EVADIS II 1 Ergonomie-Prüfer 2 Usability Heuristic evaluation 4 Inspection Methods Pluralistic usability walkthrough Inspection and design review 4 4 Cognitive walkthrough 4 Formal usability inspection 1 Questionnaires for the SUMI 1 evaluation of subject QUIS factors 2 Cognitive Workload SMEQ 1 Questionnaires MCH 2 SWORD 1 SWAT 1 NASA TLX 1 NASA RTLX 20 Performance tests DRUM Learning time measurement 50 30 Physiological measures EMG 30 of workload and stress EDA 30 EOG 30 Cardiac activity 5 Analytical evaluation SANe 5 Methods ORACLE 2 Screen analyser 10 GOMS 5 Informal approaches Focus Group 10 Interview 5 Questionnaires 15 Thinking aloud 15 Co-discovery

Page 30

Reliability of Measurements

Method

Task Analysis Checklists for s interface quality

Note: Small

Additional Cost

20 2 4 6 2 4 6

30 3 6 9 3 4 6

6 6 6 3 3

6 6 6 3 3

6 2 4 2 2 2 60 70 60 60 60 60 10 10 4 20 5 10 8 30 30

6 3 6 3 3 3 50 90 90 90 90 90 10 15 6 30 8 30 10 60 60

x x x x x x x

x

x x x x

x x x x x x x x x x x x x x

x x x x x

x x x x x x x x x x x x x x x x x x x x x x

1 1 2 2 1 1 1 1 1 1 2 1 1 1 5 1 1 1 5 5 20 30 10 3 5 5 1 10 2 2 0 5 5

x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

= Detecting deficiencies in a single design

Medium

= Comparing three designs

Extensive process

= Repeated evaluation at different stages of the design

Released Issue

Edition: 1.0



REFERENCES Bias, R. G., Mayhew, D. J. (1994). Cost-Justifying Usability. Academic Press. Boehm, B. (1981). Software Engineering Economics. Prentice-Hall. Boehm, B. W. (1988). A Spiral Model of Software Development and Enhancement. IEEE Computer 21(2): 61 – 72. Brooks, F. P., Jr. (1987). No silver bullet - essence and accidents of software engineering. IEEE Computer 20(4): 10 – 19. Cardosi, K. M., Murphy, E. D. (Eds.) (1995). Human Factors in Design and Evaluation of Air Traffic Control Systems. U. S Department of Transportation, Federal Aviation istration. DOT/FAA/RD-95/3. DOT-VNTSC-FAA-95-3. Del Balzo, J. M. (1993). Speech to the ICAO Flight Safety/Human Factors Conference. Reprinted in D. Beringer (Ed.) The Flyer, June newsletter of the Aerospace Systems Technical Group of the Human Factors and Ergonomics Society, 2-3. EATCHIP Human Resources Team (1998a). Human Factors Module – Human Factors in the Development of Air Traffic Management Systems. (HUM.ET1.ST13.4000-REP-01). Brussels: EUROCONTROL. EATCHIP Management (1998b). EATCHIP Management Handbook. Brussels: EUROCONTROL. Edwards, E. (1972). Man and Machine: Systems for Safety. In proceedings of the BALPA Technical Symposium, London. Gawron, V. J., Dennison, T. W. and Biferno, M. A. (1996). Mockups, Physical and Electronic Human Models, and Simulations. In: O’Brien, T. G, Charlatan, S. G. (Eds.) Handbook of Human Factors Testing and Evaluation. Lawrence Erlbaum Associates, Publishers, Mahwah, New Jersey. Haynes, C. (1996). Software Engineering for Usability: Integration of human factors for interfaces into the software development life cycle. Human Computer Interactions, http://www.crim.ca/hci/indiv/hayne_seu/SE_for_usability.html. ICAO. (1989). Human Factors Digest No. 1. International Civil Aviation Organization, Montreal, Canada. Circular 216-AN/131. ICAO. (1998). Working Paper on Agenda Item 6 at ICAO World Wide CNS/ATM Systems Implementation Conference. Rio de Janeiro, 11 - 15 May 1998

Edition: 1.0

Released Issue

Page 31



Kirwan, B., Evans, A., Donohoe, L., Kilner, A., Lamoureux, T., Atkinson, T. and Mackendrick, H. (1997). Human Factors in the ATM System Design Life Cycle. In proceedings of EUROCONTROL/FAA ATM R & D Seminar, 16-20 June, 1997, Bretigny, . Mantei, M. M. And T. J. Teorey (1988). Cost/benefit analysis for incorporating human factors in the software life-cycle. Communications of the ACM 31: 428 - 439. Melchior, E.-M., Bösser, T., Meder, S., Koch, A. Schnitzler, F. (1996). Usability Study - Handbook for practical usability engineering in IE projects. ECSC-EC-EAEC Brussels-Luxembourg 1996. Mitchell, C. M., Van Balen, P. M. and Moe, K. (Eds.) (1983). Human Factors Considerations in Systems Design. Nasa Conference Publication 2246, p. 17 - 20. Pressman, R. S. (1992). Software Engineering: A Practitioner’s Approach. McGraw-Hill, New York. Royce, W. W. (1970). Managing Development of Large Software Systems: Concepts and Techniques. In: Proceedings WESCON, August. Also available in Proceedings ICSE 9, Computer Society Press, 1987. Wiener, E. L. (1988). Management of Human error by Design. In: Human Error Avoidance Techniques Conference Proceedings, Society of Automotive Engineers, Inc.

Page 32

Released Issue

Edition: 1.0



GLOSSARY For the purposes of this document, the following glossary of shall apply: Acceptability: The acceptance of the system means whether the humans coming into with the system accept its existence and the way in which it operates. Cost-Benefit Analysis: An objective study in which the cost and benefits of a particular project’s option are fully quantified in economic , taking full of the times at which cost is paid and benefits accrue. Human factors: Multi-disciplinary area in which knowledge about people at work is compiled and generated, and applied to the functional relationships between people, tasks, technologies and environment, in order to produce safe and efficient human performance. Human factors engineering: Discipline that applies knowledge of human capabilities and limitations to the design of technological systems. Human factors plan: Plan for the integration of human factors in the system life-cycle. Human factors specialist: The term ‘human factors specialist’ is used widely and should be considered with caution, as there is no protection on this title. A human factors specialist usually has one of the following backgrounds: ♦

a degree in industrial psychology or cognitive psychology and some practical experience of human factors application;

♦

an engineering degree with some additional study in industrial or cognitive psychology and some practical experience of human factors application;

♦

an operational background with additional human factors training, and some practical experience.

Human performance: The extent to which goals for speed, accuracy, quality and other criteria are met by people functioning in work environments. System life-cycle: The phases a system goes through from initial concept to detailed design and implementation to installation, operation and eventually de-commissioning. Usability: Applies to systems. Means whether a system is easy to learn, pleasant to use, error-free and error-forgiving, easy to and efficient. Usability engineering: Comprises a number of methods and techniques to improve a systems usability.

Edition: 1.0

Released Issue

Page 33




Page 34

Released Issue

Edition: 1.0



ABBREVIATIONS AND ACRONYMS For the purposes of this document, the following abbreviations and acronyms shall apply:

Edition: 1.0

ATC

Air Traffic Control

ATCO

Air Traffic Control Officer/Air Traffic COntroller (UK/US)

ATM

Air Traffic Management

ATS

Air Traffic Services

CBA

Cost-Benefit Analysis

DED5

Human Resources Bureau (now DIS/HUM or HUM Unit)

DEL

DELiverable

DIS/HUM

ATM Human Resources Unit (also known as HUM Unit – formerly DED5)

DRUM

Diagnostic Recorder for Usability Measurement

EATCHIP

European Air Traffic Control Harmonisation and Integration Programme (now EATMP)

EATMP

European Air Traffic (formerly EATCHIP)

EDA

ElectroDermal Activity measurement

EEC

EUROCONTROL Experimental Centre

EMG

ElectroMyoGram

EOG

ElectroOculoGram

ET

Executive Task

EWP

EATCHIP/EATMP Work Programme

FAA

Federal Aviation istration

GOMS

Goals, Operators, Methods, Selection rules and method

HMI

Human-Machine Interface

Released Issue

Management

Programme

Page 35


Page 36


HUM

Human Resources (Domain)

HUM Unit

ATM Human Resources Unit (also known as DIS/HUM – formerly DED5)

HRT

Human Resources Team

ICAO

International Civil Aviation Organization

MCH

Modified Cooper-Harper rating scale for system workload assessment

NASA RTLX

NASA Raw Task Load indeX

NASA TLX

NASA Task Load indeX

QUIS

Questionnaire for Interface Satisfaction

REP

Report

SANe

Skill Acquisition Network

SDOE

Senior Director(ate) Operations and EATCHIP (now SDE)

SDE

Senior Director(ate) EATMP (formerly SDOE)

SHEL

Software, Hardware, Environment, Liveware

SMEQ

Subjective Mental Effort Questionnaire

ST

Specialist Task

SUMI

Software Usability Measurement Inventory

SWAT

Subjective Workload Assessment technique

SWORD

Subjective WORrkload Dominance technique

TAFEI

Task Analysis For Error Identification

TRM

Team Resource Management

Released Issue

Edition: 1.0



CONTRIBUTORS

COMMENTS of the HRT Human Factors Sub-Group (HFSG) Mr Bert RUITENBERG

IFATCA

Mr Manfred BARBARINO

EUROCONTROL DIS/HUM

Mrs Veronica HUGHES

EUROCONTROL Experimental Centre (EEC)

Mrs Dominique VAN DAMME

EUROCONTROL DIS/HUM

DOCUMENT CONFIGURATION MANAGEMENT (DCM) ASSISTANCE Ms. Carine HELLINCKX

Edition: 1.0

EUROCONTROL DIS/HUM

Released Issue

Page 37




Page 38

Released Issue

Edition: 1.0

A Business Case For Human Factors Investment 1k5b4

Overview 6h3y3j

More details h6z72

Related Documents 543cg

A Business Case For Human Factors Investment 1k5b4

Human Factors For Aviation Maitenance.pdf 3o1z57

Module 09a. Human Factors y385k

Module 9 Human Factors 3a362g

Human Factors In Aviation 5b1nv

Human Factors Question Bank 3p1c4u

More Documents from "Vitor M R Coelho" 3m4443

Nextgen Implementation Plan 2012 3k3y5r

A Empresa Social.pdf 5l255y

Doc 9156 Accident Incident Reporting Manual 473k3q

Environmental Economics For Non-economists.pdf 4rs3g

Edto (etops Casa) 516j61

A Business Case For Human Factors Investment 1k5b4