Proposal for an Evaluation Method for the Performance of Work Procedures

Abstract

Background

Noncompliance of operators with work procedures is a recurrent problem. This human behavior has been said to be situational and studied by many different approaches (ergonomic and others), which consider the noncompliance with work procedures to be obvious and seek to analyze its causes as well as consequences.

Methods

The object of the proposed method is to solve this problem by focusing on the performance of work procedures and ensuring improved performance on a continuous basis.

Results

This study has multiple results: (1) assessment of the work procedures' performance by a multicriteria approach; (2) the use of a continuous improvement approach as a framework for the sustainability of the assessment method of work procedures' performance; and (3) adaptation of the Stop-Card as a facilitator support for continuous improvement of work procedures.

Conclusion

The proposed method emphasizes to put in value the inputs of continuous improvement of the work procedures in relation with the conventional approaches which adopt the obvious evidence of the noncompliance to the working procedures and seek to analyze the cause–effect relationships related to this unacceptable phenomenon, especially in strategic industry.

1. Introduction

Work procedures (WPs), which are considered as safety rules [1], are operational principles created to protect the operator against all threats to health [2]. In spite of their important roles in health and safety at work, these WPs are not systematically respected by operators [3].

The specialized literature in health and safety at work qualifies the nonrespect of work procedures (NR-WPs) as a recurrent problem due, mainly, to the limitations of the WPs themselves [4]. For this, Vidal-Gomel [5] distinguishes three factors, which are largely discussed by different authors; these factors have widely been considered as the sources of those limitations. These factors include companies' security policies [6], [7], use conditions [8], and work conditions [9], [10].

Vidal-Gomel [5] emphasizes that these findings do not discuss a very important aspect of NR-WPs, which is operators' competence. Consequently, the problem of the NR-WPs must be approached from two points of view [5]: violation of the WPs and the regulations implemented by the operator.

Battmann and Klumb [11] consider that WP violation is an intentional action, and according to Nordlöf et al [12], it is considered as a risk-taking approach. Reason and collaborators [13] distinguish three types of WP violations: routine violations, optimization violations, and exceptional or situational violations.

All these violations, which are materialized by the deviations between what is really done and what should have been done [14], [15], [16], [17], were discussed by various authors in terms of causes [2] and consequences [18]. In other words, and broadly speaking, the question is about a set of cause–effect factors highlighted in the field of ergonomics [19], [20].

Concerning the regulations implemented by an operator, violation of WP by the operator is narrowly linked to the operator's competence [1]. According to Hale et al [21], these are the “safety initiatives” or “informal practices of safety” that are implemented by experienced operators. In this context, some authors [18] confirms that if certain WPs are systematically implemented, others are not, and they are just considered implemented according to the context.

Other authors confirm the close link between competence and experience [22]. Indeed, inexperienced operators apply the WPs more systematically because they are the only means they have [23], whereas the most experienced operators also have their experimental inheritance and the WPs are only one of these means.

To summarize the link between experience and competence, we say that experience is necessary for the development of professional competence. Consequently, studies that carried out operators' competence for the NR-WPs focused on the complementary relations and substitution between formal safety rules and operators' informal practices [24] on the one hand, and on the approaches of professional competence development [24] on the other hand.

The two points of view of the NR-WPs detailed above are also discussed according to descending and ascending approaches [21]: the first approach considers the WPs as a static tool imposed on operators, and any violation of these WPs is considered a negative behavior of the operator; the second approach is said to be constructivist because the competence of the operators is considered as his capacity to adapt the WP to the reality diversity. According to the second approach, Djebabra et al [25] consider that the respect of work procedures (R-WPs) is a key element of the safety culture.

From this fast overview of the two view points of the NR-WP problem (violation and regulation of the WPs), we conclude that the WPs are neither applied nor violated. They are used, by the operators, according to their characteristics and those of situations. Hence, in this study, the key question that arises is the following: How can we measure the performance of the WPs (P-WPs) in a well-defined context? This article attempts to answer this key question by focusing on measuring the P-WPs that seem so important to us, since they allow the development of professional competence in order to establish durability of good practices in the WPs, by the aid of a suitable tool, the “Stop-Card” of the Sonatrach Group—Algeria, which is inspired from the DuPontSTOP tool [26], [27].

2. Materials and methods

First, let us recall that the method proposed in this study aims to evaluate the P-WPs. This evaluation is based on operators' behavior vis-à-vis the hazard. On this subject, we recall that this behavior is based on a mental reasoning developed by the operator [28], which includes the following [29]: operator's knowledge, capacity to solve problems, and motivations that affect the choice of the operator's actions.

For Lancry-Hoestlandt and Laville [30], operator behavior is the link between the activity in practice and the expected performance, which can be evaluated by a technical and organizational logic (indicators of quantity, quality, and compliance), a real cost logic (penalty indicators, fatigue and stress), and a logic of the real profit (salary indicators, recognition and satisfaction).

In addition, Borman and Motowidlo [31] consider two performance categories that are complementary to each other: performance of the tasks resulting from the analysis of the workplace and contextual performance, also called “organizational citizenship” [32].

The method proposed in this study integrates the evaluation of these two types of performance:

• -Evaluation of the tasks where the importance is on the P-WPs of these tasks. This evaluation is carried out by a multicriteria approach integrating a set of suitable indicators.
• -Evaluation based on the organizational citizenship of the operators, which is materialized by their good safety initiatives. It is important to emphasize that this evaluation is often omitted from organizations' formal system of evaluation [33]. For this reason, we found it useful to make this evaluation formal by a suitable tool called the “Stop-Card.”

2.1. Task-based evaluation of the P-WPs

The evaluation process of the P-WPs in this article is inspired by the methodology of Tahon [34], which is made up of two successive steps: dimensions and performance indicators.

The first step allows structuring of the performance, which is considered as a complex concept [35], an evolutionary concept [36], and a specific concept in the context of its use [37]. The second step allows one to refine the performance dimensions in the form of observable and measurable indicators [38].

Thus for the performance dimensions, three of them are highly recommended by some authors to be integrated in any evaluation of the management system of health and safety at work [39], [40], [41]:

• -The formalization of the WP (F-PT): the more a WP is well structured and is not overloaded with unnecessary instructions in its content, the more it will be respected; resulting in fewer efforts provided by the operator and lesser constraints imposed on its compliance.

• The degree of F-WP leads us to answer some key questions such as: How do we simplify the WP put in place so that they are applied naturally? How do we standardize these WP so that they are naturally integrated into the culture of an organization? What are the provided trainings to be predicted to guarantee the involvement of operators in the progress of the R-WP? How can we strengthen the good practices on other processes to change the operators' behavior toward the R-WP?

• -The effort of maintaining the R-WP in time because a good F-WP does not require effort on the part of the operator as to his respect; but the maintenance of this R-WP in the time requires an effort on the part of the operators for a better sustainability of the R-WP.
• -The opportunities of supports (or media) to help the operators so that their commitment in the efforts to maintain the R-WP is irreversible. Among the facilitation media we include the Post Card that we will detail in the continuation of this study.

The second stage, concerning the performance indicators of the WPs, is made up of five steps (Fig. 1).

Fig. 1.

Steps of the proposed method. P-WP, performance of work procedure; WP, work procedure.

As shown in Fig. 1, after having selected a WP as well as its main steps, the approach that we suggest proposes to define the preliminary list of the performance indicators of the WPs that have been retained.

This performance indicator list is then finalized; the value of the Cronbach coefficient alpha (α) permits checking of the internal coherence of the retained indicators. In fact, the coefficient (α) must be calculated on an ordinal scale: 9 = very high inefficiency, 7 = high inefficiency, 5 = medium inefficiency, 3 = low inefficiency, and 1 = very small inefficiency.

As a reminder, this coefficient α (Eq. (1)) is a statistic used mainly in psychometrics to measure the internal coherence of the retained indicators. Its value is between 0 and 1, and it is considered as “acceptable” from a value of 0.7 [32].

$$\alpha =\frac{k}{k-1}\left(1-\frac{{\sum }_{i=1}^k{\sigma }_{Y_i}^2}{{\sigma }_X^2}\right)$$ (1)

where k is the number of items, ${\sigma }_X^2$ is the variance of the total score, and ${\sigma }_{Y_i}^2$ is the variance of item i.

The fifth step of the proposed method assesses the P-WPs using the retained indicators according to the three dimensions of performance, using a semiquantitative scale of four levels [42]: 0 = unsatisfactory, 1 = imperfect, 2 = acceptable, and 3 = effective.

This first assessment allows us to obtain a matrix called a “score matrix,” which will then be crossed with another matrix called a “weighing matrix,” in order to qualify the relative importance of the criteria of each performance [42]. The purpose of this weighing is to distinguish between the levels of relative importance of the retained indicators. In this context, a weight of “1” is assigned to the indicators judged as important, a weight of “2” to the indicators judged as very important, and finally, a weight of “3” to the criteria judged as essential.

Measurement of the P-WPs is then carried out using the following relationship:

$${\text{NP}}_i=({\text{PD}}_i,{\text{PQ}}_i,{\text{PN}}_i)\times ({\text{SD}}_i,{\text{SQ}}_i,{\text{SN}}_i)$$ (2)

where NP_i is the level of performance of an indicator “i”; PD_i, PQ_i, and PN_i are, successively, the weighing of the three performance dimensions (degree of formalization, quality of implementation, and level of ownership) for an indicator “i”; and SD_i, SQ_i, and SN_i are, successively, the scores of the three performance dimensions (degree of formalization, quality of implementation, and level of ownership) for an indicator “i”.

From the two assessments (scores and weights), we deduce the matrix of weighed scores, which serves as a basic support for the determination of the weighed score averages using the following relationship:

Average of the weighed scores = ∑weighed scores/3 (3)

Finally, the proposed method ends with the capitalization of the performance evaluation of the considered WPs using two types of representations, which are largely used in the evaluation of performance indicators: representation in the form of a radar graph [43] or prioritization of the evaluated indicators [44].

2.2. Organizational citizenship-based evaluation of WP improvement

​The goal of this evaluation of the WP improvement is to better tally the operators' behavior toward the procedures in order to make of this implication a strategic orientation based on the organizational citizenship of the operators. In this context, we retained the Stop-Card support (Fig. 2) developed by the Sonatrach Group—Algeria as a part of the Safe Behavior Program launched during the period 2007–2012 in partnership with the Norwegian company StatOïl.

Fig. 2.

Stop-Card of the “Sonatrach Group—Algeria.”

In order to maximize its chances of succeeding the operators' implication, the Stop-Card tool must be deployed in three phases:

• -The first phase introduces the content as well as objectives of the Stop-Card tool to operators. During this phase, the focus must be on the interest that the managers grant to the success of the deployment of this tool. The result achieved at the end of this phase is the introduction of a mandatory dialog between the Health, Safety and Environment (HSE) managers and operators regarding the safety regulations, and consequently the considerable improvement of the formal communication.
• -The second phase is a trial phase. During this phase, a large communication process must be implemented, which includes formations, accompaniment sessions, and briefings for the operators. At the end of this process, the operator must fill up a sample Stop-Card regarding a live situation at his/her workplace. An examination of this sample enables the validation of this phase.
• -The third phase concerns the official launch of the Stop-Card in order to implement a management policy of the Stop-Cards, which is based on the results of which the most important, for the improvement of work procedures, are the consideration of the positive initiatives and the immediate remediation of insufficiencies. During this phase, a cycle of Stop-Card must be defined (1 calendar year) in order to check to what extent the objectives are achieved. Management of the Stop-Card cycle is detailed later in this article (see the “Results and discussion” section).

In order to illustrate the interest of the proposed method, which is based on the task performance and contextual performance (safety initiative), we have applied it to an example taken from the literature [3].

2.3. Application example

The field of application of the proposed method is the oil industry. The choice of this industrial category is justified not only by its high-risk activities [45], [46], but also by the fact that both old and recent research works have indicated that the main causes of 90% of the incidents are related to professional behavior (such as the NR-WPs), while those of only 10% are related to work conditions [47], [48].

The application example is a procedure of blowing of a pipe which connects the manifold to the oil storage and treatment center on a length of 11,500 m (Fig. 3). The blowing operation, which precedes the hydrostatic tests, is performed using a foam scraper pushed by compressed air. It helps in clearing the pipe of any foreign bodies that might have been introduced during the operation.

Fig. 3.

Blowing operation of a pipeline.

The equipment used in this operation is shown in Fig. 3. This operation is performed by two operators: one operator is responsible for injecting the compressed air from the compressor into the pipeline through the blowing head, and a second operator is positioned at the other end of the pipe in order to ensure of the exit of the scraper.

The two operators remain in contact by means of communication until the end of the operation.

The blowing procedure of the pipeline is as follows:

• -
  Preparation phase of the pipeline blowing:

  • T₁—Acquisition of work authorization: the pipeline blowing operation
  • T₂—Verification of the conformity of the equipment to be used in the operation, particularly, the minute details of the compressor's technical control
  • T₃—Establishment of a preventive plan for the blowing operation of the pipeline, and preparation of intervention and evacuation means
  • T₄—Preparation of blowing equipment
  • T₅—Ensuring that neither any people who are not involved in the blowing operation nor any foreign bodies are present at the site
  • T₆—Correct mounting of the blowing head on the pipeline end
  • T₇—Establishment of the jumper hose
  • T₈—Ensuring the presence of the two operators at the two ends of the pipe
• -
  Progression phase of the pipeline blowing:

  • T₉—Operation of the compressor
  • T₁₀—Monitoring of the progression of the blow-off (in case of any problems stop the compressor immediately, rectify the problem, etc.)
• -
  Completion phase of the pipeline blowing:

  • T₁₁—Shutting down of the compressor
  • T₁₂—Disassembly of the jumper hose
  • T₁₃—Disassembly of the blowing head at the end of the pipeline
  • T₁₄—Storage of the blowing equipment
  • T₁₅—Leaving the site while ensuring its cleanliness

Let us remember that this WP is valid for all the pipeline blowing operations in the oil field.

A quick review of this blowing procedure shows that the preparation phase tasks are many. They represent 57% of all tasks. In addition, investigations of accidents and incidents show that they usually occur in this phase when the operators often seek to simplify the procedure in an attempt to reduce the workload.

A preliminary indicator list has been defined based on the task list of this work procedure; these indicators are as follows:

• I₁ = procedure speed

• I₂ = number of employee initiatives

• I₃ = Rate of employee satisfaction

• I₄ = Availability of information

• I₅ = Variation of the rhythm

• I₆ = Employee experience in the application of this WP

• I₇ = Cumulative number of training hours

• I₈ = Quality of the product (or service)

• I₉ = Procedure flexibility rate (duration of the working time organization)

• I₁₀ = Rate of allocation of the objectives associated with the procedure

• I₁₁ = Rate of change in the structure of the system or in the operating mode

• I₁₂ = Gravity rate

• I₁₃ = Precursor items

• I₁₄ = Support for the operator in case of an accident

To check these indicators' internal coherence, the coefficient α is calculated (α = 0.784). Because is higher than 0.7, which suggests that the indicator list is good and can, therefore, be considered the final list required for the performance measurement of the studied WP.

3. Results

P-WP quantification is achieved at the retained indicator level. This quantification begins with the establishment of the scores (“formalization degree,” “implementation degree,” and “ownership level”) for each indicator. The evaluators responsible for the establishment of these scores are opting for a compromise (see “score matrix” in Table 1).

Table 1.

Results of weighed scores

		Weighing matrix			Score matrix			Balanced score matrix			Average of balanced scores
		DF	QME	NA	DF	QME	NA	DF	QME	NA
Indicators	I1	1	1	1	0	0	1	0	0	1	0.33
	I2	1	1	1	1	3	2	1	3	2	2.00
	I3	3	3	2	1	1	1	3	3	2	2.67
	I4	2	3	2	1	1	0	2	3	0	1.67
	I5	2	2	2	2	1	2	4	2	4	3.33
	I6	2	2	2	2	1	1	4	2	2	2.67
	I7	3	2	1	2	1	1	6	2	1	3.00
	I8	1	1	1	1	1	0	1	1	0	0.67
	I9	2	3	2	2	1	2	4	3	4	3.67
	I10	2	3	2	2	2	2	4	6	4	4.67
	I11	3	3	2	3	1	2	9	3	6	6.00
	I12	2	2	1	2	2	1	4	4	1	3.00
	I13	2	2	2	2	1	2	4	1	4	3.00
	I14	1	1	1	2	2	1	2	2	1	1.67

DF, degree of formalization; NA, level of ownership; QME, quality of implementation.

These scores are then crossed with a weighing matrix that shows the relative importance of each indicator (see “weighing matrix” in Table 1).

The level of performance of a given indicator is determined by dividing the average of the weighted scores by the sum of the weights (Table 2).

Table 2.

Measurement of the P-WP level

		Weighing matrix			Sum of weighting	Average of the balanced scores	Performance level (in %)
		DF	QME	NA
Indicators	I1	1	1	1	3	0.33	11.00
	I2	1	1	1	3	2.00	66.67
	I3	3	3	2	8	2.67	33.37
	I4	2	3	2	7	1.67	23.86
	I5	2	2	2	6	3.33	55.50
	I6	2	2	2	6	2.67	44.50
	I7	3	2	1	6	3.00	50.00
	I8	1	1	1	3	0.67	22.33
	I9	2	3	2	7	3.67	52.43
	I10	2	3	2	7	4.67	66.71
	I11	3	3	2	7	6.00	85.71
	I12	2	2	1	5	3.00	60.00
	I13	2	2	2	6	3.00	50.00
	I14	1	1	1	3	1.67	55.67

DF, degree of formalization; NA, level of ownership; P-WP, performance of work procedure; QME, quality of implementation.

Finally, the recommended method is anticipated to:

• -prioritize the indicators by decreasing the order of their performance level:

I₁₁ > I₁₀ > I₂> I₁₂ > I₁₄ > I₅ > I₉ > I₁₃ ≥ I₇ > I₆ > I₃ > I₄ > I₈ > I₁

• -represent, on a radar graph, the different performance levels of the studied WPs (Fig. 4).

Fig. 4.

Performance of the studied WP. WP, work procedure.

The radar of Fig. 4 synthesize the P-WPs related to pipes cleaning in oil field. It clearly shows that a formalization effort of implementation or ownership must be made in priority by the HSE department of the studied oil field, particularly at the following indicators level (radar of Fig. 5): I₂, I₃, I₄, I₆, and I₈.

Fig. 5.

Improvement of the studied P-WP. P-WP, performance of work procedure.

The advantage of the proposed method is the use of the radar graph to synthesize the P-WPs. In fact, the reality on the ground shows that all the industrial companies have documented their WPs as a list of actions that are to be performed or not, without justification and without a conceptual model. Therefore, WP capitalization in a graphical form (radar, for example) allows better visualization of its relevance on the ground and consequently of its update for its respect in case of necessity. Better yet, this form of WP capitalization aids in a new WP design, and even in the comparison of the same WPs practiced in different industrial companies or in various subsidiaries of an industrial group as well. This WP capitalization constitutes a help to assess the health and safety at work conformity of a company or an industrial group.

4. Discussions

In order to improve a WP, a facilitator support must be put in place. It is about the Stop-Card; its deployment is materialized by assigning an exceptional premium of merit exclusively to the operators who submitted the best Stop-Card of the month. The purpose of this premium is to involve all the operators in the approach to accident reduction at work and more specifically in the R-WPs. Thus, each operator is encouraged to emphasize the problems encountered, in terms of near misses, during the execution of the WPs.

These near misses contribute to the improvement of these procedures. Because, the cooperation of the operators makes to reinforce the P-WP and in consequence safety at work.

Management of the Stop-Card cycle consists of the following:

• -Monthly counting of the Stop-Card
• -
  A detailed evaluation of the Stop-Card:

  • ○
    Filtering and data treatment of the Stop-Cards
  • ○
    Counting relevant proposals in order to insert them into the WP update
  • ○
    Identifying the best Stop-Card of the month
• -
  A global assessment of Stop-Cards so that the set of the provided results is synthesized in the global assessment form:

  • ○
    The number of the monthly Stop-Cards that contain relevant information (Stop-Card is of type “Yes” if its information is relevant and of type “Not” in the opposite case). Fig. 6 illustrates the results obtained for the year 2014.
  • ○
    For the Stop-Card of type “Yes,” the interest is related to the rates of the Stop-Cards containing information of the following types: partially relevant, relevant, and very relevant. Fig. 7 illustrates the results of the Stop-Cards of type “Yes” by relevance types for the year 2014.
  • It is important to state that the appreciation of these relevancies is carried out by referring back to the results obtained during the previous months from where the possibility to make the month' recommendations in form of: a projected improvement for WP, future orientations to reduce the number of Stop-Cards type “No,” and other recommendations allowing a better operators implication.

Fig. 6.

Distribution of the Stop-Card by types.

Fig. 7.

Distribution of the Stop-Card by nature of relevance.

The results presented in Fig. 6, Fig. 7 show instantaneous exploitation of the Stop-Card in an oil field of the Sonatrach Group—Algeria. These results illustrate awaited basic competence of all the operators and consequently their behavior at work.

In the same way, the results presented in Fig. 6, Fig. 7 can be exploited for 1 calendar year (2014, in our case) in order to evaluate the following: effectiveness of communication via the Stop-Card tool, individual and cultural differences of the operators, team spirit level of the operators as well as their professional behavior, and finally creation of a stimulant environment.

The assessment of these parameters allows to evaluate the Stop-Card deployment during the current year in order to better prepare the following year and thus making of this tool a process of continuous management of health and safety at work.

This article focuses on the assessment and improvement of the P-WPs. In this context, the Stop-Card tool initiated by the Sonatrach Group—Algeria allows a better analysis of the activity, which, in collaboration with the procedure users, is the keystone of a good F-WP.

The major aim of the method proposed in this article is continued knowledge construction to deduce rules of good balance, allowing better formalization of actions, which should not be only in the manuals, but also be in operators' heads. Consequently, the interest of the retained approach, compared with classical approaches (work sociology, social psychology, ergonomics, and economic models), is that in all these approaches the operator did not dare to speak of his own individual initiatives for violating the WP and whose consequences are well known by specialists in health and safety at work. Consequently, his involvement in the analysis of the NR-WP is problematic.

[39–41]Finally, the method recommended in this article has allowed us to develop a promising way to help the Sonatrach Group—Algeria, to determine the specific actions that are required to be implemented to sustain any progress approach, particularly in terms of R-WPs.

Conflicts of interest

The authors declare that they have no conflicts of interest.