Recommended Protocol for Round Robin Studies in Additive Manufacturing

Abstract

One way to improve confidence and encourage proliferation of additive manufacturing (AM) technologies and parts is by generating more high quality data describing the performance of AM processes and parts. Many in the AM community see round robin studies as a way to generate large data sets while distributing the cost among the participants, thereby reducing the cost to individual users. The National Institute of Standards and Technology (NIST) has conducted and participated in several of these AM round robin studies. While the results of these studies are interesting and informative, many of the lessons learned in conducting these studies concern the logistics and methods of the study and unique issues presented by AM. Existing standards for conducting interlaboratory studies of measurement methods, along with NIST’s experience, form the basis for recommended protocols for conducting AM round robin studies. The role of round robin studies in AM qualification, some of the limitations of round robin studies, and the potential benefit of less formal collaborative experiments where multiple factors, AM machine being only one, are varied simultaneously are also discussed.

1. Introduction

There is a long list of specific roadblocks hindering widespread adoption of additive manufacturing (AM) [1][2], but many of these can be summarized as a lack of confidence in AM materials, processes, and parts. One way to help improve that confidence is by generating more high quality data describing the performance of AM processes and parts. However, generating large sets of data is particularly complicated and expensive in the diverse, rapidly evolving arena of AM. The potential for AM lies in its ability to easily create complex, highly customized parts, not the large batches of simple shapes needed for thorough material qualification [3]. Further, there are a large number of processing parameters, procedures, and variables when creating AM parts. Add in continual vendor updates to AM hardware and software and one can see how the cost of producing hundreds or even thousands of test coupons for each variation could become prohibitively expensive.

Many in the AM community see round robin studies as a way to generate large data sets while distributing the cost among the participants, thereby reducing the cost to individual users. Standard round robin study protocols were one of the consensus-based priority action items identified in the Measurement Science Roadmap for Metals-Based Additive Manufacturing for accelerating widespread use of AM [2]. Further, ASTM International Committee F42 on Additive Manufacturing Technologies and the International Organization for Standardization (ISO) Technical Committee 261 on Additive Manufacturing identified in their Joint Plan for AM Standards Development the need for high-level round robin standards broadly impacting AM [4].

Despite this interest, there is a lack of guidance in the literature for conducting a round robin study specifically for additive manufacturing. There are existing standards for conducting interlaboratory studies of measurement methods, which will be discussed in more detail later, but certainly AM presents unique considerations. To gain further insight into round robins for additive manufacturing, researchers at the National Institute of Standards and Technology (NIST) organized and participated in several studies. This paper summarizes these efforts and examines the lessons learned along the way. These lessons learned, along with guidance from existing standards for conducting interlaboratory studies on measurement methods, form the basis for recommended standardized protocols for AM round robin studies.

2. Definitions

A round robin study, or interlaboratory study (ILS), is an experimental methodology to determine reproducibility of a “process” where tests are performed independently multiple times and the results are analyzed statistically to assess their variability. The process can be a measurement method or fabricating an artifact using a well-defined procedure (e.g., an additive manufacturing process). Another purpose of a round robin study may be to verify that results of a new process agree with that of the more established process. This type of examination is very common in evaluating the performance of measurement methods. A round robin study provides a top-down evaluation of variability because it investigates the results directly, providing visibility of variation in results when the outcomes are produced by the different participants. This differs from a bottom up approach where typically one machine user examines how variability in each input contributes to the overall variability in the output. The top down approach may not differentiate between all of the sources of variability, but it captures some that may not be visible when an individual user builds an error budget without the benefit of comparing results from other independent users.

When building AM components, a manufacturing plan specifies the production sequence, machine and processing parameters, feedstock, and post-processing used in the production run. A manufacturing plan may also be referred to as a process specification or a fixed process agreement and is typically a requirement of a quality management system. In the case of a round robin study, the manufacturing plan documents all of the procedures and parameters needed to manufacture the part by each participating entity, as well as any centralized procedures performed by the study coordinator.

3. Descriptions of NIST AM Round Robin Studies

Since 2012, NIST has led two AM round robin studies [5][6] to completion with another study ongoing, and provided significant input to a fourth study led to completion by an outside organization. The simplest and most academic of these studies focused on the geometric performance of 3D printers (MakerBot systems) used in various laboratories throughout NIST. The first material-focused round robin used a cobalt chrome (CoCr) alloy. The external study, led by EWI, focused on nickel alloy 625 (IN625). The ongoing study being led by NIST also focuses on IN625, but in a slightly different manner. In all of these studies the raw materials were provided to each participant, as well as a manufacturing plan or the specific processing parameter values (or both), in an effort to keep practices as consistent as possible between various participants. Table 1 summarizes these studies and subsequent subsections describe them in more detail.

Table 1.

Summary of additive manufacturing round robin studies

	NIST 3D printers	NIST CoCr	EWI IN625	NIST IN625
# Participants	7	8	2 labs, 3 machines	7
Machine Types	Replicator G; Replicator 2, Replicator 2×	5 EOS M270, 1 400 W PBF- laser process, 2 Arcam	EOS M270	5 EOS M270, 1 EOS M280, 1 SLM 250
Material	PLA, ABS	Cobalt-Chrome alloy	Inconel 625	Inconel 625
Measurements	Geometry	Tension (x-direction only), Microstructure	Tension, Load controlled fatigue (HCF), Strain controlled fatigue (LCF) Microstructure	Tension (x-direction only)
Goal	Learn more about round robin studies. Pilot test for geometry measurements	Learn more about conducting material round robin studies	Develop Manufacturing Plan; Seed data for S-basis design allowables	Seed data for AM materials database
Dates	May 2014 to August 2014	October 2012 to September 2014	October 2012 to September 2014	January 2014, ongoing as of May 2015

3.1 Geometry Variation with 3D Printers

Chronologically the 3D printer round robin study was the last of the studies to be initiated, but the simplicity of the study makes it a good starting point for discussion. This study was conceived to be as simple as possible to be completed in 10 weeks by a summer intern. In this study, three test parts were built by seven different NIST groups on different 3D printers and the sizes of 19 features atop a 40 mm regular octagonal base (see Figure 1) were measured three times by the same person. Variations in the sizes of the features were analyzed focusing on between-participant, between-build, and between-measurement effects [7][8]. The full report on this study can be found elsewhere [5], but highlights of the results are shown here because they are typical of results seen in other studies.

Figure 1.

Solid model of part designed and built as part of NIST 3D printer round robin.

Figure 2 shows a box-and-whisker plot summarizing measurements on one of the features that is characteristic of the results. In this plot, each box and whisker set represents one printer from one participant. The dark horizontal bar represents the median, the height of the box encompasses the middlemost half of the data, and the top and bottom of the whiskers depict the maximum and minimum measurement respectively. The heights of these boxes are relatively small compared to the range of all the boxes, demonstrating that variation between printers (i.e., between participants) is much larger than variation between builds within one printer.

Figure 2.

Box and whisker plot summarizing variation observed in measurement of the size of one feature as part of a NIST 3D printer round robin study. Note that participant A produced seven parts, participant B produced 1 part, and other participants produced three parts.

Figure 3 further demonstrates that between-participant effects dominate variation in part and feature sizes in this study. Of the nineteen features examined as part of this study, the effect of differences between participants (“printer effect”) is largest in nearly every case according to a mixed-effects model (i.e., a combination of random and non-random treatment of variables) fitted to each feature in order to perform an analysis of variance [7][8]. Further, the between-participant effect is usually at least three times larger than the within-participant (“build”) effects. Note that in most cases the measurement effect, an estimate of measurement uncertainty of the individual feature, is significantly smaller than the other effects.

Figure 3.

Plot of variability effects as determined by a mixed effects model.

3.2 NIST-led Cobalt Chrome Study

The focus of the cobalt chrome study was on the mechanical property of AM parts using five nominally identical laser-based powder-bed-fusion (PBF) machines, but other machines (including one laser-based PBF machine with a higher power laser and two electron-beam PBF machines) were included as a preliminary investigation of variability between types of machines. The parts created for this study were tension specimens oriented parallel to the machine X-axis. The parts built by each participant were all heat treated following the same procedure to relieve residual thermal stresses. The parts were built net-shape, meaning they did not require any post-process finish machining beyond removal from the build platform. Also, all tension testing was performed following the same procedure on the same machine by the same person. A full report on the study is currently being prepared and will be available [6].

The results of this study show that, once again, the between-participant variability is significantly larger than the within-participant variability. Figure 4 shows the engineering stress versus engineering strain curves for all samples tested. Clearly the samples produced on the electron-beam machines (lab 7 and lab 8) are significantly different than the samples produced on the five similar laser-based PBF machines. As such, the analysis of variance only included the samples produced on the five similar machines. Even still, with the exception of Young’s modulus, between-participant effects were still the largest contributor to variability in the tensile properties, as illustrated in the box-and-whisker plots shown in Figure 5. Note that while the tensile properties were analyzed for variation among the specimens produced for the study, their magnitudes were not compared to parts made by traditional processes.

Figure 4.

Engineering stress versus engineering strain curves for all samples tested as part of the NIST-led cobalt chrome round robin study [6].

Figure 5.

Box-and-whisker plots for the tensile properties of samples produced by each participant in the NIST-led cobalt chrome round robin study [6]. The left five (blue) entries were all from the same type of machine; the right three (red) entries were from different type machines.

3.3 EWI-led Nickel Alloy 625 Study

The EWI-led study on nickel alloy 625 took a slightly different approach than the two previously described studies. This study intended to be “akin to a round robin study,” but because of logistical issues was only able to include two participants and three machines. Further, the stated goal of this study was to develop a manufacturing plan and generate seed data for design allowable material properties. Despite these differences, the experience gained in the study was extremely valuable and certainly applicable to future round robin studies. The final report from this study is publically available [9].

A significant part of this study was the development of the manufacturing plan used by the participants to build the test parts. Seven different organizations contributed to the project and each had valuable input into the manufacturing plan. This manufacturing plan covered many aspects of producing the test parts including,

• part geometry, location in the build volume, and orientation

• machine requirements including suggested maintenance and calibrations

• raw material requirements and material handling

• building platform requirements

• machine setup, laser exposure settings, and laser path strategy

• in process monitoring

• part removal

• and post-processing of the part.

This document was certainly the most thorough and detailed of the manufacturing plans in the studies examined here. The intent of the manufacturing plan was not only to govern the builds within the study, but also to exist beyond the study to allow other users to adopt the content and structure of the plan to produce and test samples on their own, thereby adding consistent/comparable high-fidelity data to the dataset.

The EWI-led study examined horizontally-oriented and vertically-oriented tensile and fatigue specimens that were machined from larger blocks made by laser-based PBF. The blocks were heat treated and subjected to hot isostatic pressing before being machined. However, only the results of the tensile tests are applicable to the current discussion because there was no intercomparison analysis of the fatigue data. Because this was not a true round robin study, the analysis focused on hypothesis testing of sample equivalence rather than an analysis of variance. Further, because two different environments were used in the four builds, only the comparisons of like-environment samples (build 1 to build 2 and build 3 to build 4) are applicable to the current discussion. The results of the analysis are mixed, as seen in Fig. 6 showing the yield strength (YS) and ultimate tensile strength (UTS). In all cases, the vertically-oriented specimens (labeled “Z” in the figure) were statistically different than the horizontally-oriented specimens (labeled “XY” in the figure). For the horizontal specimens, build 1 was statistically similar to build 2, but the hypothesis test revealed that build 4 was statistically stronger than build 3. Looking at the vertical specimens, yield strength for build 2 is statistically larger than that of build 1, but ultimate tensile strength is statistically similar for the two. Build 4 shows larger yield strength than build 3, but the ultimate tensile strengths are similar between the two. A similar mix was found when examining percent elongation and percent area reduction.

Figure 6.

Summary of results of tension tests performed on samples as part of the EWI-led round robin study [9]. For the current discussion, build 1 should be compared to build 2 and build 3 should be compared to build 4.

3.4 NIST-led Nickel Alloy 625 Study

The NIST-led nickel alloy 625 study is still ongoing and therefore has not provided any results to analyze. However, several lessons have still been learned in conducting the study to this point. This study seeks to combine aspects of the NIST-led cobalt chrome study with aspects of the EWI-led nickel alloy study. Specifically, this study attempts to include more participants using similar equipment (all machines are laser-based PBF machines) while also providing each participant with a thorough, documented manufacturing plan.

4. Lessons Learned

We can divide what was learned from these studies into two sections: lessons learned from the study results, and lessons learned regarding procedural issues.

4.1 Results

The major takeaway from the results of the concluded studies is that between-participant variation plays a larger role than within-participant variation in the standard uncertainty of part performance. In the majority of cases studied (i.e., the individual features examined in each study), the variation in measured part characteristics between participants was statistically significant, even when using the same raw materials and machine settings. This leads to the conclusion that unknown variables exist between participants (or there are unknown machine differences, e.g., age of machine, maintenance or calibration history, etc.) that were not adequately controlled by the manufacturing plans. Looking toward future AM round robins, it appears that increasing the number of participants, or at least the number of machines, will have a larger effect on reducing the standard uncertainty in the consensus values than by having each participant produce more samples [6].

The difficulty with the results is that while the variability in the results is clear, the causes of the variability are not. Without specific knowledge of which aspects are to blame, there is no quantitative basis for modifying the manufacturing plan to produce more consistent parts. This top down approach illustrates clearly that there are unknown sources of variability [10], but if the goal is to learn more about the manufacturing process and to improve its performance, this mere illustration may be inadequate compared to error budgets or sensitivity analyses that provide more actionable results.

4.2 Procedures

One common take away from all four round robin studies is that the manufacturing plan is vitally important. It is clear from these different studies that the plans must address much more than machine processing parameters and that there are valid reasons for different studies to have different manufacturing plans. Producing a part by AM involves much more than programming the machine; there are machine calibrations and setup, raw material handling, post processing of the parts, etc. A robust manufacturing plan must address all of these areas to avoid confusion and assumptions by the different participants. Further, a very focused study using only one machine type can have a very detailed, specific manufacturing plan, but if the study includes multiple types of machines, the various parameters may need to be described more generically. Because some of the detail of the manufacturing plan may be dictated by the scope of the study, it is important to note that guidance on conducting a proper round robin study is not the same as instructions for producing a proper manufacturing plan.

The EWI-led study presented an interesting case because the development of the manufacturing plan was the most emphasized step in developing the round robin study. All study participants contributed to developing the manufacturing plan. This joint development was valuable from the standpoint that best practices from multiple users were brought together to benefit the entire plan. However, the development of the plan took nearly one year as each participant suggested best practices that often needed to be verified or validated by other participants.

Related to this issue, it was noticed that in each study it was very difficult to identify many truly “equivalent” AM systems. In the young, quickly evolving field of AM, the commercial availability of a specific machine model is rather short. But certainly similar models of machines existed in these studies. However, in nearly all cases each system used a different version of control software or file processing software. This presented some problems in programming the machines since in some cases different versions of control software offer different access to processing parameters. Further, different software versions often made it difficult to deliver consistent part files to each participant. The desire was to deliver machine program files to each participant to avoid any inconsistency in converting from computer aided design (CAD) file to stereolithography (.STL) format or from .STL to machine program file. However, in many cases the participants could not load the machine program files delivered to them because of software version incompatibility. In these cases, a more generic file was provided but then each participant needed to convert the file into the specific machine program for the software version.

A pilot run of the round robin study would help reveal many challenges, ambiguities, incompatibilities, and problems with the manufacturing plan and design file. Most often a pilot run involves a small subset of the participants executing the plan before running the experiment at full scale with all participants involved. None of the studies discussed here used a pilot run in this manner. However, a different sort of pilot run was used in the EWI-led study where each participant followed most of the manufacturing plan to run much shorter builds with simpler geometry than in the full scale run. This pilot run served two purposes: it provided a calibration part that could be used to adjust parameters on each machine, and it revealed that one participant was not executing the manufacturing plan correctly. This latter purpose is of primary importance. The pilot phase of the study is a far better time to make any adjustments or clarifications in the manufacturing plan than after commencing the full scale run when problems requiring adjustments to the plan may only present themselves after several participants have already completed their builds.

Participants should fill out a template (a process control document—PCD) with important setup and background information as they prepare to build their part(s) and return that information along with their finished parts. While the manufacturing plan describes the procedures and settings for building the parts, there is no guarantee that each participant will accurately follow the instructions. This template (PCD) will act as a checklist to remind participants of all required procedures and, more importantly, will provide meta-data for each part built in the study. This meta-data will provide essential context for interpreting the results, especially if the measured values will be stored in a material or machine performance database. Further, if there are any outliers in the data, these templates will indicate whether or not the participant deviated from the intended procedures.

A round robin study, as well as the manufacturing plan, needs to consider the entire manufacturing chain, especially post-processing of parts, which may include heat treatment, machining of specimens to final shape, surface preparation, and separating specimens from the build platform. The purpose of the study might dictate how the study handles these post-processing steps. For example, if the purpose of the test is to evaluate variability in the AM system, all steps after the completion of the AM build should be centralized. If there is variability in the post-processing, there will likely be ambiguity in analyzing the final variability in the parts: the final variability could be a result of the post-processing and not a result of the AM system. However, if the goal of the study is to examine variability in finished AM parts delivered by different suppliers, it will be more appropriate to allow each participant to handle the post-processing separately.

Heat treatment of metallic AM parts is an important consideration that presented difficulty in some of the metal material-focused studies discussed here. If a goal of the study is to produce specimens with mechanical properties similar (or superior) to properties of wrought or cast parts (or even if only a comparison to wrought or cast is desired), the details of the heat treatment must be carefully chosen since they have a profound effect on mechanical properties. Standard heat treatments for wrought or cast parts may or may not be applicable to AM parts, depending on the microstructure of the as-built AM parts. If the goal of the study is only an intercomparison of samples made within the study, the heat treatment of each specimen may only need to be consistent (and not damage the parts). Note that procedures and parameters for heat treatment (as with all other post processing steps) should be fully defined before starting any builds (i.e,, these should be part of the test plan or the manufacturing plan). However, the addition of an extra witness part to each build that will not be post-processed may help later to alleviate any concerns that test results were heavily influenced by the post-processing instead of the AM process.

Another commonality among all of the studies discussed here is that they either took longer to conduct than planned or did not fully complete their goals within the original timeframe. Of all the studies, only the 3D printer round robin took less than one year. Even for that simple study with all of the equipment on the same campus (though in different locations), all of the intended measurements and analyses could not be completed within the planned timeframe of ten weeks. One conclusion to be drawn is that more time than initially anticipated usually needs to be budgeted for studies like these. However, another direction could be to look to standardization to help streamline and clarify the protocols for round robin studies.

5. Standards for Interlaboratory Studies of Measurement Methods

Round robin studies, more formally termed interlaboratory studies, are very common in evaluating the performance of measurement methods. In fact, both ASTM International and ISO have standards detailing how to conduct interlaboratory studies to evaluate measurement methods [11][12], as well as documents summarizing these standards [13][14]. In order to extract any guidance for AM round robin studies from these measurement-focused ILSs, it is important to develop an analogy between the two. In a measurement ILS the defined control is the material, while in AM round robins the control is the design file. In a measurement ILS, the set procedure is often a standardized test method or at least a draft standard test method; in AM round robins it is the manufacturing plan. Finally, the outcome in measurement ILS is the measurement result, while in AM the result is the physical part. With an acceptable analogy established, a more detailed look at ASTM E691-13 is warranted to explore commonalities with AM and unique issues presented by AM.

5.1 Commonalities

ASTM E691-13 breaks down an ILS into three stages: planning the ILS, conducting the testing phase of the ILS, and calculation and display of the statistics. The current discussion will ignore the calculation and display of statistics because they are always some form of analysis of variance and calculation of consensus estimates, regardless of the purpose of the study. Conducting the testing phase of the ILS is broken into two subsections: the pilot run and the full scale run. The importance of the pilot run was discussed previously. The brief instructions in the full scale run section of ASTM E691-13 focus mostly on logistics and are mostly applicable to AM round robins without adjustment or can simply be removed if not applicable.

The “Planning the ILS” section of ASTM E691-13 is broken into seven subsections: ILS membership, basic design, test method, participating laboratories, materials, number of test results per material, and protocol. The ILS membership discusses the need for an ILS task force that determines the scope and details of the study, an ILS coordinator, and a statistician. This membership is also appropriate for an AM round robin. Similarly, the basic design section notes the importance of keeping the design simple and presents a few instructions that are also appropriate to AM without edit. The remaining sections require consideration specific to AM.

5.2 AM-specific Considerations

The test method subsection of ASTM E691-13 stresses the importance of a proven test method that each participant can follow as set procedures. This is certainly true in AM studies and the importance of the analogous manufacturing plan was stressed here in an earlier section. However, the special AM-related consideration for this section concerns the method of establishing that the procedures are “proven.” ASTM E691-13 states that a test method should have been subjected to ruggedness testing prior to its being used in an ILS. A ruggedness test is described in ASTM E1169-13a [15], but it essentially describes a design of experiments to analyze the sensitivity of results to various inputs. The purpose of a ruggedness test in this context is to aid in setting appropriate ranges for certain variables in the test method. However, the large number of variables present in a typical AM build (regardless of AM technology) makes a complete sensitivity analysis impractical and very unlikely. While this has several implications, the one relevant here is that there needs to be a different manner of demonstrating that the manufacturing plan is proven. The best alternative is that a pilot run can be used to test and ultimately improve the manufacturing plan. This only adds to the importance of the pilot run.

ASTM E691-13 states that 30 or more participants are desired for the ILS, but the absolute minimum for any study is 6 participants. Further, the standard suggests that participation in the ILS should not be limited to only expert practitioners because this may result in variability statistics being artificially small. When considering round robin studies for AM, there are reasons to provide alternatives or clarifications to both of these recommendations. The difficulties in finding equivalent systems (if required by the stated goal of the round robin study) were discussed previously. Compounding the problem is that there are not large numbers of metal-based AM machines currently in operation; for example the largest number of metal-based AM machines of one model in the U.S. is no more than 200 [16]. These low numbers make it impractical to recommend 30 participants in an AM round robin study. A likely alternative is to cite the observed variability revealed in existing round robin studies and emphasize the importance and implications of including as many participants a possible. However, these recommendations assume that the goal of the study is to establish a comprehensive assessment of variability of AM systems or parts, the type of assessment appropriate for a broad process qualification or part certification. If the purpose of the study is to develop data that will be used for generating design allowable material properties, adequate filters should be introduced into the protocols in order to prevent corruption of data by improper builds. Further, it may be appropriate in cases like this to require that the participants demonstrate a certain level of competence before being included in the study.

The materials subsection of ASTM E691-13 is analogous to the instructions for the part design or part file in an AM round robin study. The measurement ILS discussion focuses on the number and types of materials used in the study. In an AM study, it is certainly appropriate to consider multiple part designs, though in many cases having only one design will be sufficient. However, there are other considerations for AM studies. An AM round robin study should carefully consider the format in which the design file is delivered to the participants. Typically an AM part design evolves from the original CAD file to a .STL or additive manufacturing format (.AMF) file, to a machine file. Each file conversion presents an opportunity to lose data or part fidelity. These conversions may be particularly important for studies focused on part geometry because the conversion from CAD file to .STL file results in a faceted model that approximates the part geometry. Accuracy thresholds can be set to help control the loss of geometric fidelity, but this requires additional specific instructions in the manufacturing plan. The studies examined here delivered the machine files that describe both the design of the test part, as well as geometry of each layer and the scan strategy or tool path used for each layer in an attempt to minimize file conversions. The problems with this were mentioned previously. The problem with providing file formats that are more upstream (i.e., raw design files that are not fully processed into a format for insertion into an AM system) is that they introduce more potential sources of variability beyond the AM systems.

The “number of parts” recommendations (i.e., number of test results per material) from ASTM E691-13 apply to AM as well, especially since they stress that time and effort is better spent on a small number of samples from more participants rather than a large number of samples per participant. One additional consideration for AM is how to generate the replicate samples within each lab: within one build or over separate builds, within one machine or from several machines? In many cases, the size of the test parts will dictate that they be built in separate builds. However, if the samples are small enough to allow multiple samples in one build, much time may be saved by fabricating all of the samples in one build. However, this strategy would quantify within-build variability, which is not necessarily the same as within-lab repeatability. Similarly if a participant has multiple machines, each machine may be counted as an individual participant, depending on the stated goal of the study.

The protocol section of E691-13 concentrates on the formal communications and instructions with the participants. Again, all of this section applies to AM but with one additional consideration. A preliminary questionnaire to interested participants may be appropriate to determine whether or not a specific machine user should be considered qualified to participate in the study. This questionnaire should address the AM hardware (system model), the system control software and version, participant experience with the system or material being used in the study, as well as other essential capabilities required to complete the build.

6. Recommended Protocol for AM Round Robin Studies

These lessons learned from experience with round robin studies as well as commonalities with well-established standards on ILS lead to the following flow of events for future AM round robin studies:

1. Select the round robin task force, study coordinator, and statistician

2. Clearly define the goal and scope of the study

3. Select or develop an appropriate manufacturing plan

4. Design the round robin experiment—the raw material, required system capabilities, and number of parts and builds should be established in this step (if not in the manufacturing plan) to help inform potential participants of the time and system requirements

5. Solicit participants—send a capabilities and experience questionnaire to interested parties followed by a formal statement of intent to qualified participants

6. Prepare the part design and part files

7. Send the manufacturing plan to participants—encourage participants to examine the plan immediately and communicate any problems or feedback early

8. Execute a pilot run with select participants

9. Adjust the manufacturing plan or the experiment design if necessary

10. Send raw material to all participants (if necessary)

11. Maintain communication with the participants while they complete the required builds

12. Collect finished parts and meta-data templates (PCD) from all participants and label received parts

13. Examine all meta-data templates

14. Send parts to be post processed (if necessary)

15. Coordinate testing or measurement of all samples

16. Process, analyze, present, and store statistical data

17. Write and publish the final report

These steps will form the basis for a standard guide similar to ASTM D7778-12 [13], but specifically for the AM community.

7. Conclusions

With the proliferation of AM, the desire to better understand the processes, and the need to qualify critical components manufactured by AM, round robin studies are sure to be conducted often in the future. A standard guide for conducting these round robin studies will help ensure consistency and efficiency when conducting the studies. Consistency and efficiency will certainly benefit AM users since one of the goals of AM round robins is to reduce the cost of generating larger sets of trusted data. ASTM E691-13 and ASTM D7778-12 provide excellent templates to build from for AM-specific standard recommendations.

The AM-specific recommendations stem from the experience gained in conducting AM round robins and from analyzing the results of several concluded AM round robins. Early results indicate that better use of time and resources can be made by conducting round robin studies with a larger number of participants, each producing a smaller number of parts. Further guidance focuses on the round robin procedures, with special considerations for AM studies needed regarding part data formats, the capabilities and experience of participants, and the manufacturing plan each participant follows. It should be kept in mind that a proven, robust manufacturing plan is vital to a round robin study, but the guidance for conducting a successful round robin study is not the same as developing an appropriate manufacturing plan.

A successful round robin study will result in a quantification of the repeatability and reproducibility of a specific performance metric related to an AM test part. A successful round robin study does not ensure that the data will have low variability and therefore does not guarantee that the resulting data will be appropriate for generating design allowable material properties or for qualification of parts or materials. However, since repeatability and reproducibility are essential for a qualified manufacturing process, a well conducted round robin study will certainly aid in qualification of AM manufacturing plans or AM processes.