What is Acceptance Sampling
Acceptance Sampling is the practice of testing a sample of product from a large batch, then dispositioning (accept or reject) the entire batch (or lot) based on the results of the inspection.
In lamens terms, Acceptance Sampling will allow you to infer the overall “quality” of an entire lot while only requiring that you test a fraction of samples from the entire lot.
This definition above is generally applied to the inspection of incoming raw material, however this practice can also be applied to the inspection of material within your process, or final testing of finished product.
There are a few major techniques/concepts that every CQE should understand and be able to apply when executing as sampling plan.
This will ensure that you’re rigorously inspecting product for conformance to the products quality characteristics, these concepts are:
- Advantages & Disadvantages of Acceptance Sampling
- How to Create a Sampling Plan
- Picking your AQL
- Understand Risk Factors (α & β)
- International Sampling Standards
- Sampling Plan Types & Details
- Sampling Schemes
- Acceptance Sampling Terms & Definitions
- Plotting an Operating Characteristics (OC) Curves
- Interpreting an OC Curve
Here’s a quick video from Keith Bower that is a general overview of what Acceptance Sampling is, and how it’s used.
By the way, the two Military Standards that he references can be found just below this in step 3 of the How To Create a Sampling Plan section.
Before we get into all of the concept noted above, I want to quickly highlight the advantages and disadvantages of Acceptance Sampling. You should understand the strengths and weaknesses of this concept prior to implementation so that you’re aware of the risks, etc.
Advantages of Acceptance Sampling
The primary advantage of a sampling plan is that it is economical.
Acceptance Sampling will save you time, money and resources when evaluating the overall quality of your product.
This savings can take the form of reduced labor, reduced part movement and a shorter product throughput.
Unless you’ve hired Lucy and Ethel, then you’re in real trouble!
A Sampling Plan is also required anytime you have to test for a requirement that is destructive in nature.
Disadvantages of Acceptance Sampling
The disadvantage of Acceptance Sampling is that it can introduce risk. Anytime you’re making inferences about the overall quality of a lot, and thus make a decision to accept or reject the entire lot, you run the chance of making the incorrect decision.
For example, a consumer may accept a lot of product that they believe to be of high quality, that does not actually meeting their Quality Standards, this is called Consumer Risk.
Another form of risk is that a consumer may reject a lot of product that actually does meeting their Quality Standards, however their sampling plan has led them to make the wrong decision, this is called Producers Risk. Both of these risks are discussed in more detail below.
One last disadvantage of Acceptance Sampling is that it can lead to a false sense of security when it comes to ensuring quality. Quality can never be inspected into your product, and process controls should always been taken to prevent or appraise product for Quality.
A Sampling Plan should not be used as a substitute to process monitoring or process improvement. By the time the product is being inspected, the quality (high or low) is built into the product.
How to Create an Acceptance Sampling Plan
I think the best way to understand how acceptance sampling works is to walk you through the process of creating an actual sampling plan.
Below I’ll walk you through the 6 step process of fully defining your Acceptance Sampling Plan.
Step 1 – Determine your AQL
When developing your Acceptance Sampling Plan it’s important to review the defects associated with your device and classify those defects.
In doing this you can define a different AQL level for each defect type (minor, major & critical), then inspect for each defect over the entire sample size and ensure an acceptable rate for each defect type (minor, major & critical).
For example, critical defects should have a lower AQL than a minor defect; but both can be inspected for during your sampling plan.
When picking your AQL you can also group defects into one AQL level or you can break our each defect into its own AQL limit. For example, perhaps your product has 3 major defects, all of these can be lumped into 1 bucket, or each can have it’s own Accept/Reject value.
Lumping them all into 1 bucket drives a tighter inspection because it allows fewer overall defects. This decision must be based on your knowledge of the product and its different defects and how those defects interact.
Also, when you’re sampling for multiple defect types (minor, major) at different AQL levels then the sampling can be thought of as an AND function.
Your sampling plan can result in no more than 1 critical defect AND no more than 3 major defects AND no more than 10 minor defects. All conditions must be met before the lot is received.
In general, the following AQL’s are applied to Critical, Major & Minor Defects:
- Critical – 0.1
- Major – 2.5
- Minor – 4.0
If you’re sampling material that has been purchased from a supplier, you will want to gain agreement with the supplier an an acceptable AQL. In this way, they can inspect for the agreed upon AQL prior to shipment. This will also establish an agreed upon baseline for lot rejection.
Another item you’ll want to establish with your supplier when creating a sampling plan for raw material is your Risk Factors (α & β).
Step 2 – Pick Your Risk Factors (α & β)
Because you’re only testing a fraction of the total product, there is a risk that your conclusion about the overall quality level (AQL) of the total product is wrong. There are 2 risks in this situation; one for the Producer of the product, and one for the Consumer.
Consumer Risk is the risk that your sampling plan will lead you to accept a lot of product that does not actually meet your quality standards.
The Consumer Risk level is defined by the symbol β and for calculation purposes is normally set at 10% or 0.10. This means that there is a 10% chance of of accepting a lot that is actually performing at the Rejectable Quality Level (Unacceptable).
Producers Risk is the risk that a sampling plan may lead to the rejection of a lot that actually does meet all quality standards.
Producers Risk is defined by the symbol α and for calculation purposes is normally set at at 5% or 0.05. The Producers Risk is interpreted to mean that there is a probability α (5% traditionally) of being rejected even if the full lot quantity is acceptable.
Step 3 – Know Your Data Type
Today there are 2 widely utilized sampling plans which were originally developed in the 1940′s as Military Standards; these are ANSI Z1.4 & ANSI Z1.9.
To choose the correct sampling plan, you must understand the type of data you’re collecting. Your data will either be Attribute (pass/fail) or Variable.
If you have the opportunity to collect either types of data, I would suggest collecting Variable data.
3.1 – Sampling Plan Standard for Attribute Data (ANSI/ASQ Z1.4)
This standard is basically a replicate of MIL-STD-105A and should be used when you are collecting Attribute Data (Pass/Fail).
Sampling Plans for attribute data are constructed such that you will test a sample quantity(n) of product from the overall population (N) and compare the number of non-conformances you observe (d) against a predefined acceptance number (a) & rejection number (r).
3.2 – Sampling Plan Standard for Variable Data (ANSI/ASQ Z1.9)
This standard is basically a replicate of MIL-STD-414 and should be used when you are collecting Variable Data.
While collecting variable data can be more complicated, the required sample sizes for sampling variable data is much smaller than that of attribute data.
These Sampling plans assume that your data is distributed Normally and rely on statistical calculations like the sample Average, Standard Deviation or Range.
These plans are set up such that you will compare your calculated statistical values (Average & Standard Deviation) against pre-set critical values (k) which will reveal the proper Accept/Reject decision.
Using the Standard Deviation Method, you will calculate Qu & QL and compare that against k. For example if Qu < k or QL > k then the lot does not meet the acceptance criteria. This assumes your data has a 2-tail distribution.
I’m not including any tables for the variable sampling plans as they are used less frequently and are a bit more complicated to explain.
Step 4 – Determine Which Sampling Plan You Want to Use
In general, there are 4 different, predefined types of sampling plans you can utilize.
Each one will have a different sent of acceptance/rejection rules and will also require a different number of samples to be tested.
Another difference between these plans is how they are administered, which can affect the cost associated with the testing.
Below are a handful of the most common sampling plans that you can utilize:
- Single Sampling Plan
- Double Sampling Plan
- Multiple Sampling Plan
- Continuous Sampling Plan
Here’s a Video from Clint Stevenson at NC State University where he does an excellent job of explaining the ideas of a single and double sampling plan.
4.1 Single Sampling Plan
The single sample plan is similar to the example above and is fully characterized by the following features (at an AQL of 1.0):
|N||The Total Lot Size||The total quantity of product contained within the lot you are testing||N = 7000|
|n||The Sample Size||The sample size of your Acceptance Sampling Plan||n = 200|
|acc||Acceptance Number||A pre-defined number (maximum limit) of allowable non-conformances that can be observed during your testing before you reject the lot||a = c = 5|
|rej||Rejection Number||A pre-defined number or limit where the full lot will be rejected if this many non-conformances is observed during testing.||r = 6|
|d||Number of observed Non-conformances||This is the actual number of measured non-conformances that you observe during your Acceptance testing.||d = 4|
In this example you receive a lot of 7,000 (N) widgets and you sample 200 (n), you can compare to total number of rejects (d = 4) with your acceptance number (ac = 5). Since d is less than ac, you can accept this lot.
The advantage of this type of sampling plan is that it is easy to administer. The technician only has to pull one set of samples and perform 1 round of testing.
The disadvantage of this sampling plan is that over the long run, it results in a higher Average Sample Number. This means that the cost associated with this sampling plan ends up being higher than the others.
4.2 Double Sampling Plan
For the double sampling plan, the inspector is required to pull & inspect a sample, then based on your predefined acceptance and rejection criteria the inspector can make one of three decisions, to accept, reject or continue testing.
Here is a table of the key features of a Double Sampling plan with some example values.
|N1 = 30||Ac1 = 0||Rej1 = 2|
|N2 = 30||Ac2 = 1||Rej2 = 3|
So let’s take the example above and we pulled 30 (n) samples to test.If no nonconformances are found, the lot can be accepted. This is where the double sampling plan can save you time and money.
Additionally if the number of nonconformances of those first 30 samples exceeds your reject number (rej1 = 2), then you can also reject the lot. If you’ve got 1 non-conformance then you can then perform the 2nd round of sampling by testing an additional 30 samples.
For this 2nd round of testing it’s important that you don’t return the original test samples back to the full lot, as you want to be testing the remaining product from the full lot.
The advantage of the double sampling plan is that over the long run, your ASN is lower than a single sampling plan. The trade off for the double sampling plan is the complexity of administering the test and any time that is wasted retrieving additional samples, etc.
4.3 Multiple Sampling Plan
The multiple sampling plan is just an extension of the double sampling plan, where you can plan up to 7 different rounds of product testing, each with their own acceptance and rejection criteria.
This has the same advantages and disadvantages as the double sampling plan where on average, your sampling number per acceptance lot will go down, however the plan can become difficult to administer when the person performing the testing has to pull 7 different lots of product to test.
Here’s an example of a Multiple Sampling Plan at an Sample Size Code of J and an AQL of .65.
|N1 = 20||acc1 =*||rej1 = 2|
|N2 = 20||acc2 = *||rej2 = 2|
|N3 = 20||acc3 = 0||rej3 = 2|
|N4 = 20||acc4 = 0||rej4 = 3|
|N5 = 20||acc5 = 1||rej5 = 3|
|N6 = 20||acc6 = 1||rej6 = 3|
|N7 = 20||acc7 = 2||rej7 = 3|
In this example the sample sizes are all the same, for your application each new sample size and accept/reject criteria can be different. Similar to the multiple sampling plan is the sequential sampling plan.
The major difference being that the testing in a sequential plan can go on continuously until an acceptance or rejection decision is clear or until the entire lot has been tested. In this plan there is always a range of values between the acceptance and rejection values such that further testing can always be chosen.
4.4 Continuous Sampling Plan
If you work in an industry where the production of certain products are continuous, then a Continuous Sampling Plan can be implemented.
The Continuous Sampling Plan can be fully characterized by 2 features:
- f – the frequency of sampling a unit; normally expressed as a fraction like 1/50, which implies that 1 in 50 units is inspected.
- i – the clearing number; normally expressed as a whole integer like 50, which implies that 50 units must be defect free before sampling can begin.
A continuous sampling plan works like this, when the lot first starts up, the inspection begins at 100% frequency until i (the clearing number) of units are found to be defect free. At that point, the inspection plan changes to a frequent inspection.
The sampling will continue at this pre-defined frequency until a defect is found at which point the inspection will switch back to 100% until i (clearing number) units are found to be defect free.
A distinction for this type of plan is that real time samples are taken from the line at set intervals, providing test data that is highly representative of the full lot quantity. With other sampling plans the the picking of samples can be more biased.
4.5 Average Sample Number
One major difference between these different plans is the Average Sample Number. This is the average total number of samples required to test to make an accept/reject decision.
Below is a comparison of 3 different sampling plans at an AQL of 6.5 and a sample size code of E.
Because double and multiple sampling plans are able to accept/reject lots without testing the full cumulative sample population, utilizing these two plans will, over time, reduce the total number samples that you inspect.
Step 5 – Determine the Correct Sampling Quantity (n)
Ok, so you’re almost done! At this point you’ve determined your AQL’s and that you’re going to be measuring attribute data using a single sampling plan.
The next thing you’ve gotta do is determine the correct number of sample to test. This is easily looked up using the MIL-STD-105A in the Sample Size Code Letter Table. Below is a copy of the table from the retired Military Standard.
|MIL-STD-105E : Table 1 - Sample size code letters|
|Lot Size||Special Inspection Levels||General Inspection Levels|
What you do is take your total lot Quantity, N, and find the corresponding Sample Size Code Letter. For example if your full lot quantity is 40,000 pieces and you’re using the General Inspection level II (Which most people do), then your sample size code letter would be N.
Step 6 – Determine Your Accept & Reject Numbers
Once you’ve determined your sample size code letter you can then look up your Sample Size, Accept (a or acc) Number & Reject (r or rej) Number on the appropriate ANSI or MIL-STD table.
Regardless of which table you use, you will usually start with your code letter on the left, which will indicate which Sample Size you need.
Then your Ac & Re numbers will be present at the intersection on your AQL and Sample Size Code Letter.
For example, using the Single Sampling Plan Table at a Normal Inspection Level below, you can up your Sample Size Code Letter, N, and find that your sample size is 500.
Then at an AQL of 2.5 your accept/Reject numbers are 21 & 22.
This means you would reject the entire lot if 22 or more defects are found and you would accept the lot if 21 or fewer defects were found.
Here’s a good explainer video on how to use these existing Sampling Standards to create an Acceptance Sampling Plan.
Sampling Schemes – Switching Rules & Inspection Level
So instead of just setting up 1 sampling plan, you can actually get real fancy and set up a whole sampling scheme. A Sampling Scheme is a combination of sampling plans that use Switching Rules that change the level of inspection that you perform.
These Switching Rules allow you to tighten or reduce your inspection level based on the Quality Level of previous lots of product that you’ve inspected.
Generally, most Sampling Schemes start at a Normal Inspection Level.
If subsequent inspections are all successful it is assumed that the product being inspected is being produced under a state of statistical control. This rationale allows you to switch to a reduced inspection plan.
In the reduced inspection plan you are allowed to test a smaller quantity of sample. If a non-conformance is observed during the reduced inspection plan, you can switch back to the normal inspection plan.
Similarly if many non-conformances are observed during the normal inspection plan, you can switch to a heightened inspection plan. The heightened inspection plan will have more stringent acceptance numbers, increasing lot rejections and forcing a quality improvement upstream of the inspection.
Once the overall quality level has been improved at the upstream process, the plan can switch back to the normal inspection plan. Here’s a graphic that will help you understand the switching rules associated with a sampling scheme.
Additionally, you can also implement sampling plans based on Special Inspection Levels that are characterized as sampling plans with very small sample sizes. Because this situation represents a low number of samples, it increases the level or Consumer & Producer Risk.
Special Inspection Levels typically exist when the cost associated with testing is high and the risk of an being wrong is low.
Sampling Terms & Definitions
|AQL||Acceptance Quality Limit||The quality level that is the worst tolerable (ISO 2859 standard).|
|LTPD||Lot Tolerance Percentage Defective||The worst quality level in an individual lot that should still be acceptance. This is also know as the Rejectable Quality Level (RQL) or Limiting Quality Level (LQL).|
|IQL||Indifference Quality Level||The level of quality where the consumer has a 50% chance of both accepting or rejecting the full lot.|
|AOQ||Average Outgoing Quality||The Average quality of a finished good, after all inspection, testing, rework, etc. It is also the proportion of product that is still nonconforming|
|AOQ Curve||Average Outgoing Quality Curve||The graph that shows the Average Outgoing Quality for various values of incoming quality.|
|AOQL||Average Outgoing Quality Limit||The Average Quality Limit of any specific lot as determined by the sampling plan that would be deemed acceptable and released to the customer or for futher processing.|
|ASN||Average Sample Number||The average number of items in the sample needed to make a decision on the lot quality.|
|-||Sampling Scheme||A set of sampling plans that use the switching rules between normal, heightened or reduced inspection.|
|-||Sampling Plan||A plan that dictates the sample size and acceptance criteria for inspection.|
|-||Special Instruction Level||A special sampling plan that has a very small sample size when compared to the overall lot quantity.|
|-||Normal Inspection||A set of Sampling Plans as defined by the ANSI/ASQ Standard for general level II inspection. (e.g. N = 5,000, n = 100, a = 2)|
|-||Tightened Inspection||A set of Sampling Plans that are tightened as compared to the Normal Inspection Level. This typically means a lower acceptance number, a, for the same number of samples , n. (e.g. N = 5,000, n = 100, a = 0)|
|-||Reduced Inspection||A set of Sampling Plans that are reduced as compared to the Normal Inspection Level. This typically means a reduced sample size, n, but the sample acceptance number, a. (e.g. N = 5,000, n = 35, a = 2)|
|-||Switching Rules||A set of requirements for the switching between the different inspection plans; Normal, Heightened, Reduced.|
|β||Consumer Risk||The risk that your sampling plan will lead you to accept a lot of product that does not actually meet your quality standards.|
|α||Producer Risk||The risk that a sampling plan may lead to the rejection of a lot that actually does meet all quality standards.|
Average Outgoing Quality
So there’s one particular topic within Acceptance sampling that could use it’s own paragraph and that is Average Outgoing Quality.
So for every Acceptance Scheme that you create, you will ensure that the average outgoing quality level is better than the incoming quality level. This is because a certain number of lots will be rejected and their quality levels will be improved (rework, sorting, etc) before you will accept them.
Quality experts have created an Average Outgoing Quality Curve to represent the Average Outgoing Quality as it varies based on the incoming quality level.
The X axis on this graph is the Incoming Quality Level and the Y is the Outgoing Quality Level.
As you can see the graph of Average Outgoing Quality is small but increasing, this is because lots with a very good incoming quality level (low number of non-conformances) will likely be accepted based on your sampling plan.
The graph eventually peaks at a point where your sampling plan will likely accept the greatest number of non-conformances before it begins rejecting lots.
The maximum value on this graph is called the Average Outgoing Quality Limit as this is the worst possible Average Outgoing Quality that your sampling plan will result in. This is an important concept, different sampling plans result in different Average Outgoing Quality Limits.
The graph then begins to dip because the incoming quality level is so bad that the sampling plan will be reject the lot forcing a quality improvement (rework or reinspection) in the final outgoing quality.
The AOQL can be calculated using the following equation provided by the NIST where p is the incoming quality level and pa is the probability of accepting the incoming lot.
AOQ is then calculated at different theoretical levels of p to create the curve.
Acceptance Sampling – Generally Accepted Practices
There are a number of generally accepted practices that you must understand when implementing a sampling plan to ensure that you’re executing the test properly.
The first thing you must do is ensure that the samples you are testing are random. Sample Randomization ensures that your testing is representative of the total population and is as unbiased as possible.
Bias in sample selection can represent a large source of error that can impact your final disposition of the product.
For example if you wanted to determine the average home price in America by taking a sample, your data would be very biased if you only called homeowners who lived in Manhattan or Hollywood.
Sampling Without Replacement
Another generally accepted practice for sampling plans when multiple groups of samples are tested (double or multiple sampling plans) is that you must not return the tested product back into the general population of product.
This is a concept known as sampling without replacement and it’s a fundamental assumption used in creating many Sampling Plans, so you must abide by it.
There are a few plans that allow sampling with replacement, but these are for plans with very high sample sizes which reduces the likelihood of drawing the same sample.
The Operating Characteristic Curve
As we discussed above, there is always a risk that your acceptance sampling will result in an incorrect decision to accept or reject.
To understand the acceptance probability associated with your acceptance plan, you can create an OC Curve to plot the acceptance probability versus the fraction non-conforming.
The Operating Characteristic Curve is a plot of the probability of accepting a lot against the a theoretical incoming quality level, p(% nonconforming). The Probability of Acceptance Pa is plotted on the Y Axis, with the Theoretical Incoming Lot Defect Rate on the X Axis.
Plotting the OC Curve
The 2 most common distributions used to calculate the probability of acceptance, pa, are the Binomial Distribution and the Poisson Distribution.
For the Binomial Distribution, the Probability of Acceptance can be calculated using the following equation:The Binomial distribution should used to create your OC Curve when your lot size is very large because the Binomial distribution assumes that the probability associated with all samples are equal (sampling with replacement).
The second most popular distribution utilized in creating OC Curves is the Poisson Distribution.
The following equation can be used to calculate the probability of exactly x defects or defective parts in a sample n.
Interpreting an OC Curve
Every OC curve is unique and specific to the combination of your sample size and acceptance number.
For example when you increase your acceptance number (a) from 0 to 4, you put a “shoulder on your OC curve, increasing the AQL (Acceptance Quality Level) for any given sample size, n. This is because your plan now fully accepts 4 defects during inspection.
Similarly, If you increase your sample size, n, you effectively make the downward slope of the OC curve greater. Put differently, when you sample more product, you increase the ability of the plan to distinguish the full lot quality.
Additionally, by varying the sample size and acceptance criteria together, you can create similar OC Curves.
In the example below I’ve created 1 curve at n = 50, a = 1 and a second curve at n = 25, a = 0. As you can see these OC curves are very similar with the n = 25 plan being more stringent at low incoming lot quality and less stringent at worse incoming lot qualities.
Here’s a decent video that walks you through some of the basic concepts of OC Curves.
Here are a few questions that I’ve put together that should challenge your knowledge, understanding and application of the major topics associated with Acceptance Sampling:
References, Resources & Links
- ANSI Standard Tables
- http://www.itl.nist.gov/div898/handbook/pmc/section2/pmc2.htm Test Product for Acceptability: Lot Acceptance Sampling
- Zero Acceptance Number Sampling Plans, Fifth Edition
- The Certified Quality Engineer Handbook, Third Edition
- Juran’s Quality Handbook: The Complete Guide to Performance Excellence
- Acceptance Sampling on Wikipedia – Contains a full list of all ISO & ASTM standards associated with Acceptance Sampling
- http://www.qualityinspection.org/what-is-the-aql/ What is the “AQL”, and when it is applicable?