There is a lot of confusion around Test and Measurement… especially where Qualification and Manufacturing Pedigree is concerned. I will break it down here into 3 categories and explain why R&D testing is cheap and why costs go up when you step into production.
R&D, Qualification, and Production Testing
There are few controls around this sort of test equipment and cheap hobby tools usually do just fine. A $200 all-in-one USB based device that offers Analog Input, Digital I/O, timers, and maybe some Analog Output can provide all the stimulus and data collection you need to ring out bugs and make good progress toward a proof of concept.
Generally speaking these tools are not calibrated, they are not modular or expandable (for re-use on the line), they are not rugged or reliable, they depend on a PC being connected, they are not capable of holding Precision (hardware timing), they are limited in sampling rate, and generally speaking they are not reliable or fit for making statements about Precision and Accuracy.
Hobby grade tools are more than sufficient for R&D work… because the risks involved are relatively low and any engineering facility will have some quality lab equipment (possibly shared) to validate test results. A good Meter (Fluke) and a high speed scope (ideally isolated) are the real tools of the trade… but they are expensive and often shared… so folks set up with USB boxes at their desk. The Take Home Concept is that the cost of mistakes starts going up by Orders of Magnitude as you come out of R&D and into Production…
When we talk about Qualification Testing… we are talking about stepping out of R&D. We are talking about freezing a design and developing metrics to make statements about Accuracy and Precision such that we can confidently state that if we make 1,000 of these… we wont end up doing rework operations multiplied by 1,000.
A qualification tester should be Calibrated or of a known quality standard. It should be reconfigurable… as its life cycle is usually short. It should be hardware timed (precise). It should be high resolution (accurate) and isolated…. such that it is capable of picking up low level noise and does not appreciably affect the circuit under test.
A Qualification tester is the tool used to develop confidence in a design before committing to volume production. Generally it is best if a third party develops a Qual tester so as to uncover blind spots that may be intentionally or unintentionally missed by the R&D Engineer. More specifically… a Qualification tester is built strictly to a set of requirements. Think of Requirements as a series of transfer functions. A black box with input, behavior, and output… such that a series of behaviors and functions can be listed and tested. Requirements are negotiated and can be lax or rigorous.
There are infinite ways to define requirements… and infinitely more ways to test to those requirements… but the job of a Qual tester is to state unequivocally that a set of production intent hardware meet or exceed the stated requirements (yes… I understand that is a bit circular). I can go on ad nauseum but I will spare you.
Qualification is where R&D lets go of a design… where any problems which were previously masked are uncovered… and where we build absolute confidence in our readiness for Production.
Sometimes folks mix Qualification into Production Testing… but really… a Production tester should be the simplest, cheapest, way to display a RED LIGHT or a GREEN LIGHT such that a part can move on to next assembly or be sent back for rework.
Production testers look for things that change… like impurities in conformal coatings which develop as leakage currents. Parts which come in out of spec due to counterfeit production or poor storage. PCB’s which have missing, extra, or inconsistent electrical connections. A thousand solder points. … Its the job of the Production Department to drive costs down as far as possible… and if production is doing their job… at some point unacceptable quality will be reached. Its the production tester that sets the boundaries for this exercise.
Yield … there is no such thing as 100% on an efficient production line. Some number of widgets will always go back for rework. Its the job of the production tester to provide a “troubleshooting guide” to speed up this process. At a minimum a production tester should be able to indicate the test or tests which failed. The rework tech, with time, can interpret these results, identify problems in process, and make corrections to improve yield while keeping costs down.
UI (User Interface or User Experience) is very important in a production tester. It has to be absolutely dummy proof. It has to be BOMB PROOF… such that a guy can be picked up at a temp agency, put to work, and no impact on quality can be measured. There must be no correlation between the test operator and the final product.
A production tester must scale well. If production is to improve it either must run faster (in relation to the rest of the processes such that it is not a big impact on the critical path) or it must be able to be cheaply and quickly duplicated.
Again I could go on ad nauseum.
Every engineer who is successful already has their preferred set of R&D tools. Usually this is something they became familiar with in school, at home, or at a previous job. The tools work and everything is great.
Where things get hairy and confusing is when its time to Qualify a design and move into Production. This is the point where most costly mistakes are made.
R&D/Proof of Concept, Functional Prototype/Production Intent Hardware, Short Run Production/Full Production
The cost of mistakes goes up by an order of magnitude as we break out of R&D. A $10 mistake becomes a $100 mistake… then the same sort of mistake becomes a $1000 mistake… MULTIPLIED BY UNITS IN PRODUCTION!
I cant stress this enough… and every business minded lean-machine out there thinks they can jump the gap and hover directly from R&D into successful Production. I am not a “hater” – its possible – but it is also highly likely that any attempt to avoid due diligence will just result in the money being spent anyway… then spent again. Infinite examples available…
OK – So what do I buy to make this magic happen and how much does it cost?
Anyhow – my point was originally to introduce you to some of the hardware tools that I use. Lets skip R&D tools… since that is really a personal preference… and review proper tooling for Quality.
This breaks down into two major categories: Big Budget and Small Budget
Big Budget is a laboratory setting where progress is measured in years and quality is non-negotiable. Small budget is probably where you are… where you just need to get something going for as little investment as possible.
Big Budget looks like NI PXI… and $50k is the sort of spend you are looking at.
Small Budget looks like NI cRio… and $5k is actually a reasonable spend for a system (if you buy used and are thrifty)
Before I call out hardware… lets take a moment to clarify an assumption.
<It is my belief that a system should be configured as follows: Windows UI linked to a Real Time or FPGA control loop.
User Interface: This can run on a standard windows box and generally everything is fine. This part of the system really does no work other than taking commands and displaying results. This machine can be local or remote…. headless and integrated or user facing. This machine protects the Real Time hardware from non-deterministic behavior. Its a buffer… and an interface.
Test Hardware: This part of the system MUST be deterministic to be reliable. At the slow end… maybe a 1ms RTOS (Real Time Operating System) control loop. At the fast end… maybe a 1uS FPGA control loop. This means that all outputs and all inputs are guaranteed to be precise and perfectly repeatable over time. It means that sufficient buffering is in place and that even if there is some sort of lag or delay in the UI… that the actual UUT (Unit Under Test) is under tight control and results can be compared apples to apples over hundreds or thousands of cycles.
So tying back in… PXI is a high end chassis that allows this sort of thing. cRio is a smaller/cheaper/more rugged version of this – and that is what we will focus on for the final chapter of this nerd-rant.
Without turning this into some sort of National Instruments commercial… It can be described as a back-plane or chassis with a built in controller. There are 4 or 8 slots where cards can be plugged in (AI, AO, DI, DO, II, you name it… these are measurement heads… with 1-32 ports….) and the chassis is expandable as needed to host more than 8 cards.
At the head of the chassis is either an embedded FPGA, an RTOS controller, or both. This is the key to success… as the system can run headless with no external connection to a PC or network – therefor little to no chance of latency related issues… which are the sort of bugs that pop up after hours, days, months, or years to bite you in the ass.
The RTOS or FPGA system is programmed with a behavior loop. Said loop runs hell or high water. The system can take ques and commands… and can provide buffered data.
- A chassis is purchased ($500)
- Appropriate cards are purchased ($50 – $500 each)
- A windows box or laptop is purchased ($400)
- Licenses to run the UI, RTOS, and FPGA are obtained (Negotiable – $2k to $5k… but that is something that I buy and use over and over for different jobs.)
Buying these things new is a killer… but … the used market is looking GREAT. Here are some examples from Ebay.
(Best value is in buying 2 or 3 package deals then breaking the systems up as needed)
cRio Chassis with a Real Time controller and 7 I/O cards – $2,100.00 USD
8 slots and a controller – $500.00 USD
8 slots and a controller – $600.00 USD
Controller – $100 USD
32 Channel, +/-10V, 16bit, 250kS/s Analog Input – $230.00 USD
RTOS, FPGA, 8 slots – $450.00 USD
So… the Angel Investor hurtle to launch this sort of activity is about $10k….. $5k to get into the hardware and $5k to get a maxed out license. A license works forever but can be upgraded as new features are released. Legacy hardware is supported on the timetable of decades.
The cost of initial setup and support on a system like this is reverse expo. Once a system is in place a Junior Engineer with minimal oversight should be able to expand, duplicate, or otherwise modify at a burn of about $50/hr. Initial setup one time burns at $100/hr. There are A LOT of experts out there who can pick up on a system and modify it in short order… that’s why NI is so popular… because it is *VERY EASY TO FOLLOW IN THE FOOTSTEPS* of the previous engineer.
Until next time… Eat your broccoli and brush your teeth. I am off to get paid to test stuff.