Find all your DIY electronics in the MakerShed. 3D Printing, Kits, Arduino, Raspberry Pi, Books & more!

When we first started having our Pulse Sensor (featured in MAKE magazine Volume 29) manufactured, we ordered them in limited batches of 500 pieces, so we could get volume pricing on parts and assembly costs, and update the hardware quickly. After a second run of 500, we decided that we understood the design and engineering enough to commit to a larger production run. We were also running out of Pulse Sensors. So, we prepared for an order of 2000 Pulse Sensors. This was big time for us.

(As a reminder, the Pulse Sensor shines light into your skin, then looks for how much light reflects back. The more blood it hits, the less light reflects back. Our Arduino program then uses this data to determine your heart rate.)

For previous runs, we ordered all of the electronic components from suppliers like Mouser or Digi-Key, then bundled them up and shipped them to our production facility in southern China. The factory in China fabricates the Printed Circuit Boards, PCBs, and assembles (places and solders) the parts we send them onto the PCBs. As you can imagine, that process incurs a lot of shipping and duty costs, transit time, and hassle for us. You need a spreadsheet to track it all. We were ready for a better method, and our fabrication facility offered us a turnkey solution for a small added cost, where they would find, buy, and receive all of the parts. This was ideal for us. So we gave them our Bill Of Materials (BOM), they returned an acceptable quote, and we were on our way.

Everything was going fine. We sat back and had a beer. We congratulated ourselves on a fine job. Then had another beer. At this point, all we had to do is pay the factory, and check the top of production, the first small batch of parts. If the top of production samples were all good, all we had to do is wait for our finished goods. What could go wrong? We ran out of beer and switched to green tea. In a state of knowing Zen we waited for our production samples to come in. When they came, they looked gorgeous. All we had to do was a quick routine test, which surely would be fine, and “green light” the factory to make all 2000 Pulse Sensors. Drinking our tea in cozy confidence, we plugged the production samples into Arduino, put the unit on our fingertip, and casually looked at the screen for the Processing sketch to perfectly render our heartbeat. (Cue first spit-take now.)

The sensors did not work. In fact, all of them did not work, in exactly the same way. This was actually good news, since a consistent problem is easier to find and fix. Immediately we were in Problem Solving Mode. The first two rules of problem solving are Check Everything. We made sure we had the right code, then verified the soldering job and made sure the right parts were placed correctly. We confirmed that all the parts were correct, the circuit connections were correct, and the soldering was good. They did not work, but they looked exactly the same as our last batch. Just to make sure we weren’t going insane, we tested a sample from the previous production run, and it worked great.

Then Joel noticed that the green LED was not shining in quite the right color – more of a teal than a super-bright green. The lower intensity and difference in wavelength was sufficient enough to kill the accuracy of the Pulse Sensor, if not its functioning altogether.

Our contact at the factory seemed responsive to our problem. They confirmed that they ordered our part number from a reputable company, and sent a photo of the reel of parts they received. The photo confirmed that they were indeed the LEDs we specified for production. It had the manufacturer’s label (Kingbright USA) and correct part number on them. At this point, we were nervous. Were we insane? Did Kingbright change the color of their green LED? Was the reel part of a bad batch? Or maybe the new green color was within an accepted tolerance range of manufactured lots? This was not good!

We made a quick call to Kingbright, and asked them why their new green LED’s where not like their old green LEDs. (Cue second spit-take now, over the phone.) Kingbright asked for photos of the parts we used. After reviewing the photo, an engineer at Kingbright responded quickly, and solved the mystery at last. That reel of LEDs in the photo from our manufacturer bore a reference number that Kingbright didn’t use, and contained 500 more pieces than they supply per reel. Even the shape and style of the plastic reel was not the same as the kind Kingbright uses. Our factory’s supplier had bought counterfeit LEDs!

LED reel from China

Mystery solved. Our engineering prowess was exonerated, and our factory seemed free of any malfeasance. A presumably well-known and trusted parts supplier (according to our contract manufacturer) had sold counterfeit parts, knowingly or not. The gritty details of who punked who was beyond us. Meanwhile, we needed to ship Pulse Sensors, and ship them yesterday.

To get our working units, we could simply order 2000 LEDs in the USA and ship them to our factory. The problem was, in six days, all of China would shut down for the two week celebration of Chinese New Years Festival, and there was not enough time for the parts to even clear customs. This did not help us “ship yesterday.” We had to act fast! Luckily, we had stockpiled 1000 pieces of the correct LEDs at Joel’s place in Brooklyn. If the factory could assemble the Pulse Sensors with all the parts except for the LED, we could hand-solder the correct LEDs in Brooklyn ourselves. This was the quickest way to fix the problem, and we received the LED-less Pulse Sensors the day before Chinese New Year.

When we received the Pulse Sensors, we brewed a pot of green tea, and started soldering. One LED soldered, and 999 more to go. So much for turnkey production. It took three days for the two of us to solder 1000 completed Pulse Sensors. For your enjoyment, we made a time-lapse video of the process. Now we just have have 1000 more to go.


UPDATE: A previous version of this story identified the counterfeit parts supplier as Mouser China. This is NOT correct. We apologize to Mouser for this mistake.

We have determined that someone somewhere in the chain mischaracterized Mouser’s operations in China and falsely pointed to them. MAKE, Joel Murphy, and Kevin Hess of Mouser have talked and we’re all satisfied that Mouser was in no way to blame here. Below is a response to the original post, from Kevin Hess of Mouser:

Mouser was definitely not the source of the counterfeit components.

While Mouser does have branches in China, all shipments are from our US warehouse.  Mouser obtains every product directly from the component manufacturer (or factory-authorized sources).  We can also confirm that this part number was not shipped by Mouser to any customer in mainland China.  Therefore, Mouser is not the source of the counterfeit component.

Mouser is certified to SAE aerospace standard AS9120A for high-level traceability, product handling and anti-counterfeit control.   It is a known fact that the most common sources of counterfeit components are unauthorized brokers and unscrupulous contract manufacturers, who obtain the products in China.

Kevin Hess
Vice President Technical Marketing
Mouser Electronics, Inc

Learn about the Pulse Sensor and how to use it in MAKE Volume 29:

MAKE Volume 29

We have the technology (to quote The Six Million Dollar Man), but commercial tools for exploring, assisting, and augmenting our bodies really can approach a price tag of $6 million. Medical and assistive tech manufacturers must pay not just for R&D, but for expensive clinical trials, regulatory compliance, and liability — and doesn’t help with low pricing that these devices are typically paid for through insurance, rather than purchased directly. But many gadgets that restore people’s abilities or enable new “superpowers” are surprisingly easy to make, and for tiny fractions of the costs of off-the-shelf equivalents. MAKE Volume 29, the “DIY Superhuman” issue, explains how.

BUY OR SUBSCRIBE!

Joel Murphy

Hardware, baby, hardware.


Related

Comments

  1. Rich Brunner says:

    I just toured a local engineering and assembly shop in Maine here. They don’t do pcb boards but all the component adding to the boards they do. They have a completely automated smd soldering machine and a wave soldering machine. They hand solder anything else that can’t be done on the machines. If I ever make a product I will probably go through them.

    1. Craig C. Kipst says:

      Don’t leave us hanging, Rich. Whose site did you tour?

  2. Edward Vogel says:

    I ran into a similar problem in 1988. We sent the proper materials to a manufacturer in China and they (allegedly) sold the specially hardened steel and used a cheaper softer steel they had on hand. End result high speed solenoid valves that worked for 10 cycles instead of 1 billion.

    This after my manager crowed about how great the work ethic in China was compared to my manufacturing line.

    1. Joel Murphy says:

      Crazy story Edward. Working with overseas companies to find the ‘alleged’ fraud can turn into a trip down a rabbit hole of obfuscation and misinformation. We were lucky that our contact there is forthright and very responsive to problems when they arise.

  3. That was an interesting and edifying story. Thank you guys for being so candid about a process that probably most in your situation wouldn’t have shared.

  4. mathcampbell says:

    I’m still thinking that it’s better business practice to go local for anything possible;
    The further things are away, the more it damages the environment to ship it (negligible I know, but still), and in addition, whilst it may be *cheaper* to have it outsourced, the only reason it’s cheaper is becasue *everyone* gets things outsourced; standard of living is nowhere close to the West in most of China, but even so, it’s still increasing by the bucketload, and on top of that, add on export fees, import fees, shipping, regulation etc. and the possibility that you might end up getting shafted (which I know can happen locally to, but at least you can go down there and sort it – not so if they’re in China, at least not as easy), and it adds up that local-ism seems to be better; eventually labour-costs will even out and then we’ll be stuck in a situation where no-one in the west can manufacture anything, China can and they win again.

    Just my tuppence though, and I’m sure you had good reason to outsource (and no disparagement is meant to the Chinese who 99% of the time do a great job – it’s just it’s a dog-eat-dog world, and I don’t want eaten by China in ten years when they’re economy is still booming, they have al the service stuff AND the manufacturing. What will we be left with?)

  5. Jim says:

    Ironic that the reel from Mouser-USA says ‘Made in China’ while the reel from Mouser-China doesn’t.

  6. Ingo says:

    This happens all the time, even with domestic contract manufacturers. We have had counterfit processors (who would have thought) and power diodes on our boards that we have discovered. Who knows what else that we have not caught. The CMs substitute cheaper caps/resistors all the time, which is hard to detect,except for higher failure rates. We even know of a case where a CM sub’ed an electrolytic aluminum can style cap, where the actual cap was a smaller cheaper one “inside” the correctly labelled can ! They’ll do whatever they can get away with and that’s one reason why really critical stuff like airplane parts and medical equipment is so tightly controlled.

    1. Joel Murphy says:

      Wondering how much money they saved after putting the can inside the can! Almost sounds like someone would do that kind of fraud for the thrill of it. Our device is so simple that I never thought there would be a problem. The facility we use claims that they only purchase turnkey parts from reputable western companies because of the high incidence of counterfeit goods. Yet here we are. Thanks for telling your story.

  7. neamerjell says:

    That’s what you get for outsourcing to China! Same old story I have heard and read about many times before – counterfeit this, low quality that, something else that spontaneously combusts when current is applied to it.

    Serves you right. That’s what you get for denying your fellow citizens jobs.

  8. Laen says:

    Why doesn’t MAKE manufacture in the USA? Especially for electronics manufacturing, there are several excellent US choices, and it’s the right thing to do.

  9. Kid I says:

    I think Make and all other DIY / Open Source Hardware companies should start a movement of doing 100% of the process in the USA.

    1. Joel Murphy says:

      That’s a very interesting challenge, Kid I. Though there are some things that can’t be domestically sourced for competitive pricing (all of the components we use are made in Malaysia or Japan or Taiwan for example). Though I could foresee a 100% domestic fabrication and assembly mandate giving rise to more fraud, as it is difficult to prove one way or the other where the work gets done until a problem like this arises. And the competitive nature of this business encourages folks to keep their business practices close to their chests. This is a discussion worth having.

  10. ka1axy says:

    Sad, but from my experience with manufacturing in China, not unexpected at all. What surprised me was that you had used the Chinese branch of Mouser (didn’t even know they had one) and you would expect their product to be legit. Sounds like someone in the Mouser China procurement department has some ‘splainin’ to do.

    Yeah, manufacturing in China has many pitfalls to catch the trusting and naive who think that the cost is lower. The *initial* cost is lower, but, like programming work outsourced to India, the rework costs aren’t counted in that equation…

    1. Joel Murphy says:

      I’ve had a 6 year relationship with the company that I work with, and the only problems that I have had with them prior to this was a result of a design flaw on my end. When the work is done right, it arrives on time and for the price that can’t be beat. After this experience, we will be re-evaluating our production process and updating our bids from domestic manufacturers.

  11. unclemymy says:

    Three years ago, I pulled my company’s molds from China, because our critical opaque plastic assemblies were being made out of cheap reground plastic. They even sent fake certification sheets when asked about it. We had to replace a lot of them that were practically disintegrating. After all was said and done, our parts were actually a little cheaper having them made right in Wisconsin, and from the properly certified American plastic. Please take note, future engineers, getting things made “cheaper” in China may now just be a myth that your dad’s generation is perpetuating.

    1. I put together proposals for large scale manufacturing equipment, and I’ve had more than a few customers either a) openly ask me to commit to *not* manufacturing in China (we hardly do) or b) submit a completely separate bid for any equipment manufactured in China. We’ve also had equipment that we’ve had manufactured in China suddenly “clone” itself instead of receiving a second order to supply that equipment. (In my line of work, fabricators can quickly get themselves in trouble if they try to simply work off of/modify existing prints. They don’t have the engineering capability to do the actual design nor learn what we can from previous projects.) As a result, in those cases where we do use Chinese sources, we’re very careful to limit the supply and piece it out to different suppliers so no single one can figure out what we’re doing.

      The fact is the basic underpinnings for reliable business (trust, easy access to an impartial judiciary when needed) are simply not there. I know I can order equipment almost site unseen in the US; ordering from China requires me to throw in extra money for inspections/trips/project management hours to handle all the inevitable problems. In the US, I can plan to assign the vast majority of fabrication to a single shop with no concerns that shop is going to screw me over in the future or use my own detail drawings to try to undercut me on repeat business. I can’t do that in China.

      I’m sure others have had success in China; people wouldn’t be sending stuff there if they didn’t. But before committing to manufacture there, remember that something like half of all outsourcing efforts do *not* provide the expect returns/profits. They end up costing you more money than if you’d just done it elsewhere for a few dollars more.

    2. crazybutable says:

      What company did you use in Wisconsin?

  12. lynnewu says:

    Would love to see Make and makers use US facilities. Consider trying http://www.ueinc.com , a Wisconsin-based contract electronics manufacturer. ObDisclaimer: they’re a client

  13. rahereJel says:

    It’s not as if there hasn’t been enough warning to all concerned.

  14. Hank says:

    Well it seems the challenge has been laid. Perhaps MAKE Magazine should look to its own community of Makers before it looks to off-shore.

  15. I work for a contract electronics manufacture on the Wisconsin – Illinois boarder http://www.logicalproducts.com we also donate inactive inventory, and scrap electronic assemblies to the Milwaukee Makers space.

  16. China has, in effect, pulled a “Wal-Mart” on US manufacturing. They’re going to run a loss-leading economy until they’re the only source of products and services and then they’re in the position to do whatever they please.

    As an above poster said China being a cheaper source of parts and labor is going to become a myth in this decade.

  17. Meat Lover says:

    Did Mouser confirm that they shipped these parts to your contract manufacturer? Did your CM provide proof of purchase like copies of the PO, invoice and packing slip?

    You’re throwing Mouser under the bus with out giving them any chance to respond. Incomplete and shoddy reporting.

    1. Joel Murphy says:

      Hi Meat Lover,
      First of all, I’m not a reporter, I’m trying to make Pulse Sensors. Also, we are not putting the blame on anyone because we don’t know where the fraud originated and likely never will. Once we discovered that we had *real* counterfeit parts, our work was cut out for us in solving the problem first for our customers, and then for our future business. We will likely not be using a turnkey solution with our CM in the future. It is beyond our scope and budget to chase the perpetrator of this fraud down the rabbit hole. Kingbright and Mouser were both made aware of the fraud and I am not privy to their actions in response.

  18. Sue W says:

    Good for you guys! Very thorough work. I really appreciate the hard work you did on this. I’ll be ordering a couple of these today!

  19. Stephen says:

    I work for a contract manufacturer in Northern Ohio, (TL Industries, Northwood Oh) and counterfeit parts are always a concern, even in the US, but resolution of the these type of problems is easier here than halfway around the world. I guess my point is look into local CM it might be worth it:)

  20. mathcampbell says:

    It’s not a slur on the chinese manufacturers; after all, a lot of these components end up being made in china to start with (Or Taiwan, Singapore, etc.) so you’re not gaining much in terms of shipping etc.
    But what you are gaining, as someone else said, is oversight. You use a US company (or in my case, staying local, a Scottish company) and what you’re getting is not only the same standard of work as you *can* get in China, but a legal system and culture that backs that up, and likely in the same jurisdiction as you are.
    If I ordered parts from a company here in Scotland, and they turn out fake, or bad, or they crew me around etc. I can go to my local Trading Standards (UK government agency dealing with, well, trading standards) and seek redress, and if that all fails, a small filing fee and a trip to my local Sherriff’s Court, and I can sue them.

    I get the same problems from a company in China, and by the time you add up the costs to get there, deal with their anarchic bribery, corruption and communist legal system, chase the company down whatever rabbit hole they’ll have scurried at the first sign of trouble, and you’re basically looking at throwing that money down the drain. Cheaper just to make a new order, locally. So why not just do that from the get go.

    I think it would be a great idea, if in future, MAKE puts out an open-call for new contracts, with an emphasis on local manufacturers. Don’t know how that would gel with your business model, but it would be an inspiring sign for the no-doubt struggling US companies that would love a slice of your manufacturing business. Of course, you could come to Scotland as well ;)

  21. Similar thing happened to Sparkfun a while back: http://www.sparkfun.com/news/395
    What was amazing was the level of hiding that went on. It would be interesting to see how this turns out if you also specify a test scheme to be performed by the CM before they ship to you. How the heck do other companies get their stuff built to proper spec in these situations?

  22. Brian says:

    Every time I get something injection molded in China I have to have the material tested back here in the US. About 50% of the time the molders flat out lie about the material they are using. I mean, as long as it melts, and fills the mold, who cares what it’s made out of, right? And it’s not just one company – everyone seems to be doing it.

  23. David Gonzalez M.D. says:

    Is it possible that some of the defective sensors shipped to customers? I just received my sensor this week and can not get a wave-form with an amplitude greater than 1 or 2 (and most of the time it’s 0). It is sensing light just fine, but doesn’t seem to sense any variability when placed on a finger or ear lobe. I can’t find any way to report a potentially defective product on MakerShed.

    1. Joel Murphy says:

      Hello David,
      Please go to http://www.pulsesesnsor.com and follow the links to the getting started guide and also the forum. There you will be able to receive help with your problem.

  24. [...] PCB’s are made overseas, for the most part, when nothing goes wrong.  But the rest of this kit is assembled by our hands, into a nicely compacted form.  I’ve [...]

  25. [...] PCBs are made overseas, for the most part, when nothing goes wrong. But the rest of this kit is assembled by our hands, into a nicely compacted form. I’ve [...]

  26. [...] PCBs are made overseas, for the most part, when nothing goes wrong. But the rest of this kit is assembled by our hands, into a nicely compacted form. I’ve always [...]

  27. [...] PCBs are made overseas, for the most part, when nothing goes wrong. But the rest of this kit is assembled by our hands, into a nicely compacted form. I’ve always [...]

  28. […] For the board on the left, [Thomas] managed to release some smoke from the components during the first test. As a learning experience he decided to recreate an open hardware pulse sensor. It is made up of an op-amp reading from a photo sensor, paired with an LED to light up the tip of your finger. He laid out his own board in Eagle and put in a $2 order with OSH Park. The image above shows [Thomas] reflowing the photo sensor using his clothes iron. After the rest of the assembly was complete he fired it up — producing the oft mentioned magic blue smoke. A second run on the board ended in folly also, perhaps because he reused components from the initial “smoked” board. One thing that this failure turned up was an interesting article on counterfeit parts. […]

  29. […] For the board on the left, [Thomas] managed to release some smoke from the components during the first test. As a learning experience he decided to recreate an open hardware pulse sensor. It is made up of an op-amp reading from a photo sensor, paired with an LED to light up the tip of your finger. He laid out his own board in Eagle and put in a $2 order with OSH Park. The image above shows [Thomas] reflowing the photo sensor using his clothes iron. After the rest of the assembly was complete he fired it up — producing the oft mentioned magic blue smoke. A second run on the board ended in folly also, perhaps because he reused components from the initial “smoked” board. One thing that this failure turned up was an interesting article on counterfeit parts. […]

  30. […] For the board on the left, [Thomas] managed to release some smoke from the components during the first test. As a learning experience he decided to recreate an open hardware pulse sensor. It is made up of an op-amp reading from a photo sensor, paired with an LED to light up the tip of your finger. He laid out his own board in Eagle and put in a $2 order with OSH Park. The image above shows [Thomas] reflowing the photo sensor using his clothes iron. After the rest of the assembly was complete he fired it up — producing the oft mentioned magic blue smoke. A second run on the board ended in folly also, perhaps because he reused components from the initial “smoked” board. One thing that this failure turned up was an interesting article on counterfeit parts. […]

  31. […] For the board on the left, [Thomas] managed to release some smoke from the components during the first test. As a learning experience he decided to recreate an open hardware pulse sensor. It is made up of an op-amp reading from a photo sensor, paired with an LED to light up the tip of your finger. He laid out his own board in Eagle and put in a $2 order with OSH Park. The image above shows [Thomas] reflowing the photo sensor using his clothes iron. After the rest of the assembly was complete he fired it up — producing the oft mentioned magic blue smoke. A second run on the board ended in folly also, perhaps because he reused components from the initial “smoked” board. One thing that this failure turned up was an interesting article on counterfeit parts. […]