3D Printing & Imaging
How to Evaluate the 2015 Make: 3DP Test Probes

Want to see how your 3D printer stacks up to the competition? Download our test-print files and follow along.

For more on 3D printing, check out Make: Volume 42.  Don't have this issue? Get it in the Maker Shed.
More 3D printing in Make: Vol. 42.

Printing:

For evaluating overall system (machine, software, and materials) performance, print the following at default or “medium” settings. For most machines, this translates to a layer height of 0.2mm and default speeds and temperatures.

If evaluating multiple machines, it is imperative to use the same material, preferably from the same spool (unspooled lengths of filament can be used for individual prints to facilitate this). The Make: Magazine 3DP testing team used Ultimachine orange PLA for all PLA machines.

DownloadDOWNLOAD AND PARTICIPATE!

The 2015 3DP Test Geometries, created by Andreas Bastain, are available from Make:’YouMagine and Thingiverse accounts. Print them yourself and report your settings and scores!

How Much Filament?

  1. dimensional_accuracy_test.stl requires approximately 0.99m of filament
  2. bridging_test.stl requires approximately 1.54m of filament
  3. overhang_test.stl requires approximately 2.00m of filamentDSC04368
  4. negative_space_tolerance_test.stl requires approximately 2.54m of filament
  5. retraction_performance_test.stl requires approximately 0.52m of filament
  6. XY_resonance_test.stl requires approximately 0.95m of filament (if your slicer cannot do 0.5mm walls, use XY_resonance_1.0mm_walls.stl)
  7. Z_resonance_test.stl requires approximately 2.64m of filament 
  8. The Maker Faire Robot Action Figure (with supports) requires approximately 3.58m of filament

Here’s how to evaluate and score your prints.

1. Dimensional Accuracy

Using digital calipers, measure the second from the bottom tier of the print (with a target diameter of 20mm) across the X and Y directions, following the guide on the bottom of the print. Differences between X and Y measurements indicate the magnitude of backlash present in the system.

DA_probe

  • Assign the print a “1” if average deviations in X or Y are greater than 0.4mm.
  • Assign the print a “2” if average deviations in X or Y are between 0.4 and 0.3mm.
  • Assign the print a “3” if average deviations in X or Y are between 0.2 and 0.3mm.
  • Assign the print a “4” if average deviations in X or Y are between 0.1 and 0.2mm.
  • Assign the print a “5” if average deviations in X or Y are between 0 and 0.1mm.

2. Bridging Performance

Inspect the five bridges for dropped perimeters and infill.

bridgfin

  • Assign a “1” if any bridge has dropped infill.
  • Assign a “2”if only the longest two bridges have dropped infill.
  • Assign a “3” if none of the bridges have dropped infill, but all have dropped perimeters.
  • Assign a “4” if the shortest two bridges compiled without any dropped perimeters.
  • Assign a “5” if all bridges compiled without any dropped perimeters (drooping of less that 2mm is acceptable).

3. Overhang Performance

Inspect the 30, 45, 60, and 70 degree overhangs, looking for drooping perimeters, wobbling extrusions, and infill hemorrhaging.

O_probe

  • Assign a “1” if the printer did not compile any of the individual overhangs. Assign a “2” if the printer compiled the geometry but dropped loops and infill on the 60 and 70 degree overhangs.
  • Assign a “3” if the printer only dropped loops on the 70 degree overhang.
  • Assign a “4” if the printer didn’t drop any perimeters and the surface of the 60 and 70 degree is only slightly different from the surface of the 30 and 45 degree overhangs.
  • Assign a “5” if there is little distinguishable difference in surface structure between the four overhang angles.

4. Negative Space Tolerances

Remove the captive pins by hand without using tools.

NTT_prove

  • Assign the print a “0” if no pins can be removed.
  • Assign the print a “1” if the 0.6mm radial tolerance pin can be removed.
  • Assign the print a “2” if the 0.6mm and 0.5mm radial tolerance pins can be removed.
  • Assign the print a “3” if the 0.6, 0.5, and 0.4mm radial tolerance pins can be removed.
  • Assign the print a “4” if the 0.6, 0.5, 0.4 and 0.3mm pins can be removed.
  • Assign the print a “5” if all pins can be removed.

5. Fine Positive Space Features Performance

Evaluate based on the quality of deposition composing the spires:

PFF_probe

  • Assign the print a “1” if the spires did not compile due to extruder jam/lack of material flow.
  • Assign the print a “2” if the spires compiled but are densely connected by strands of material.
  • Assign the print a “3” if the spires compiled and there are some connecting strands but the main deviation from target geometry is due to volume flow issues (under- or over-extrusion).
  • Assign the print a “4” if the spires compiled, there are no connecting strands, but there are volume flow issues.
  • Assign the print a “5” if the spires compiled and there are no connecting strands and no stepping or ridging due to volume flow issues.

6. Mechanical Resonance in XY

This test evaluates both resonance in the XY gantry, deposition control during linear extrusions, and deposition control at layer changes. As resonance is difficult to measure quantitatively, this is a binary test.

XY_probe

  • If there is any rippling at the corners or at the midpoint of the print wall with the inset, assign the print a “fail” value of “0”.
  • If there is no rippling, assign the print a “pass” value of “2”.

While expressly designed for evaluating resonance, the print can also be used to evaluate deposition control in a more qualitative manner (inconsistent extrusion width, standing wave effects).

7. Mechanical Resonance in Z

This test exposes resonance in the Z axis if present and is subject to a binary evaluation.

Z_probe

When illuminated from above, if there is a noticeable loss of layer registration in the top half of the print, manifesting as horizontal ridging, assign the print a “fail” value of “0”.

If there is no loss of layer registration with increasing Z height, assign the print a “pass” value of “2”. In addition to evaluating Z resonance and layer registration, this print can expose misalignment in the Z-axis if there are consistent ridges of the same pitch as the leadscrew.

To get the total mechanical resonance score, add the XY and Z scores.

Documentation and Sharing:

Please use the “I Made One” button on your file sharing site of choice to share your results! Make sure to include the following information with your photo(s) of your completed test prints:

  • Machine make and model
  • Slicer and slicing settings (layer height, number of shells, print temperature, extrusion multipliers, speeds)
  • Print time – Start with a room temperature extruder and platform.  Begin timing when you start the print, and include the preheat sequence. Keep timing through any post-print sequence, like the extruder or platform returning to a homing position.
  • Filament source

64 thoughts on “How to Evaluate the 2015 Make: 3DP Test Probes

  1. Andreas, I love this. Thanks for releasing it. The perfect complement would be a post on what to do to your printer to improve your scores on each test. Thanks!

  2. Once again very basic tests, what about a comparison of support removal. These tests are greatgreat if you print iPhone cases or vases that don’t require support.

      1. I posted this response over on the “What is Print Quality?” post (https://makezine.com/magazine/what-is-print-quality/), but I’ll repost here, as this post is heavily commented.
        —————-
        We felt that the current crop of machines should be able to handle bridging and minor overhangs without support. Many could. As Andreas mentions, we tested to failure so the tests are intentionally overkill.

        We ran one long overnight sculptural print with support material on all the printers (just to get a rough idea) – but did not quantify those prints. I can tell you that most software currently does a terrible job of creating easily removable supports and surface pockmarks after removal were rampant. At this juncture, I would recommend creating custom supports in MeshMixer over any automatically generated support.

      2. Support is tricky and proper execution lies far more in the domain of the slicing engine than the hardware. Given that some of the machine/slicer combinations could barely handle the bridging and overhang tests and that so few slicers have executed single-material support well, we did not look closely into the matter. It is worth a chapter in longer term investigations of slicing engines though.

        I’ve used the overhang probe for testing basic support algorithms, but I’ll go ahead and design a proper probe that covers, concave (most difficult), and convex (easier than planar) overhangs in addition to planar ones.

        Next year we’ll do a more in depth benchmarking of slicing engines and test support algorithms.

        1. “…so few slicers have executed single-material support well, we did not look closely into the matter”. So in other words, because only two printers on the market, which (with their integrated software) are actually able to do something (generate easily-removable support) that is absolutely essential in printing a vast category of objects (those that require support), we’ll just ignore that parameter. That’s a bit like reviewing every aspect of a car except whether or not it actually goes forward.

          1. Implementing support is not impossible in Repetier Host/Slic3r or Cura. In most its just a checkbox, like Cura. But it is so wasteful I try to be smarter about design and orientation. If I need It, I like mesh mixer the best. I’ve heard good things about Simplify3D but I’m not paying $140 for a slicer when I print outstanding with Cura. It’s an open-source principle for me personally. It also isn’t too late for anyvbody to do these tests if they own the printers, or the manufacturers. Actually I fully expect each to do so and post profiles and settings for everyone to benefit from.

          2. Julia, I wouldn’t go quite that far. Different users require different things from their machines. A sculptor might not care about dimensional accuracy but will place a high value on the the lack of resonance effects that disrupt surface finish. An engineer might prioritize fine feature resolution and dimensional accuracy above bridging performance. As an engineer designing mechanical parts for fabrication on FDM systems, I solve the problem by designing parts to never require support or I model it is manually. With the shoot out, we just provide the data and let people assign their own weights to how important it is to them.

            It’s a shame that more slicers don’t have better support algorithms– when single-material support is more widely implemented, we’ll benchmark it.

          3. It is indeed a shame that more slicers can’t generate easily removable support, but it is the reality. Those printers that don’t work with, or don’t have, a slicer that can do so should be penalized in ratings versus those that do, rather than pretending that the vast performance gulf separating open-source from integrated systems in this respect doesn’t exist. I am also an engineer designing mechanical parts, and any engineer who says they can design all their parts so that support is never needed is either being disingenuous, designs only trivial parts, compromises designs due to no-support limitations, or spends an inordinate amount of time creating workarounds for this printer limitation.

            I can only shake my head when I see so many people always agonizing over designing things so as to not require support, or whining that something “is not printable” because it needs support. People who use slicer/printer/filament systems that generate easily-removable support don’t give it a second thought. It’s time to stop apologizing for crappy slicers just because they’re open-source.

            And really, how many slicers are we talking about here? Five? Two of them generate support that for the most part can be popped off with your fingers, while the other three generate support that in many cases cannot be removed by any means, without damaging the part. Should a printer that can only be used with the latter slicer really be rated as the equal of one that can use the former slicer? Is five slicers really so many that a printer’s slicer compatibility can’t be taken into account when rating the printer, especially when it so dramatically affects what you can and can’t print?

  3. Not sure how the ultimaker 2 beat the zortrax m200? Also not sure why makerbot is listed in any category other than terrible. Guess it boils down to which company pays the most money or the crew that review the printers don’t know what they are doing. Andreas Bastian however knows his stuff can’t speak for the rest of the nobodys

    1. Ouch! No money exchanges hands. Printer manufacturers send their machines and we review them. On models tested, when machines were run at the default settings, the Ultimaker 2 scored higher than the Zortrax.

      From your comments, it sounds like you’ve read the issue. If so, then you know that the Zortrax scored in the top 5 machines tested and received a very favorable review.

      If you have a Zortrax and have very dialed-in settings or tips you’d like to share – we’d love to hear them!

      1. I do own a Zortrax M200 and the ultimaker 2 has no where near the reliability, easy support releasing or quality for every print and requires no adjustment of retraction, flow rate and feed rate. I’m assuming that you did testing in the past before the 0.0.9 update from Zortrax however with the new software patch the print quality is night and day view here http://forum.zortrax.com/index.php?/topic/596-comparison-of-3d-printers/page-3
        You can also see here the comparison on previous pages how the zortrax has much better surface quality than the ultimaker 2.

        I understand that they make you use their filament which is ABS but their filament is cheap and consistent and more reliable than PLA.You also used a filament which was not Z-ABS which voids the warranty and goes against testing the Zortrax on fair terms. The extruder on Zortrax runs hot similar to Afinia at about 268c most ABS filament likes to run with 210 to 240 tops.

        I’m just saying that when you create tests you should consult with experts on the forums in regards to testing the printers as you may not be utilizing the machine appropriately, in this case it was the filament and I can understand that you wrote the article most likely before the software update but it does increase surface detail tremendously

        1. If you read the summary section of the PDF, you will see some more details about how we tested regarding filament exceptions.

          For machines that did not claim to be able to handle PLA or used chipped, proprietary or special high temp filament – or were unable to print in Ultimachine PLA without jamming – we used the OEM filament. It is noted in the comparison chart and reviews summary. We also note the good quality and inexpensive price of the Zortrax filament in the review.

          Zortrax does not claim PLA amongst its materials and the software does not allow for temperature control, so Ultimachine ABS to run the test prints.

          Most of our testers have been printing since the very early desktop days. Several have been testing printers for three consecutive Shootout events. During our Shootouts, we are doing out-of-the-box testing, with out-of-the-box-settings – but we always contact support and use the forums if we run into trouble.

          I’m glad you like the Zortrax, we liked it too! Software is always improving, it’s a moving target. Improvements in digital fabrication tools, as long as your hardware is fairly solid, are mostly software. That’s good news for us all!

          I didn’t personally review or test the Zortrax, so I’m not sure which software version was used. I’ll ask the reviewer (Nick Parks) to chime in here.

          1. Hi Anthony, before using Ultimachine ABS, I ran each print with Zortrax Z-Ultrat Primium filament. With the exception of the prints being stronger and support being easier to remove (when applicable), the quality was identical. We did not use firmware version 0.0.9, since the newest firmware available at the time of the shootout was 0.0.8. As Zortrax is a young company developing an incredible product, I imagine that their prints will continue to become significantly better with every firmware update. I’d love to see your prints with firmware version 0.0.9!

            Thanks,
            -Nick

          2. Ah – yes! I was just making edits to my response, you beat me to it. You are correct, it was Ultimachine ABS, which printed very well, according to our reviewer who has also used the Z-Filaments. I’d love to see the prints run in Zortrax’s Z-Filaments, can you run them and post the pictures? Materials are a big part of how a printer preforms, I’m sure we’d all love to see the results!

          3. Anna you are very kind but you contradict yourself. Using a third party filament violates the warranty which you stated you did not violate the warranty. Additionally you stated care was taken when selecting print temps for filaments however you used ultimachine abs which prints at 240c which is a much lower temp than what the extruder runs on the zortrax remember 268c is its temp for Z-ABS with the Zortrax which is 28c off that’s quite a bit, when you exceed temp you diminish quality. PLA is a cop out let’s be honest the only difficulty with PLA is retraction but that can be dialed in easily and PLA does not require a heated platform or enclosure and prints on virtually anything. Ultimaker 2 does terrible with abs read their forums. If the Zortrax can print so easily with abs with all the variables required can you imagine PLA? At the end of the day you guys labeled this article as ultimate printing guide but did not take into account simple variables. Doing this for 3 years does not make anyone an expert. I’ve been around 3d printing since the makerbot cupcake and the original reprap darwin but that doesn’t make me an expert just means I know all the little quirks of 3d printing. Everyone who purchases this guide will now believe that the zortrax is a happy medium when it is at the top. I’ll print those pieces by the end of the week and attach them here.

        2. Actually, with models sliced by Z-Suite 0.0.8 or 0.0.9, the extrusion temperature of the Zortrax M200 is 280C (0.0.7 used 270C). Generic ABS will definitely not perform optimally at this temperature.

      2. “Favorable review” – Happy Medium is an insult for Zortrax especially placing it together with ditto. When just another reprap got Rookie of the year “Bee” sorry what is special there or just because it is Open Source. Ultimaker 2 as number one, maybe if is about just printing dumb figurines but also there Zortrax can win without any tuning. All tests prepared for RepRap’ish printers now is 2014 not 2008 Support generation and ability to survive 48h print with failsafe extruder is important not bridging and tinkering, makerbot 5th anywhere in the top is a next joke. (I do own – Zortrax, Ultimaker 1/2 and Makerbot 5th gen) you guys can foolish peoples which never used this 3D printers but not professional users.

        1. Hi Mark, as I discussed with Anthony, the printers were ranked via a blind test based based on their print quality alone. In the review I spoke very highly of the Zortrax’s abilities, it is truly an amazing printer. As new software is released, I’m sure this machine will only print better and better. 6th of 26 isn’t anywhere close to the a “Happy Medium”, this machine was a top performer, and I think those who read the review’s of the top printers will realize how good this machine is.

          Thanks,
          -Nick

    2. Hi Anthony, I wrote the review on the Zortrax M200. Let me start by saying after printing through at least 20lbs of filament, I am throughly impressed with the machine. I expressed the quality of this machine in the review, it prints extremely reliably, and accurately. Support comes off well and prints are very durable. That being said, it can only print in ABS and as beautiful as the prints were, we did see significant resonance in the x y and z axes and found that the printer requires support material when bringing most gaps. I ran the test prints in both Ultimachine and Zortrax filament and the results were the same.

      After printing each print, we placed a coded indicator on the bottom and put them into piles of prints with those from other machine sorted by print model. We then conducted a blind study based purely on print quality and sorted the prints from one to five without knowing which machine they came from.

      The Zortrax rated very well, but was held back by it’s bridging ability and and resonance. I know that by tuning the settings in this machine, the prints would have come out better, but to keep things equal we printed every print on every printer with the default settings. We then tallied the points to rank the printers.

      If you read the Zortrax M200 review, you’ll see that we thought it was a wonderful machine. Like everything I’m sure that our testing methods could be improved but please believe me when I say, we worked very hard to avoid any bias and rate machines solely on their print quality. If you have suggestions to improve the next shootout, please let Anna or I know and we’ll definitely consider them.

      Thanks,
      -Nick

      1. Nick,
        I’m glad you love the printer but take in mind a couple of things. ABS is a much harder to print well filament vs PLA that sticks to anything and virtually has no warp and does not need a heated enclosure. ABS is like painting with oil based (no pun intended) professional paint and PLA is the cheap water color paint that is simple and easy to use but at the end of the day it’s a paint for kids. Most machines now a days print with PLA but that’s a cop out and any print that degrades in your car when it’s hot outside shouldn’t be used as a filament. Stratasys industrial machines don’t offer PLA why do you think that is? Next lets talk about bridging, bridging is an excuse for a printer that can’t print with support or “easy to remove support shall I say” so most printers rely on bridging because they can’t do supports well even people who moderate at ultimaker forum recommend not to ( http://umforum.ultimaker.com/index.php?/topic/6387-removing-support-from-prints/?hl=%2Bavoid+%2Boverhangs#entry58807). You don’t see stratasys providing bridging examples because that’s a limitation as supports provide better surface quality so bridging shouldn’t be a test to begin with and did you increase the fan to 100% when you tried to bridge? The resonance in the x, y, and z axis was do to the fact that you didn’t pay attention to nozzle diameter and requirements from the print manufacture.
        the 0.5mm box for the xy resonance test is below the required thickness from zortrax and would require a nozzle of .2mm to print effectively, a user from the zortrax forum printed the 1mm box and it printed extremely well without resonance which is aka moirè. The z axis has to be a lie as well look at this vertical piece http://forum.zortrax.com/index.php?/topic/596-comparison-of-3d-printers/ and that was printed on .0.0.8 firmware.

        Key items here
        -PLA requires no difficulty to print and is not a challenging
        factor what’s so ever.
        -Bridging is a term used only for printers that can’t generate proper
        support so they must compensate.
        -XY resonance test was not done correctly on the zortrax m200

        Ultimaker 2 does nothing but PLA well and when it prints ABS the surface quality is terrible because it shows imperfections much more prevalent.

        Ultimaker 2 should not of won and Zortrax was not accurately and properly tested.

        1. I appreciate you responding so quickly. The test prints that we used were meant to test the machines to their limits. In the case of z-resonance, that meant thin and tall, non of the results were faked and Make is not paid by any 3D Printer manufactures or anyone else for that matter to produce the reviews. All of the printers had the same models and most printers even have the same .4mm nozzle size (including the Ultimaker). In most cases, you’re right, bridging is not required, but some things that are nice about bridging besides users not having to remove support material are that they yield shorter print times, and also consume less filament and energy.

          One thing we did do is layout the results so people could see what categories the printers did well in. Those who don’t need bridging will probably disregard the it’s bridging capability.

          For most engineer’s, ABS would be preferable, but for those who keep printers inside their homes, they may be bothered by the smell of ABS. Also, PLA is nice because it is bio degradable. I did the bridging test a few times once with the additional fan speed at 100%, it did help a little bit but was mostly consistent.

    3. Anthony,

      The Zortrax M200 is a good printer but it has 3 serious limitations that I hope Zortrax will fix on their next model:

      1. The printer can not print PLA.
      2. The warranty is void if third-party filament is used.
      3. The printer does not have open source hardware and software.

      Zortrax’s next model needs the ability to control the temperature of the nozzle so it can print PLA and other materials. Ideally, Zortrax should offer this as an upgrade to their current customers. If there are third-party filaments that are known to damage the printer, they should be listed on Zortrax’s web site. Third-party filaments that have been tested and found to work fine should also be listed on Zortrax’s web site.

      Makers need and want the freedom to modify the hardware and software of our printers. It is much more difficult for us to make improvements and modifications with a closed source printer like the Zortrax M200. I suggest that Zortrax use a non-commercial open source license for their hardware and software like Ultimaker.

      Make’s resonance tests are really an indication of the printer’s default print speed. Zortrax could slow down the default print speed if they want to pass Make’s resonance tests. I personally don’t care about the bridging score either. If Zortrax fixes the 3 serious limitations above and improves their results on the negative space tolerance test, Zortrax will have a good chance of having the best printer in next year’s shootout.

      1. Not quite sure if you have used a 3D printer before but open sourced printers are garbage. I’ve owned the Mendal, Solidoodle 3, Afinia H480, Pegasus SLA 3d printer and the Zortrax. All except the Afinia H480 and the Zortrax have amazing print quality and reliability both packaged together, Both printers are closed sourced and both printers have extruders that run hot. When I had the open sourced printers I had nothing but issues with printing. You couldn’t just drag, drop and print. Parameters had to be adjusted based on each print such as the speed, flow, retraction and feed rate. Marlin has never been solid and all the slicers except for simplify 3d had slow import times on importing STL and/or Slicing the STL to gcode. I couldn’t print lets say a bolt, gear and bust all at once because each required different settings to print correctly with any decent quality. With closed sourced printers you can drag and drop any type of STL whether it’d be a bolt, gear, bust, mount you name it and it would print it beautifully and successfully 98 percent of the time. I’m not sure what everyone’s fascination with PLA is but for any engineer or designer you want a material that will not warp in the sun or doesn’t brake while you apply any tension. PLA is extremely brittle and although it has little warp there are techniques that can be applied such as a heated enclosure that reduce warp and other material such as z-hips that has highly reduced warp properties on it’s own. The way I see it having owned a shop is that the tools you use industrial or home use all are closed source because they are professional products such as a stratasys 3d printer or the Matsuura VMC I owned and you trust the engineer as you get successful results with the tools you are using. Zortrax has done nothing but deliver an amazing product and until you own one you will never really understand the beauty and peace of mind that it brings you because it truly is a print and go printer unlike the ultimaker 2. I don’t have to mess with feed speed, retraction, temp or anything else I simply drag and drop and select my infill and go. If you want to tinker and play with a toy then go with a reprap, delta or ultimaker but if you want a professional product that works and works every time you get a Zortrax M200. Filament is cheap from them so don’t understand that and open sourced only means something when a product is in it’s developing stages but there is nothing to develop with the Zortrax.

  4. I’ve been printing with shape-ways for a while now, and now I’m curious how their printing in various materials stacks up against these metrics… >_>

  5. hi, amazing work and test method!
    we have a 3D printing lab in Milan (Italy) and we’re working on some test method for materials instead of printers. Some tests can be overlaid and be a good test method both for printers and materials.
    Since we think that in many cases parameters drive print quality more than the printer itself can you please share the settings you used for each print/printer?
    in this way we can check the full method and have a better understanding of the review.
    For our material tests we are running a matrix test of temperature-speed combination to evaluate surface finish, cooling time, layer adhesion, and they could also be run on yours models to check for example the bridging ability of a filament.
    Thanks and keep it up!

      1. When the individual printer reviews are published on Makezine.com, how about providing a link to the web page that contains the parameters used? I haven’t been able to find any default parameters for the da Vinci on us.xyzprinting.com. As can be seen in the post by Chuck Hellebuyck, without knowing the parameters, it is impossible to know why two people got different results on the same test. Enough parameters need to be specified so that two people can get the same results on the same test.

  6. I have a few questions about the meaning of the print scores.

    1. For the accuracy score, the article says “measure the second from the bottom tier of the print (with a target diameter of 20mm) across the X and Y directions.” The article also says “Assign the print a 5 if average deviations in X or Y are between 0 and 0.1mm.” Does “average deviations in X or Y” mean

    a) the average of the deviation in X and the deviation in Y
    b) the maximum of the deviation in X and the deviation in Y
    c) the average of the deviations in X and the deviations in Y for each of the different tiers in the model
    d) the maximum of the deviations in X and the deviations in Y for each of the different tiers in the model
    e) none of the above

    In my opinion, it does not make sense to average deviations. A printer with a deviation in X of .4mm and a deviation in Y of 0mm is clearly worse than a printer with a deviation of .2mm in both X and Y. The average deviation is the same in these two cases. I think the accuracy score should be based on the maximum deviation across all tiers in the model with the diameter of each tier measured in 4 places. It would be useful to report the range of target diameter minus measured diameter for each printer. A range of [-.2mm, .2mm] would indicate more accuracy but less precision than a range of [.3mm, .4mm], for example. Wikipedia has a good description of accuracy and precision:

    http://en.wikipedia.org/wiki/Accuracy_and_precision

    2. Does a backlash score of 5 mean the absolute value of the measured size in the X direction minus the measured size in the Y direction is 0 to .1mm?

    3. What is the meaning of the numeric values for the “Surface Curved” and “Surface General” scores listed in the magazine?

    Overall, I think these test models are excellent. I especially like the Negative Space Tolerance test because there is a clear connection between test results and what can be made with a particular printer. I agree with a comment made by jstults that it would be more informative to report the actual measurements where possible. The measurements can be binned to create a score so that scores for different tests can be combined. The actual measurements are more useful and understandable than a score. Next year, I suggest listing the actual measurements for each test and a total score for each printer.

  7. I just ran the tests on my davinci 1.0. I have corrected the loose glass and repaired a broken bearing mount in the past ( as seen in my YouTube videos: http://www.youtube.com/beginnerelectronics
    Excellent mode: 0.2 layer, 10% fill, standard heat, shells, no support.
    I recorded the following:
    Accuracy – 5
    Bridging – 2 ( but is 5 with supports which are easy to breakaway on Davinci)
    Overhang – 4
    Fine Features – 4
    Tolerance – 2
    XY Resonance -Pass
    Z Resonance -Pass

    These numbers put my Davinci in top 10 which is amazing for a sub $500 fully assembled printer.

    1. On the accuracy test, what did you get for the length of the diameter in the X direction and the Y direction? Why do you think you scored better than Make on the accuracy and Z resonance tests?

  8. Thanks for this benchmark! It has been useful for me to benchmark my Robo3D printer!
    What I’m missing is some pictures of the most common failure modes and I’d also appreciate a clarification to the question posted by David H below.

    Thanks again, Daniel

    1. Would you mind sharing your test results? I just got a Robo3D R1 for my birthday and am debating returning it for a Printrbot Simple Metal. I would love to have some quantitative data to guide my choice.

      Thanks.

  9. In the description of the bridging test, what is the meaning of “dropped infill” versus “dropped perimeters”? Does it mean “there is a gap in the bridge” versus “the bridge is drooping”? Does the phrase “bridges compiled” refer to something the software did or something the hardware did? Is the description suppose to say “drooped perimeters” (the perimeters are sagging) instead of “dropped perimeters” (the perimeters are missing)? How is it possible to have dropped or drooped perimeters without also having dropped or drooped infill? Please post a few pictures to clarify these situations.

    The description of the bridging test says:

    Assign a 1 if any bridge has dropped infill.
    Assign a 2 if only the longest two bridges have dropped infill.

    Based on this description, it appears that the score would be 1 if only the longest bridge has dropped infill. Since higher scores are better, I guess what is meant is:

    Assign a 1 if more than two bridges have dropped infill.
    Assign a 2 if one or two bridges have dropped infill.

    Thank you for making these great test models and for sharing them.

  10. Make Magazine measured greater than .4mm of backlash for the da Vinci 1.0 while Chuck Hellebuyck measured 19.95 – 19.89 = .06mm of backlash for this printer. Can anyone make a guess about why Make and Chuck got such different backlash measurements on the same model of printer? The only explanation I can think of is maybe the printer Make tested got damaged in shipping. Make’s review said “Our package clearly had some serious international miles on it, but everything was well packed and in good order.”

    I would not expect differences in software settings to change the amount of backlash. Backlash is related to the amount a stepper motor has to be told to move before the nozzle actually moves at all when the direction of motion changes. When the direction of motion changes, the printer software can tell the stepper motor to move a little extra in the new direction to make up for gaps between pairs of gears. What the printer software can’t compensate for is the variability in these gaps between different printers.

    It will be interesting to see the test results from other da Vinci owners. On the accuracy/backlash test, I suggest that people report the measured values of the diameters in the X direction and the Y direction so that there will be a clearer picture of where the results are different.

  11. The manufacturers of closed-source printers (MakerBot, Zortrax,…) should reconsider the benefits of open-source for both the customer and the manufacturer. I will only buy open-source printers because open-source gives me the flexibility to fix my own problems instead of being dependent on the manufacturer.

    There are plenty of good open-source printers. Six out of ten of the top performing printers in Make’s annual guide (what Make called “The Standouts”) use open-source software. Four out of ten have open-source hardware. If MakerBot Replicator 5th Generation and Zortrax M200 were open-source, maybe a customer could figure out how to improve their test results.

  12. I see one flaw with this evaluation set and that is that it does not score the ability to print easily removed support. Some real word designs that we want to print cannot be printed without some kind of support. No printer can print cantilevers without support. Printers like the Zortrax and Up/Afinia with snap away support or those offering a second disolvable filament are invaluble for printing anything complex and I think should be rewarded in the scoring for such. I have a collection of repraps for my use and one Up Plus for the prints that present as too gravity defying to print unsupported. I really wish the open Slicers would get this worked out!

  13. For the XY resonance and Z resonance tests in the next 3D printer shootout, I suggest printing the test prints at up to 5 different speeds and derive the score from the shortest print time that passes the resonance test and has all longer print times also pass the test. In the table below, the print times are labeled A(longest) to
    E(shortest) and

    P = Pass
    F = Fail
    X = Don’t Care

    ______Print Times
    Score__A B C D E
    0______F X X X X
    1/A____P F X X X
    1/B____P P F X X
    1/C____P P P F X
    1/D____P P P P F
    1/E____P P P P P

    This resonance score indicates the number of test prints that can be printed in a unit of time without getting a resonance failure. Higher scores are better. The resonance scores for all printers can be multiplied by the same arbitrary scale factor to make the scores human friendly numbers. The speeds used could be 20%, 40%, 60%, 80% and 100% of a printer’s fastest speed. If a printer is only able to print at 3 different speeds, it is fine to use those 3 speeds and only use the A, B and C columns in the table above. It is not necessary for each printer to be tested at the same speeds when the resonance tests are scored as described here.

    This resonance score is more informative than a pass/fail score because it characterizes the rigidity and damping of a printer’s mechanics while being independent of the default print speed. With a pass/fail score, a printer with a fast default speed could fail the test even though it may be able to pass the test at a faster speed than another printer that passed the test. Giving a printer with a fast default speed a score of fail and a printer with a slow default speed a score of pass is misleading, especially when print times and layer heights are not reported. Printer manufacturers should not be encouraged to slow down the default print speed of their printers so that they can correctly print structures that rarely occur in practice.

    In addition to the resonance score, it would be useful to report the value of the print speed software setting that resulted in the shortest print time that passed the resonance test and had all longer print times also pass the test. When a person wants to minimize resonance with the least impact on print time, this would tell them a good print speed to try.

    Ideally, the software for a printer should figure out where the printer needs to slow down to avoid resonance. Printer manufacturers could put a 2 or 3 axis accelerometer on the print head. If the accelerometer indicates that the print head moved somewhere it shouldn’t have, the printer could slow down on the next layer in that area. The printer could also save this information to the SD card so that the next time the same object is printed, the print quality would be better. These accelerometers are used in cell phones and they are very inexpensive.

  14. In the next 3D printer shootout, I suggest replacing the bridging score with these two numbers:

    1. Length of longest bridge without dropped infill where all shorter bridges also don’t have dropped infill (possible name: Completed Bridge)

    2. Length of longest bridge with less than 2mm of droop where all shorter bridges also have less than 2mm of droop (possible name: Straight Bridge)

    These scores are useful when designing objects and they are easier to interpret than an integer from 1 to 5. With the current bridging test print, the possible values for each score would be: 6cm.

    In the table below,
    P = bridge passed (minimal droop or no dropped infill)
    F = bridge failed (excessive droop or dropped infill)
    X = don’t care

    _______Bridge Length (cm)
    Score_____2 3 4 5 6
    6cm_____P P P P P

    Typically, bridges longer than a bridge that failed would also fail but the table above handles all possible cases. For the purpose of summing scores of multiple tests to get an overall score, the scores above would be replaced with an integer from 0 to 5.

    When possible, it would be best to have all printers print at approximately the same speed when printing the accuracy, bridging, overhangs, fine features and negative space tolerance tests. One approach would be to do each test twice, once at the default speed and once at approximately the speed of the slowest printer. An alternative would be to print Samuel Bernier’s robot at the default speed. Based on the print time of the robot, the print speed for the other tests would be scaled down so all printers do those tests at approximately the speed of the slowest printer. Printers that don’t have an adjustable print speed would be tested at their default speed. The print times for each test print should be reported.

    The software version number and the most important software settings (print speed, layer height, infill %, quality mode) that were used for each printer should be specified. Please provide pictures of each test print on makezine.com when the review of each printer is published on the web site.

    As others have pointed out, the accuracy and backlash scores would be easier to interpret if they were reported as a distance instead of an integer from 1 to 5 (for example, “0.15mm” instead of “4”).

    On printers that can print both PLA and ABS, it would be ideal if all tests were performed with both types of material. It is probably not fair to compare PLA results from one printer with ABS results from a different printer. Testing with both PLA and ABS when possible would provide a more complete picture of a printer’s capabilities.

  15. Here is a Google spreadsheet of the test results and printer features:

    https://docs.google.com/spreadsheets/d/1EKsDga2PVD_H9HI2MJbPXCey6bYFEIWErOsAHKHZ3GU/edit?usp=sharing

    This spreadsheet makes it easy to sort the printers by whatever tests and features are most important to you. The negative space tolerance (minimum gap), accuracy and backlash scores were converted to their corresponding measurements to make them easier to interpret (“.2 – .3” means between .2 and .3mm). The initial view of the spreadsheet is sorted by the 2nd to 5th columns. To sort by different columns:

    1. Highlight all columns in rows 4 to 27 with the left mouse.
    Be sure to include the bottom row of labels (row 4) in the highlighted data.

    2. Click Data>Filter views>Create new temporary filter view.

    3. Click the boxed triangle in the column you want to sort and select Sort A->Z for
    increasing numbers or Sort Z->A for decreasing numbers.

    4. To sort by multiple columns, repeat step 3 doing the least important column first
    and the most important column last.

    Google Sheets is not able to sort columns on some older browsers, such as Firefox 3.6. Google Sheets will display a message at the top of the page if you are using an unsupported browser. Unsupported browsers can still display, print and download the spreadsheet.

    A good way to print this spreadsheet is to click the printer icon below “File” and choose “No gridlines”, “Actual size” and “Landscape”. Then click “Print”. If using Chrome, on the next page, select “+ More settings” and uncheck “Fit to page”. If using Firefox or Internet Explorer, a PDF will be created when you print the spreadsheet. Choose “Actual size” and “Landscape” when printing this PDF.

    Feel free to modify this spreadsheet and share your modifications. If logged into a Google account, you can use File>Make a copy. To save the spreadsheet to a local machine, use File>Download as. To make the spreadsheet fit on one 8.5″ x 11″ page in Excel, use Page Layout>Orientation>Landscape and
    Page Layout>Margins>Custom Margins. Set the top and bottom margins to .5″ and set the left and right margins to .2″ when using Excel.

    1. Here are some observations derived from the spreadsheet in my previous post:

      1. DeltaMaker, Ultimaker 2 and Replicator 5th Generation were the only printers that printed a removable pin in a hole with a .2mm radial gap. Despite this excellent performance on gaps, DeltaMaker and Replicator 5th Generation did poorly on the fine features test (the spires did not print or were densely connected by strands of material).

      2. DeltaMaker, Printrbot Simple Metal, TAZ 4 and Type A 2014 Series 1 were the only printers that had both accuracy and backlash under .1mm. Of these 4 printers, DeltaMaker was the only one that did well on both the surface curved and surface general tests.

      3. Ultimaker 2, Zortrax M200, TAZ 4 and Felix 3.0 were the only printers that got perfect scores on the fine features test. Of these 4 printers, Ultimaker 2 and TAZ 4 were the only ones that did well on the bridging test.

      4. Ultimaker 2 and Zortrax M200 were the only printers that got perfect scores on both the surface curved and surface general tests.

      5. Printrbot Simple Metal is about 1/4th the price of Ultimaker 2 and it did well on all tests except the fine features, surface curved and XY resonance tests. Printrbot Simple Metal was the only printer that got perfect scores on both the bridging and overhangs tests. The main limitation of Printrbot Simple Metal is the small build volume of 150mm x 150mm x 150mm (5.9″ x 5.9″ x 5.9″).

      6. Printers that do auto-leveling as currently implemented will probably have difficulty printing large flat surfaces parallel to the XY plane, but none of the tests evaluated this. The 5 printers in this survey that do auto-leveling compensate for bed tilt by adjusting coordinates while printing. If the bed is tilted, this will produce small steps on flat surfaces parallel to the XY plane. Poor surface quality will hide this defect. Zortrax M200 is unusual in that it can compensate for bed tilt by adjusting coordinates but it also can use its tilt sensor to tell the user how to turn knobs to level the bed. This hybrid of auto-leveling and assisted leveling seems like an excellent idea.

      7. Among printers that printed a gap of .4mm or less, about 3/4ths of them printed the X-direction and Y-direction diameter of a cylinder within .1mm of the same size (backlash test).

      8. About half of the printers failed the XY resonance test. It would be interesting to know what can be done in the design of a model and/or in the software settings to compensate for this weakness in a printer. Slowing down the print speed should help.

      9. If you want a model to print on all of the printers in this survey, the maximum build volume is 140mm x 135mm x 125mm (5.5″ x 5.3″ x 4.9″). The next step down in build volume is the Printrbot Simple, which has a build volume of 100mm x 100mm x 100mm (3.9″ x 3.9″ x 3.9″).

      10. Andreas Bastain’s tests do an admirable job of probing different characteristics of 3D printers. As Andreas pointed out, these tests give users and vendors something to experiment with so we can try to find changes in software settings, software algorithms, hardware and filament materials that improve the results.

  16. how about you tell us how to fix these issues rather than telling us how to figure out that there is an issue we already knew about.

  17. Curious… It says you used the same material for all Pla machines. What about the ABS machines? What material did you use for all of them if they are suppose to be the same…and what about the machines that use their own special ABS materials because the machine is calibrated to them and 3rd party filaments don’t print as well?

Comments are closed.

Andreas Bastian researches, designs, and builds new types of 3D printers. He has worked with FDM, SLS, DMLS, and SLA technologies and aims to make all more accessible. Currently a 3D printing research scientist at Autodesk, he is also active in the e-NABLE 3D printing prosthetics community.

View more articles by Andreas Bastian