If there’s one thing that’s the same about everyone’s broadband connection, it’s that it’s slow. Usually slower than it was advertised to be when you got it. But slow isn’t as irritating as sporadic, when you get constant drops and outages in your internet connection, it can drive you to frustration.
It drove one man in Washington D.C. to monitor his broadband connection with a Raspberry Pi, and automatically tweet Comcast when his connection drops to a fraction of advertised speed.
This is actually something I’ve been doing myself for a couple of years, also using a Raspberry Pi stuffed in a corner of my network closet. Well, not the bit where I tweet my broadband provider. Instead my script is a bit more direct: It automatically submits a trouble ticket into their support queue.
Over the time I’ve been doing this I’ve used various methods to measure the latency and speed of my broadband connection. Most recently I’ve switched to using speedtest-cli, a command line interface to the speedtest.net servers written in Python.
Not sure if Raspberry Pi is right for you? Make:’s interactive Board Guide lets you dial into the field to find the best board for your needs.
It’s really easy to install. Just open up a terminal window on your Raspberry Pi and type the following at the command line,
$ sudo apt-get install python-pip $ sudo pip install speedtest-cli
this will install pip — a package management system for Python — if you don’t already have it installed, and then the speedtest-cli package from the pip repositories.
Once installed it’s rather easy to grab and measure your broadband speed.
$ speedtest-cli Retrieving speedtest.net configuration... Retrieving speedtest.net server list... Testing from Acme Broadband Provider. (XXX.XXX.XXX.XXX)... Selecting best server based on latency... Hosted by Foo Limited (Metropolis) [2.52 km]: 35.27 ms Testing download speed........................................ Download: 14.47 Mbit/s Testing upload speed.................................................. Upload: 1.46 Mbit/s $
Although I’ve disguised other details, that really is my average reported broadband speed. I have an ADSL2+ Annex M connection, unfortunately FTTC hasn’t quite made it to my parts of the world.
While this is my average speed, it’s well below what I’m paying for, which is 25 Mbit/s down and 2.5 Mbit/s up. However I’ve more or less given up on obtaining that… these days I’m just happy if it’s stable. Because, every so often, I go through patches where my latency rises to several hundred milliseconds, and my bandwidth drops to 1 or 2 Mbit/s. These periods can go on for days.
Which makes me rather jealous of our friend in Washington, who is complaining to Comcast when his 150Mbit/s connection drops to a low of 50Mbit/s.
In any case, now that we have our command line tool installed, we can run it automatically using cron, which allows you to schedule commands to run at specified times (every hour), and log the output to a file.
The easiest way to do this is to create a quick script, let’s call it speedtest-cron.sh, which will log the date and the output of the test to a file,
#!/bin/bash date >> /home/pi/speedtest.log /usr/local/bin/speedtest --simple >> /home/pi/speedtest.log
Then go ahead and edit your crontab file, adding an entry to run the script test once an hour. You can do that by typing,
$ crontab -e
at the command line, and then adding the line,
0 * * * * /home/pi/speedtest-cron.sh
to the crontab file before saving it. This will run the test once an hour, at the top of the hour, appending the date-stamped output of the speedtest command to a log file.
$ cat speedtest.log Sun Jan 31 19:49:01 GMT 2016 Ping: 34.961 ms Download: 14.44 Mbit/s Upload: 1.41 Mbit/s $
However the output of the speedtest-cli package, even in its “simple” mode, is pretty messy. We could go into the package and fix things so that the output is somewhat more useful, CSV format perhaps, or we could rewrite the whole thing in Perl. But since this is a quick hack, it’s probably easiest just to fix things with a bash script.
Which is exactly what the speedtest-cli-extras script does. It captures the output of the script, reformats it, and outputs it on a single line with time stamps and values separated by semicolons,
$ git clone https://github.com/HenrikBengtsson/speedtest-cli-extras.git $ cd speedtest-cli-extras/bin $ ./speedtest-csv 2016-01-31 17:00:33;2016-01-31 17:01:27;Acme Broadband Provider;XXX.XXX.XXX.XXX;Foo Limited (Metropolis);2.52 km;34.768 ms;14.43 Mbit/s;1.31 Mbit/s;http://www.speedtest.net/result/XXXXXXXXXX.png $
again with names changed to protect the guilty. As you can see the output is much more useful, especially if we want to create graphs, than we had previously. Running this every hour from cron is going to start to pile up evidence in a nicely formatted data file, hopefully allowing us to have a satisfying argument with our broadband provider inside a few days.
Except that, while I’m on the road, I don’t want to have to SSH into the Raspberry Pi connected to my ADSL modem, grab the CSV file, and manually post the trouble ticket I’ve filed with my broadband provider.
This is where IFTTT‘s Maker Channel, introduced towards the middle of last year, comes in handy. I went ahead and created a recipe on IFTTT to take the data passed to a Maker Channel event called “speedtest” and automatically fill a Google Sheet with the output of speedtest-cli script.
The easiest way to get the data from our Raspberry Pi to IFTTT at this point is to modify the speedtest-cli-extras script. Instead of printing out the output to a log file, we’ll make a POST web request with the event name and our secret key — the key is assigned when you connect the channel — of the form,
https://maker.ifttt.com/trigger/speedtest/with/key/{secret_key}
with a JSON body consisting of three values — the latency, download, and upload speeds — to be passed on to the action in the recipe.
After you download it you should go ahead and substitute your own SECRET_KEY before testing it out by running the modified script manually at the command line. If all goes well you should see something like this,
$ ./speedtest-ifttt.sh Congratulations! You've fired the speedtest event $
Go ahead and check your Google Drive, there should be a new Sheet called “Speedtest.”
This Sheet should have a single row with four columns populated. The first is the date stamp on the IFTTT Maker Channel event, the second is the ping time in ms (latency of the connection), the third the download speed in Mbit/s, with the final column being the upload speed in Mbit/s.
January 26, 2016 at 02:49AM | 34.039 | 12.56 | 1.32 |
If that’s worked, all we have to do is modify our speedtest-cron.sh script to run our new script when it’s called at the top of the hour by cron,
#!/bin/bash date >> /home/pi/speedtest.log /home/pi/speedtest-ifttt.sh >> /home/pi/speedtest.log echo "" >> /home/pi/speedtest.log
and each hour we’ll now add another row to the Google Sheet.
At this point you now have a Raspberry Pi-based monitoring system in place to measure your broadband speed once an hour, and automatically log it to the cloud. This should hopefully give you enough evidence to wave at your broadband provider when things are, let’s say, less than optimal with your connection.
In my own case if I have three low (<2Mbit/s) download speed measurements in a row, the script also goes on to make a second HTTP POST request to my broadband provider’s helpdesk system, and files an automated trouble ticket — containing a pointer to my Google Sheet — to tell them my connection is having problems. But, since your broadband provider almost certainly uses an entirely different helpdesk system than mine, we’ll leave that as an exercise for the reader.
The real question is, when you submit your trouble tickets does Comcast actually do anything?
Right? You can have all the monitoring on your side that you want, graphs, data out the ears, and they’re still going to tell you to connect your Windows machine directly to the router and also have you tried rebooting. Then they’ll blame wifi interference.
That said, I really like the addition of the logging, just for personal use. I’m going to have to look into copying that into my own setup.
6❝my neighbor’s mate is getting 98$. HOURLY. on the internet❞….A few days ago new McLaren F1 subsequent after earning 18,512$,,,this was my previous month’s paycheck ,and-a little over, 17k$ Last month ..3-5 h/r of work a day ..with extra open doors & weekly. paychecks.. it’s realy the easiest work I have ever Do.. I Joined This 7 months ago and now making over 87$, p/h.Learn More right Here➤➤➤➤➤ http://GlobalSuperEmploymentVacanciesReportsJobs/GetPaid/98$hourly… .❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦,❦.❦,❦vf……..
Over the time I’ve been doing this I’ve used various methods to measure the latency and speed of my broadband connection. Most recently I’ve switched to using speedtest-cli, a command line interface to the speedtest.net servers written in Python.
Comcast? Probably not. If you’re not a Comcast for business customer then no. I know one chap, he’s got the fastest connection ever, and when there ar problems, he’s got them on it, at what seems to be faster then light speeds.
Pingback: Use Raspberry Pi to Measure Broadband Speeds, Dropouts to Hold Your ISP Accountable | 懒得折腾
If the bandwidth you’re paying for is more than 100Mbps, how does a RPi measure it? Don’t get me wrong, I love this idea, but the RPi is always going to be see the bandwidth capped at 100Mbps or less.
If you have broadband with >100Mbps throughput, I don’t think you have much to complain about in the first place?
Or you could run the script on a linux box that’s running a 1gE.
Pingback: DIY ISP Network Speed Monitor | DIYGadgets
I used a Raspberry Pi to send the data to Loggly: you just need a two line script then! http://blog.scphillips.com/posts/2015/05/monitoring-broadband-speed-with-loggly/
Nice!
Thanks for that. I followed your instructions and it’s working like a charm!
i can’t seem to find the secret key
HI guys.
I made exactly the same a few months ago.
The only difference is that the error report goes to your ISP Twitter Account to put a ticket, using IFTTT.
May be is useful for someone:
https://github.com/esturniolo/SpeedTest_Boton
It’s not finish and has bugs, but is usable.
I would want to use the Perl script because I’ve heard that the Python compiler allows things it shouldn’t, which often causes Python programs to be faulty. As much as people might complain about Java’s security, the compiler is rather strict and will at least point you to the line which has faulty code if it can’t give you a description of the problem.
Is the script somehow able to determine when a computer, mobile device, or set-top box is downloading something, so as to prevent a false readout? If it can’t, this script could be reporting that the Internet access speed is lower than it should be when it can’t get an accurate measurement as a result of a large file being downloaded on the network?
No. But unless you’re saturating the available bandwidth the throughput, i.e. the broadband speed measurement, shouldn’t drop by that much. It’ll drop, but not significantly unless you have a really bad piece of copper.
In that case you don’t even need the higher bandwidth? I know if I download something big the bandwidth won’t be anywhere near where it should be. It should detect if there is any significant downloads going on. Someone running netflix will surely throw the measurements off.
People often talk about bandwidth and throughput as if they’re the same thing, they aren’t. That said, yes. A for a lot of home broadband connections, having HD video streaming simultaneously will through the speed (throughput) measurements off quite a bit.
My concern is that if you tell the script that you’re supposed to be getting 15 megabits/second, the moment you start watching a YouTube video, or upload a photo to Flickr, your Internet speed will go down, and then the Raspberry Pi will end up falsely notifying your ISP of a drop in service.
Hi to all, i want to make this, but im lost on the secret key…. where can i see my secret key?
thanks in advanced for some replies
best regards
Your secret key is generated when you connect the IFTTT maker channel, and is shown here, https://ifttt.com/maker
Hi to all , i have a strange problem when i run ./speedtest-ifft.sh it works and spreadsheet get populated with correct info but at top of the hour all fields are empty only with date stamp, someone know whats possible is wrong?
thanks in advanced
Sounds like you have a path problem with the script called from cron and, while the script is running you’re not actually doing the speed test. Check the path to the speedtest-cli program in you speedtest-ifttt.sh script is correct. It’s possible that on your system it’s not installed in /usr/local/bin.
Hi Alasdair, first thanks for the support, really appreciated, afther checked seems to me the path is correct , but the log file is populated only with ; “Congratulations! You’ve fired the speedtest event” every time the script run … really i dont know what is wrong , any more help is really appreciated.
thanks in advanced
so… i have pick up the problem ….. my speedtest-ifttt.sh was only with : query speedtest: speedtest-cli
no i add /usr/bin/speedtest-cli and all works fine
thanks for help and for great post.
cheers
That worked for me as well, thanks.
I was linking to an older version of the gist. I’ve now updated the link in the text to point to the latest version.
I was having this same issue. It’s because the download for the speedtest-ifttt.sh that you have linked above is missing the full path. The code you have above includes the correct path. So basically, anyone that downloads the file you linked is going to have this problem.
Thanks. Was linking to an older version of the gist, now fixed.
Hello mister
I have the same problem. When I run the ./speedtest-ifttt.sh it runs and populates the sheet just fine. But when the cron job runs only the time stamp populates.
Well I guess you answered this already but I didn’t quite think that was the reason and I was wrong. I had the script in the home/pi folder. So I think I’ll go good now.
ITT: youre paying for 25 megabytes not bits and they just blame the advertising
8 bits to a byte: megabytes > megabits paying for megabits likely, not megabytes
Pingback: Monitor the Consistency of Your Broadband Speeds with a Raspberry Pi and IFTTT | Parallel Feed
FYI.. Line 54 of the “download” speediest-iffttt.sh is incorrect compared to the line above (the line above works fine). It’s missing the full path.
Thanks. The download link was to an older version of the gist. Now fixed.
Maybe a noob question (maybe not) but couldnt this be done on a normal PC? I have one that runs 24/7 and would be interested to check how my speeds fluctuate during the course of a day. If anything from the above changes when doing this on a PC, could anyone point it out?
Are you running a Linux distro? I don’t have much experience with IronPython (it does support pip I think), but I assume you could get speedtest-cli to run on Windows, scheduling the task without cron is going to look different, though.
If you have Linux then it’s pretty much the same steps
Do you need the full raspberry Pi for this, or would a zero be capable, been considering jumping into the pool, and this project might just be the push I need …
A Zero would have plenty of horsepower for this, but you’d need to get a micro-usb ethernet adapter for it, they are around $10, or a micro-usb-OTG adapter and a usb wifi adapter, < $15 together.
Pingback: Monitor the Consistency of Your Broadband Speeds with a Raspberry Pi and IFTTT | ..:: Frog in the box ::.. | ..:: Frog in the box ::..
This was a fun little project on a snowy day. My ping times are much higher than running from speedtest.net. the DL and UL speeds are close but latency is like 900ms on cli and 20-30ms on web…just curious if anyone else had that too. Haven’t figured it out just yet.
There are several reasons why you might be getting different results. The Speedtest.net servers have been migrated to using pure socket tests instead of HTTP based tests, different version of Python will execute thing slightly differently, and CPU, memory capacity, and speed (or other machines and how your network is connected together) can affect the result.
That said a difference of 20-30ms and 900ms is pretty extreme. You’re not using powerline Ethernet by any chance are you? I’ve seen this sort of problem happen when that gets disrupted by a spike from a washer-dryer or some other high current device.
Thanks for the reply! I am not using powerline ethernet and consider the infrastructure pretty simple with modem>router>switch>endpoints (wifi and ethernet). My guess is its just that old gen1 raspberry pi and even with it headless and no usb devices connected that the bus is just slow. I am gonna build a quick linux box and set up speedtest-cli to compare. My gut says its at the Pi level tho.
Could be, I was using a Pi 2. Wouldn’t have thought it’d make a difference, would be surprised in fact, but I could be wrong.
Interesting. I’m going to have to look into this one. I think I want to be sure I can control the test server. I don’t want it to be owned by my ISP (comcast). I get consistently faster results when I get with them. It seems less realistic to me.
You can force the server choice with a command line option on speedtest-cli. Run it from the command line with “speedtest-cli –help” for a full list of options.
Great mini-project you have here :D I have two questions:
1- how hard is to make some sort of live graphic with that info?
2- sometimes the upload field returns somehow a date. a few minutes ago I saw a log with “2016/01/20” in it. What is the cause of this?
Create a header row in the Google Sheet by inserting one at the top of the sheet and grabbing and dropping the separator and dragging it down to make it a header rather than a normal cell row. Then select the entire C and D column by clicking on C and shift-clicking D. Then insert a graph using the menu bar item. Since the entire column is selected the graph will update each time another row is appended to the sheet.
How did you get this to work since you use . and not , as decimal?
I cannot get the . to work and have to manually change it to , to get the graph.
Is there somewhere in the script I can get it to type it with a , instead?
No idea about the date issue. Haven’t seen that.
I have something like this, but I send data to an Emoncms server (using Mqtt protocol). Than Emoncms does live graphics and I can also use this data into my Openhab “home automator”. I send also Raspberry informations. This is not so hard to do. If you want something easy to do, you can take a look at the following project. It gets connection data, but also some modem data.
http://cdfreitas.github.io/blog/2013/04/19/raspberry-pi-seu-provedor-de-banda-larga-entrega-o-prometido/
Nice!
Hi,
I had the same problem with the date. I found out that google sheet formats automatically the numbers. For example, for me, it changed 1.01 to 01/01/2016. You can change that bym not formatting automatically number. (format/number/…)
Hope it helps
If you have a Windows machine that could access the file you could have a scheduled task to run a PowerShell scrip that calls a macro in Excel to build a chart from the file export it as a PNG file and upload it to a webserver. I know Libre Office supports Python scripting, but I’m not sure of it’s capabilities to do the same. This will all get quite a bit easier in July when “all” of Ubuntu userland is added to Windows 10 Redstone.
I’ve created the script and set the cron job (running every minute for troubleshooting purposes) but it seems that it’s throwing an error about permissions. The output mail from the mail handler says “/bin/sh: 1: /home/pi/speedtest-cron.sh: Permission denied” What did I miss? (I did just update file permissions on the .log file, but that wasn’t the solution)
Check the ownership of the script. It’s possible that you copied it in as root and cron running as the pi user can’t access it?
Thanks – it wasn’t ownership, it was permissions – not executable. I’m logging to the .log file, onto the next step!
I am trying to get this to work except when I run the speedtest-csv executable, it completes and tells me congrats and all that. However, It never runs the recipe on IFTTT. I replaced the SECRET_KEY on the line “secret_key=”SECRET_KEY”” with “secret_key=”key_I_got_from_maker_site””.
Any ideas?
dis-regard. When I setup the recipe I named it “Speedtest” and the script is trying to run “speedtest”. FYI Capitalization matters :)
Yup, capitalisation matters!
When I run speedtest-ifttt.sh, it runs correctly and reports that I’ve fired the spreadsheet event, but I don’t have a new spreadsheet in my Google Drive. Any suggestions?
Have you copied the IFTTT recipe into your account, and activated on the remote end? Both ends need to be set up for things to work.
You have to name the Event in IFTTT as “speedtest”, notice the lowercase
Pingback: Monitor the Consistency of Your Broadband Speeds with a Raspberry Pi and IFTTT – Iconic Moment
Just done a comparative test using the command line and web site on the same Ubuntu machine. On the website I got a latency of 2ms, DL 269.13 Mbps and UL 306.44 Mbps and then, using the command line, latency 12.943 ms, DL 155 Mbps and UL 94.18 Mbps.
Not sure that command line tool is working very well?!
There are several reasons why you might be getting different results. The Speedtest.net servers have been migrated to using pure socket tests instead of HTTP based tests, different version of Python will execute thing slightly differently, and CPU and memory capacity and speed (or other machines and how your network is connected together) can affect the result. If you want really accurate results, this isn’t the way to do it. You should be using iperf to your ISP’s servers (if they allow that).
Do exist any script of iperf to ifttt ?
Thanks
Pingback: Liquidmatrix Security Digest Podcast - Episode 63 | Liquidmatrix Security Digest
This can only work with a PI? anyway to run such a scriot on a windows server?