Project Byzantium Aims to Avoid Internet Outages

Project Byzantium Aims to Avoid Internet Outages

With this post, we welcome R. Mark Adams as a makezine.com contributor. I met Mark when he gave an awesome presentation at Dorkbot DC on creating music from DNA data (he’s a computational biologist by day). He’s also actively involved in HacDC. When I think of a sort of Platonic ideal of a maker, I think of folks like Mark. He’s smart, generous, enthusiastic, resourceful, there’s an artist mixed in there with the scientist/engineer, and he’s just an all-around great guy. That Platonic ideal extends to his family, too, with a fellow geek wife (who’s campaigning to get hackerspaces into libraries) and two awesome daughters who are always in the thick of it. At Dorkbot meetings, when we go around and introduce ourselves and talk about what we make, they’re never shy about chiming right in (and complaining about their homework). Please help me in welcoming Mark. -Gareth

Project Byzantium team working

At Washington, DC’s hackerspace, HacDC, a team led by the enigmatic “Doctor” seeks to develop a simple system for quickly deploying an ad-hoc internet in the event of internet outages. With all of the recent disasters (both natural and human) which have severed access to the net when it is most needed, groups all over the world have been developing solutions which allow alternate internets to be brought up quickly and effectively. HacDC’s “Project Byzantium” team (named after the fault tolerance based on the Byzantine Generals problem) is working hard on implementing a mesh network which requires minimal effort to configure and which any wireless-enabled device can make use of, whether or not it is running mesh networking software. The goal is something that a smart high school freshman could get going with hardware from the local mall. Participation is welcome – a good place to get started is the project wiki. The Doctor himself will be demonstrating the system at ContactCon in October.

Here’s hoping that the next step is for hackerspaces everywhere to start experimenting with this and other related technologies, prove that it works, and share it with local communities and beyond – a robust internet is your friend!

A quick update (10/12)– for those who are interested in the project documentation, code or participating in the development process, the Project Byzantium repository can be found here.

More:

38 thoughts on “Project Byzantium Aims to Avoid Internet Outages

  1. Anonymous says:

    I would be more excited about this, if it didn’t seem like it was yet-another-project trying to reinvent the wheel. 

    I like the general idea, but wireless meshes are not a panacea for connectivity.  They are often worse than other options that seem cruder, but actually get the job done more reliably.  Perhaps y’all have solved it, in which case, kudos.  But I’d really like to see it in action first, in a large scale deployment.  It’s funny how that last mile of scalability is often a killer. 

    Also, it seems like there are several projects like this one, all flailing away at this concept and not pooling knowledge.  Or resources. 

    The Internet (TCP/IP) is already designed to function in the manner needed to achieve the goal of resilient widespread connectivity.  The challenge here is simply a lightweight manner to provide the systems support (i.e. DNS, routed gateways) and ground coverage at the edge needed to make a functional package rapidly deployable.  System security would also be a plus, given that in some cases political elements could honeypot the system to find and kill dissidents.  Or disrupt it by subverting it with their own dns servers etc. 

    For all that to be effective as a system, you need a disciplined and standards-based approach to minimalistically design the platform for both the minimum of hardware (getting easier every day) and configuration (not getting any easier last I checked).  A big problem there is that auto-configuration is difficult to do reliably, particularly if you have no coherent centralized control of the system.  (Wireless makes this even worse, generally). 

    I do wish them the best of luck, and I hope they’re smarter than me, as they’ll need to be to really get this to work where many others have failed. 

    1. Riley Porter says:

      I kind of see your point.  However, I would say that “standards” and having the “one way” to do this is kind of backwards.  I mean if everyone did this sort or project the same way.  Then wouldn’t they all suffer from the same weaknesses?

      I see the need for people to be working together to solve this, But I think modular-ity I guess is the true answer.. Meaning.. Cell towers jammed?  Piggy back sat signals.  WIFI CELL dead?  Find a hardline and broadcast via rf mesh.. 

      Cool post mark thanks for the info!

      1. Anonymous says:

        That would be a simplistic version of a standard.  Use of abstraction layers (i.e. OSI layer model) means that you can skin the cat anyway you like, as long as you make it possible for your cat-skin-o-matic-9000 to play nice with others.  Standards enable a variety of options and provide the necessary coherence to enable interoperability.  They are a tool, not a mental subjugation. 

        I’d also say that modularity is intrinsic to the Internet, that’s what enabled it’s success.  So I don’t disagree that it’s necessary, but I also don’t think that these projects are *socially* interconnected enough to make the impact desired.  It’s fun and exciting to solve technical problems.  But frankly, the networking technologies already exist, and can be used pretty much as-is to solve this.  So trying to come up with yet another clever way to skin that cat is a waste of time. 

        What is more of a challenge is streamlining it at a deployment and systemic level to a new economy of scale that would allow an irresistible level of deployment saturation.  If you focus on making the system lightweight and reliable, and elegant, then I’ll be far more interested.  If it costs $10 a node, that may actually be too expensive, can it be driven down further?  Can you make it so ubiquitous that it’s just not possible to block any more?  And how do you bridge the system to the outside world?  UN relays? 

        The first thing that most think of for these projects is a wireless mesh, to solve the connectivity issues, as if that will just magically work at any real scale.  Yes, it can be made to work, but so far only by people who know what they’re doing, not as a self-regulating amorphous blob of devices.  The routing problems become quite unwieldy for a low-resource microprocessor, and I’ve yet to see a truly elegant solution to that, that didn’t cost a fortune. 

        I have personally deployed wireless meshes, on a professional level, so I have some experience with this.  They have improved, but it’s not as easy as many would think.  The performance isn’t anything to write home about either, RF being what it is. 

        Using simpler diverse technologies(physical media-wise), and simpler topologies would probably work better, faster and cheaper in the end.  Not as sexy, but probably more effective. 

        1. The Doctor says:

          Actually, a lot of our work lies in making it easy and fast to set up a general purpose mesh network.  Take a look at our wiki (which, I might add, has links to our source code repository and mailing list archives): http://wiki.hacdc.org/index.php/Byzantium_Live_Distro

          As for bridging a mesh to the global Net, we’re making use of lessons already learned by the Battlemesh hackers and including sets on instructions for improvising long-haul links, like those used in Egypt, Syria, Afghanistan, and Pakistan right now.  Unfortunately, we can’t package wok-fi or cantennas with our .iso images (it would be cool if we could) but we can tell people how to build it themselves if they don’t know already.  If you have “personally deployed wireless meshes” as you say you have, then by all means get in touch with us through our project page, we’d love to talk with you about what you’ve learned, and we can always use another engineer on the project!

          1. Anonymous says:

            Well, looking at the wiki many of my concerns are clearly already thought of.  Nice to see, it’s more fleshed out than I expected.  I presume that you’re going to implement DNSSec?  It’s non-trivial, but solves a lot of problems and would be on my “must” list for this project.  I didn’t see it on my first pass, but maybe I missed it. 

            My point about wireless meshes still stands, while battlenet is an impressive experiment, there is a long distance between enthusiasts experimenting with technology and what your stated goals are.  If you can do it, great (prove me wrong, please!), but I’d advise a method-agnostic approach, so it generally doesn’t matter if you’re using a mesh, point to point, or other topology or media. 

            If you can come up with a good general convention for deployment, that may be even more helpful than a brilliant protocol solution.  In short, I’m just saying go with what works, not necessarily what’s most technologically attractive.  Maybe it’s a mesh, maybe it’s not.  Perhaps this is already obvious to you, in which case I apologize for the bandwidth waste. 

            I run a very large (~60K node) and extremely heterogeneous network, with a focus on reliability.  So admittedly my standards for performance are very high, perhaps higher than is applicable for this project’s goals.  Best effort would make a big difference when your trying to defeat a total blackout after all. 

            Also, I’m not in the habit of saying things that aren’t true, thank you very much.  I was involved with a prototype installation of the old Cisco outdoor mesh AP’s that used the old Richochet cases(and apparently still do, which is somewhat hilarious).  I wasn’t impressed, particularly at layer 3.  Apparently that’s evolved into the aironet 1510 now, but I suspect that it’s an entirely different animal at this point.  The prototypes, even with good antennas, had scalability and reliability issues that quickly outweighed the rest.  Power and connectivity to the traditional network still had to be solved, so convenience of deployment was somewhat limited, particularly with the power requirements.  But that was in a campus environment with fiber available, so obviously a different situation with different priorities.  And it was pre-N, so that changes the landscape somewhat as well on the RF side, where there is the most vulnerability for failure.  (And fail it did, frequently). 

            One of the most valuable lessons I’ve learned is that redundancy is not a panacea for reliability.  In fact, it often counter-intuitively decreases reliability, particularly in situations where control is less than total for configuration changes.  So inherently, there is a significant technical challenge to getting a mesh to work consistently without undesirable dynamic behavior, particularly with many nodes.  It looks like there is now a lot of interesting work being done to solve those issues, and I do look forward to seeing the results at scale.  But I also encourage the simplest approach to serve a need, whatever that ends up being. 

          2. haxwithaxe says:

            I’m one of the developers on byzantium. For now at least we are completely ignoring DNS within the mesh. The lack of a central authority to say who gets what domain throws a giant wrench into the process. We will have nodes running services share what services they are running and those will appear in a webpage that each node has local to itself.
            The problems arise when you have two (or more) mesh networks that begin separately and two (or more) people choose the same hostname at which point you have two valid hostnames that collide with each other when the networks meet at a later time and no good way to resolve them without making the hostnames nonhuman-friendly or making the user decide which one they want manually.
            It’s a deceptively hard problem :/

          3. Anonymous says:

            I was thinking more of securing the external dns service to the hosts than dns within/for the mesh, but your point is still well taken.  Ignoring DNS entirely within the mesh is probably a great idea, given the challenges.  I’ve just thought of several possible solutions, and they all kinda suck.  (time hash word pickers etc etc). 

          4. haxwithaxe says:

            right now we are focused on the local supernode (the one(s) that are hosting services and connecting from one mesh to another) functionality. It’s were most of the work is, and the software part of connecting a mesh to the internet is well documented at this point so we haven’t done much with mesh to internet part. i’ll add a feature request in our github for the dnssec stuff. it won’t likely make it in as more than a package before 2.0 (where we will be focusing on operation inside hostile networks), but it’s definitely a good idea :)

          5. Anonymous says:

            Well, looking at the wiki many of my concerns are clearly already thought of.  Nice to see, it’s more fleshed out than I expected.  I presume that you’re going to implement DNSSec?  It’s non-trivial, but solves a lot of problems and would be on my “must” list for this project.  I didn’t see it on my first pass, but maybe I missed it. 

            My point about wireless meshes still stands, while battlenet is an impressive experiment, there is a long distance between enthusiasts experimenting with technology and what your stated goals are.  If you can do it, great (prove me wrong, please!), but I’d advise a method-agnostic approach, so it generally doesn’t matter if you’re using a mesh, point to point, or other topology or media. 

            If you can come up with a good general convention for deployment, that may be even more helpful than a brilliant protocol solution.  In short, I’m just saying go with what works, not necessarily what’s most technologically attractive.  Maybe it’s a mesh, maybe it’s not.  Perhaps this is already obvious to you, in which case I apologize for the bandwidth waste. 

            I run a very large (~60K node) and extremely heterogeneous network, with a focus on reliability.  So admittedly my standards for performance are very high, perhaps higher than is applicable for this project’s goals.  Best effort would make a big difference when your trying to defeat a total blackout after all. 

            Also, I’m not in the habit of saying things that aren’t true, thank you very much.  I was involved with a prototype installation of the old Cisco outdoor mesh AP’s that used the old Richochet cases(and apparently still do, which is somewhat hilarious).  I wasn’t impressed, particularly at layer 3.  Apparently that’s evolved into the aironet 1510 now, but I suspect that it’s an entirely different animal at this point.  The prototypes, even with good antennas, had scalability and reliability issues that quickly outweighed the rest.  Power and connectivity to the traditional network still had to be solved, so convenience of deployment was somewhat limited, particularly with the power requirements.  But that was in a campus environment with fiber available, so obviously a different situation with different priorities.  And it was pre-N, so that changes the landscape somewhat as well on the RF side, where there is the most vulnerability for failure.  (And fail it did, frequently). 

            One of the most valuable lessons I’ve learned is that redundancy is not a panacea for reliability.  In fact, it often counter-intuitively decreases reliability, particularly in situations where control is less than total for configuration changes.  So inherently, there is a significant technical challenge to getting a mesh to work consistently without undesirable dynamic behavior, particularly with many nodes.  It looks like there is now a lot of interesting work being done to solve those issues, and I do look forward to seeing the results at scale.  But I also encourage the simplest approach to serve a need, whatever that ends up being. 

          6. Anonymous says:

            Well, looking at the wiki many of my concerns are clearly already thought of.  Nice to see, it’s more fleshed out than I expected.  I presume that you’re going to implement DNSSec?  It’s non-trivial, but solves a lot of problems and would be on my “must” list for this project.  I didn’t see it on my first pass, but maybe I missed it. 

            My point about wireless meshes still stands, while battlenet is an impressive experiment, there is a long distance between enthusiasts experimenting with technology and what your stated goals are.  If you can do it, great (prove me wrong, please!), but I’d advise a method-agnostic approach, so it generally doesn’t matter if you’re using a mesh, point to point, or other topology or media. 

            If you can come up with a good general convention for deployment, that may be even more helpful than a brilliant protocol solution.  In short, I’m just saying go with what works, not necessarily what’s most technologically attractive.  Maybe it’s a mesh, maybe it’s not.  Perhaps this is already obvious to you, in which case I apologize for the bandwidth waste. 

            I run a very large (~60K node) and extremely heterogeneous network, with a focus on reliability.  So admittedly my standards for performance are very high, perhaps higher than is applicable for this project’s goals.  Best effort would make a big difference when your trying to defeat a total blackout after all. 

            Also, I’m not in the habit of saying things that aren’t true, thank you very much.  I was involved with a prototype installation of the old Cisco outdoor mesh AP’s that used the old Richochet cases(and apparently still do, which is somewhat hilarious).  I wasn’t impressed, particularly at layer 3.  Apparently that’s evolved into the aironet 1510 now, but I suspect that it’s an entirely different animal at this point.  The prototypes, even with good antennas, had scalability and reliability issues that quickly outweighed the rest.  Power and connectivity to the traditional network still had to be solved, so convenience of deployment was somewhat limited, particularly with the power requirements.  But that was in a campus environment with fiber available, so obviously a different situation with different priorities.  And it was pre-N, so that changes the landscape somewhat as well on the RF side, where there is the most vulnerability for failure.  (And fail it did, frequently). 

            One of the most valuable lessons I’ve learned is that redundancy is not a panacea for reliability.  In fact, it often counter-intuitively decreases reliability, particularly in situations where control is less than total for configuration changes.  So inherently, there is a significant technical challenge to getting a mesh to work consistently without undesirable dynamic behavior, particularly with many nodes.  It looks like there is now a lot of interesting work being done to solve those issues, and I do look forward to seeing the results at scale.  But I also encourage the simplest approach to serve a need, whatever that ends up being. 

      2. Anonymous says:

        That would be a simplistic version of a standard.  Use of abstraction layers (i.e. OSI layer model) means that you can skin the cat anyway you like, as long as you make it possible for your cat-skin-o-matic-9000 to play nice with others.  Standards enable a variety of options and provide the necessary coherence to enable interoperability.  They are a tool, not a mental subjugation. 

        I’d also say that modularity is intrinsic to the Internet, that’s what enabled it’s success.  So I don’t disagree that it’s necessary, but I also don’t think that these projects are *socially* interconnected enough to make the impact desired.  It’s fun and exciting to solve technical problems.  But frankly, the networking technologies already exist, and can be used pretty much as-is to solve this.  So trying to come up with yet another clever way to skin that cat is a waste of time. 

        What is more of a challenge is streamlining it at a deployment and systemic level to a new economy of scale that would allow an irresistible level of deployment saturation.  If you focus on making the system lightweight and reliable, and elegant, then I’ll be far more interested.  If it costs $10 a node, that may actually be too expensive, can it be driven down further?  Can you make it so ubiquitous that it’s just not possible to block any more?  And how do you bridge the system to the outside world?  UN relays? 

        The first thing that most think of for these projects is a wireless mesh, to solve the connectivity issues, as if that will just magically work at any real scale.  Yes, it can be made to work, but so far only by people who know what they’re doing, not as a self-regulating amorphous blob of devices.  The routing problems become quite unwieldy for a low-resource microprocessor, and I’ve yet to see a truly elegant solution to that, that didn’t cost a fortune. 

        I have personally deployed wireless meshes, on a professional level, so I have some experience with this.  They have improved, but it’s not as easy as many would think.  The performance isn’t anything to write home about either, RF being what it is. 

        Using simpler diverse technologies(physical media-wise), and simpler topologies would probably work better, faster and cheaper in the end.  Not as sexy, but probably more effective. 

      3. The Doctor says:

        Technically we do have a standard – we’re using RFC 6126 (https://tools.ietf.org/html/rfc6126 – better known as Babel) as our mesh routing protocol.  However, routers by definition can also act as protocol translators, and Byzantium can (and has!) been used to bridge meshes and networks running different routing protocols.  We also aim for modularity by making it easy for someone running a node to set up, configure, and operate different services as well as bridging between networks if they so choose.  We are likely going to automate this functionality for the beta release (alpha release is slated for the end of October of 2011).

    2. Riley Porter says:

      I kind of see your point.  However, I would say that “standards” and having the “one way” to do this is kind of backwards.  I mean if everyone did this sort or project the same way.  Then wouldn’t they all suffer from the same weaknesses?

      I see the need for people to be working together to solve this, But I think modular-ity I guess is the true answer.. Meaning.. Cell towers jammed?  Piggy back sat signals.  WIFI CELL dead?  Find a hardline and broadcast via rf mesh.. 

      Cool post mark thanks for the info!

    3. The Doctor says:

      Mesh networking is not a panacea for connectivity, but on the other hand what happens when the ISPs are ordered to shut down as they did in Egypt earlier this year?  The services that people were using to organize were rendered largely uncontactable, thus, one of our goals is to provide functional replacements to work around this.

      To our knowledge, no one is making mesh networking easy or fast to set up – that is one of our design goals.  As for large scale mesh deployments, may I point out that Battlemesh (http://battlemesh.org/) regularly builds meshes that span entire countries?  We’ve been learning from both their successes as well as their failures, and making our results and code public so that everyone can benefit from them.

      As to your statement about the Internet being resilient, may I again point out that earlier this year Egypt all but vanished from the global Net because their government commanded that a handful of routers in a pair of IXPs be configured to stop announcing routes?  The Internet is not resilient, not as long as orders can be cut to cause ISPs to stop carrying out the task of sending and recieving traffic or single runs of fibre be physically disconnected (as happened in California a few short years ago).  It has also been shown that even domain registrations may be subverted without so much as a by-your-leave.

      As for our ways and means, we invite you to browse our wiki and source code to see how we do things.  By all means, send us patches (all of our software is open source, under the GPLv3)!

    4. The Doctor says:

      Mesh networking is not a panacea for connectivity, but on the other hand what happens when the ISPs are ordered to shut down as they did in Egypt earlier this year?  The services that people were using to organize were rendered largely uncontactable, thus, one of our goals is to provide functional replacements to work around this.

      To our knowledge, no one is making mesh networking easy or fast to set up – that is one of our design goals.  As for large scale mesh deployments, may I point out that Battlemesh (http://battlemesh.org/) regularly builds meshes that span entire countries?  We’ve been learning from both their successes as well as their failures, and making our results and code public so that everyone can benefit from them.

      As to your statement about the Internet being resilient, may I again point out that earlier this year Egypt all but vanished from the global Net because their government commanded that a handful of routers in a pair of IXPs be configured to stop announcing routes?  The Internet is not resilient, not as long as orders can be cut to cause ISPs to stop carrying out the task of sending and recieving traffic or single runs of fibre be physically disconnected (as happened in California a few short years ago).  It has also been shown that even domain registrations may be subverted without so much as a by-your-leave.

      As for our ways and means, we invite you to browse our wiki and source code to see how we do things.  By all means, send us patches (all of our software is open source, under the GPLv3)!

  2. something l33t says:

    With all due respect, rocketryguy, it’s not about the technology. The technology has existed for years and it’s been proven to work in several large installations. We really are at the point where we can download some software and buy some commodity hardware and everything (pretty much) magically happens.

    The only thing standing in the way of a project like this becoming successful is the drive of those involved. You need to pool resources and organize minions to finish polishing different bits, then eventually focus on logistical issues, testing, etc. It’s time-consuming and laborious but definitely feasible.

    What’s weird is, right now is the perfect time to get people to focus on these kinds of projects because of how they can help people. But will it be the same environment in 10 years? Will people still focus on trying to improve communications for people who need them *without* focusing on dollar signs? I hope so, but i’m doubtful.

    ( P.S. success, like genius, is also 1% inspiration and 99% perspiration… they are probably not smarter than you, but they’re willing to give it a go anyway. wanna help? http://hacdc.org/cgi-bin/mailman/listinfo/byzantium )

  3. something l33t says:

    With all due respect, rocketryguy, it’s not about the technology. The technology has existed for years and it’s been proven to work in several large installations. We really are at the point where we can download some software and buy some commodity hardware and everything (pretty much) magically happens.

    The only thing standing in the way of a project like this becoming successful is the drive of those involved. You need to pool resources and organize minions to finish polishing different bits, then eventually focus on logistical issues, testing, etc. It’s time-consuming and laborious but definitely feasible.

    What’s weird is, right now is the perfect time to get people to focus on these kinds of projects because of how they can help people. But will it be the same environment in 10 years? Will people still focus on trying to improve communications for people who need them *without* focusing on dollar signs? I hope so, but i’m doubtful.

    ( P.S. success, like genius, is also 1% inspiration and 99% perspiration… they are probably not smarter than you, but they’re willing to give it a go anyway. wanna help? http://hacdc.org/cgi-bin/mailman/listinfo/byzantium )

  4. The Doctor says:

    Hey, Mark – could you please link to the project’s website?  It seems that a lot of the misunderstanding is due to the fact that there is only a link to ContactCon, and not our documentation or source code.

  5. Andrew Q Righter says:

    Excellent choice in contributors Gareth! Mark is an absolute gem, I am very lucky to have known him at DorkbotDC (where HacDC was formed!), HacDC and as a friend. May you have many interesting posts! :D 

    <3,

Comments are closed.

Discuss this article with the rest of the community on our Discord server!
Tagged

Biomolecular cryptologist, robotics hacker, bad skateboarder and all-around mad artist.

View more articles by R. Mark Adams

ADVERTISEMENT

Maker Faire Bay Area 2023 - Mare Island, CA

Escape to an island of imagination + innovation as Maker Faire Bay Area returns for its 15th iteration!

Buy Tickets today! SAVE 15% and lock-in your preferred date(s).

FEEDBACK