Data Expedition, Inc.
News
Articles, events, announcements, and blogs
Data Expedition, Inc.
News
Articles, events, announcements, and blogs
When I first began to encounter the speed limitations of TCP, it seemed like that should be an easy problem to solve. It was obvious that data wasn't moving as fast as it should. If I could see that the speed should be faster, how difficult could it be to create an algorithm that figured it out?
Years of experience and research revealed the answer: very difficult.
As is often the case when translating human intuition into algorithms, we humans have an advantage. We know things the computers can't, like how fast our internet connection "should" be. At least, we think we know. But in working with many hundreds of IT professionals over the years, I've found that the majority do not have an accurate idea of their system's performance capabilities. "Should" is a dangerous assumption.
Without reliable human input to say how fast a path should be, algorithms must depend on their own observations to figure out an ideal transport speed. The challenge is that there is very little to observe. The internet is effectively a black box: you drop a packet in and... it's gone. The only feedback you can hope for is to receive a different packet at some indefinite time in the future.
Working out what's happening to those packets after you drop them into the internet is a lot like the children's pool game Marco Polo. The "it" player is trying to find their way to a moving target while blind-folded, hobbled by water, and surrounded by yelling. Their only information is to shout out and listen through the noise for replies. This is a surprisingly apt metaphor for internet data transport protocols. Like the Marco Polo player, an algorithm trying to figure out how fast data can move has only the patterns of call and response to guide it.
Over the years I've spoken to participants from more than a dozen projects that started out confident they could come up with a novel data transport solution with just a few months of development work. Some managed to solve an edge case, usually one where human knowledge could give the algorithms a "peek" at their targets. But all gave up on the idea of quickly finding a scalable, general purpose solution.
Like kids playing in a pool, the environment of the internet is noisy and ever changing. Figuring out how to navigate that environment while blind and hobbled is difficult. There "should" be an algorithmic solution; and there is. The Multipurpose Transaction Protocol (MTP/IP) is the result of decades of experience and research, distilled into a set of tools that anyone can use. When you find yourself looking at a data transfer and thinking, "I know it should go faster," test that "should" with our software.