Morphex's blogologue (Life, technology, music, politics, business, mental health and more)

This is the blog of Morten W. Petersen, aka. morphex in various places. I blog about my life, and what I find interesting and/or important. This is a personal blog without any editor or a lot of oversight so treat it as such. :)

My email is

I am

An OGG/Vorbis player, implemented in Javascript.

My Kiva bragging page
My shared (open source) code on GitHub

Morphex's Blogodex

Me on Instagram

Zope hosting by Nidelven IT

Morten Petersen on Linkedin

Morten Petersen on Facebook

Morten Petersen on SoundCloud

Morten Petersen on MixCloud

Blogologue on Twitter

Older entries

Atom - Subscribe - Categories

Facebook icon Share on Facebook Google+ icon Share on Google+ Twitter icon Share on Twitter LinkedIn icon Share on LinkedIn

An eventful coding week for Simple TCP proxy

I've been mailing a bit on the python-users mailing list about STP this week, and there were some discussions there that were unfruitful - but I got some useful tips in the process:

This TCP proxy needs to scale and perform reasonably well, so the last thing I worked on was making sure that downloading large files via the proxy works reasonably well, in these commits:

In was interesting, and relieving to see that I could speed up the download process 3-4x by multiplying the "transfer buffer" by 8. A local download of the test.tar file was around 1.3 GB/s via lighttpd, and 122 MB/s via STP. After the buffer change, the speed increased to 449 MB/s. Which is more than what a gigabit connection to the internet can handle.

So, so far so good, and good enough.

The previous commit:

I got a tip for, and reduced the thread stack size to the minimum, which is from what I gather, necessary to ensure that it will run reliably on different platforms - as Linux has a rather big thread size and that could break on others if you're reliant on it.

I also implemented a "fall through", so that the thread sending data back and forth doesn't stop and wait for .N number of seconds, if data has just been sent back and forth.

And in the commit before that, I adjusted the sleep time for threads "polling" on whether or not they have become active:

which also increased the speed at which STP could deal with the connection queue.

There has been some questions on the Python list as to exactly what I'm building and why, but I think a fair description would be that it is a simple to understand, and run, TCP proxy which does some useful things out of the box.

Coming from the web development and hosting world, I'm tempted to add some HTTP protocol features as well, but that's something that belongs in another project, maybe using STP as the frame which it plugs into.

[Permalink] [By morphex] [A simple TCP Proxy (Atom feed)] [31 Jul 20:08 Europe/Oslo]