Share on Facebook
Share on Google+
Share on Twitter
Share on LinkedIn
An eventful coding week for Simple TCP proxy
I've been mailing a bit on the python-users mailing list about STP this week, and there were some discussions there that were unfruitful - but I got some useful tips in the process:https://mail.python.org/pipermail/python-list/2022-July/9070...
This TCP proxy needs to scale and perform reasonably well, so the last thing I worked on was making sure that downloading large files via the proxy works reasonably well, in these commits:
https://github.com/morphex/stp/commit/3328a45d5314040abd37fa...
https://github.com/morphex/stp/commit/5e477bf5756e2763db799c...
In was interesting, and relieving to see that I could speed up the download process 3-4x by multiplying the "transfer buffer" by 8. A local download of the test.tar file was around 1.3 GB/s via lighttpd, and 122 MB/s via STP. After the buffer change, the speed increased to 449 MB/s. Which is more than what a gigabit connection to the internet can handle.
So, so far so good, and good enough.
The previous commit:
https://github.com/morphex/stp/commit/5e477bf5756e2763db799c...
I got a tip for, and reduced the thread stack size to the minimum, which is from what I gather, necessary to ensure that it will run reliably on different platforms - as Linux has a rather big thread size and that could break on others if you're reliant on it.
I also implemented a "fall through", so that the thread sending data back and forth doesn't stop and wait for .N number of seconds, if data has just been sent back and forth.
And in the commit before that, I adjusted the sleep time for threads "polling" on whether or not they have become active:
https://github.com/morphex/stp/commit/9910ca8c80e9d150222b68...
which also increased the speed at which STP could deal with the connection queue.
There has been some questions on the Python list as to exactly what I'm building and why, but I think a fair description would be that it is a simple to understand, and run, TCP proxy which does some useful things out of the box.
Coming from the web development and hosting world, I'm tempted to add some HTTP protocol features as well, but that's something that belongs in another project, maybe using STP as the frame which it plugs into.
[Permalink] [By morphex] [A simple TCP Proxy (Atom feed)] [31 Jul 20:08 Europe/Oslo]
A simple TCP Proxy
So as I was working on the SMPS, simple message passing system, I looked at ways of protecting it on the public net.Port knocking was one thing I looked at, and although it looked fine, I thought this was an opportunity to hone my Python/programming skills, and freshen up a bit on programming.
So I ended up writing a simple TCP proxy, and a working copy is here:
https://github.com/morphex/stp/tree/e781e84eee7b510879593d1a...
The test_http.sh script will setup a proxy in front of a local webserver.
The STP seems a bit sluggish when a lot of connections are made, but I'll have a look at how to optimize that performance.
[Permalink] [By morphex] [An SSL TCP client/server implementation in Python (Atom feed)] [26 Jul 23:52 Europe/Oslo]
Encountering Python crash on numerous client connects to socketserver
On the SMPS package, I've refactored a bit since the last blog post, and I also encountered what looks like a bug in Python.Python can randomly crash if too many client connections are made to the socketserver, and it crashes in such a way that it is difficult for me as a regular Python programmer to understand what the problem could be.
So although I was tempted to do a bit of C debugging and poke around, I do have limited time and energy, so I did the smart thing and patched it up so that it is easy to reproduce the problem:
https://github.com/morphex/smps/commit/59fbc78ebddcc3a68c6ea...
and then I created an issue in the Python bug tracker:
https://github.com/python/cpython/issues/95204
[Added later:] Aha. So it turns out I was quitting threads, and this gave an unclear error message. Thanks to Jeff Epler for pointing that out.
That got worked on in this commit:
https://github.com/morphex/smps/commit/9b539075627dd2016fca6a98f70913f21671e072
and the parent commit. Now with > 400 test connections I get an SSL handshake timeout, but, that's in ssl.c and probably due to some configuration setting somewhere.
[Permalink] [By morphex] [An SSL TCP client/server implementation in Python (Atom feed)] [24 Jul 18:13 Europe/Oslo]
Thread-safe message database for SMPS
I hacked a little more on the SMPS project today, which resulted in the following commit:https://github.com/morphex/smps/commit/8c565c561c0a9593f4d5e...
A pretty straight-forward message passing queue, that expires messages after n number of seconds. I guess the only thing left now before it can be put into practical use, is adding some sort of handshake with a secret, so that it is possible to protect the message queue from unauthorized access.
[Permalink] [By morphex] [An SSL TCP client/server implementation in Python (Atom feed)] [18 Jul 17:37 Europe/Oslo]
An SSL TCP client/server implementation in Python
Related to work I found myself needing a solution to pass messages, so I started working on something simple, just to get the basics down.I started out with the idea of doing it over HTTPS, but after a bit, I decided to go for a more minimalistic solution, using just TCP over SSL.
I've implemented a simple example of how to implement an SSL protected client/server message passing solution in Python 3.9. I googled a bit, and couldn't find a good, solid, simple example, so I figured the first iteration of the Simple Message Passing System could serve as an example, for others as well.
The committed state of the files [edit: for that first iteration are] here:
https://github.com/morphex/smps/tree/5e519cbbdf254c09ae3fe4d...
[Permalink] [By morphex] [Python and web (Atom feed)] [17 Jul 20:33 Europe/Oslo]
Some more easy sunday accounting hacks
I added some code to the ravencoin-taxman tool today:https://github.com/morphex/ravencoin-taxman/commit/55f47b67f...
which enables the calculation of earnings in a local currency. So if you're mining for example, it is possible to get the entire earnings in a year, into the local currency, which should be sufficient for accounting and tax purposes for that year.
The commit also included a bit of refactoring, of I guess you can call it standard debug stuff, to lessen the amount of code necessary to write later. Support for CSV files using ; as a separator was also added, the Norwegian bank uses that as a separator in its datafiles, as , is the decimal separator in the Norwegian locale.
I also added a test, to illustrate the command line and how to operate it:
https://github.com/morphex/ravencoin-taxman/commit/e08301ba0...
[Permalink] [By morphex] [Ravencoin and ethereum (classic) accounting tools (Atom feed)] [19 Jun 20:46 Europe/Oslo]
Comparing apples and oranges
So finally for this sunday hacking, I added some tests for the ravencoin-taxman project, and before that, I added RVN-USD calculation for transactions; giving (an acceptable) USD valuation of any given transaction:https://github.com/morphex/ravencoin-taxman/commit/170f2167b...
The code of this project looks cleaner than then one for the ethereum-classic-taxman project; probably because I'm getting back into the groove of programming, but also because they are fairly similar, and this could be called a re-write of sorts.
Refactoring seems to be a key part of productive programming, and necessary to manage any software project with more than a few pages of code. I see now that there will be at least a 3rd project between ethereum-classic-taxman and ravencoin-taxman, with code utilized by both.
[Permalink] [By morphex] [Ravencoin and ethereum (classic) accounting tools (Atom feed)] [05 Jun 19:46 Europe/Oslo]
A little sunday refactoring
Making a little progress in the ravencoin-taxman accounting script today, refactored date parsing into its own function in this commit:https://github.com/morphex/ravencoin-taxman/commit/a6b8089cd...
There are some that might argue that a function (in any programming language) shouldn't take up more than one console window - if it's bigger than that it should be broken up.
I kind of agree, it's a good guiding principle. In this case, the function was reduced to less than half the size, and in the process, I guess you can say it become better documented as well, because the old function is easier to read and the new parse_date function also has a docstring with a little information.
[Permalink] [By morphex] [Ravencoin and ethereum (classic) accounting tools (Atom feed)] [05 Jun 11:00 Europe/Oslo]
Exchange rate file parsing for ravencoin taxman
I hacked a little more on thehttps://github.com/morphex/ravencoin-taxman/
Ravencoin accounting tool today, it takes a lighter approach than ethereum-classic-taxman, downloading a CSV file of transactions instead of running through all the blocks in a blockchain node to find relevant transactions.
Since it does that, it makes things a bit less mentally demanding to work with, and I need to get FIFO sorted out for accounting purposes, so that it is possible to calculate loss and gains for capital tax.
Ravencoin-taxman works with data from different sources, one part is the Ravencoin transactions, another is the rate in USD for RVN at any given time (day) and finally a local currency is often necessary, in my case NOK, Norwegian kroner, to calculate things correctly for accounting and tax purposes.
Working a bit with CSV files, it is easy to appreciate their simplicity, on the other hand, they're not what you can call "well-defined" formats, for example the date fields in them can have different meanings based on where the CSV file is generated. YY/MM/DD or YY/DD/MM or even DD/MM/YY.
So I worked on that today. I think there is a little CSV tool that can be refactored into a separate Python package for working with CSV files, or data that isn't strictly declared or formatted.
CSV files are great in their simplicity and utility, but as this code shows:
https://github.com/morphex/ravencoin-taxman/commit/62fe3c364...
It takes quite a bit of work to make sure that data from CSV files is interpreted correctly, and if you're working with a limited amount of data, it is possible for that code to interpret a date wrong, switching the month and day fields the wrong way around.
[Permalink] [By morphex] [Ravencoin and ethereum (classic) accounting tools (Atom feed)] [29 May 14:40 Europe/Oslo]
JSON viewer for JSON database
I was looking to get a little done on the ethereum-classic-taxman accounting tool today, and thought a bit outside-the-box, what could I need in there that isn't a direct priority.The tool uses JSON databases, I switched a little while back because there could be security issues related to using a Python pickle database as the database backend.
An added benefit of using JSON is that its content is easy to view, and for example for debugging purposes, I thought it could be a good thing to have a tool that creates a view of the data that is easy to navigate and view. For example for debug purposes.
So I created this little script:
https://github.com/morphex/ethereum-classic-taxman/blob/main...
There are graphical JSON viewers on Ubuntu for example, but this little script can also have its output piped into a file, so that a database can be edited by hand in an editor. Or it could be piped to less on Linux/UNIX for viewing and searching.
On a related note, I saw some people lost their savings on the recent Luna/Terra crash. On the upside, I guess now is a chance to make a bet that the new variant with a massively higher amount of coins minted will succeed.
[Permalink] [By morphex] [Python and web (Atom feed)] [15 May 21:33 Europe/Oslo]
Ravencoin and ethereum (classic) accounting tools
So I've been hacking away at the accounting tool for ethereum (classic) today, adding support for converting the USD rate for ETH/ETC to another currency, based on data provided in the CSV format.It's mostly here:
https://github.com/morphex/ethereum-classic-taxman/commit/08...
I had to code my way around one thing in that respect, and that's because the USD-NOK rate for example is provided by the central bank, and they don't provide rates for Saturday, Sunday etc. So when transactions are made on days like that, the nearest, previous USD-NOK rate is used.
It seems reasonable to solve it that way, because something tangible (known) is better and more logical that whatever rate might come the day or even days after the transaction.
I also started work on another crypto currency accounting tool today
https://github.com/morphex/ravencoin-taxman/
Ravencoin Taxman. This takes a different approach from ethereum-classic-taxman, as it works with CSV data retrieved from a block explorer service. It doesn't download transactions and blocks from a node.
It takes much less (programming) time and (computer) resources to do it that way, but I also noticed that it was difficult to get CSV exports of ETC transactions on major / well known ETC block explorers today. This shows that it is useful to have a tool that is independent of for example block explorers, something that will work with just a node available.
The ravencoin-taxman code will need to have some crypto (RVN-USD) rate and fiat rates databases as well, I think I might just copy over most of the code for that from the ethereum-classic-taxman project. It doesn't seem like there will be much if any changes to those code bits in the future, and I don't see any obvious way now of making a package that will fit for both projects.
So I guess the only major bit left for both projects is getting the valuation (crypto currency * crypto currency rate in USD) combined with FIFO correct. That would pave the way later for example for tax reduction later, choosing the crypto-currency with the highest value when received for outgoing transactions. Which leads to less taxes due.
[Permalink] [By morphex] [Python and web (Atom feed)] [07 May 23:56 Europe/Oslo]
Some more work on an Ethereum (classic) accounting tool
So, I've hacked some more on the tool I'm building for accounting purposes.I guess since the last time I've posted on it, there are mainly two things I've been working on, one is valuation of crypto currency, the other is correctness of generated CSVs.
I've followed a simple principle when it comes to valuation of the crypto currency; and that is, first in, first out. Other options could be first in, highest value out, or first in, lowest value out.
It looks like in Norway, there is also the option of first in, whichever you want out; meaning you can receive crypto, and for tax purposes, choose which crypto currency you sell first. Which could be useful for tax purposes, to increase or decrease the tax owed due to dealings with crypto currency.
But it's a bit to keep track of, and I think I've gotten most of it done.
Another part however, is making sure the values in the CSV / Spreadsheet are correct, and I noticed that somewhere along the line of transactions, the account balance was off, compared to the state in etherscan.io for example. So I started looking, and figured out that it was due to a transaction that was registered, but didn't complete, because it ran out of gas. The max fee to complete the transaction was too low.
But as the transaction is still registered, you still have to pay the gas fee, so for that transaction, the value of the crypto transferred to another account is zero, and the gas fee still needs to be deducted.
You can see more about this on the go-ethereum issue tracker
https://github.com/ethereum/go-ethereum/issues/24768
there is also information in the commit:
https://github.com/morphex/ethereum-classic-taxman/commit/1a...
I think this shows, really well, how good Python integrates with the command line, and how easy it is to get something done in Python. A handful of changed lines, and it is possible to manually exclude a set of comma-separated transactions.
Of course this is also due to knowledge of how Python works, but yes, a great scripting and prototyping language Python is.
[Permalink] [By morphex] [Ethereum (classic) accounting data generator for Python (Atom feed)] [27 Apr 15:57 Europe/Oslo]
Ethereum (classic) accounting data generator for Python
Here's a python tool to generate CSV files suitable for accounting purposes, for incoming and outgoing Ethereum (classic) transactions:https://github.com/morphex/ethereum-classic-taxman
It will start with one main address, and then generate a CSV with transactions for that address, and one CSV for each address that receives ETH/ETC from that main address.
Crypto currencies were hyped up for a while, but given how much time has gone by, it looks like they are here to stay. I got some .com vibes from this crypto hype, but given that crypto currencies and related technologies will make finance-related activities such as loans cheaper and more accessible for the lender, and transactions in general cheaper and more flexible, there is little doubt left in my mind that it will work.
Betting on the right horse(s) is another matter however.
Regardless, taxes have to be paid, so I created this script that will connect to a geth/core-geth node and download all transactions related to a given address. I saw there were different services that offered x amounts of queries for y amount of money, and that's nice, but as a brogrammer and a small business owner I liked the idea of being able to be completely independent of such services.
The script comes with a test and test address 0x222f266B603e4fa6D5605b5ec1F1C86E35C19F7D (not owned by anyone I know), and running it generates the following CSV:
From,To,Hash,Value,Value USD,GAS Price ETH,Gas price USD,GAS Price Gwei,Block,Timestamp,Timestamp date,Timestamp time,Exchange rate
0x45d022c169c1198c29F9CBe58C666fc8D1Bb41f1,0x222f266B603e4fa6D5605b5ec1F1C86E35C19F7D,0xbdf64741321bf426aaf6c8903069b907335856c6a5756f0429d3f504d8b52cb0,0.806234,22.030682,0.000021,0.000574,1000000000,14671158,1646599200,2022-03-06,21:40:00,27.325415
Which looks right to me. I've also tested it against other ETH addresses and the calculations look correct. The precision of the last column rate was too high and it created a number which wasn't real, so just now I added a 6-point precision to floats generated in the CSV. This is also just enough to capture the standard low fee for ETC transactions.
[Permalink] [By morphex] [Python and web (Atom feed)] [03 Apr 10:21 Europe/Oslo]
A python script to calculate placement of poles for a roof
I've been working on the cabin, lately clearing out a tree behind the cabin that was a threat to the roof.Some blog posts about some of the work on the cabin here: http://blogologue.com/search?category=1591541232X85
Anyway, I need to setup some new poles for the porch / balcony, as there is a lot of rot in the parts below the roof, and those parts support the roof. So I can't just rip it out.
I wrote a script some time ago, and was going to use that today as I took finer measurements of the new beam that will carry the roof and how long an extended beam will have to be. As I was reading the script I couldn't make sense of it, so I re-wrote it, and posted a new one today:
https://github.com/morphex/misc/blob/master/holesforpoles3.p...
The old one is also in the same repository as holesforpoles.py, but you'll have to dig through the version history for that one.
I think it shows that in construction work, you have to cut once, measure twice, calculate thrice and maybe re-write the code quadrice. I wrote a holesforpoles2.py program today as well, but that was also discarded, more as a tool in the thinking process I guess.
[Permalink] [By morphex] [Python and web (Atom feed)] [13 Jun 17:14 Europe/Oslo]
An Open Source license for scripts, small code bits and programs
I have some miscellaneous code here:https://github.com/morphex/misc
Which hasn't been given a license yet, and I was wondering what license to give it. Just to make things clear, and make it easier to make use of the software. Using imapsync I was reminded of this.
It's been put up there for sharing and re-use obviously, so what are the best suggestions for a license? I was thinking BSD, GPL or LGPL.
Not sure whether to use GPL 2 or 3 though.
My email is morphex@gmail.com
[Update on the 22nd of May]
So I landed on the GPL, version 3. I think it's a good choice for me, as it is more explicit on what it grants and easier to use for others, and also makes it difficult for someone to make something closed-source and make money off it, without paying me.
I read a bit on GNU.org about the GPL version 3 and I liked what I read, one thing we don't need is more lawyers to get something useful done, or make it harder for small enterprises to innovate, or favor large companies or duopolies/monopolies.
[Permalink] [By morphex] [Python and web (Atom feed)] [16 May 20:17 Europe/Oslo]
An IMAP migration script
So, last December I got an email from the email hosting provider for Nidelven IT that the email server would be taken down in 6 months time.I didn't like the timing, as I was in court process, the third one in 7 years about my kids, but understand that things are expensive to maintain, a potential security hole etc. when they age.
So I wrote a little script that pretty much would do what was necessary.
Then after some thinking, it struck me that this is something others would need to do, and it wasn't completely straightforward. So I decided I could model a script based on the process I was using.
Here's the script:
https://github.com/morphex/misc/blob/master/migrate-imap.py
I found the imapsync script:
https://github.com/imapsync/imapsync
Which can be used to do the heavy lifting. I read the license file for that project, and although I'm not a lawyer, it seems straightforward enough that I can use it for my needs. It might've been a better choice to use a known license, but whatever, it is very minimalist and straightforward in its wording.
The script just lists folders for now, then I guess it could build a shell script file which calls imapsync, and that can be inspected and executed.
I was scratching my head a bit as I was writing the script, as the print() statement printed parentheses, then I saw I was running it with python 2 and not 3.
Other than that, I wasn't able to figure out a way to parse command line options for the script using just getopt, am I missing something or is there another module?
[Update on the 13th of May]
The script is now more or less complete. Gilles also responded to an email, saying imapsync imapsync also has --justfolderlists.
I couldn't quite understand the getopt module, haven't used it much before.
[Update on the 15th of May]
I'm now using this script to run imapsync, and imapsync is chugging away, at around 5-6 messages per second.
After posting the previous update I looked over the script a few times, and spotted a print() statement too much, in the generation of the shell script. That goes to show that just looking over code is useful.
Latest commit here: https://github.com/morphex/misc/commit/bcf34c85e93237e79f1920a7184bf0f4e7f5032f
I also made SSL mandatory, it's the kind of mistake someone could make, not using SSL, and it's easy to edit the script file afterwards to remove it, if you know what you're doing.
[Update on the 16th of May]
So the build migration script script is working, and imapsync looks like a sturdy piece of software, it ran through hundreds of thousands of messages in one run. Had to add a command line flag to copy header-less messages, imapsync suggested it might be Draft messages etc. and was on about a Message-ID. A second pass copying over remaining messages was uneventful.
[Permalink] [By morphex] [Python and web (Atom feed)] [09 May 19:53 Europe/Oslo]
A small script to find suitable, recurring dates
So, I recently was in a long process, to find an arrangement between me and the kids mom regarding the kids.We made a court settlement, and the setup is that I see them every 8th week. Through my volunteer work, I have commitments during Christmas and Easter, and I also plan to partake in a mini triathlon in the middle of July each year. A useful goal, to keep me motivated for exercise. I also meet family around this triathlon, and it's good to keep connected, and get connected.
I've come quite a long way physically, I feel much lighter and nimbler today than I did a year ago, five years ago.
So, I created this small script:
https://github.com/morphex/misc/blob/master/togetherness.py
Using the Python 3 datetime module. I haven't used the formatting before, so I'm not sure if I did it right, but it looks right on the console anyway.
I thought about adding checks for whether dates were around Christmas or Easter, but it didn't look like the datetime module could automatically figure out what period Easter falls on each year.
And having run the script, I found a usable start date just checking Easter dates manually.
[Updated 03/02/2021 10:30] Mark Lawrence chimes in on email, mentioning the dateutil module:
https://dateutil.readthedocs.io/en/stable/
When I think about it, the Python library documentation could do with a section at the end of each module, which briefly summarizes known, trusted modules and packages and their contents.
It's not directly under PSF control, but it would be useful to have nonetheless.
[Permalink] [By morphex] [Python and web (Atom feed)] [01 Feb 19:36 Europe/Oslo]
Adding some reporting functionality
I just added some code to the surveil app, the beginnings of what will be a reporting feature.https://github.com/morphex/surveil
(report.sh, report.py)
I found that the surveil app uses quite a bit of bandwidth when sending video, on the mobile network anyway, so I added an option to just store the videos locally a while ago.
That works, but then the surveillance app can stop making videos for whatever reason and nobody's the wiser.
So I figured it would be nice with a daily report of the videos created, so that it is possible to keep an eye on things, even if videos aren't mailed.
Another thing I'd like to add to the report is the file size and checksum of the video, so that it is possible to see if someone has tampered with the video.
But almost as soon as I got started I got tired, so I'll finish it off later.
Oh and I moved the find command out to a separate file; subprocess.Popen and shell command line arguments turned out to be a bit too complicated.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [03 Oct 21:30 Europe/Oslo]
Python quick-fix of broken router
I have a router which seems to "take the day off" every once in a while, and this started after I filled up all 4 Ethernet ports.Rebooting, the only fix I've found so far, fixes the problem, so that all 4 Ethernet ports start working again.
Rebooting the router gets boring and annoying after a while, so I decided to write a script to automatically reboot the router every hour.
That script is here:
https://github.com/morphex/netgear_reboot/
Didn't need to write any Python 2 or 3 specific hacks to make it work on either Python 2 or 3 either.
I was reading a Python 2-3 cheat sheet here:
http://python-future.org/compatible_idioms.html
And after getting used to using for example print(1), I like the idea of making things simpler and more consistent, an easier to learn syntax.
An added benefit is that I, and other who use two or more keyboard layouts, don't have to remember where the backticks are located on no_nb and en_us keyboard layouts.
[Permalink] [By morphex] [Python and web (Atom feed)] [15 Dec 15:51 Europe/Oslo]
Focusing on the simple things
This morning the internet became unavailable, after also being unavailable this weekend for several days.So I decided to take a look at my demo board which does surveillance with a webcam using the surveil app, surveil is here:
https://github.com/morphex/surveil
Well, one thing lead to another (...), and I locked myself out of the demo board.
Which was all-in-all a good thing, because when I decided to make things easy for myself, I instead ran the surveil app on my laptop, with the webcam attached there.
I was a bit surprised and embarrassed when the surveil script which should have given a helpful error message on the wrong command-line arguments, instead failed with a TypeError, because I had forgotten a comma.
So I fixed that, and noticed that the contents of the surveil directory (images taken with the webcam that could contain sensitive data) was included in the commit.
This was a big deal, and I included the surveil and longterm data storage directories in the .gitignore file.
Finally, I made the video capture device a configure option, as I don't use the webcam integrated in the laptop, but rather /dev/video1 - which is the device the USB Webcam gets when attached.
A commit of these changes is here:
https://github.com/morphex/surveil/commit/42743c4f3785e1e9dd...
Last week I drifted off in an interesting conversation on the Python-User list:
https://mail.python.org/pipermail/python-list/2018-November/...
Which I guess shows that I could've spent the time thinking about an interesting concept on more pragmatic things, like testing the surveil script on another machine.
Finally, I'm looking for a way to do testing; and I'm wondering of a good way to test that the command-line interface functions as expected as well.
I guess that's more of a functional test, but maybe there is a package which integrates unit and functional tests / integration tests.
[Update..]
I found Integrate which I'll be testing. Bonus points for supporting Python 2.7, as I have for example the software powering this blog (Issue Dealer, issuedealer.com) using Zope 2 and Python 2.7.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [07 Dec 14:52 Europe/Oslo]
Taking a look at my Python surveil(lance) app
So, I created this surveillance app in Python, to surveil (https://github.com/morphex/surveil) the room where I spend most of my time, just to make sure that nobody else visits it, without my approval.Before I wrote this app, I saw there were different applications out there, that could do some sort of surveillance, but I guess I recognized early on that I could easily mail images to myself, and that this was a good approach as it kind of disconnects the surveil app from outside dependencies, at least it doesn't have to have an internet connection up absolutely all the time, to function.
Another feature of mailing myself images compiled into videos, is that as soon as it comes into (in my case) GMail's system, there is a record of the video, and it is because of that, difficult to manipulate data, when a mail has been delivered.
Python was the language of choice, because I wanted to make things easy for myself, and Python is the language I've worked with the most, and it is easy to read and write things in Python.
Before this, I had dabbled a bit with ffmpeg, playing around with videos, adding effects to them and so on.
I'd say fortunately I dabbled with ffmpeg, because it is quite a powerful video processing package. The command line is not intuitive and user friendly, but once the command line is right, ffmpeg seems to be pretty stable.
I read about Python and the GIL (Global Interpreter Lock) today, and I remember this from years back. I guess over 90% of the programming work I've done to date, is in Python and Zope 2. Zope 2 had a way around the GIL and exploiting all CPU cores, and that was running a main database server, and then having several database clients/applications on each of their on process, which effectively went around the GIL, as each application was one process to the operating system. This worked well as the system was built for few writes to the database and many reads.
So, fortunately I dabbled with ffmpeg before this project, because I soon realized that threading could be a bit of a headache, so I opted for running ffmpeg as a subprocess of the surveil app, and using shell scripting and files to pass messages; so when ffmpeg is done processing a set of images into a video, it creates a file, and a thread with a loop in the surveil app monitors for these files, then mails the video and deletes the files related to that video afterwards.
Python is simple and intuitive, and that's the main reason I think, that it became my programming language of choice. It was easy to get started, interesting to explore Python via the interpreter, and there was an advanced web system which used Python as the main programming language.
Years back, I was pretty much obsessed with cross-platform portability, but with this surveil app, I'm creating a temporary (RAM-based) file system which is Linux-specific, and I'd be happy if it runs on *nix/POSIX systems. It's all in keeping things simple.
So I guess the point of this post was to underline that, yes, Python has a GIL, but in most instances it is a non-issue, and if there are CPU-intensive things, these can be forked out to a command-line tool, or you could even have an extension in C which does the heavy lifting. If I hadn't learnt ffmpeg, I could easily have spent a lot more time writing this app.
Another point that is worth mentioning, is the config.py file for this project, is easy to read, and goes with the Python style of simplicity, and goes one step beyond being a config file, it also has some (easy to understand) logic.
This project is a mix of different systems; Python to glue it all together, *nix file systems and scripting in a middle layer, and the ffmpeg application for the heavy-duty work. As far as I can tell, this is all good and stable, although the surveil app reboots about every 24 hours, so it is hard to know how well it runs with a 100% uptime.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [23 Oct 15:26 Europe/Oslo]
Adding (mandatory) SMTP authentication to my surveil app
So, I was thinking a bit lately, about adding a small feature to my surveillance app, so that it would send a mail whenever it was started.Went about to add that this evening, when I discovered that there were large gaps in time between mailed videos in my Surveillance folder/label.
After some searching and debugging, I found that the SMTP server (GMail's in this case), was rejecting emails from my IP. I guess that's just an automated response, when over time, I sent emails to myself, morphex@gmail.com, from myself and at some point that gets flagged as spam, because I wasn't logged in before sending emails.
Anyway, I guess it was naive to try to run something without logging into an SMTP with spam being what it is, so I added (mandatory) support for logging into the outgoing SMTP server today.
In addition to this, I guess it is nice to have the ability to reboot the host regularly, as a host system might for some reason become bogged down. So I added a config option to specify the amount of time between each reboot, in config.py:
https://github.com/morphex/surveil/blob/d1ff83091d9f33533ce9...
I think the config file is quite neat, because Python is neat; one can add config options, and even some logic to deal with config options in one file, and it seems natural.
At the end of the config file as it was on that commit, there is a statement that adds 5-60 minutes of delay to the reboot interval specified above - so that it is a bit harder to predict when a surveillance camera is rebooted.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [15 Oct 00:57 Europe/Oslo]
Changes to my Python surveillance (webcam, web camera) app
So, I created a surveillance app a week ago, because I felt it would be comforting to be able to see if somebody had been into my room.Since then, I had to make the mailer code, the codes that mails a compiled video to the given email address, a bit more robust:
https://github.com/morphex/surveil/commit/a02dbc05d78b71aea2...
As I one day discovered that the last mail sent to me with a video, was sent at 03:34, and looking at the log for surveil, I could see that DNS had stopped working.
Today I added another mailer feature, which is simply moving the mailer code into its own function, and then running that code as a loop forever, in a separate thread:
https://github.com/morphex/surveil/commit/146f1fe88511f94358...
Other notable changes are a separate configuration file, as well as parsing the output from fswebcam when a picture is taken, and if an error is detected, re-get the image:
https://github.com/morphex/surveil/commit/8c8b84fc46fd0863e7...
Finally, I added a script that can be started from Cron, so that the surveil app starts running as soon as the laptop/desktop/demoboard - whatever, boots up.
https://github.com/morphex/surveil/commit/6291603045579228fc...
I added the script, because it's easy to forget to include the PATH etc. - which leads to confusion, irritation and so on, when automating things.
Of all the things I did summarized here, I think the mailer hack is the best; I simply moved some code around and fired up another thread, so that images are taken and videos created and mailed, without delays which could create gaps in time where the room was not surveilled.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [24 Sep 15:01 Europe/Oslo]
A surveillance app (Python tree here I come!)
So, lately I've been taking it very easy on the activities front, since I've had a court-related matter to get to see the kids.So I've had some ideas, urges etc. to do technical stuff, artsy stuff etc. - and today I wrote myself a script that will run on a PC, or a demo board like Raspberry Pi, Orange Pi etc. - and take pictures on a webcam, and merge these pictures taken over time into a video, and mail them.
I started with Python (2), but figured that now is the time to take the plunge to Python 3 and don't look back.
I have to say, Python 3 isn't difficult, there are some changes in conventions etc. but nothing big.
So I was able to be productive in Python 3 straight away, and wrote this script:
https://github.com/morphex/surveil
Which will do what I mentioned above. And here's the source code tree as it was when I wrote this post:
https://github.com/morphex/surveil/tree/1cb5ceed7657e2635cca...
This video was generated from this script:
http://blogologue.com/surveil_out.webm
And impressively enough, each image the video was generated from, was 55-78 KB, while the entire video with 18 images was ~120 KB. I guess VP9 is effective at compressing, at the same time, a lot of the objects in the video are static.
[Permalink] [By morphex] [Python and web (Atom feed)] [17 Sep 00:08 Europe/Oslo]
Fixing up an old blog, transparent PNG embedded in Python code
So I've been fixing up on my blog lately, adding some iframe and javascript code, as well as backend code, to make it play my playlist from SoundCloud automatically.This blog is running on Zope 2, and the blog software was written by me.. it's not been maintained well though, so I've been fixing up some minor things here and there.
One of the things I needed to fix was that on the search page, some images were referenced that for some reason wasn't approved by the weblog layer, and raised old-style HTTP authentication boxes.
So I figured I'd fix this in an easy and quick way. I created a new image in The GIMP, 1x1 transparent, and saved it as a PNG with as little metadata as possible and level 9 compression.
Looking at the file afterwards in Python, it looks like this:
-rwxrwx---+ 1 Morphex None 117 aug 2 14:26 1x1_transparent.png
-rwxrwx---+ 1 Morphex None 68 aug 2 14:36 1x1_transparent2.png
Morphex@Morphex-PC /cygdrive/C/Users/Morphex.Morphex-PC/Documents
$ python
Python 2.7.10 (default, Jun 1 2015, 18:17:45)
[GCC 4.9.2] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f=open('1x1_transparent2.png')
>>> f.read()
'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4<MY BLOG LAYOUT LINE BREAK>
\x89\x00\x00\x00\x0bIDAT\x08\xd7c`\x00\x02\x00\x00\x05\x00\x01\xe2&\x05\x9b\x00\x00\x00\x00IEND\xaeB`\x82'
>>>
I updated the IssueDealerWeblog code to return this:
security.declarePublic('image')
def image(self, id=None, REQUEST=None, RESPONSE=None):
"""Returns an image related to the published issue."""
result = self.catalog_search(id=self.get_published_ids(),
get_local_image_ids=id,
meta_type='Issue')
if result:
return base.base.image.im_func(self, id=id, REQUEST=REQUEST, RESPONSE=RESPONSE)
else:
RESPONSE.setHeader('content-type', 'image/png')
RESPONSE.setBody('\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4<MY BLOG LAYOUT LINE BREAK>
\x89\x00\x00\x00\x0bIDAT\x08\xd7c`\x00\x02\x00\x00\x05\x00\x01\xe2&\x05\x9b\x00\x00\x00\x00IEND\xaeB`\x82', lock=True)
return
And now the search page returns without raising any authentication boxes. It's not the most purist way to just fail silently like this, but a quick fix that helps with the appearance and user-friendliness for a regular user.
[Edit: Adding paragraph..] And in case you're wondering, that image binary is released to the public domain, so you can use the Python string here anywhere. Or download the image
[Permalink] [By morphex] [Python and web (Atom feed)] [02 Aug 14:38 Europe/Oslo]