Morphex's blogologue (Life, technology, music, politics, business, mental health and more)

This is the blog of Morten W. Petersen, aka. morphex in various places. I blog about my life, and what I find interesting and/or important. This is a personal blog without any editor or a lot of oversight so treat it as such. :)

My email is morphex@gmail.com.

An OGG/Vorbis player, implemented in Javascript.

My Kiva bragging page
My shared (open source) code on GitHub

Morphex's Blogodex

News
Me on Instagram
Slashdot

Zope hosting by Nidelven IT

Morten Petersen on Linkedin

Morten Petersen on Facebook

Morten Petersen on SoundCloud

Morten Petersen on MixCloud

Blogologue on Twitter



Older entries



Atom - Subscribe - Categories


A python script to calculate placement of poles for a roof

I've been working on the cabin, lately clearing out a tree behind the cabin that was a threat to the roof.

Some blog posts about some of the work on the cabin here: http://blogologue.com/search?category=1591541232X85

Anyway, I need to setup some new poles for the porch / balcony, as there is a lot of rot in the parts below the roof, and those parts support the roof. So I can't just rip it out.

I wrote a script some time ago, and was going to use that today as I took finer measurements of the new beam that will carry the roof and how long an extended beam will have to be. As I was reading the script I couldn't make sense of it, so I re-wrote it, and posted a new one today:

https://github.com/morphex/misc/blob/master/holesforpoles3.p...

The old one is also in the same repository as holesforpoles.py, but you'll have to dig through the version history for that one.

I think it shows that in construction work, you have to cut once, measure twice, calculate thrice and maybe re-write the code quadrice. I wrote a holesforpoles2.py program today as well, but that was also discarded, more as a tool in the thinking process I guess.


[Permalink] [By morphex] [Python and web (Atom feed)] [13 Jun 17:14 Europe/Oslo]

An Open Source license for scripts, small code bits and programs

I have some miscellaneous code here:

https://github.com/morphex/misc

Which hasn't been given a license yet, and I was wondering what license to give it. Just to make things clear, and make it easier to make use of the software. Using imapsync I was reminded of this.

It's been put up there for sharing and re-use obviously, so what are the best suggestions for a license? I was thinking BSD, GPL or LGPL.

Not sure whether to use GPL 2 or 3 though.

My email is morphex@gmail.com

[Update on the 22nd of May]

So I landed on the GPL, version 3. I think it's a good choice for me, as it is more explicit on what it grants and easier to use for others, and also makes it difficult for someone to make something closed-source and make money off it, without paying me.

I read a bit on GNU.org about the GPL version 3 and I liked what I read, one thing we don't need is more lawyers to get something useful done, or make it harder for small enterprises to innovate, or favor large companies or duopolies/monopolies.

[Permalink] [By morphex] [Python and web (Atom feed)] [16 May 20:17 Europe/Oslo]

An IMAP migration script

So, last December I got an email from the email hosting provider for Nidelven IT that the email server would be taken down in 6 months time.

I didn't like the timing, as I was in court process, the third one in 7 years about my kids, but understand that things are expensive to maintain, a potential security hole etc. when they age.

So I wrote a little script that pretty much would do what was necessary.

Then after some thinking, it struck me that this is something others would need to do, and it wasn't completely straightforward. So I decided I could model a script based on the process I was using.

Here's the script:

https://github.com/morphex/misc/blob/master/migrate-imap.py

I found the imapsync script:

https://github.com/imapsync/imapsync

Which can be used to do the heavy lifting. I read the license file for that project, and although I'm not a lawyer, it seems straightforward enough that I can use it for my needs. It might've been a better choice to use a known license, but whatever, it is very minimalist and straightforward in its wording.

The script just lists folders for now, then I guess it could build a shell script file which calls imapsync, and that can be inspected and executed.

I was scratching my head a bit as I was writing the script, as the print() statement printed parentheses, then I saw I was running it with python 2 and not 3.

Other than that, I wasn't able to figure out a way to parse command line options for the script using just getopt, am I missing something or is there another module?

[Update on the 13th of May]

The script is now more or less complete. Gilles also responded to an email, saying imapsync imapsync also has --justfolderlists.

I couldn't quite understand the getopt module, haven't used it much before.

[Update on the 15th of May]

I'm now using this script to run imapsync, and imapsync is chugging away, at around 5-6 messages per second.

After posting the previous update I looked over the script a few times, and spotted a print() statement too much, in the generation of the shell script. That goes to show that just looking over code is useful.

Latest commit here: https://github.com/morphex/misc/commit/bcf34c85e93237e79f1920a7184bf0f4e7f5032f

I also made SSL mandatory, it's the kind of mistake someone could make, not using SSL, and it's easy to edit the script file afterwards to remove it, if you know what you're doing.

[Update on the 16th of May]

So the build migration script script is working, and imapsync looks like a sturdy piece of software, it ran through hundreds of thousands of messages in one run. Had to add a command line flag to copy header-less messages, imapsync suggested it might be Draft messages etc. and was on about a Message-ID. A second pass copying over remaining messages was uneventful.

[Permalink] [By morphex] [Python and web (Atom feed)] [09 May 19:53 Europe/Oslo]

A small script to find suitable, recurring dates

So, I recently was in a long process, to find an arrangement between me and the kids mom regarding the kids.

We made a court settlement, and the setup is that I see them every 8th week. Through my volunteer work, I have commitments during Christmas and Easter, and I also plan to partake in a mini triathlon in the middle of July each year. A useful goal, to keep me motivated for exercise. I also meet family around this triathlon, and it's good to keep connected, and get connected.

I've come quite a long way physically, I feel much lighter and nimbler today than I did a year ago, five years ago.

So, I created this small script:

https://github.com/morphex/misc/blob/master/togetherness.py

Using the Python 3 datetime module. I haven't used the formatting before, so I'm not sure if I did it right, but it looks right on the console anyway.

I thought about adding checks for whether dates were around Christmas or Easter, but it didn't look like the datetime module could automatically figure out what period Easter falls on each year.

And having run the script, I found a usable start date just checking Easter dates manually.

[Updated 03/02/2021 10:30] Mark Lawrence chimes in on email, mentioning the dateutil module:

https://dateutil.readthedocs.io/en/stable/

When I think about it, the Python library documentation could do with a section at the end of each module, which briefly summarizes known, trusted modules and packages and their contents.

It's not directly under PSF control, but it would be useful to have nonetheless.

[Permalink] [By morphex] [Python and web (Atom feed)] [01 Feb 19:36 Europe/Oslo]

Adding some reporting functionality

I just added some code to the surveil app, the beginnings of what will be a reporting feature.

https://github.com/morphex/surveil

(report.sh, report.py)

I found that the surveil app uses quite a bit of bandwidth when sending video, on the mobile network anyway, so I added an option to just store the videos locally a while ago.

That works, but then the surveillance app can stop making videos for whatever reason and nobody's the wiser.

So I figured it would be nice with a daily report of the videos created, so that it is possible to keep an eye on things, even if videos aren't mailed.

Another thing I'd like to add to the report is the file size and checksum of the video, so that it is possible to see if someone has tampered with the video.

But almost as soon as I got started I got tired, so I'll finish it off later.

Oh and I moved the find command out to a separate file; subprocess.Popen and shell command line arguments turned out to be a bit too complicated.

[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [03 Oct 21:30 Europe/Oslo]

Python quick-fix of broken router

I have a router which seems to "take the day off" every once in a while, and this started after I filled up all 4 Ethernet ports.

Rebooting, the only fix I've found so far, fixes the problem, so that all 4 Ethernet ports start working again.

Rebooting the router gets boring and annoying after a while, so I decided to write a script to automatically reboot the router every hour.

That script is here:

https://github.com/morphex/netgear_reboot/

Didn't need to write any Python 2 or 3 specific hacks to make it work on either Python 2 or 3 either.

I was reading a Python 2-3 cheat sheet here:

http://python-future.org/compatible_idioms.html

And after getting used to using for example print(1), I like the idea of making things simpler and more consistent, an easier to learn syntax.

An added benefit is that I, and other who use two or more keyboard layouts, don't have to remember where the backticks are located on no_nb and en_us keyboard layouts.

[Permalink] [By morphex] [Python and web (Atom feed)] [15 Dec 15:51 Europe/Oslo]

Focusing on the simple things

This morning the internet became unavailable, after also being unavailable this weekend for several days.

So I decided to take a look at my demo board which does surveillance with a webcam using the surveil app, surveil is here:

https://github.com/morphex/surveil

Well, one thing lead to another (...), and I locked myself out of the demo board.

Which was all-in-all a good thing, because when I decided to make things easy for myself, I instead ran the surveil app on my laptop, with the webcam attached there.

I was a bit surprised and embarrassed when the surveil script which should have given a helpful error message on the wrong command-line arguments, instead failed with a TypeError, because I had forgotten a comma.

So I fixed that, and noticed that the contents of the surveil directory (images taken with the webcam that could contain sensitive data) was included in the commit.

This was a big deal, and I included the surveil and longterm data storage directories in the .gitignore file.

Finally, I made the video capture device a configure option, as I don't use the webcam integrated in the laptop, but rather /dev/video1 - which is the device the USB Webcam gets when attached.

A commit of these changes is here:

https://github.com/morphex/surveil/commit/42743c4f3785e1e9dd...

Last week I drifted off in an interesting conversation on the Python-User list:

https://mail.python.org/pipermail/python-list/2018-November/...

Which I guess shows that I could've spent the time thinking about an interesting concept on more pragmatic things, like testing the surveil script on another machine.

Finally, I'm looking for a way to do testing; and I'm wondering of a good way to test that the command-line interface functions as expected as well.

I guess that's more of a functional test, but maybe there is a package which integrates unit and functional tests / integration tests.

[Update..]

I found Integrate which I'll be testing. Bonus points for supporting Python 2.7, as I have for example the software powering this blog (Issue Dealer, issuedealer.com) using Zope 2 and Python 2.7.

[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [07 Dec 14:52 Europe/Oslo]

Taking a look at my Python surveil(lance) app

So, I created this surveillance app in Python, to surveil (https://github.com/morphex/surveil) the room where I spend most of my time, just to make sure that nobody else visits it, without my approval.

Before I wrote this app, I saw there were different applications out there, that could do some sort of surveillance, but I guess I recognized early on that I could easily mail images to myself, and that this was a good approach as it kind of disconnects the surveil app from outside dependencies, at least it doesn't have to have an internet connection up absolutely all the time, to function.

Another feature of mailing myself images compiled into videos, is that as soon as it comes into (in my case) GMail's system, there is a record of the video, and it is because of that, difficult to manipulate data, when a mail has been delivered.

Python was the language of choice, because I wanted to make things easy for myself, and Python is the language I've worked with the most, and it is easy to read and write things in Python.

Before this, I had dabbled a bit with ffmpeg, playing around with videos, adding effects to them and so on.

I'd say fortunately I dabbled with ffmpeg, because it is quite a powerful video processing package. The command line is not intuitive and user friendly, but once the command line is right, ffmpeg seems to be pretty stable.

I read about Python and the GIL (Global Interpreter Lock) today, and I remember this from years back. I guess over 90% of the programming work I've done to date, is in Python and Zope 2. Zope 2 had a way around the GIL and exploiting all CPU cores, and that was running a main database server, and then having several database clients/applications on each of their on process, which effectively went around the GIL, as each application was one process to the operating system. This worked well as the system was built for few writes to the database and many reads.

So, fortunately I dabbled with ffmpeg before this project, because I soon realized that threading could be a bit of a headache, so I opted for running ffmpeg as a subprocess of the surveil app, and using shell scripting and files to pass messages; so when ffmpeg is done processing a set of images into a video, it creates a file, and a thread with a loop in the surveil app monitors for these files, then mails the video and deletes the files related to that video afterwards.

Python is simple and intuitive, and that's the main reason I think, that it became my programming language of choice. It was easy to get started, interesting to explore Python via the interpreter, and there was an advanced web system which used Python as the main programming language.

Years back, I was pretty much obsessed with cross-platform portability, but with this surveil app, I'm creating a temporary (RAM-based) file system which is Linux-specific, and I'd be happy if it runs on *nix/POSIX systems. It's all in keeping things simple.

So I guess the point of this post was to underline that, yes, Python has a GIL, but in most instances it is a non-issue, and if there are CPU-intensive things, these can be forked out to a command-line tool, or you could even have an extension in C which does the heavy lifting. If I hadn't learnt ffmpeg, I could easily have spent a lot more time writing this app.

Another point that is worth mentioning, is the config.py file for this project, is easy to read, and goes with the Python style of simplicity, and goes one step beyond being a config file, it also has some (easy to understand) logic.

This project is a mix of different systems; Python to glue it all together, *nix file systems and scripting in a middle layer, and the ffmpeg application for the heavy-duty work. As far as I can tell, this is all good and stable, although the surveil app reboots about every 24 hours, so it is hard to know how well it runs with a 100% uptime.

[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [23 Oct 15:26 Europe/Oslo]

Adding (mandatory) SMTP authentication to my surveil app

So, I was thinking a bit lately, about adding a small feature to my surveillance app, so that it would send a mail whenever it was started.

Went about to add that this evening, when I discovered that there were large gaps in time between mailed videos in my Surveillance folder/label.

After some searching and debugging, I found that the SMTP server (GMail's in this case), was rejecting emails from my IP. I guess that's just an automated response, when over time, I sent emails to myself, morphex@gmail.com, from myself and at some point that gets flagged as spam, because I wasn't logged in before sending emails.

Anyway, I guess it was naive to try to run something without logging into an SMTP with spam being what it is, so I added (mandatory) support for logging into the outgoing SMTP server today.

In addition to this, I guess it is nice to have the ability to reboot the host regularly, as a host system might for some reason become bogged down. So I added a config option to specify the amount of time between each reboot, in config.py:

https://github.com/morphex/surveil/blob/d1ff83091d9f33533ce9...

I think the config file is quite neat, because Python is neat; one can add config options, and even some logic to deal with config options in one file, and it seems natural.

At the end of the config file as it was on that commit, there is a statement that adds 5-60 minutes of delay to the reboot interval specified above - so that it is a bit harder to predict when a surveillance camera is rebooted.


[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [15 Oct 00:57 Europe/Oslo]

Changes to my Python surveillance (webcam, web camera) app

So, I created a surveillance app a week ago, because I felt it would be comforting to be able to see if somebody had been into my room.

Since then, I had to make the mailer code, the codes that mails a compiled video to the given email address, a bit more robust:

https://github.com/morphex/surveil/commit/a02dbc05d78b71aea2...

As I one day discovered that the last mail sent to me with a video, was sent at 03:34, and looking at the log for surveil, I could see that DNS had stopped working.

Today I added another mailer feature, which is simply moving the mailer code into its own function, and then running that code as a loop forever, in a separate thread:

https://github.com/morphex/surveil/commit/146f1fe88511f94358...

Other notable changes are a separate configuration file, as well as parsing the output from fswebcam when a picture is taken, and if an error is detected, re-get the image:

https://github.com/morphex/surveil/commit/8c8b84fc46fd0863e7...

Finally, I added a script that can be started from Cron, so that the surveil app starts running as soon as the laptop/desktop/demoboard - whatever, boots up.

https://github.com/morphex/surveil/commit/6291603045579228fc...

I added the script, because it's easy to forget to include the PATH etc. - which leads to confusion, irritation and so on, when automating things.

Of all the things I did summarized here, I think the mailer hack is the best; I simply moved some code around and fired up another thread, so that images are taken and videos created and mailed, without delays which could create gaps in time where the room was not surveilled.


[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [24 Sep 15:01 Europe/Oslo]

A surveillance app (Python tree here I come!)

So, lately I've been taking it very easy on the activities front, since I've had a court-related matter to get to see the kids.

So I've had some ideas, urges etc. to do technical stuff, artsy stuff etc. - and today I wrote myself a script that will run on a PC, or a demo board like Raspberry Pi, Orange Pi etc. - and take pictures on a webcam, and merge these pictures taken over time into a video, and mail them.

I started with Python (2), but figured that now is the time to take the plunge to Python 3 and don't look back.

I have to say, Python 3 isn't difficult, there are some changes in conventions etc. but nothing big.

So I was able to be productive in Python 3 straight away, and wrote this script:

https://github.com/morphex/surveil

Which will do what I mentioned above. And here's the source code tree as it was when I wrote this post:

https://github.com/morphex/surveil/tree/1cb5ceed7657e2635cca...

This video was generated from this script:

http://blogologue.com/surveil_out.webm

And impressively enough, each image the video was generated from, was 55-78 KB, while the entire video with 18 images was ~120 KB. I guess VP9 is effective at compressing, at the same time, a lot of the objects in the video are static.


[Permalink] [By morphex] [Python and web (Atom feed)] [17 Sep 00:08 Europe/Oslo]

Fixing up an old blog, transparent PNG embedded in Python code

So I've been fixing up on my blog lately, adding some iframe and javascript code, as well as backend code, to make it play my playlist from SoundCloud automatically.

This blog is running on Zope 2, and the blog software was written by me.. it's not been maintained well though, so I've been fixing up some minor things here and there.

One of the things I needed to fix was that on the search page, some images were referenced that for some reason wasn't approved by the weblog layer, and raised old-style HTTP authentication boxes.

So I figured I'd fix this in an easy and quick way. I created a new image in The GIMP, 1x1 transparent, and saved it as a PNG with as little metadata as possible and level 9 compression.

Looking at the file afterwards in Python, it looks like this:

-rwxrwx---+ 1 Morphex None 117 aug 2 14:26 1x1_transparent.png
-rwxrwx---+ 1 Morphex None 68 aug 2 14:36 1x1_transparent2.png

Morphex@Morphex-PC /cygdrive/C/Users/Morphex.Morphex-PC/Documents
$ python
Python 2.7.10 (default, Jun 1 2015, 18:17:45)
[GCC 4.9.2] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f=open('1x1_transparent2.png')
>>> f.read()
'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4<MY BLOG LAYOUT LINE BREAK>
\x89\x00\x00\x00\x0bIDAT\x08\xd7c`\x00\x02\x00\x00\x05\x00\x01\xe2&\x05\x9b\x00\x00\x00\x00IEND\xaeB`\x82'
>>>

I updated the IssueDealerWeblog code to return this:

    security.declarePublic('image')
    def image(self, id=None, REQUEST=None, RESPONSE=None):
        """Returns an image related to the published issue."""
        result = self.catalog_search(id=self.get_published_ids(),
                                        get_local_image_ids=id,
                                        meta_type='Issue')
        if result:
            return base.base.image.im_func(self, id=id, REQUEST=REQUEST, RESPONSE=RESPONSE)
        else:
            RESPONSE.setHeader('content-type', 'image/png')
            RESPONSE.setBody('\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4<MY BLOG LAYOUT LINE BREAK>
\x89\x00\x00\x00\x0bIDAT\x08\xd7c`\x00\x02\x00\x00\x05\x00\x01\xe2&\x05\x9b\x00\x00\x00\x00IEND\xaeB`\x82', lock=True)
            return

And now the search page returns without raising any authentication boxes. It's not the most purist way to just fail silently like this, but a quick fix that helps with the appearance and user-friendliness for a regular user.

[Edit: Adding paragraph..] And in case you're wondering, that image binary is released to the public domain, so you can use the Python string here anywhere. Or download the image No description available.

[Permalink] [By morphex] [Python and web (Atom feed)] [02 Aug 14:38 Europe/Oslo]