Share on Facebook
Share on Google+
Share on Twitter
Share on LinkedIn
Adding some reporting functionality
I just added some code to the surveil app, the beginnings of what will be a reporting feature.https://github.com/morphex/surveil
(report.sh, report.py)
I found that the surveil app uses quite a bit of bandwidth when sending video, on the mobile network anyway, so I added an option to just store the videos locally a while ago.
That works, but then the surveillance app can stop making videos for whatever reason and nobody's the wiser.
So I figured it would be nice with a daily report of the videos created, so that it is possible to keep an eye on things, even if videos aren't mailed.
Another thing I'd like to add to the report is the file size and checksum of the video, so that it is possible to see if someone has tampered with the video.
But almost as soon as I got started I got tired, so I'll finish it off later.
Oh and I moved the find command out to a separate file; subprocess.Popen and shell command line arguments turned out to be a bit too complicated.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [03 Oct 21:30 Europe/Oslo]
Focusing on the simple things
This morning the internet became unavailable, after also being unavailable this weekend for several days.So I decided to take a look at my demo board which does surveillance with a webcam using the surveil app, surveil is here:
https://github.com/morphex/surveil
Well, one thing lead to another (...), and I locked myself out of the demo board.
Which was all-in-all a good thing, because when I decided to make things easy for myself, I instead ran the surveil app on my laptop, with the webcam attached there.
I was a bit surprised and embarrassed when the surveil script which should have given a helpful error message on the wrong command-line arguments, instead failed with a TypeError, because I had forgotten a comma.
So I fixed that, and noticed that the contents of the surveil directory (images taken with the webcam that could contain sensitive data) was included in the commit.
This was a big deal, and I included the surveil and longterm data storage directories in the .gitignore file.
Finally, I made the video capture device a configure option, as I don't use the webcam integrated in the laptop, but rather /dev/video1 - which is the device the USB Webcam gets when attached.
A commit of these changes is here:
https://github.com/morphex/surveil/commit/42743c4f3785e1e9dd...
Last week I drifted off in an interesting conversation on the Python-User list:
https://mail.python.org/pipermail/python-list/2018-November/...
Which I guess shows that I could've spent the time thinking about an interesting concept on more pragmatic things, like testing the surveil script on another machine.
Finally, I'm looking for a way to do testing; and I'm wondering of a good way to test that the command-line interface functions as expected as well.
I guess that's more of a functional test, but maybe there is a package which integrates unit and functional tests / integration tests.
[Update..]
I found Integrate which I'll be testing. Bonus points for supporting Python 2.7, as I have for example the software powering this blog (Issue Dealer, issuedealer.com) using Zope 2 and Python 2.7.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [07 Dec 14:52 Europe/Oslo]
Taking a look at my Python surveil(lance) app
So, I created this surveillance app in Python, to surveil (https://github.com/morphex/surveil) the room where I spend most of my time, just to make sure that nobody else visits it, without my approval.Before I wrote this app, I saw there were different applications out there, that could do some sort of surveillance, but I guess I recognized early on that I could easily mail images to myself, and that this was a good approach as it kind of disconnects the surveil app from outside dependencies, at least it doesn't have to have an internet connection up absolutely all the time, to function.
Another feature of mailing myself images compiled into videos, is that as soon as it comes into (in my case) GMail's system, there is a record of the video, and it is because of that, difficult to manipulate data, when a mail has been delivered.
Python was the language of choice, because I wanted to make things easy for myself, and Python is the language I've worked with the most, and it is easy to read and write things in Python.
Before this, I had dabbled a bit with ffmpeg, playing around with videos, adding effects to them and so on.
I'd say fortunately I dabbled with ffmpeg, because it is quite a powerful video processing package. The command line is not intuitive and user friendly, but once the command line is right, ffmpeg seems to be pretty stable.
I read about Python and the GIL (Global Interpreter Lock) today, and I remember this from years back. I guess over 90% of the programming work I've done to date, is in Python and Zope 2. Zope 2 had a way around the GIL and exploiting all CPU cores, and that was running a main database server, and then having several database clients/applications on each of their on process, which effectively went around the GIL, as each application was one process to the operating system. This worked well as the system was built for few writes to the database and many reads.
So, fortunately I dabbled with ffmpeg before this project, because I soon realized that threading could be a bit of a headache, so I opted for running ffmpeg as a subprocess of the surveil app, and using shell scripting and files to pass messages; so when ffmpeg is done processing a set of images into a video, it creates a file, and a thread with a loop in the surveil app monitors for these files, then mails the video and deletes the files related to that video afterwards.
Python is simple and intuitive, and that's the main reason I think, that it became my programming language of choice. It was easy to get started, interesting to explore Python via the interpreter, and there was an advanced web system which used Python as the main programming language.
Years back, I was pretty much obsessed with cross-platform portability, but with this surveil app, I'm creating a temporary (RAM-based) file system which is Linux-specific, and I'd be happy if it runs on *nix/POSIX systems. It's all in keeping things simple.
So I guess the point of this post was to underline that, yes, Python has a GIL, but in most instances it is a non-issue, and if there are CPU-intensive things, these can be forked out to a command-line tool, or you could even have an extension in C which does the heavy lifting. If I hadn't learnt ffmpeg, I could easily have spent a lot more time writing this app.
Another point that is worth mentioning, is the config.py file for this project, is easy to read, and goes with the Python style of simplicity, and goes one step beyond being a config file, it also has some (easy to understand) logic.
This project is a mix of different systems; Python to glue it all together, *nix file systems and scripting in a middle layer, and the ffmpeg application for the heavy-duty work. As far as I can tell, this is all good and stable, although the surveil app reboots about every 24 hours, so it is hard to know how well it runs with a 100% uptime.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [23 Oct 15:26 Europe/Oslo]
Adding (mandatory) SMTP authentication to my surveil app
So, I was thinking a bit lately, about adding a small feature to my surveillance app, so that it would send a mail whenever it was started.Went about to add that this evening, when I discovered that there were large gaps in time between mailed videos in my Surveillance folder/label.
After some searching and debugging, I found that the SMTP server (GMail's in this case), was rejecting emails from my IP. I guess that's just an automated response, when over time, I sent emails to myself, morphex@gmail.com, from myself and at some point that gets flagged as spam, because I wasn't logged in before sending emails.
Anyway, I guess it was naive to try to run something without logging into an SMTP with spam being what it is, so I added (mandatory) support for logging into the outgoing SMTP server today.
In addition to this, I guess it is nice to have the ability to reboot the host regularly, as a host system might for some reason become bogged down. So I added a config option to specify the amount of time between each reboot, in config.py:
https://github.com/morphex/surveil/blob/d1ff83091d9f33533ce9...
I think the config file is quite neat, because Python is neat; one can add config options, and even some logic to deal with config options in one file, and it seems natural.
At the end of the config file as it was on that commit, there is a statement that adds 5-60 minutes of delay to the reboot interval specified above - so that it is a bit harder to predict when a surveillance camera is rebooted.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [15 Oct 00:57 Europe/Oslo]
Changes to my Python surveillance (webcam, web camera) app
So, I created a surveillance app a week ago, because I felt it would be comforting to be able to see if somebody had been into my room.Since then, I had to make the mailer code, the codes that mails a compiled video to the given email address, a bit more robust:
https://github.com/morphex/surveil/commit/a02dbc05d78b71aea2...
As I one day discovered that the last mail sent to me with a video, was sent at 03:34, and looking at the log for surveil, I could see that DNS had stopped working.
Today I added another mailer feature, which is simply moving the mailer code into its own function, and then running that code as a loop forever, in a separate thread:
https://github.com/morphex/surveil/commit/146f1fe88511f94358...
Other notable changes are a separate configuration file, as well as parsing the output from fswebcam when a picture is taken, and if an error is detected, re-get the image:
https://github.com/morphex/surveil/commit/8c8b84fc46fd0863e7...
Finally, I added a script that can be started from Cron, so that the surveil app starts running as soon as the laptop/desktop/demoboard - whatever, boots up.
https://github.com/morphex/surveil/commit/6291603045579228fc...
I added the script, because it's easy to forget to include the PATH etc. - which leads to confusion, irritation and so on, when automating things.
Of all the things I did summarized here, I think the mailer hack is the best; I simply moved some code around and fired up another thread, so that images are taken and videos created and mailed, without delays which could create gaps in time where the room was not surveilled.
[Permalink] [By morphex] [A surveillance app (Python tree here I come!) (Atom feed)] [24 Sep 15:01 Europe/Oslo]
A surveillance app (Python tree here I come!)
So, lately I've been taking it very easy on the activities front, since I've had a court-related matter to get to see the kids.So I've had some ideas, urges etc. to do technical stuff, artsy stuff etc. - and today I wrote myself a script that will run on a PC, or a demo board like Raspberry Pi, Orange Pi etc. - and take pictures on a webcam, and merge these pictures taken over time into a video, and mail them.
I started with Python (2), but figured that now is the time to take the plunge to Python 3 and don't look back.
I have to say, Python 3 isn't difficult, there are some changes in conventions etc. but nothing big.
So I was able to be productive in Python 3 straight away, and wrote this script:
https://github.com/morphex/surveil
Which will do what I mentioned above. And here's the source code tree as it was when I wrote this post:
https://github.com/morphex/surveil/tree/1cb5ceed7657e2635cca...
This video was generated from this script:
http://blogologue.com/surveil_out.webm
And impressively enough, each image the video was generated from, was 55-78 KB, while the entire video with 18 images was ~120 KB. I guess VP9 is effective at compressing, at the same time, a lot of the objects in the video are static.
[Permalink] [By morphex] [Python and web (Atom feed)] [17 Sep 00:08 Europe/Oslo]