Tomboy Notes & WebDAV
So I mentioned previously that I was starting to use Tomboy notes on Ubuntu for keeping notes and syncing them. I was basically just saving them in my Dropbox folder, and as alluded to below I'm not really comfortable with all my stuff being on someone else's server without encryption.
I finally got around to setting up WebDAV on my apache server, and thought I would point Tomboy at it. Turns out not so easy. Turns out you need some unpackaged installs (and none of the instructions around work with the latest versions), then I hit this bug, then this one.
Working instructions to install WDFS/FUSE in 11.10 are as follows:
$ sudo apt-get install libneon27 libneon27-dev checkinstall libfuse-dev libglib2.0-dev libneon27-gnutls-dev
$ wget http://noedler.de/projekte/wdfs/wdfs-1.4.2.tar.gz
$ tar xzvf wdfs-1.4.2.tar.gz
$ cd wdfs-1.4.2
$ ./configure
$ make
$ sudo make install
$ sudo gpasswd -a user fuse
$ sudo chgrp fuse /dev/fuse
However once that is done you will find that Tomboy pumps out errors relating to the gnome keyring. This is because it is trying to use a deprecated environment variable holding the socket of the keyring daemon. A fix is apparently somewhere upstream, however there is a relatively simply workaround in that I can manually mount the remote WebDAV (using wdfs) and then simply tell Tomboy that is a local directory.
mkdir /mnt/disc
sudo wdfs http://disc0tech.com:90/webdav/ /mnt/disc0 -o allow_other
The I simply go into Tomboy preferences and setup local directory sync pointing at the above mount point. Bingo.
Last but not least I added this to fstab, to remount in future.
wdfs#http://disc0tech.com:90/ /mnt/wd2 fuse allow_other,uid=0,gid=500,nonempty,username=admin,umask=007,locking=simple,noauto
Posted at 11:09AM Apr 21, 2012 by disc0tech in tech | Comments[0]
Mail me when there is an ssh login on my server
Thought I would share. I wanted to be alerted to any shells opened on my server and was impressed with how straightforward it was to setup in /etc/profile.
I added some commands which use mailx to provide a dump of the relevant IP addresses, usernames and last logins in the email body.
touch /tmp/user
echo "USER LOGGED IN:" > /tmp/user
whoami >> /tmp/user
echo "****************" >> /tmp/user
echo "REMOTE CONNECTION DETAILS" >> /tmp/user
netstat | grep ssh >> /tmp/user
echo "****************" >> /tmp/user
echo "LOCAL WAN DETAILS" >> /tmp/user
ifconfig | grep inet >> /tmp/user
echo "****************" >> /tmp/user
echo "LAST LOGINS">> /tmp/user
last >> /tmp/userecho "User logged in" | mailx -s "SSH Login alert: mysite-DOT-com" me-AT-mysite.com
Posted at 03:32PM Jan 14, 2012 by disc0tech in tech | Comments[1]
ubuntu 11.10
Installed Ubuntu 11.10 and reduced my PC to a frozen boot "Booting system without full network configuration..."
Good work, Ubuntu.
Posted at 07:23PM Nov 12, 2011 by disc0tech in tech | Comments[0]
Dropbox vs Evernote
Following on from my previous post, I've been using Dropbox for the last week instead of Evernote. The main reason for this being data security, proprietary lock-in and cross-platform support. Here is my findings so far:
Platform support so far seems much better, it runs well on Ubuntu Maverick, with system tray integration and no noticeable performance overhead. Unlike Evernote it is also integrated directly into the file system - so I simply copy files to /home/adam/Dropbox rather than muck around attaching files to notes. Even email integration is in place (albeit via a third party, and it isn't possible for me to tag emails, they all go to the same folder).
It also seems to work fine on my iPhone, I downloaded an mp3 on my laptop, copied it to my dropbox folder and then was able to listen to it on my iPhone, it does seem less effective than Evernote at persisting data on my phone. Much like Spotify I need to mark things as favourites before they are synced locally, and the local storage appears to be capped at 1GB - which may not be enough for me.
Security is much better, I have successfully used TrueCrypt to encrypt a subset of the data. Whilst I need to ensure that the TrueCrypt software is running, and accept that the entirety of my encrypted volume will sync as one blob rather than incrementally, it's pretty much transparent. As an aside the encryption and features offered by TrueCrypt are the best I have seen for personal security.
I feel a lot less locked in with Dropbox, all my data is accessible on my filesystem. I can access my data using WebDAV, I can use it for my filesystem backups. In fact one of my next tasks is to setup the CLI version on my server and some cron jobs to copy to the Dropbox directory. Once I've finished doing that I think I will be a rather heavy user!
OK so now to the downside, and there is only one downside, SEARCH. The search facilities are basic, really basic. Compared to Evernote's advanced search syntax, image text recognition, and server side indexing - I feel like I have lost a major piece of functionality. A key use case for me is need to know something like a reference number for a train journey and being able to call it up quickly on my phone without going online. Dropbox isn't offering me this. For me this is a killer feature, but is mostly limited to a single use case - travel. For now I'm going to continue to use Evernote for travel and use Dropbox for everything else. Whilst I haven't done any research, I expect to be able to find a solution for Ubuntu that full-text searches my data, but for my Iphone I think I will have to live without it.
As a final note, I'm posting this from the Sapsen train between Moscow and St Petersburg, kudos Russia for having free wifi!
Posted at 11:35AM Jan 23, 2011 by disc0tech in tech | Comments[0]
Fuzzing & Automated QA
I've been reflecting for some time on automated test design and analysis. Most automated testing tools available focus on executing to a test specification, rather than on inventing new tests... the reasons for this are obvious, automatically generated tests do not know or care about your business priorities and put limited focus on happy path execution. So why do them?
Well, the answer is don't, at least not for testing new functionality. However, the benefit of using automatically generated tests for regression testing is greater. Theoretically, even the most obscure changes in application behaviour can be detected. Imagine testing an XML interface containing a free text field which is later persisted to a database, 90% of manual testers would only conduct boundary value / equivalence partitioning tests, without considering some of the very obscure possibilities - what about exhaustively testing non-escaped special characters, not just ASCII but multi-byte UTF formats? The benefit of that seems low - however there are a few scenarios where it starts to become very interesting...
Fuzz Testing is a concept that has been around since 1988, and it is defined by wikipedia as a software testing technique that provides invalid, unexpected, or random data to the inputs of a program. However the practical applications seem to be constrained to two specific areas... negative and security testing. If I need high levels of reliability I need to know that odd inputs aren't going to throw exceptions. For instance I need to ensure that if I pass more characters than the underlying data type or database field can handle, then my program will fail gracefully rather than throw a nasty exception. Consider a 20 character free text field parsed into with a VARCHAR(50) in the underlying schema, a manual tester may test 21 characters, but they are unlikely to test 51 unless they are a white-box tester.
Penetration testing is actually the only area where there seems to be significant active research into input generation techniques. These are being used to identify ways of crashing or otherwise manipulating applications. This 2009 webinar indicates significant enterprise uptake for finding vulnerabilities. Interestingly, dumb-fuzzing (simply random data) can be enhanced with computer-readable application specifications, or generic models to lead to 'smart-fuzzing'.... in the example above this could manifest through the injection of the text ";drop table users;".
I have been able to find little evidence of fuzzing extending into the functional regression arena, but I find this extremely interesting. Microsoft Research seems to be moving down this path with Pex, but is focussing on leveraging the IDE's inherent knowledge of inputs and outputs, which is essentially white-box testing and naturally limited in its scope. This is very interesting but is a fundamentally different principle. Which is probably why MS has called it automated exploratory testing rather than fuzzing.
My interest in this was initially piqued from working on a complex batch application with limited 'happy-path' documentation, and no user interface, simply thousands of static XML documents. Common human errors led to interesting and sometimes surprising behaviour. The solution I envisaged for regression testing was similar to a smart-fuzzing plugin framework but also moved into results analysis and automated specifications. When consistent fuzz tests are used across releases this could change the way we think of regression testing
My aspirational test framework, would have models for common components or interfaces, models for input generation techniques such as BVA, EP and pairwise algorithms, and more interestingly results comparison and evaluation techniques - not to determine whether the behaviour was correct, but whether it was the same as the last build.
Whilst this could conduct a massive amount of testing starting just with a non-fuzzed file, the first problem I envisaged was the unmanageable amount of data a test would generate and the limited value of many of the tests. The only way to overcome this problem would be to automatically refine the tests and isolate the trigger condition that led to a failure, a similar concept is reflected in this short python script written in 2010, which selectively un-fuzzes a fuzzed file until it isolates the cause of a SEG fault.
Moving away from testing and into the wider QA domain... having recently been reading on behaviour-driven testing, and in particular the TDD concept of tests as requirements and as an executable specification I see no reason why this technique cannot be extended to use fuzz testing to derive an executable technical specification. I vary the nomenclature here because executable specifications seem to focus on the desired functionality, rather than a holistic definition of the behaviour of the application. In simple terms I view this as using fuzz testing results to define the actual behaviour of the application, rather than defining the desired outcome in the tests and validating it. This is delving into an area I know less about, and even with common standards such as XML the computer readable specifications such as XSD are very limited (think cross field validation, I can't reflect in an XSD that the salutation cannot be 'Mrs' if a gender field holds 'Male'). There seems to be some effort to address this with OASIS, and a Java schema IDE and framework which allows the resulting schema to be ported back into the application for input validation.
The tools and techniques I have described could be used to create a very powerful QA framework, not just automating certain testing but creating a feedback loop into both documentation and programmatic input validation in a production environment. I'm interested in exploring this more with some like-minded QA professionals and building a proof of concept.... if this appeals to you please get in touch: adam-AT-disc0tech-DOT-com
Posted at 12:31PM Jan 22, 2011 by disc0tech in testing | Comments[0]
Goodbye Evernote...
I'm a heavy premium user of Evernote, but more than anything else it has given me a taster of the service I want, but can't have. Evernote isn't keeping up with my requirements, and I want to level up while I only have a year's worth of data to migrate.
I use Evernote in several ways:
- I forward important reference emails to a special email address, so I can read attachments later on Evernote on my IPhone or any computer. For example if I'm travelling I will always forward itineraries and booking confirmations.
- I make quick notes via the IPhone or desktop client
- I access data from other people's computers via a web client
- I take photos using my IPhone of things like business cards and notes written on the back of a fag packet. Then Evernote automatically stores it as a note.
There are a few things, that Evernote isn't doing for me:
SUPPORTING ALL MY DEVICES
About six months ago I got rid of Windows for good and switched to Linux. The only reason I had been holding back on doing this for some time was Evernote, being the only application I couldn't live without or run under Wine. This is surprising from a company which has the slogan "Remember Everything; Access Anywhere". Competitors support Linux, and given they already support Macs it should be an easy port.
There has clearly been demand for the last two years and this Facebook thread shows 260 comments. Whilst Evernote argue that the market share of Linux is not worth it, as Ian Forrester points out they happily developed a Windows Phone 7 version despite Microsoft's likely failure to be able to catch up with IOS and Android.
There is an open source alternative, NeverNote, which uses the Facebook API. However it is clunky and unstable, in my experience. Also when Evernote 4 was released some refactoring seemed to fix the problems with Wine, and after a bit of buggering around I can now run it under Wine on Linux... but the user interface performance is abysmal. The web access is slightly faster but it isn't good enough for my regular desktop use.
ENCRYPTING MY DATA
I'm obviously nervous about protecting my personal data, access to my Evernote data store would give someone an insane amount of detail about my life, personality, thoughts and friends. I recognise that the weakest physical link in the chain is likely to be access to my devices, rather than Evernote's security precautions. Data sent over the wire between my devices and the Evernote servers is SSL encrypted, so difficult to intercept. This only comes with the premium membership, which is fair enough I think.
My growing concern is that Evernote themselves have access to the data, and that there is an increasing of precedent of organisations (rather than individuals) successfully gaining access to internet based data.
As an example, the US criminal investigation into Wikileaks has subpoenaed Twitter, Google and Facebook to provide significant private data about a list of people, including an Icelandic MP (which is not going down well, incidentally). For users with a heavy online presence, which the recipients on the subpoena most likely are, the information that will be available can be data mined to an unparalleled extent. To some extent there is lots of data on Twitter and Facebook that was originally intended for the public domain, and therefore isn't a concern, but this is certainly not true of Evernote. Everything in my Evernote repo is intended to viewed only by me.
Evernote admittedly has provided some built-in encryption functionality but it is only really usable for an occasional note. What I want is for all the data to be encrypted by default, and my Evernote client applications to transparently decrypt the data, without requesting any passwords from me. There are a few reasons why Evernote aren't addressing it, to quote them from an interesting thread on their blog "Unfortunately, if we only stored an opaque encrypted backup of your database file on our server, we couldn't provide most of the features that our users love."
I don't agree with Evernote on a few things they think would be impossible, but recognise that any functionality which is performed server side needs to be done without the necessary encryption - this would definitely prevent searching images for text, transcribing audio notes as text and lots of partner applications would presumably add a lot less value.
I fundamentally do not want my data to be accessible at a location outside of my control, if compute power is the problem (which I am unconvinced of) I am happy for *my* devices to act as a peer to peer compute cloud.
IT'S NOT OPEN
I increasingly see the benefit of open source software in moving products forward. If Evernote was GPL'ed then we would have had a Linux client two years ago. We would probably also have encryption as an open source application would integrate with open standards like WebDAV. I should point out the Evernote does provide an API which can be used by consumers, but I don't really want to write my own application.
I've researched / tried a number of alternatives, including git (designed by Linus Torvalds as a peer to peer source control system)... right now I am writing this in Tomboy Notes which syncs with Ubuntu One across my Linux devices, I can access Ubuntu One notes and files via their website, and ironically Ubuntu have developed a Windows client - so I can use it cross platform. This might solve my cross platform issues, but it still isn't encrypted.
I have previously looked at Dropbox but I wrote it off initially as it doesn't support notes anywhere nearly as nicely as Evernote, nor does it have search. It does however have Iphone, Windows and Linux support. The tipping point for me was when I realised Tomboy natively integrates with both Dropbox and WebDAV.
Digging into the solution a bit further, I can apparently can mount it as a device in Linux and put an encryption layer in front, so all remotely persisted data is transparently secure. This solution is exactly what I want though, client driven encryption so even Dropbox cannot access my data, whilst they claim they already can't it seems unlikely me they can't reverse engineer my password.
It looks like there are multiple options, and pitfalls with the options available. Dropbox looks to have some awesome features and I'm going to try it out with TrueCrypt and Tomboy. I'm not sure how I will be able to get decryption working on my IPhone, so I may have to settle for only encrypting parts of the repo.
Posted at 04:34PM Jan 15, 2011 by disc0tech in tech | Comments[1]
Alternative to dynamic DNS
After watching this amusing defcon presentation I decided to beef up my own security.
I was in the middle of signing up for an expensive dynamic DNS service the other night so I could always locate the external IP of any of my devices (say if they were stolen like in the video... and then logged on to the perps own home internet...), when I realised that my use of it would be so limited I would be better off coming up with something a little more crude.
I knocked up the below to pull my ip and hostname and dump it into a local file with a message like "yourhost was last seen at 82.45.238.116 at Wed Jan 05 20:59:45 +0000 2011". This is regenerated and copied to a remote web server every hour..
require 'rubygems'
require 'open-uri'
require 'socket'
$hostname = Socket.gethostname
srcFile = "/tmp/#{$hostname}"
tgtFile = "/var/www/ip/#{$hostname}"
$ip = open("http://myip.dk") { |f| /([0-9]{1,3}\.){3}[0-9]{1,3}/.match(f.read)[0].to_a[0] }
$hostname = Socket.gethostname
$time = Time.now.inspect
File.open("/tmp/#{$hostname}",'w') {|f| f.write("#{$hostname} was last seen at #{$ip} at #{$time}") }
Posted at 09:11PM Jan 05, 2011 by disc0tech in tech | Comments[0]
State of the Art
I had forgotted about this for years, the amiga demo scene (which I was introduced to through fellow geeks, but ended up sending me away from technology) was a confluence of assembly code, top notch amiga graphics technology and electronic music. State of the Art is the seminal example of this, and I just rediscovered it *thanks youtube
Posted at 12:53PM Nov 21, 2010 by disc0tech in tech | Comments[0]