## Stats on #rtmstats

Once in a while — well, at the end of a year, actually — Remember the Milk gently suggests you might want to brag to the community how many tasks you have completed in the passing year. Some treat it as a PR plug, some don’t, do as you want. What is interesting is the question: how many tasks others completed?

So I wrote a bunch of simple scripts to search Twitter regularly for #rtmstats, save the results and later parse the numbers from the tweets at large. The code is on the Github.

Below are two graphs showing the gather reports between 27 December 2012 and yesterday. Not much changes recently so I think those will remain accurate.

The first graph shows reports in time. It is a bit harsh, but when you change the roll period value in the field in lower left (i.e. the number of data points to average over) you will see the trend: the earlier the report, the larger number of tasks has been completed. This is of course expected, for when you complete large number of task you probably visit RTM more often and report faster.

The second graph shows the reports sorted by the number of completed tasks. The X axis describes the “top percentage” of the report, e.g. (45.7; 762) means that roughly that if you completed at least 762 tasks, you are in “top 45.7%”.

## Yoxos under Ubuntu 12.04

Just a quick note to myself and potential users of Yoxos under Ubuntu 12.04, which seems not to work in the default installation.

The problem is that after starting Yoxos Launcher and selecting “Advanced mode”, empty dialog appears and the application hands.

The cause is the missing library for rendering web pages, to put things simple. In previous (pre-11.10 I think) Ubuntu releases probably xulrunner was used to do so, but now that it’s missing from repos, SWT can’t find a library to handle the rendering.

In 12.04 the simplest solution is to install WebKit, as this can also be used by SWT, e.g.:

sudo aptitude install libswt-webkit-gtk-3-jni


## User certificates on Samsung Galaxy S Plus

Recently I bought a new Android phone and as I’m using it in WPA2-Enterprise network, I needed to import a client certificate.

All the tutorials about installing user certificates on your Android — both official ones and user-authored — mention to take your PKCS#12 file, put it on the root folder of your SD card and import in Android from Settings > Location and security > Install encrypted certificates menu option.

The problem is that no matter how hard I tried converting certs and keys in all the possible ways, I failed. My phone was constantly refusing cooperation, happily announcing that

No certificate file found in the SD card.

It was driving me crazy because I knew it used to work on my old phone. Searching the web I spot a fragment in Galaxy Nexus User’s Guide about importing certificates (page 104):

To install a certificate from your phone’s internal storage:
1. Copy the certificate or key store from your computer to the root of your device’s internal storage (that is, not in a folder).
2. From a Home or All Apps screen, touch the Settings icon.
3. Go to Personal > Security > Credential storage > Install from storage

Did you notice the shift from “SD card” to “Internal storage”? Well, Nexus is 4.0-based and my Galaxy S Plus works on 2.3.4, but what the heck… I tried and it worked!

So, if you have Samsung Galaxy S Plus (I9001) and your device can’t find certificates on SD card, just try to put them into internal storage.

I wonder whether this is the Samsung mistake of copying too much from their 4.0 codebase to older phones’ software, or other phones also exhibit this behavior?

## Getting started with jEdit hacking: plugin compilation

In the first installment of jEdit hacking series I described how to get jEdit running from the repo. However, jEdit wouldn’t be what it is without plugins. So, this time I’ll try to add a plugin. Compiled on our own, of course, not from the Plugin Central!

As described on the mailing list, there’s a problem with Sessions plugin in recent compilations of jEdit. The cause is probably API changes. The bug is fixed in the repo. I wanted the plugin, so plugin compilation happened.

First, let’s obtain the plugin. No surprise here, as the plugin is in the same repo, as jEdit itself:

mkidr plugins
cd plugins
svn co http://jedit.svn.sourceforge.net/svnroot/jedit/plugins/Sessions/trunk Sessions


The only thing we need to do to compile Sessions is to know the location of jedit.jar. After successful compilation of basic jEdit it should be in its build directory. If you followed the structure of directories presented in my tutorials (jEdit next to plugins), relative path is ../../jEdit/build/jedit.jar. For one-shot compilation the required property can be given on the command line:

ant -Djedit.install.dir=../../jEdit/build/


And… that’s it! One level above (in plugins) you should have Sessions.jar. After installation of Sessions from Plugins Central, close jEdit (make sure it is not running in the background!) and put the compiled JAR into ~/.jEdit/jars (see http://plugins.jedit.org/install.php for details, also on Windows), replacing the downloaded one (you can make backup if you want). Start jEdit again and check in Plugin Manager if the version is greater than the one in the repo (1.5.1 vs. 1.4.3 from the repo, at the time of writing).

## Getting started with jEdit hacking

I really like jEdit. It’s not my choice for the environment of multi-file, complicated projects, but I use it very often for ad-hoc text, code and XML editing. So I decided to give back to the community a little and hack jEdit a bit.

In a few articles I’ll write my way through the mysteries of jEdit hacking, from the point of view of a newcomer.

## Fetching the sources

Because I prefer Git to SVN, I decided to convert jEdit repo to local Git repos. So git svn to the rescue!

git svn init http://jedit.svn.sourceforge.net/svnroot/jedit --branches=jEdit/branches/ --tags=jEdit/tags/ --trunk=jEdit/trunk/
git svn fetch


Well, because SourceForge likes to drop connection randomly on large SVN transfers, it’ll be much easier to wrap the last line in a while loop:

while ! git svn fetch; do
echo "*** Retrying ***"
done


If you’re OK with only SVN, just checkout the repo:

svn co http://jedit.svn.sourceforge.net/svnroot/jedit/jEdit/trunk jEdit


The docs says you’ll also need build-support module, but for building plain jEdit (without modules), you don’t need it. If you ever need it, use the above instructions for fetching jEdit, replacing jEdit in the URL and path with build-support. Make sure to put build-support dir next to jEdit one.

## Building

jEdit build system is based on Ant. In Ubuntu required packages can be installed with:

aptitude install ant ant-optional


plus you will need a Java compiler — I won’t go into details here, leave a comment if you have problems.

README.SRC.txt is pretty straight-forward about the required steps to build the system. First, you must configure jEdit build system. Copy build.properties.sample to build.properties and set up what necessary. Most defaults are fine, at the first shot I only setup the deprecation and warning policy (jEdit compilation gives a lot of warnings…):

# warning settings
build.nowarn=true
build.deprecation=false
build.compilerarg=


Now, just compile:

ant build
# or just ant


## Running

If you want to run the just-compiled code, do it with

ant run


For serious hacking you will probably need debugger set up. I prefer Eclipse for my daily programming jobs, so we’ll use that.

In Eclipse, create new Java project, name it ‘jEdit’ (surprise!) and place it anywhere you want. Now, remove the default src folder and create a linked folder to the place where jEdit lives.

jEdit doesn’t use a separate source folder, sources are scattered in the main folder. Fortunatelly, there are not so many top-level packages, so not so many sub-folders.

Configure the project builder to use the just linked folder as the source folder. Configure inclusion/exclusion filter — we only need org.* packages. Resulting setting look like this:

This will allow us to setup breakpoints, navigate and read the code easily. I don’t care about Eclipse compilation as long as there are no errors.

The easiest way to debug jEdit is to run it outside Eclipse, with debugging agent, and to run remote debugging in Eclipse. To run jEdit with debugging server use:

ant run-debug


jEdit virtual machine will stop waiting for a connection from Eclipse (with one minute timeout). In Eclipse, create debug configuration (RunDebug Configurations…Remote Java ApplicationNew launch configuration) and point to localhost on port 5005. Click Debug and here you go, debugging jEdit. Good luck!

## The power of sets

A few weeks ago I’ve participated in Code Retreat in Poznań. The main theme of the workshop was not to learn some specific programming techniques, but rather to try working in pairs, in fact in many pairs, five or six, each for less than an hour.

The goal was to program Conway’s Game of Life, where each cell on rectangular board is born, lives or dies depending on the number of live neighbors (the live cell with 2 or 3 live neighbors survives, the dead cell with 3 live neighbors becomes alive).

The most common model is a two dimensional array. However, this approach has a drawback of the need to expand the array in all directions when new cells are to be born. Coding it correctly, with all test cases in less than an hour is not easy.

A friend in one of the pairs had a great idea: forget about an array and keep a set of live cells. Then, in each evolution step, expand this set with newly born cells and remove the ones which don’t survive.

We coded the solution in Ruby (without tests) and Java (with tests, in later session). Below is my personal try in Perl.

The concept can be neatly described in set theory. Let S be the set of live cells, neigh(c) is a function returning all neighbors of cell c. The set of candidates (potentially live cells in next round) is

$$C = \bigcup_{c\in S} \mathrm{neigh}(c)$$

then the live set in the next round is

$$S’ = \{ c\in C : |\mathrm{lneigh}(c)|=3 \vee c\in S \wedge |\mathrm{lneigh}(c)|=2 \}$$

$$\mathrm{lneigh}(c)$$ being the set of live neighbors of c

$$\mathrm{lneigh}(c) = \mathrm{neigh}(c) \cap S$$

The first version of code used plain hashes as sets, but using Set::Scalar gives cleaner code. There are some caveats, however.

use warnings;
use strict;

use Test::More;
use Set::Scalar;

# well, this should be default...
*Set::Scalar::Base::_strval  = sub { "$_[0]" }; my @D = ( [ -1, -1 ], [ -1, 0 ], [ -1, 1 ], [ 0, -1 ], [ 0, 1 ], [ 1, -1 ], [ 1, 0 ], [ 1, 1 ] ); my$h_rod = Set::Scalar->new(Seq::from_arr([-1,0], [0,0], [1,0]));
my $v_rod = Set::Scalar->new(Seq::from_arr([0,-1], [0,0], [0,1])); my$set = Set::Scalar->new(Seq::from_arr([1, 2], [3, 4]));
ok $set->has(Seq->new(1,2)); # this wouldn't work without _strval redefinition$set = $h_rod;$set = step($set); is$set, $v_rod;$set = step($set); is$set, $h_rod; done_testing; sub step { my ($live) = @_;
my $candid = Set::Scalar->new(map { neigh($_) } $live->members); return Set::Scalar->new(grep { my @c_live_neigh = grep {$live->has($_) } neigh($_);
@c_live_neigh == 3 || @c_live_neigh == 2 && $live->has($_);
} $candid->members); } sub neigh { my ($o) = @_;
map { Seq->new($o->[0] +$_->[0], $o->[1] +$_->[1]) } @D;
}

package Seq;
use overload '""' => sub {
return join ",", @{$_[0]}; }; sub new { my$class = shift;
return bless [ @_ ], $class; } sub from_arr { return map { Seq->new(@$_) } @_;
}


As Jarkko writes in the docs of Set::Scalar:

Using references (or objects) as set members has not been extensively tested. The desired semantics are not always clear: what should happen when the elements behind the references change? Especially unclear is what should happen when the objects start having their own stringification overloads.

Well, this is the case here. As we insert into the set the array refs, we need to convince Set::Scalar that two different refs are the same set element, if only the referred arrays have the same content.

However, stringification overload is not enough, as Set::Scalar additionally uses the refs addresses in its own stringification sub, _strval. So we need to redefine the sub to provide the set with our way of comparing things.

In fact, this was a big surprise to me: none of checked CPAN set distributions allowed the use of own identity methods. Even plain old hash-as-a-set is better in this regard, as stringification is performed to check for key presence:

my %hsh = map { $_ =>$_ } ( Seq->new(1,2), Seq->new(3,4), Seq->new(1,2) );
is values(%hsh), 2;


## From sound to notes

Some time ago I was given a link to a nice youtube video. Around 00:04 I recognized “Stairway to heaven” intro. Friends told me I was wrong, they just sound alike. Challenge accepted! Let’s analyze it.

Extract the sound with

ffmpeg -t 10 -i video.flv suspected.wav


First 10 secs is enough.

Then load the file into Audacity and begin the sound processing.

Because we are interested in a very specific sounds, let’s filter out frequencies which are above the scale of a guitar. We’ll use “Low-pass filter” with rolloff of 48dB/octave, filter quality of 0.7 and cutoff frequency of 700Hz. This will allow us to better hear where the guitar plays.

Now, the hard work. What I want to do is to find beginning of the notes, and for every suspected fragment analyze its’ spectrum. Looking at the spectrum peeks, I’ll assume the biggest ones (in guitar tune frequencies) are the tones we are looking for.

The good practical method to find the beginning of a tone is to listen to a fragment in slow motion. Also you may try to capture the correct moment by selecting a fragment and lowering its end until you hear no tone change at the end of this fragment. E.g. select a fragment from 3.4s to 3.9s and play it. You will hear two tones. Then, lower the end of the fragment until you hear only the first note. For me it was around 3.7s (a hint: text input for selection is very recommended). This is the beginning of the second tone.

Proceed with all the following sounds. Take into account, that previously played tones can co-sound, so analyze changes in spectrum for adjacent fragments.

My choices for times of the beginnings of tones, found peek frequencies and corresponding notes are:

3.44    293     D4
3.70    348     F4
3.95    441     A4
4.24    587     D5
4.48    662+277 E5 + C#4
4.74    350     F4
5.02    443     A4
5.36    659     E5
5.65 701        F5
6.05    440     A4
6.35    396     G4
6.60    357     F4


Then the guitar player goes to another theme. But this certainly IS “Stairway to heaven”! You can play it like this:

E|----------------10---|-12-------------12---|-13------------------|
B|-----------10--------|-----------10--------|------10-------------|
G|------10-------------|------10-------------|-----------10--------|
D|-12------------------|-11------------------|----------------10---|
A|---------------------|---------------------|---------------------|
E|---------------------|---------------------|---------------------|


while the common version you will find on the internet goes like:

E|----------5--7--------7-8--------8--2--------2-0---------0-
B|-------5--------5----------5-----------3---------1-----1---
G|----5--------------5----------5-----------2---------2------
D|-7-----------6----------5-----------4----------------------
A|-----------------------------------------------------------
E|-----------------------------------------------------------


Beside the transposition and a small variance in 2nd tact (2 notes reversed), they are the same.

Now, it would be very nice, if the computer could do the whole analysis. E.g. analyze the changes in spectrum in time and draw conclusions, when new sounds appear and what tones they are. Do you know such a tool, or should I start writing my own?

## Mapping terminal colors

Recently I started to work more intensively under Windows. Being a Linux convert, I installed MSYS to have bash and other UN*X tools. Although MSYS works nice, I had problems with proper console behavior. Both the Windows default one (cmd.exe) and Console2 lack some terminal capabilities, so I stick with MSYS’ rxvt.

One thing I didn’t like about rxvt was the colors. I have Tango colors set in Gnome Terminal, so I tried to copy the palette.

Rxvt allows you to set an ANSI color with -colorX options, but accepts only X11 color names, while Gnome Terminal gives you RGB values.

So I wrote a simple script which reads an RGB triple from the input and finds the closest matching colors in the palette. Here it is:

use warnings;
use strict;

use 5.010;
use Color::Similarity::HCL qw( distance );

my ($fname) = @ARGV; die "Usage:$0 /etc/X11/rgb.txt" unless defined $fname; my @palette; open my$xcol_fh, '<', $fname or die "Can't open$fname: $!"; while (<$xcol_fh>) {
chomp;
next if /^\s*!/;
my ($r,$g, $b,$name) = $_ =~ /^\s*(\d+)\s+(\d+)\s+(\d+)\s+(.*)/; push @palette, { name =>$name, r => $r, g =>$g, b => $b }; } close$xcol_fh;

while (my $l = <STDIN>) { chomp$l;

my @triple = $l =~ /^\s*(\d+)\s+(\d+)\s+(\d+)/;$_->{dist} = distance([ @triple ], [ @$_{qw( r g b )} ]) for (@palette); say "Best matches for (", join(qq(, ), @triple), ") are: "; my @srt = sort {$a->{dist} <=> $b->{dist} } @palette; for my$col (@srt[0..9]) {
say "$col->{name} (", join(qq(, ), @$col{qw( r g b )}), "): $col->{dist}"; } }  or even simpler, using Convert::Color::X11, as suggested by LeoNerd: use warnings; use strict; use 5.010; use Convert::Color::X11; use Color::Similarity::HCL qw( distance ); my @palette = map { { name =>$_, rgb => [ Convert::Color::X11->new($_)->rgb8 ] } } Convert::Color::X11->colors; while (my$l = <STDIN>) {
chomp $l; my @triple =$l =~ /^\s*(\d+)\s+(\d+)\s+(\d+)/;
$_->{dist} = distance([ @triple ],$_->{rgb}) for (@palette);

say "Best matches for (", join(qq(, ), @triple), ") are: ";
my @srt = sort { $a->{dist} <=>$b->{dist} } @palette;
for my $col (@srt[0..9]) { say "$col->{name} (", join(qq(, ), @{$col->{rgb}}), "):$col->{dist}";
}
}


For me, rxvt best emulates Tango palette with the following options:

-color0 black -color1 red3 -color2 chartreuse4 -color3 gold3 -color4 DodgerBlue4 -color5 plum4 -color6 turquoise4 -color7 honeydew3 -color8 gray34 -color9 firebrick2 -color10 chartreuse2 -color11 khaki -color12 SkyBlue3 -color13 plum3 -color14 cyan3 -color15 gray93


## Per-tags feeds in MovableType

In Movable Type it is possible to generate a feed for entries matching a given tag only. For this to happen, just select some tag, e.g. from tags cloud and take a look at link named “Feed of results tagged …”. For example, for this blog’s Perl tag, the link looks like:

 http://tu.wesolek.net/cgi-bin/mt/mt-search.cgi?tag=Perl&Template=feed&IncludeBlogs=2&limit=20 

and you can use this URL as a feed, e.g. for Ironman challenge.

There is a problem with Ubuntu package, however (Karmic at least). Packaged MT is missing search_templates link in /usr/lib/cgi-bin/movabletype. I reported this bug, and you can apply the simple solution on your own.

## Perlmonks on giving ready solutions

I read Perlmonks irregularly, sometimes I even post something. I give hints, thoughts or bigger chunks of code. But for the first time I got as much as 40% of downvotes on an entry that I think is rather neutral.

I understand that some people don’t like to give others ready solutions, they prefer “to teach others to fish than to give them a fish”. But is it really something that bothers them so much to give thumbs down to the ones who don’t mind? No feelings hurt, I’m just curious…

Find recent content on the main index or look in the archives to find all content.

• Przemek Wesołek: Thanks LeoNerd, now it looks neater. read more
• LeoNerd: That can probably be made much shorter by Convert::Color. First read more