Getting started with jEdit hacking: plugin compilation

| No Comments | No TrackBacks

In the first installment of jEdit hacking series I described how to get jEdit running from the repo. However, jEdit wouldn’t be what it is without plugins. So, this time I’ll try to add a plugin. Compiled on our own, of course, not from the Plugin Central!

As described on the mailing list, there’s a problem with Sessions plugin in recent compilations of jEdit. The cause is probably API changes. The bug is fixed in the repo. I wanted the plugin, so plugin compilation happened.

First, let’s obtain the plugin. No surprise here, as the plugin is in the same repo, as jEdit itself:

mkidr plugins
cd plugins
svn co http://jedit.svn.sourceforge.net/svnroot/jedit/plugins/Sessions/trunk Sessions

The only thing we need to do to compile Sessions is to know the location of jedit.jar. After successful compilation of basic jEdit it should be in its build directory. If you followed the structure of directories presented in my tutorials (jEdit next to plugins), relative path is ../../jEdit/build/jedit.jar. For one-shot compilation the required property can be given on the command line:

ant -Djedit.install.dir=../../jEdit/build/

And… that’s it! One level above (in plugins) you should have Sessions.jar. After installation of Sessions from Plugins Central, close jEdit (make sure it is not running in the background!) and put the compiled JAR into ~/.jEdit/jars (see http://plugins.jedit.org/install.php for details, also on Windows), replacing the downloaded one (you can make backup if you want). Start jEdit again and check in Plugin Manager if the version is greater than the one in the repo (1.5.1 vs. 1.4.3 from the repo, at the time of writing).

Getting started with jEdit hacking

| No Comments | No TrackBacks

I really like jEdit. It’s not my choice for the environment of multi-file, complicated projects, but I use it very often for ad-hoc text, code and XML editing. So I decided to give back to the community a little and hack jEdit a bit.

In a few articles I’ll write my way through the mysteries of jEdit hacking, from the point of view of a newcomer.

In this article we will download, compile and debug with Eclipse the basic jEdit, without plugins.

Fetching the sources

Because I prefer Git to SVN, I decided to convert jEdit repo to local Git repos. So git svn to the rescue!

git svn init http://jedit.svn.sourceforge.net/svnroot/jedit --branches=jEdit/branches/ --tags=jEdit/tags/ --trunk=jEdit/trunk/
git svn fetch

Well, because SourceForge likes to drop connection randomly on large SVN transfers, it’ll be much easier to wrap the last line in a while loop:

while ! git svn fetch; do
    echo "*** Retrying ***"
done

If you’re OK with only SVN, just checkout the repo:

svn co http://jedit.svn.sourceforge.net/svnroot/jedit/jEdit/trunk jEdit

The docs says you’ll also need build-support module, but for building plain jEdit (without modules), you don’t need it. If you ever need it, use the above instructions for fetching jEdit, replacing jEdit in the URL and path with build-support. Make sure to put build-support dir next to jEdit one.

Building

jEdit build system is based on Ant. In Ubuntu required packages can be installed with:

aptitude install ant ant-optional

plus you will need a Java compiler — I won’t go into details here, leave a comment if you have problems.

README.SRC.txt is pretty straight-forward about the required steps to build the system. First, you must configure jEdit build system. Copy build.properties.sample to build.properties and set up what necessary. Most defaults are fine, at the first shot I only setup the deprecation and warning policy (jEdit compilation gives a lot of warnings…):

# warning settings
build.nowarn=true
build.deprecation=false
build.compilerarg=     

Now, just compile:

ant build
# or just ant

Running

If you want to run the just-compiled code, do it with

ant run

For serious hacking you will probably need debugger set up. I prefer Eclipse for my daily programming jobs, so we’ll use that.

In Eclipse, create new Java project, name it ‘jEdit’ (surprise!) and place it anywhere you want. Now, remove the default src folder and create a linked folder to the place where jEdit lives.

jEdit doesn’t use a separate source folder, sources are scattered in the main folder. Fortunatelly, there are not so many top-level packages, so not so many sub-folders.

Configure the project builder to use the just linked folder as the source folder. Configure inclusion/exclusion filter — we only need org.* packages. Resulting setting look like this:

eclipse-project-settings.png

This will allow us to setup breakpoints, navigate and read the code easily. I don’t care about Eclipse compilation as long as there are no errors.

The easiest way to debug jEdit is to run it outside Eclipse, with debugging agent, and to run remote debugging in Eclipse. To run jEdit with debugging server use:

ant run-debug

jEdit virtual machine will stop waiting for a connection from Eclipse (with one minute timeout). In Eclipse, create debug configuration (RunDebug Configurations…Remote Java ApplicationNew launch configuration) and point to localhost on port 5005. Click Debug and here you go, debugging jEdit. Good luck!

The power of sets

| No Comments | No TrackBacks

A few weeks ago I’ve participated in Code Retreat in Poznań. The main theme of the workshop was not to learn some specific programming techniques, but rather to try working in pairs, in fact in many pairs, five or six, each for less than an hour.

The goal was to program Conway’s Game of Life, where each cell on rectangular board is born, lives or dies depending on the number of live neighbors (the live cell with 2 or 3 live neighbors survives, the dead cell with 3 live neighbors becomes alive).

The most common model is a two dimensional array. However, this approach has a drawback of the need to expand the array in all directions when new cells are to be born. Coding it correctly, with all test cases in less than an hour is not easy.

A friend in one of the pairs had a great idea: forget about an array and keep a set of live cells. Then, in each evolution step, expand this set with newly born cells and remove the ones which don’t survive.

We coded the solution in Ruby (without tests) and Java (with tests, in later session). Below is my personal try in Perl.

The concept can be neatly described in set theory. Let S be the set of live cells, neigh(c) is a function returning all neighbors of cell c. The set of candidates (potentially live cells in next round) is

$$ C = \bigcup_{c\in S} \mathrm{neigh}(c) $$

then the live set in the next round is

$$ S’ = \{ c\in C : |\mathrm{lneigh}(c)|=3 \vee c\in S \wedge |\mathrm{lneigh}(c)|=2 \} $$

\( \mathrm{lneigh}(c) \) being the set of live neighbors of c

$$ \mathrm{lneigh}(c) = \mathrm{neigh}(c) \cap S $$

The first version of code used plain hashes as sets, but using Set::Scalar gives cleaner code. There are some caveats, however.

use warnings;
use strict;

use Test::More;
use Set::Scalar;

# well, this should be default...
*Set::Scalar::Base::_strval  = sub { "$_[0]" };

my @D = (
    [ -1, -1 ], [ -1, 0 ], [ -1, 1 ],
    [  0, -1 ],            [  0, 1 ],
    [  1, -1 ], [  1, 0 ], [  1, 1 ]
);

my $h_rod = Set::Scalar->new(Seq::from_arr([-1,0], [0,0], [1,0]));
my $v_rod = Set::Scalar->new(Seq::from_arr([0,-1], [0,0], [0,1]));

my $set = Set::Scalar->new(Seq::from_arr([1, 2], [3, 4]));
ok $set->has(Seq->new(1,2));  # this wouldn't work without _strval redefinition

$set = $h_rod;

$set = step($set);
is $set, $v_rod;

$set = step($set);
is $set, $h_rod;

done_testing;


sub step {
    my ($live) = @_;
    my $candid = Set::Scalar->new(map { neigh($_) } $live->members);

    return Set::Scalar->new(grep {
        my @c_live_neigh = grep { $live->has($_) } neigh($_);
        @c_live_neigh == 3 || @c_live_neigh == 2 && $live->has($_);
    } $candid->members);
}

sub neigh {
    my ($o) = @_;
    map { Seq->new($o->[0] + $_->[0], $o->[1] + $_->[1]) } @D;
}


package Seq;
use overload '""' => sub {
    return join ",", @{$_[0]};
};

sub new {
    my $class = shift;
    return bless [ @_ ], $class;
}
sub from_arr {
    return map { Seq->new(@$_) } @_;
}

As Jarkko writes in the docs of Set::Scalar:

Using references (or objects) as set members has not been extensively tested. The desired semantics are not always clear: what should happen when the elements behind the references change? Especially unclear is what should happen when the objects start having their own stringification overloads.

Well, this is the case here. As we insert into the set the array refs, we need to convince Set::Scalar that two different refs are the same set element, if only the referred arrays have the same content.

However, stringification overload is not enough, as Set::Scalar additionally uses the refs addresses in its own stringification sub, _strval. So we need to redefine the sub to provide the set with our way of comparing things.

In fact, this was a big surprise to me: none of checked CPAN set distributions allowed the use of own identity methods. Even plain old hash-as-a-set is better in this regard, as stringification is performed to check for key presence:

my %hsh = map { $_ => $_ } ( Seq->new(1,2), Seq->new(3,4), Seq->new(1,2) );
is values(%hsh), 2;

From sound to notes

| No Comments | No TrackBacks

Some time ago I was given a link to a nice youtube video. Around 00:04 I recognized “Stairway to heaven” intro. Friends told me I was wrong, they just sound alike. Challenge accepted! Let’s analyze it.

First of all, let’s download the video from youtube. Visit e.g. http://keepvid.com and enter video URL: http://www.youtube.com/watch?v=_erQi563tTU. Download the result (I chose MP4 480x360).

Extract the sound with

ffmpeg -t 10 -i video.flv suspected.wav

First 10 secs is enough.

Then load the file into Audacity and begin the sound processing.

Because we are interested in a very specific sounds, let’s filter out frequencies which are above the scale of a guitar. We’ll use “Low-pass filter” with rolloff of 48dB/octave, filter quality of 0.7 and cutoff frequency of 700Hz. This will allow us to better hear where the guitar plays.

Now, the hard work. What I want to do is to find beginning of the notes, and for every suspected fragment analyze its’ spectrum. Looking at the spectrum peeks, I’ll assume the biggest ones (in guitar tune frequencies) are the tones we are looking for.

The good practical method to find the beginning of a tone is to listen to a fragment in slow motion. Also you may try to capture the correct moment by selecting a fragment and lowering its end until you hear no tone change at the end of this fragment. E.g. select a fragment from 3.4s to 3.9s and play it. You will hear two tones. Then, lower the end of the fragment until you hear only the first note. For me it was around 3.7s (a hint: text input for selection is very recommended). This is the beginning of the second tone.

Proceed with all the following sounds. Take into account, that previously played tones can co-sound, so analyze changes in spectrum for adjacent fragments.

My choices for times of the beginnings of tones, found peek frequencies and corresponding notes are:

3.44    293     D4
3.70    348     F4
3.95    441     A4
4.24    587     D5
4.48    662+277 E5 + C#4
4.74    350     F4
5.02    443     A4
5.36    659     E5  
5.65 701        F5
6.05    440     A4
6.35    396     G4
6.60    357     F4

Then the guitar player goes to another theme. But this certainly IS “Stairway to heaven”! You can play it like this:

E|----------------10---|-12-------------12---|-13------------------|
B|-----------10--------|-----------10--------|------10-------------|
G|------10-------------|------10-------------|-----------10--------|
D|-12------------------|-11------------------|----------------10---|
A|---------------------|---------------------|---------------------|
E|---------------------|---------------------|---------------------|

while the common version you will find on the internet goes like:

E|----------5--7--------7-8--------8--2--------2-0---------0-
B|-------5--------5----------5-----------3---------1-----1---
G|----5--------------5----------5-----------2---------2------
D|-7-----------6----------5-----------4----------------------
A|-----------------------------------------------------------
E|-----------------------------------------------------------

Beside the transposition and a small variance in 2nd tact (2 notes reversed), they are the same.

Now, it would be very nice, if the computer could do the whole analysis. E.g. analyze the changes in spectrum in time and draw conclusions, when new sounds appear and what tones they are. Do you know such a tool, or should I start writing my own?

Mapping terminal colors

| 2 Comments | No TrackBacks

Recently I started to work more intensively under Windows. Being a Linux convert, I installed MSYS to have bash and other UN*X tools. Although MSYS works nice, I had problems with proper console behavior. Both the Windows default one (cmd.exe) and Console2 lack some terminal capabilities, so I stick with MSYS’ rxvt.

One thing I didn’t like about rxvt was the colors. I have Tango colors set in Gnome Terminal, so I tried to copy the palette.

Rxvt allows you to set an ANSI color with -colorX options, but accepts only X11 color names, while Gnome Terminal gives you RGB values.

So I wrote a simple script which reads an RGB triple from the input and finds the closest matching colors in the palette. Here it is:

use warnings;
use strict;

use 5.010;
use Color::Similarity::HCL qw( distance );

my ($fname) = @ARGV;
die "Usage: $0 /etc/X11/rgb.txt" unless defined $fname;

my @palette;
open my $xcol_fh, '<', $fname or die "Can't open $fname: $!";
while (<$xcol_fh>) {
    chomp;
    next if /^\s*!/;
    my ($r, $g, $b, $name) = $_ =~ /^\s*(\d+)\s+(\d+)\s+(\d+)\s+(.*)/;
    push @palette, { name => $name, r => $r, g => $g, b => $b };
}
close $xcol_fh;

while (my $l = <STDIN>) {
    chomp $l; 

    my @triple = $l =~ /^\s*(\d+)\s+(\d+)\s+(\d+)/;
    $_->{dist} = distance([ @triple ], [ @$_{qw( r g b )} ]) for (@palette);

    say "Best matches for (", join(qq(, ), @triple), ") are: ";
    my @srt = sort { $a->{dist} <=> $b->{dist} } @palette;
    for my $col (@srt[0..9]) {
        say "$col->{name} (", join(qq(, ), @$col{qw( r g b )}), "): $col->{dist}";
    }   
}

or even simpler, using Convert::Color::X11, as suggested by LeoNerd:

use warnings;
use strict;

use 5.010;
use Convert::Color::X11;
use Color::Similarity::HCL qw( distance );

my @palette = map { { name => $_, rgb => [ Convert::Color::X11->new($_)->rgb8 ] } } Convert::Color::X11->colors;                                                                                                                              

while (my $l = <STDIN>) {
    chomp $l; 

    my @triple = $l =~ /^\s*(\d+)\s+(\d+)\s+(\d+)/;
    $_->{dist} = distance([ @triple ], $_->{rgb}) for (@palette);

    say "Best matches for (", join(qq(, ), @triple), ") are: ";
    my @srt = sort { $a->{dist} <=> $b->{dist} } @palette;
    for my $col (@srt[0..9]) {
        say "$col->{name} (", join(qq(, ), @{$col->{rgb}}), "): $col->{dist}";
    }   
}

For me, rxvt best emulates Tango palette with the following options:

-color0 black -color1 red3 -color2 chartreuse4 -color3 gold3 -color4 DodgerBlue4 -color5 plum4 -color6 turquoise4 -color7 honeydew3 -color8 gray34 -color9 firebrick2 -color10 chartreuse2 -color11 khaki -color12 SkyBlue3 -color13 plum3 -color14 cyan3 -color15 gray93

Per-tags feeds in MovableType

| No Comments | No TrackBacks

In Movable Type it is possible to generate a feed for entries matching a given tag only. For this to happen, just select some tag, e.g. from tags cloud and take a look at link named “Feed of results tagged …”. For example, for this blog’s Perl tag, the link looks like:

http://tu.wesolek.net/cgi-bin/mt/mt-search.cgi?tag=Perl&Template=feed&IncludeBlogs=2&limit=20

and you can use this URL as a feed, e.g. for Ironman challenge.

There is a problem with Ubuntu package, however (Karmic at least). Packaged MT is missing search_templates link in /usr/lib/cgi-bin/movabletype. I reported this bug, and you can apply the simple solution on your own.

Perlmonks on giving ready solutions

| No Comments | No TrackBacks

I read Perlmonks irregularly, sometimes I even post something. I give hints, thoughts or bigger chunks of code. But for the first time I got as much as 40% of downvotes on an entry that I think is rather neutral.

I understand that some people don’t like to give others ready solutions, they prefer “to teach others to fish than to give them a fish”. But is it really something that bothers them so much to give thumbs down to the ones who don’t mind? No feelings hurt, I’m just curious…

Compiling Perl 5.10.1 under Ubuntu

| No Comments | No TrackBacks

Note: Since 5.10.1 is not upstream, there might be changes in packages versions, so downloading patches as described below might not work. I’ll try to update this entry to reflect the correct links, but in case of problems just go to http://patch-tracker.debian.org/package/perl, select the current version of Perl 5.10.1 and download mod_paths.diff manually.


After reading encouragingly simple instructions on compiling Perl 5.10.1, I decided to compile Perl — for the first time — on my own.

Everything went fine, but after installation it occurred that the order of directories on @INC is “core, vendor, site”, where the expected — after using Debian’s Perl for years — was “site, vendor, core”. Apparently a bug (for most), with no chance to be corrected due to backward compatibility.

So, I applied a patch from Debian, which corrects the order. Note, that besides correcting the order, the patch adds /etc/perl at the beginning of @INC and /usr/local/lib/site_perl at the end. This is fine with me, but you can remove it from the code if you wish.

mod_paths.diff:

Subject: Tweak @INC so that the ordering is:

    etc (for config files)
    site (5.8.1)
    vendor (all)
    core (5.8.1)
    site (version-indep)
    site (pre-5.8.1)

The rationale being that an admin (via site), or module packager
(vendor) can chose to shadow core modules when there is a newer
version than is included in core.


---
 perl.c |   62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 62 insertions(+), 0 deletions(-)

diff --git a/perl.c b/perl.c
index 94f2b13..5a6744a 100644
--- a/perl.c
+++ b/perl.c
@@ -4879,9 +4879,14 @@ S_init_perllib(pTHX)
     incpush(APPLLIB_EXP, TRUE, TRUE, TRUE, TRUE);
 #endif

+#ifdef DEBIAN
+    /* for configuration where /usr is mounted ro (CPAN::Config, Net::Config) */
+    incpush("/etc/perl", FALSE, FALSE, FALSE, FALSE);
+#else
 #ifdef ARCHLIB_EXP
     incpush(ARCHLIB_EXP, FALSE, FALSE, TRUE, TRUE);
 #endif
+#endif
 #ifdef MACOS_TRADITIONAL
     {
    Stat_t tmpstatbuf;
@@ -4906,11 +4911,13 @@ S_init_perllib(pTHX)
 #ifndef PRIVLIB_EXP
 #  define PRIVLIB_EXP "/usr/local/lib/perl5:/usr/local/lib/perl"
 #endif
+#ifndef DEBIAN
 #if defined(WIN32)
     incpush(PRIVLIB_EXP, TRUE, FALSE, TRUE, TRUE);
 #else
     incpush(PRIVLIB_EXP, FALSE, FALSE, TRUE, TRUE);
 #endif
+#endif

 #ifdef SITEARCH_EXP
     /* sitearch is always relative to sitelib on Windows for
@@ -4954,6 +4961,61 @@ S_init_perllib(pTHX)
     incpush(PERL_VENDORLIB_STEM, FALSE, TRUE, TRUE, TRUE);
 #endif

+#ifdef DEBIAN
+    incpush(ARCHLIB_EXP, FALSE, FALSE, TRUE, FALSE);
+    incpush(PRIVLIB_EXP, FALSE, FALSE, TRUE, FALSE);
+
+    /* Non-versioned site directory for local modules and for
+       compatability with the previous packages' site dirs */
+    incpush("/usr/local/lib/site_perl", TRUE, FALSE, FALSE, FALSE);
+
+#ifdef PERL_INC_VERSION_LIST
+    {
+   struct stat s;
+
+   /* add small buffer in case old versions are longer than the
+      current version */
+   char sitearch[sizeof(SITEARCH_EXP)+16] = SITEARCH_EXP;
+   char sitelib[sizeof(SITELIB_EXP)+16] = SITELIB_EXP;
+   char const *vers[] = { PERL_INC_VERSION_LIST };
+   char const **p;
+
+   char *arch_vers = strrchr(sitearch, '/');
+   char *lib_vers = strrchr(sitelib, '/');
+
+   if (arch_vers && isdigit(*++arch_vers))
+       *arch_vers = 0;
+   else
+       arch_vers = 0;
+
+   if (lib_vers && isdigit(*++lib_vers))
+       *lib_vers = 0;
+   else
+       lib_vers = 0;
+
+   /* there is some duplication here as incpush does something
+      similar internally, but required as sitearch is not a
+      subdirectory of sitelib */
+   for (p = vers; *p; p++)
+   {
+       if (arch_vers)
+       {
+       strcpy(arch_vers, *p);
+       if (PerlLIO_stat(sitearch, &s) >= 0 && S_ISDIR(s.st_mode))
+           incpush(sitearch, FALSE, FALSE, FALSE, FALSE);
+       }
+
+       if (lib_vers)
+       {
+       strcpy(lib_vers, *p);
+       if (PerlLIO_stat(sitelib, &s) >= 0 && S_ISDIR(s.st_mode))
+           incpush(sitelib, FALSE, FALSE, FALSE, FALSE);
+       }
+   }
+    }
+#endif
+#endif
+
 #ifdef PERL_OTHERLIBDIRS
     incpush(PERL_OTHERLIBDIRS, TRUE, TRUE, TRUE, TRUE);
 #endif
-- 
tg: (daf8b46..) debian/mod_paths (depends on: upstream)

So, my modified compilation instructions with Ubuntu’s “taste” look like:

wget http://www.cpan.org/modules/by-authors/id/D/DA/DAPM/perl-5.10.1.tar.bz2
wget http://patch-tracker.debian.org/patch/series/dl/perl/5.10.1-7/debian/mod_paths.diff
tar xjf perl-5.10.1.tar.bz2
cd perl-5.10.1
patch -b -p1 <../mod_paths.diff
perl Configure -de -Dprefix=${HOME}/local -Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Doptimize=-O2
make
make test
make install

Perl test in MT

| No Comments | No TrackBacks

Here’s a little test of Perl formatting with SyntaxHighlighter.

#!/usr/bin/perl

use strict;

print "Hello there!";

I’ll write a bit more on my Movable Type config soon, as there are some interesting quirks.

Compiling with distcc and ccache

| No Comments | No TrackBacks

In my working environment a firewall is set up that prevents connecting with remote server do do distributed computing with distcc. Here’s a short note on how I work around it and generally, how my compilation chain is constructed.

First, I setup SSH tunnel to avoid firewall restrictions:

ssh SERVER -L 3633:localhost:3632

and run a build on the local machine

DISTCC_HOSTS='localhost:3633' CCACHE_PREFIX='distcc' CXX='ccache g++' make -j8

The compilation chain is as follow:

  1. ccache is run instead of g++.
  2. ccache checks if it has a locally (on the client) cached version of compilation result. If so, the result is returned.
  3. If not, distcc is called and performs compilation on the SERVER.
  4. Compilation result is stored in ccache’s cache and “returned back”.