
A	How do I remove the first line from a FASTA format file during input?
A	Permission denied when i exec Postfix reload in a perl script
A	How to pass array and hash data types to subroutines arguments in perl?
A	Perl and Complex SOAP Request
A	Escaping xml tags inside xml

Adding Stackoverflow user feed to your homepage 16-12-2012 16:48:00

Following on from a comment on a previous post, this post will demonstrate how to use XML::Feed and Template::Toolkit to format your recent Stackoverflow activity, suitable for including on your own web page. Aptly, this was aided by a stackoverflow question!

This time we can use XML::Feed both to get and parse the feed:

perl

Line number Off | Hide | Select all

use XML::Feed;
my $feed = XML::Feed->parse(URI->new('http://stackoverflow.com/feeds/user/1691146'));

We need to adapt the template to account for XML::Feed's differences:

perl

Line number Off | Hide | Select all

my $template = <<"TEMPLATE";
[% feed.title %]
Activity:
[% FOREACH item = feed.items %]
[% item.issued.dmy %] [% item.issued.hms %]\t[% item.title %]
[% END %]
TEMPLATE

And the template toolkit invocation:

perl

Line number Off | Hide | Select all

$tt->process( \$template, { 'feed' => $feed } )
or die $tt->error();

Full example:

perl

Line number Off | Hide | Select all

#!/usr/bin/perl
use strict;
use warnings;
use XML::Feed;
use Template;
my $stackoverflow_id = 1691146;
my $stackoverflow_url = "http://stackoverflow.com/feeds/user/$stackoverflow_id";
my $template = <<"TEMPLATE";
[% feed.title %]
Activity:
[% FOREACH item = feed.items %]
[% item.issued.dmy %] [% item.issued.hms %]\t[% item.title %]
[% END %]
TEMPLATE
my $tt = Template->new( 'STRICT' => 1 )
or die "Failed to load template: $Template::ERROR\n";
my $feed = XML::Feed->parse(URI->new($stackoverflow_url));
$tt->process( \$template, { 'feed' => $feed } )
or die $tt->error();

This is still a really simple script with added support for Atom feeds.

Original post blogged on LessThanDot.

Create HTML from output of Perl::Critic 22-09-2012 07:46:00

Perl::Critic is great. If you haven't tried it, you should. It can help you improve the quality of your code no end.

I wrote this script to make it easier to view the feedback in the context of the code itself. When run in a directory containing Perl code it'll create a critic_html directory containing index.html summarizing the results:

critic_html/index.html

Clicking through to any of the files will give you all the violations found by Perl::Critic, inline with the code:

In addition to Perl::Critic, the script uses Template::Toolkit to format the output. The code below and templates can be found in the attached critic_html.zip

perl

Line number Off | Hide | Select all

#!/usr/bin/perl
use strict;
use warnings;
use Perl::Critic;
use Perl::Critic::Utils;
use Template;
use Cwd qw(abs_path);
use File::Basename;
use English qw(-no_match_vars);
use constant SEVERITY => 1; # Include all violations.
use constant SKIP_GOOD => 0; # Skip files with no violations?
mkdir 'critic_html';
mkdir 'critic_html/src';
use autodie;
# Analyse all perl files below the current directory.
my @files = Perl::Critic::Utils::all_perl_files(q{.});
my @summary = (); # Store statistics on each file processed.
foreach my $file (@files) {
# Create a new Critic for per file statistics.
my $critic = Perl::Critic->new( '-severity' => SEVERITY );
my $file_safe = $file;
$file_safe =~ s/[\W]/_/g;
my @violations = $critic->critique($file);
next if (!@violations && SKIP_GOOD);
push @summary, { 'filename' => $file,
'link' => "src/$file_safe.html",
'stats' => $critic->statistics() };
open my $FH, '<', $file;
my @lines = <$FH>;
close $FH;
# Attach all violations to the line they were found on.
my @violations_by_line = ();
my $line_number = 1;
foreach my $line (@lines) {
# Get all the violations for the current line.
my @line_violations = grep { $_->line_number() == $line_number} @violations;
push @violations_by_line, { 'number' => $line_number,
'content' => $line,
'violations' => \@line_violations };
$line_number++;
}
write_html("critic_html/src/$file_safe.html", 'codefile', { 'title' => "Critic Analysis of $file",
'lines' => \@violations_by_line } );
}
write_html('critic_html/index.html', 'index', { 'title' => 'Perl::Critic::HTML Summary',
'files' => \@summary });
sub write_html {
my ($filename, $template, $data) = @_;
# Include templates from the install directory.
my $tt = Template->new({ 'INCLUDE_PATH' => dirname(abs_path($PROGRAM_NAME)).'/templates' } );
print "Writing $filename\n";
open my $FILE, '>', $filename;
$tt->process($template, $data, $FILE);
close $FILE;
return;
}

Original post blogged on LessThanDot.

Taming RSS Feeds with XML::RSS and Template::Toolkit 14-09-2012 11:01:00

This is a remarkably simple trick which I've found very handy. With a few lines of Perl you can take any RSS feed and format it to your liking.

Get the Feed

You can do this using LWP::Simple:

perl

Line number Off | Hide | Select all

use LWP::Simple;
my $feed_url = 'http://feeds.bbci.co.uk/news/rss.xml';
my $feed = get($feed_url)
or die ("Failed to fetch feed.");

Process the Raw Result

Using XML::RSS, convert the raw feed into a more manageable hash.

perl

Line number Off | Hide | Select all

use XML::RSS;
my $rss = XML::RSS->new();
$rss->parse($feed);

Format to Your Liking

Template::Toolkit can take in a template and a hash reference of values to substitute into the template.

perl

Line number Off | Hide | Select all

# Define a template
my $template = <<"TEMPLATE";
[% channel.title %]
Headlines:
[% FOREACH item = items %]
[% item.pubDate %]\t[% item.title %]
[% END %]
TEMPLATE

This simple template will take the BBC news feed from above and print out a list of headlines with publication dates.

perl

Line number Off | Hide | Select all

my $tt = Template->new()
or die ("Failed to load template: $Template::ERROR\n");
# Combine the template with the processed RSS feed.
$tt->process ( \$template, $rss )
or die $tt->error();

Putting it All Together

perl

Line number Off | Hide | Select all

#!/usr/bin/perl
use strict;
use warnings;
use XML::RSS;
use LWP::Simple;
use Template;
##################
# Configuration:
#
##################
my $feed_url = 'http://feeds.bbci.co.uk/news/rss.xml';
my $template = <<"TEMPLATE";
[% channel.title %]
Headlines:
[% FOREACH item = items %]
[% item.pubDate %]\t[% item.title %]
[% END %]
TEMPLATE
##################
##################
my $tt = Template->new()
or die ("Failed to load template: $Template::ERROR\n");
my $feed = get($feed_url)
or die ('Failed to fetch feed.');
my $rss = XML::RSS->new();
$rss->parse($feed);
$tt->process ( \$template, $rss )
or die $tt->error();

bash

Line number Off | Hide | Select all

rob@arrakis:~/public_html/rss-reader$ perl rss-reader.pl
BBC News - Home
Headlines:
Fri, 14 Sep 2012 12:15:22 GMT Kate privacy invasion 'grotesque'
Fri, 14 Sep 2012 11:49:10 GMT US missions on film protest alert
Fri, 14 Sep 2012 12:25:25 GMT Alps attack girl returning to UK
Fri, 14 Sep 2012 10:17:03 GMT Woman is held after car body find

In this example the RSS feed and template are defined in code but they can just as easily be defined in files or a database allowing for changes/additions without deploying new code.

Original post blogged on LessThanDot.

Using a Perl Singleton to Share Values Between Objects 12-09-2012 14:54:00

It's been quite a while since I last posted here. Since my last entry I got a new job, got married, went on honeymoon (got back yesterday!) and spent quite a lot of time maintaining legacy Perl code.

One thing I see a lot is sharing objects by passing them to the constructor of a new object. This is fine until you're dealing with complicated sets of objects within objects.

perl

Line number Off | Hide | Select all

#!/usr/bin/perl
use strict;
# Read some variables out of a configuration file.
my $config = get_config($config_file);
# Create a new object which contains an object which uses some values from the config.
my $object = MyObject->new($config);
$object->do_something();
package MyObject;
sub new {
my $class = shift;
my $config = shift;
# Store the config incase we create any objects which need it.
my $self = { 'config' => $config };
return bless $self, $class;
}
sub do_something {
my $self = shift;
my $object2 = MyObject2->new($self->{'config'});
return $object2->do_something_else();
}

MyObject is now holding a reference to the config object which it doesn't need directly. This will only get worse if MyObject2 needs another object which requires the config object, etc.

Instead we could use a singleton object:

perl

Line number Off | Hide | Select all

package MyConfigObject;
use strict;
use Carp;
my $config = undef;
sub new {
if(!$config) {
my $class = shift;
my $config_file = shift;
croak unless $config_file;
$config = bless {}, $class;
$config->parse_file($config_file);
}
return $config;
}

The example above changes to:

perl

Line number Off | Hide | Select all

#!/usr/bin/perl
use strict;
# Create a new object to hold the values from our config file.
my $config = MyConfigObject->new($config_file);
# Create a new object which contains an object which uses some values from the config.
my $object = MyObject->new();
$object->do_something();
package MyObject;
sub new {
my $class = shift;
return bless {}, $class;
}
sub do_something {
my $self = shift;
my $object2 = MyObject2->new();
return $object2->do_something_else();
}
package MyObject2;
sub new {
# Get the config object
my $config = MyConfigObject->new();
# Do some stuff with it!
}

Whenever we need to use a value from the config file, we create a new MyConfigObject and get the instance we already created without having to reparse the config file or maintain complicated links through parent objects.

Original post blogged on LessThanDot.

Manipulating Test Data with Sed 22-05-2011 12:41:00

Following on from my previous blog about sed, this blog will detail how I use sed most often: manipulating test data when running demonstrations of new features. Let's say we've developed a web service which creates users from the following XML document:

xml

Line number Off | Hide | Select all

<user>
<username>USERNAME</username>
<password>PASSWORD</password>
</user>

We might want to demonstrate this web service using a number of combinations of usernames and passwords, for instance:

robearl/password
rob_earl/pass_word
rob'earl/pass'word
rob-earl/pass-word
etc.

To do these tests we might create four XML documents containing each username/password pair. Every different combination we wanted to demonstrate would require a new document.

Instead, we can use sed. Create a single document like the one above and pass it through sed to replace the variables before passing it to the web service using curl:

bash

Line number Off | Hide | Select all

cat single_document.xml | sed -e 's/USERNAME/robearl/' -e 's/PASSWORD/password/' | curl <options> -d @- <web service>

Using this technique, we avoid having to create lots of XML documents and it's obvious what data is being passed without having to open up each document during the demonstration.

Original post blogged on LessThanDot.

Ignoring SVN Directories at the Command Line 14-05-2011 13:20:00

A colleague of mine told me this neat trick which I had to share:

bash

Line number Off | Hide | Select all

echo "FIGNORE=.svn" >> ~/.bashrc
source ~/.bashrc

Now when you're moving around subversion controlled directories with only one subdirectory, hitting tab won't try to autocomplete to ".svn/".

Original post blogged on LessThanDot.

Bulk Editing With Sed 30-04-2011 17:51:00

This week, after refactoring some XML based Java classes, I was left with ~50 XML documents (used for testing) with the wrong root element. After a few seconds pondering opening a text editor and find/replacing 50 times I had a better idea: sed.

sed can be used to perform regular expressions on each line of a file (or files). To solve the problem all it took was:

text

Line number Off | Hide | Select all

sed -i s/oldroot/newroot/ *
-i: edits the files in place instead of printing to stdout

All 50 documents updated, simple as that.

Original post blogged on LessThanDot.

Catching Key Conflicts with Perl's Class::DBI 31-01-2011 09:52:00

When inserting to a database I find it more convenient to catch a key conflict than to perform a select in order to check for duplicates. Defining a column as unique will ensure data integrity at the lowest level.

sql

Line number Off | Hide | Select all

CREATE TABLE customer (
id int AUTO_INCREMENT,
name varchar(20),
address varchar(50),
email varchar(50),
constraint PRIMARY KEY(id),
UNIQUE(email)
);

Catching a user attempting to register with an email address which is already in the database is fairly simple when using DBI:

perl

Line number Off | Hide | Select all

my $row = $dbh->do(qq{INSERT INTO customer ( name, address, email ) VALUES (?,?,?);}, undef, ("Rob", "Rob's House", "my\@email.com")); # Returns undef on error.
&error($dbh->errstr) unless $row; # Forward to error handler.

Class::DBI is a database abstraction Perl module. It wasn't immediately apparent to me how to handle a key conflict when using it. After a little investigation I found key violations will trigger an exception which makes it fairly simple to handle:

perl

Line number Off | Hide | Select all

eval
{
$new_cust = Customer->insert({name=>"Rob", address=>"Rob's House", email=>"my\@email.com" }); # Insert using Class::DBI constructor.
}; &error($@) if $@; # Forward to error handler.

The eval block will catch the thrown exception, allowing you to gracefully handle it through your own subroutine.

Original post blogged on LessThanDot.

Writing Perl Tests with Test::More 15-01-2011 14:29:00

Writing tests for code is just as important as writing the code itself. Time spent writing tests is less time spent tracking down and fixing bugs, making it a great investment. Despite knowing this it can sometimes be hard to force yourself to stop coding and write tests. Fortunately, Perl has some modules to make it pretty simple. Here's the module we're going to be testing:

perl

Line number Off | Hide | Select all

#!/usr/bin/perl
package MyMaths;
use strict;
sub new
{
my $proto = shift;
my $class = ref($proto) || $proto;
my $self = {};
bless($self, $class);
return $self;
}
sub add
{
my $self = shift;
my $total = 0;
foreach my $factor (@_)
{
$total += $factor;
}
return $total;
}
1;

As you can see, we have a simple class with one method, add, which adds together all the arguments passed to it. We now need to define a test file for this module, called mymaths.t:

perl

Line number Off | Hide | Select all

#!/usr/bin/perl
use Test::More tests=>6;
use MyMaths;
$mymaths = new MyMaths;
is( $mymaths->add(1,2,3), 6, "1 + 2 + 3 = 6");
is( $mymaths->add(6,2), 8,"6 + 2 = 8");
is( $mymaths->add(1,2,3,4), 10, "1 + 2 + 3 + 4 = 10");
is( $mymaths->add(1,2), 3, "1 + 2 = 3");
is( $mymaths->add(2), 2, "2 = 2");
is( $mymaths->add(2,-1), 1, "2 + -1 = 1");

This test file uses the is() method of the builtin Test::More module. This method takes 3 arguments: the test, the expected result and a description. If a test fails, is() will give some feedback on the test.

edit: To run the tests use the `prove` utility which is part of the Test::Harness package:

prove -v mymaths.t

Run all tests in the current directory with:
prove -v .

~~To run the tests we use another builtin module, Test::Harness. Create tests.pl:~~

perl

Line number Off | Hide | Select all

#!/usr/bin/perl
use Test::Harness qw(&runtests);
@tests = @ARGV ? @ARGV : <*.t>;
runtests @tests;

~~This script can either be run with a list of test files as arguments, or with no arguments to run all test files. Running the script results in:~~

text

Line number Off | Hide | Select all

./mymaths.t ..
1..6
ok 1 - 1 + 2 + 3 = 6
ok 2 - 6 + 2 = 8
ok 3 - 1 + 2 + 3 + 4 = 10
ok 4 - 1 + 2 = 3
ok 5 - 2 = 2
ok 6 - 2 + -1 = 1
ok
All tests successful.
Files=1, Tests=6, 0 wallclock secs ( 0.01 usr 0.02 sys + 0.02 cusr 0.01 csys = 0.06 CPU)
Result: PASS

Everything went fine, as expected. Now go back to the original class definition and make a typo on line 22, so it reads = instead of += and rerun the tests:

text

Line number Off | Hide | Select all

Most of the tests failed and gave good feedback to help find the problem.

Summary

Although writing tests can be a chore it is worth doing. If you write tests a little at a time:

When you design a class.
When you add a method to a class.
When you find a bug.

You'll develop a good library of tests without too much effort.

Further reading: Test::More perldoc

Original post blogged on LessThanDot.

My First Android Application (Ubuntu Linux) 05-01-2011 17:46:00

This post details my first adventure into developing for the Android platform and the steps taken to get the HelloWorld application to run.

Download the Linux SDK from: http://developer.android.com/sdk/index.html
Extract the archive and take note of the directory
Install Eclipse (`apt-get install eclipse`)
Install Eclipse plugin
1. Help->Install Software
2. Add repository: https://dl-ssl.google.com/android/eclipse
3. Select Developer Tools
4. Read and accept licenses
Configure Eclipse
1. Windows->Preferences->Android
2. Set SDK Location
3. Select OK
Setup SDK and Virtual Device
1. From Eclipse: Window->Android SDK and AVD Manager->Available Packages
2. Select Android Repository->Install Selected
3. Select Virtual devices->New: Name the device and select the target API version
Create Android Project in Eclipse
1. Project name: HelloAndroid
2. Application name: Hello
3. Package name: com.example.helloandroid
4. Create Activity: HelloAndroid
5. Build Target: Same as the target as selected for Virtual Device
Add the code below to src/com.example.helloandroid/HelloAndroid.java
Create a new Runtime configuration
1. Run->Run Configurations->Android->New
2. Project: HelloAndroid
3. Select Target tab and choose the Virtual Device configured earlier
Run!
After a couple of minutes the emulator should boot and display the HelloWorld application (You may need to unlock the screen first)

HelloAndroid.java

java

Line number Off | Hide | Select all

package com.example.helloandroid;
import android.app.Activity;
import android.os.Bundle;
import android.widget.TextView;
public class HelloAndroid extends Activity {
/** Called when the activity is first created. */
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
TextView tv = new TextView(this);
tv.setText("Hello, Android");
setContentView(tv);
}
}

For more information see the Android development documentation:

http://developer.android.com/sdk/installing.html
http://developer.android.com/sdk/eclipse-adt.html#installing
http://developer.android.com/resources/tutorials/hello-world.html

Original post blogged on LessThanDot.