oylenshpeegul's blog

The Most Wonderful Time of the Year

It's Advent and you know what that means...Perl calendars!

• Perl
• Perl 6
• Japan
  • Casual
  • Hacker
• Dancer
• OX

Also, there is Sys Advent!

Happy coding!

 

I just spotted Nuba Princigalli's tweet on pretty colorful JSON. Neat!

My first thought was that Perl could do the download without curl

perl -MLWP::Simple -MDDP -MJSON -e 'p decode_json get shift'
     http://www.reddit.com/r/perl.json

My next thought was that Mojolicious could do the whole thing

perl -Mojo -E 'say r j g(shift)->body'
     http://www.reddit.com/r/perl.json

The g does the HTTP GET, the j decodes the JSON, and the r dumps it out. But it uses Data::Dumper. How to get it to use Data::Printer instead? I guess we could just use Data::Printer directly

perl -Mojo -MDDP=alias,dump,output,stdout -e 'dump j g(shift)->body'
    http://www.reddit.com/r/perl.json

Both Mojo and DDP are defining a p, so I'm asking DDP to use dump instead. And DDP uses stderr by default, so I'm asking it to use sdtout instead.

Hrm. Not very satisfying. I feel like I'm fighting both modules. Is there a Data::Printer plugin for Mojolicious that I'm not aware of?

asynchronicity in Go

Recently, I talked about asynchronous page loads in Perl three different ways (using AnyEvent, Mojolicious, and HTTP::Async).  Commenters fixed the Mojo example and added an IO::Lambda example. Another poster created a Perl 6 example.

Today, I did it in Go, which has concurrency built in to the language. That is, no special modules are needed to load the pages asynchronously;  if we wrote a program to load the pages one after another instead, the import lines would be the same. 

package main

import (
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "time"
)

func loadpage (url string, c chan<- string) {
    start := time.Now()
    response, error := http.Get(url)
    if error != nil {
        c <- fmt.Sprintln(error)
        return
    }
    length, _ := io.Copy(ioutil.Discard, response.Body)
    response.Body.Close()
    c <- fmt.Sprintf("%-25s has length %8d and loaded in %.2f s\n",
                     url, length, time.Since(start).Seconds())
}

func main() {
    urls := []string{
        "https://www.google.com",
        "http://www.windley.com/",
        "https://www.bing.com",
        "http://www.example.com",
        "http://www.wetpaint.com",
        "http://www.uh.cu",
    }

    start := time.Now()

    c := make(chan string)

    for _, url := range urls {
        go loadpage(url, c)
    }

    for i := range urls {
        fmt.Print(i, " ", <-c)
    }

    fmt.Printf("Total elapsed time %.2f s\n",
               time.Since(start).Seconds())
}

Running this gives

$ go run apl.go
0 Get https://www.bing.com: certificate is valid for a248.e.akamai.net,
*.akamaihd.net, *.akamaihd-staging.net, not www.bing.com
1 https://www.google.com    has length    12400 and loaded in 1.01 s
2 http://www.windley.com/   has length    50178 and loaded in 1.04 s
3 http://www.wetpaint.com   has length    74952 and loaded in 1.06 s
4 http://www.example.com    has length     2966 and loaded in 1.09 s
5 http://www.uh.cu          has length    57304 and loaded in 2.97 s
Total elapsed time 2.97 s

Nice! I'm still new at Go, but so far I really like it.

YAPC::NA 2012

I've just returned home from Madison, Wisconsin, where I attended YAPC::NA (Yet Another Perl Conference, North America). It was fantastic!

I heard Larry Wall answer questions

 

I saw mst talk about automation

 I met garu (the guy who wrote Data::Printer)

 

I saw Tom Christiansen talk about Unicode

 

and lots more

 

A lot of the talks are online, so if you missed them live, you can still catch them on video. I think I'm going to go check out some of the ones I missed (there are four or five talks at a time, so you can't see everything).

 

I'm afraid I dropped the ball on the Knitting BOF. I happened to meet Kelley and auggy, just by coincidence, but I missed Charlton. I laid a bunch of Perl-related knitting out on the table on game night to try to attract the attention of any other knitters who may have been lurking, but my apologies to anyone expecting a more formal BOF session. Even without a BOF, I got quite a few rows done on this hat, in addition to the mittens I made while traveling. Also, I was pleased to be able to give away a couple of Perlish dishcloths while I was there.

asynchronicity

I've never played much with asynchronous page loads before and then it's come up three times this month.

First, I saw this post on AnyEvent, so I gave that try

#!/usr/bin/env perl

use v5.14;
use warnings;
use AnyEvent;
use AnyEvent::HTTP;
use Time::HiRes qw(time);

my $cv = AnyEvent->condvar( cb => sub { warn "done";});

my @urls = (
    "https://www.google.com",
    "http://www.windley.com/",
    "https://www.bing.com",
    "http://www.example.com",
    "http://www.wetpaint.com",
    "http://www.uh.cu",
);

my $start = time;
my $result;

$cv->begin(sub { shift->send($result) });

foreach my $url (@urls) {
    $cv->begin;
    my $now = time;
    my $request;
    $request = http_request(
        GET => $url,
        timeout => 2, # seconds
        sub {
            my ($body, $hdr) = @_;
            my $u   = sprintf "%-25s", $url;
            my $et  = sprintf "%5.3f", time - $now;
            my $len = sprintf   "%8s", $hdr->{'content-length'};
            push @$result, ($hdr->{Status} =~ /^2/)
                ? "$u has length $len and loaded in $et s"
                : "Error for $u: ($hdr->{Status}) $hdr->{Reason}";
            undef $request;
            $cv->end;
        }
    );
}
$cv->end;
warn "End of loop\n";
my $foo = $cv->recv or die;
say for @$foo;
my $tet = sprintf "%5.3f", time - $start;
say "Total elapsed time: $tet s";

$ ./anyevent.pl
End of loop
done at ./anyevent.pl line 11.
https://www.bing.com      has length    31711 and loaded in 0.826 s
http://www.example.com    has length     2966 and loaded in 0.854 s
https://www.google.com    has length    32384 and loaded in 1.767 s
http://www.windley.com/   has length    91898 and loaded in 2.120 s
http://www.wetpaint.com   has length    93698 and loaded in 2.506 s
http://www.uh.cu          has length    18493 and loaded in 4.269 s
Total elapsed time: 4.284 s

Jesse Shy saw that as well, which resulted in this Mojolicious solution

#!/usr/bin/env perl

use v5.14;
use warnings;
use Mojo::UserAgent;
use Mojo::IOLoop;
use Time::HiRes qw(time);

my @urls = (
    "https://www.google.com",
    "http://www.windley.com/",
    "https://www.bing.com",
    "http://www.example.com",
    "http://www.wetpaint.com",
    "http://www.uh.cu",
);

my $ua = Mojo::UserAgent->new;

my $start = time;

foreach my $u (@urls) {
    my $now = time;
    $ua->get($u => sub {
        my($ua, $tx) = @_;
        if (my $res = $tx->success) {
            my $url = sprintf "%-25s", $u;
            my $et  = sprintf "%5.3f", time - $now;
            my $len = sprintf "%8s", length($tx->res->body);
            say "$url has length $len and loaded in $et s";
        }
        else {
            my($message, $code) = $tx->error;
            say "Error for $u : ($code) $message";
        }
        Mojo::IOLoop->stop;
    });
    Mojo::IOLoop->start;
}

my $tet = sprintf "%5.3f", time - $start;
say "Total elapsed time: $tet s";

$ ./anymojo.pl
https://www.google.com    has length    11287 and loaded in 0.521 s
http://www.windley.com/   has length    91898 and loaded in 0.590 s
https://www.bing.com      has length        0 and loaded in 0.117 s
http://www.example.com    has length        0 and loaded in 0.079 s
http://www.wetpaint.com   has length    93698 and loaded in 1.489 s
http://www.uh.cu          has length    18530 and loaded in 2.993 s
Total elapsed time: 5.792 ms

Then Naveed Massjouni mentioned HTTP::Async just by coincidence, so I had to try that to compare

#!/usr/bin/env perl

use v5.14;
use warnings;
use HTTP::Async;
use HTTP::Request;
use Time::HiRes qw(time);

my @urls = (
    "https://www.google.com",
    "http://www.windley.com/",
    "https://www.bing.com",
    "http://www.example.com",
    "http://www.wetpaint.com",
    "http://www.uh.cu",
);

my $start = time;
my $async = HTTP::Async->new;

# These ids are just a one-up numbers, starting at 1? Why not 0?
my @ids = $async->add( map {HTTP::Request->new(GET => $_)} @urls );

my $now = time;
while (my($response, $id) = $async->wait_for_next_response) {
    my $url = sprintf "%-25s", $urls[$id-1];
    my $len = sprintf   "%8s", length $response->content;
    my $et  = sprintf "%5.3f", time - $now;
    say "$url has length $len and loaded in $et s";
    $now = time;
}

my $tet = sprintf "%5.3f", time - $start;
say "Total elapsed time: $tet s";

$ ./httpasync.pl
https://www.bing.com      has length    32157 and loaded in 0.157 s
http://www.example.com    has length     2966 and loaded in 0.106 s
http://www.windley.com/   has length    83915 and loaded in 0.154 s
https://www.google.com    has length    32379 and loaded in 0.052 s
http://www.uh.cu          has length    18593 and loaded in 1.464 s
http://www.wetpaint.com   has length    92333 and loaded in 0.556 s
Total elapsed time: 3.800 s

All three seem to work, but the AnyEvent timings are the only ones that really make sense. I'm not quite sure what to do about that.

---

Update: In addition to Blipsofadoug's improvements to the Mojo solution and Dimitry Karasik's IO::Lambda solution down in the comments below, don't miss ttjjss's Perl 6 solution over here!

What's wrong with v?

This morning, I came upon this tweet. I don't understand what is blowing the considerable mind of RJBS. I'm guessing he considers this post to have settled the matter. But just because he's willing to manually convert his version numbers to decimal format, that doesn't mean we all are. I write modules like this

package RJBS v1.2.3 {
    our $year = 2012;
}

and scripts like this

#!/usr/bin/env perl

use v5.14;
use warnings;
use RJBS v1.2.3;

say "I'm using RJBS version ", RJBS->VERSION;
say "It's $RJBS::year, for pete's sake!";
say "I'm not writing ", version->parse(RJBS->VERSION)->numify,
    " any more. Forget it.";

I don't see the problem.

$ ./rjbs.pl
I'm using RJBS version v1.2.3
It's 2012, for pete's sake!
I'm not writing 1.002003 any more. Forget it.

I guess if you still care about older perls, it might be an issue. But I don't. Heck, I don't even assign to $VERSION directly anymore. Indeed, I think that feature alone is reason to upgrade to Perl 5.12 (and the sexy new package syntax is worth upgrading to Perl 5.14 for).

I put "use v6" in my Perl 6 scripts and "use v5.14" in my Perl 5 scripts, so--- yeah--- when I care about the version number, I want to put "use Foo v1.2.3". If that is an accident of history, then it's a happy one.

Dumping data in the Perl REPL

Recently, I mentioned dumping data in Perl. More recently, in the comments we learned about the "print_escapes" property. I decided I usually wanted that,  so I added it to my .dataprinter file

# -*- perl -*-
{
    print_escapes => 1,
}

I also added the DataPrinter plugin to my .re.pl/repl.rc file

# -*- perl -*-

use feature qw(say);
use Term::ANSIColor;

my @plugins = qw(
    Colors
    DataPrinter    ### THIS IS THE ONLY THING I ADDED
    FancyPrompt
);

sub myprompt {
    my $color = shift // 'black';
    my $string = shift // '> ';
    sub {color('reset').color($color).$string.color('reset')};
}

$_REPL->load_plugin($_) for @plugins;
$_REPL->fancy_prompt(myprompt('green', 'perl5> '));
$_REPL->fancy_continuation_prompt(myprompt('blue', 'perl5* '));

# Don't leave the terminal like that!
END {say '';}

So now I get a much more colorful REPL experience.

 

Note the other thing from garu's comment: the 'carriage return' gets quoted because I used a space instead of an underscore for it this time; the other hash keys do not get quoted.

 

Barack Obama's coming to the DC-Baltimore Perl Workshop?

No, probably not. But I wonder how he'd get there if he were. Well, I've just read about Google::Directions, so let's find out!

#!/usr/bin/env perl

use v5.14;
use warnings;
use Google::Directions::Client;
use Mojo::DOM;

my $origin = shift // die "\n\tUsage: $0  starting_address\n\n";

my %params = (
    origin      => $origin,
    destination => '800 S Rolling Rd, Catonsville, MD 21228',
);

my $response = Google::Directions::Client->new->directions(%params);
die $response->status unless $response->status eq 'OK';

my $leg = $response->routes->[0]->legs->[0];

my $miles = sprintf "%.1f", $leg->distance / 1609.344;
my $minutes = int 0.5 + $leg->duration / 60;
say "Approximately $miles miles (about $minutes minutes)";

while (my($i, $step) = each @{$leg->steps}) {
    printf "%3d. ", $i+1;
    say Mojo::DOM->new( $step->html_instructions )->all_text;
}

I've just hard coded in the address to DCBPW and used Mojo::DOM to scrub the HTML directions. I thought about using Term::ReadLine to prompt for the starting address, but I decided it was just as easy to grab it off the command line.

$ ./dcbpw.pl '1600 Pennsylvania Ave, Washington, DC'
Approximately 34.9 miles (about 52 minutes)
  1. Head north on 17th St NW toward H St NW
  2. Take the 1st right onto H St NW
  3. Turn left onto New York Ave NW
  4. Continue onto K St NW
  5. Turn left onto 7th St NW
  6. Take the 2nd right onto New York Ave NW
  7. Continue onto US-50 E Entering Maryland
  8. Take the Balt-Wash Pkwy exit on the left toward Baltimore
  9. Merge onto MD-295 N
 10. Take the exit onto I-195 W toward I-95/Catonsville
 11. Exit onto S Rolling Rd Destination will be on the left

See you in Catonsville!

Hello, GNOME 3!

Recently, I mentioned I installed Crunchbang on my laptop and I'm loving it.  But my desktop machine is still running Ubuntu and today I did a bunch of upgrades. Well, little did I know, this batch of upgrades included updating the desktop to GNOME 3. That was unexpected! I knew about GNOME 3, but I figured it would be in the upcoming Ubuntu 12.04.

The first thing I noticed was that the focus was not follwing the mouse. I don't like having to click to focus, so I fired up the Compiz settings manager and--- sure enough--- "click to focus" was checked. So I unchecked it and that seemed to work.

The next thing I noticed was that the mouse wheel no longer changed workspaces. Everybody else in the world--- Xfce, KDE, LXDE, Openbox--- can change workspaces with the mouse wheel by themselves, but for some reason the mighty GNOME needs help from Compiz. So, back to the settings manager and, again, the settings for "workspace next" and "workspace prev" were back to "disabled". I set them to "Button 4" and "Button 5", respectively.

And...nothing. No effect. Then I noticed that the GNOME 3 workspace configuration was unrelated to the settings in the settings manager.

So, even though the "click to focus" check box worked, the workspace settings are being quietly ignored. The way to change workspaces in GNOME 3 is with C-M-up_arrow an C-M-down_arrow (there is no left and right...the workspaces grow downwards as needed).

Thankfully,  I discovered GNOME Shell extensions¹.  These are not an official part of GNOME Shell, I guess, but it seems we can't live without them. For Ubuntu, they live in a different repository.

sudo add-apt-repository ppa:ferramroberto/gnome3
sudo apt-get update
sudo apt-get install gnome-shell-extensions-common

You could log out and back in again, but I just reloaded the GNOME shell with

M-F2 r 

GNOME Shell extensions are a way for folks to submit their own customizations. In particular, someone named markos created Desktop Scroller. I installed that and now I can change workspaces with the mouse wheel if I move the pointer over to the right edge of the screen. Good enough. I usually have enough stuff in each workspace that I have to move the pointer near an edge to find an exposed chunk of root window anyway. And the comments suggest improvements are coming, so soon it might do exactly what I want. Thanks, markos!

---

[1] This only works in Firefox, apparently. My usual browser is Google Chrome, but when I visit extensions.gnome.org with that it says, "You do not appear to have an up to date version of GNOME3. You won't be able to install extensions from here. See the about page for more information".

Dumping data in Perl

We often need to dump data in Perl. The standard library has Data::Dumper and Dumpvalue, but when I have good access to CPAN, I prefer to use Data::Dump or YAML. Last year, I heard about Data::Printer and used that for a while, but switched back to Data::Dump because I couldn't distinguish between a blank string and a string with just nulls in it. If I had data that looked like

my $stuff = {
    blank => "",
    null => "\0",
    undefined => undef,
    zed => 0,
    zero => '0',
};

Data::Printer gave me

\ {
    blank       "",
    null        "",
    undefined   undef,
    zed         0,
    zero        0
}

whereas Data::Dump gave

{ blank => "", null => "\0", undefined => undef, zed => 0, zero => 0 }

and YAML gave

---
blank: ''
null: "\0"
undefined: ~
zed: 0
zero: 0

Yes, the Data::Printer output is very colorful, but I really need to see that null! I was thinking that Data::Printer was eating it, but today I had occasion to run a Data::Printer program inside EShell and I noticed the null was still there!

\ {
    blank       "",
    null        "^@",
    undefined   undef,
    zed         0,
    zero        0
}

So I just need to get my terminal to show it to me. Or get Data::Printer to output something that my terminal is happier showing me. First, I tried adding a filter to Data::Printer like so

use Data::Printer
    filters => {SCALAR => sub { ${$_[0]} =~ s/\0/\\0/gr;}};

but that called my SCALAR filter instead of the built-in SCALAR filter. I couldn't figure out how to call my SCALAR filter and then hand off to the built-in one. So I hacked the the Data::Printer source code to change the nulls in the built-in SCALAR filter.

I ended up surveying a bunch of different data dumpers with a few more whitespace characters added to the mix.

#!/usr/bin/env perl

use v5.14;
use warnings;

my $stuff = {
    blank => "",
    null => "\0",
    undefined => undef,
    zed => 0,
    zero => '0',
    tab => "\t",
    newline => "\n",
    carriage_return => "\r",
    form_feed => "\f",
};

use Data::Dumper;
$Data::Dumper::Sortkeys = 1;
say "\nData::Dumper";
say Data::Dumper::Dumper $stuff;

use Dumpvalue;
say "\nDumpvalue";
say Dumpvalue->new->dumpValue($stuff);

use Data::Dump qw(pp);
say "\nData::Dump";
pp $stuff;

use YAML;
say "\nYAML";
say YAML::Dump $stuff;

use Data::Printer;
say "\nData::Printer";
p $stuff;

use JSON;
say "\nJSON";
say to_json $stuff, {pretty => 1};

use XML::Dumper;
say "\nXML::Dumper";
say pl2xml $stuff;

use Data::Dumper::Concise;
say "\nData::Dumper::Concise";
say Data::Dumper::Concise::Dumper $stuff;

use Data::Dump::Streamer;
say "\nData::Dump::Streamer";
Data::Dump::Streamer::Dump $stuff;

use Data::PrettyPrintObjects;
say "\nData::PrettyPrintObjects";
say PPO $stuff;

use Data::TreeDumper;
say "\nData::TreeDumper";
say DumpTree $stuff;

Running this (with the unmodified Data::Printer) gives

Data::Dumper
$VAR1 = {
          'blank' => '',
          'carriage_return' => '
',
          'form_feed' => '',
          'newline' => '
',
          'null' => '',
          'tab' => '    ',
          'undefined' => undef,
          'zed' => 0,
          'zero' => '0'
        };


Dumpvalue
'blank' => ''
'carriage_return' => "\cM"
'form_feed' => "\cL"
'newline' => '
'
'null' => "\c@"
'tab' => "\cI"
'undefined' => undef
'zed' => 0
'zero' => 0


Data::Dump
{
  blank => "",
  carriage_return => "\r",
  form_feed => "\f",
  newline => "\n",
  null => "\0",
  tab => "\t",
  undefined => undef,
  zed => 0,
  zero => 0,
}


YAML
---
blank: ''
carriage_return: "\r"
form_feed: "\f"
newline: "\n"
null: "\0"
tab: '  '
undefined: ~
zed: 0
zero: 0


Data::Printer
\ {
    blank             "",
    carriage_return   "",
    form_feed         "
                       ",
    newline           "
",
    null              "",
    tab               "  ",
    undefined         undef,
    zed               0,
    zero              0
}


JSON
{
   "null" : "\u0000",
   "undefined" : null,
   "carriage_return" : "\r",
   "zed" : "0",
   "tab" : "\t",
   "blank" : "",
   "zero" : "0",
   "form_feed" : "\f",
   "newline" : "\n"
}


XML::Dumper
<perldata>
 <hashref memory_address="0x185ede8">
  <item key="blank"></item>
  <item key="carriage_return">
</item>
  <item key="form_feed"></item>
  <item key="newline">
</item>
  <item key="null"></item>
  <item key="tab">  </item>
  <item key="undefined" defined="false"></item>
  <item key="zed">0</item>
  <item key="zero">0</item>
 </hashref>
</perldata>


Data::Dumper::Concise
{
  blank => "",
  carriage_return => "\r",
  form_feed => "\f",
  newline => "\n",
  null => "\0",
  tab => "\t",
  undefined => undef,
  zed => 0,
  zero => 0
}


Data::Dump::Streamer
$HASH1 = {
           blank   => '',
           carriage_return
                   => "\r",
           form_feed
                   => "\f",
           newline => "\n",
           null    => "\0",
           tab     => "\t",
           undefined
                   => undef,
           zed     => 0,
           zero    => 0
         };

Data::PrettyPrintObjects
{
  blank           => '',
  carriage_return => '
',
  form_feed       => '',
  newline         => '\n',
  null            => ,
  tab             => '  ',
  undefined       => undef,
  zed             => 0,
  zero            => 0
}


Data::TreeDumper

|- blank =   [S1]
|- carriage_return = [\r]  [S2]
|- form_feed =   [S3]
|- newline = [\n]  [S4]
|- null =   [S5]
|- tab = [\t]  [S6]
|- undefined = undef  [S7]
|- zed = 0  [S8]
`- zero = 0  [S9]

I'm not sure if I'd rather look at whitespace or escape sequences for things like tabs and newlines, but I'm positive I want to be able to see the difference between an empty string (which is false in Perl) and a string containing a null (which is true in Perl).

---

Update: (22 Jan 2012) garu applied my patch, so now Data::Printer shows the nulls as \0

$ cpanm Data::Printer
--> Working on Data::Printer
Fetching http://search.cpan.org/CPAN/authors/id/G/GA/GARU/Data-Printer-0.27.tar.gz ... OK
Configuring Data-Printer-0.27 ... OK
Building and testing Data-Printer-0.27 ... OK
Successfully installed Data-Printer-0.27 (upgraded from 0.26)
1 distribution installed
$ perl -MDDP -e '$h = {null => "\0"}; p $h'
\ {
    null   "\0"
}

---

Update: (23 Jan 2012) garu spruced it up even more (see comments)!

$ cpanm Data::Printer
--> Working on Data::Printer
Fetching http://search.cpan.org/CPAN/authors/id/G/GA/GARU/Data-Printer-0.28.tar.gz ... OK
Configuring Data-Printer-0.28 ... OK
Building and testing Data-Printer-0.28 ... OK
Successfully installed Data-Printer-0.28 (upgraded from 0.27)
1 distribution installed

$ perl -MDDP -e '$h = {null => "\0"}; p $h'
\ {
    null   "\0"
}

$ perl -MDDP -e '$h = {foo => "\t"}; p $h'
\ {
    tab   "    "
}

$ perl -MDDP=escape_chars,0 -e '$h = {foo => "\t"}; p $h'
\ {
    tab   "\t"
}