Foreign Function Interface and Perl

FFI::Raw - 2014-12-10

libffi is a Foreign Function Interface (FFI) used by a number of scripting languages and virtual machine platforms to call native code. It doesn't require a compiler, and as long as dynamic libraries can be found, development packages aren't necessary either.

For Perl, this means that FFI::Raw (Perl bindings for libffi) provides a viable alternative to the traditional Foreign Function Interface, known as XS. There are lots of reasons why you might not want to implement something using XS. For me the motivation is not having to delve into perlapi or perlxs, both of which are a fine prescription for madness. There are even some good reasons why you might want to use libffi instead of XS; it should work with any language that generates machine code, so go ahead and write your extensions in assembly or rust!

Calling a function

FFI::Raw has a constructor that takes the name of the library, the name of the function that you want to bind to, the return type for that function, and a list of the argument types. For example, I can call the C puts function on my Linux system like this:

use FFI::Raw;
my $lib = '/lib/x86_64-linux-gnu/libc.so.6';

my $puts = FFI::Raw->new(
  $lib, 'puts',
  FFI::Raw::int,  # return value type
  FFI::Raw::str,  # argument types
);

$puts->call("hello world");

I usually put the return value type and argument types on separate lines, as above, to differentiate them.

Recent versions of FFI::Raw have a shortcut where if you pass undef as the library argument it will search the currently running process for symbols. This is a good way to call functions in the standard C library, which usually has a different name on every platform.

my $puts = FFI::Raw->new(
  undef, 'puts',
  FFI::Raw::int,  # return value type
  FFI::Raw::str,  # argument types
);

Finding a library

For the rest of this article I am going to use libmagic as an example of how to create useful bindings. This library is commonly available on Unix systems and it provides an interface for determining the type of a file by its contents.

The first thing that we need to do is find the path to the libmagic library. To do that we will use FFI::CheckLib.

use FFI::CheckLib;
my($lib) = find_lib( lib => 'magic' );

The bindings

Now we need to create the bindings for the libmagic library like we did for puts before.

my $open = FFI::Raw->new(
  $lib, 'magic_open',
  FFI::Raw::ptr,
  FFI::Raw::int,
);
my $load = FFI::Raw->new(
  $lib, 'magic_load',
  FFI::Raw::int,
  FFI::Raw::ptr, FFI::Raw::str,
);
my $buffer = FFI::Raw->new(
  $lib, 'magic_buffer',
  FFI::Raw::str,
  FFI::Raw::ptr, FFI::Raw::ptr, FFI::Raw::int,
);
my $close = FFI::Raw->new(
  $lib, 'magic_close',
  FFI::Raw::void,
  FFI::Raw::ptr,
);

We also need some constants that are defined in magic.h:

use constant MAGIC_NONE => 0x000;
use constant MAGIC_MIME => 0x410;

Unfortunately there is no getting around this. Since we are not using a compiler, there is no reliable way of parsing the C header files, except for by implementing a C parser. Also requiring the C header files may mean installing development packages (if you recall one of the advantages of libffi is that we don't need those installed).

If the library that you are targeting changes its constants frequently then FFI::Raw may not be the tool that you want to use. On the other hand, if your library is frequently changing its interface like that then that library may not be the tool that you want to use either.

Using the bindings

Now we can write a useful program using the bindings that we have created.

# Create a handle for interacting with libmagic
# replace MAGIC_NONE with MAGIC_MIME to get the
# mime type instead of a description
my $magic = $open->call(MAGIC_NONE);

# Load the magic definitions file.  undef gets
# translated into NULL, which means use the 
# system default.
$load->call($magic, undef);

# read in the content of a file and convert it
# into a pointer
open my $fh, '<', "unknownfiletype";
my $content = do { local $/; <$fh> };
close $fh;
my $ptr = FFI::Raw::MemPtr->new_from_buff(
  $content, length $content,
);

# pass the buffer into libmagic and print the results!
say $buffer->call($magic, $ptr, length $content);

# free up the resources used by the magic handle
$close->call($magic);

In this example we use FFI::Raw::MemPtr to construct a memory pointer and copy the content of the file into that pointer. If you know that the library that you are using is not going to do any funny busines, like write into your buffer, then you can also skip the memory copy by using FFI::Util's scalar_to_buffer function:

my($ptr, $size) = scalar_to_buffer($content);
say $buffer->call($magic, $ptr, $size);

It returns a pointer to the data stored by the scalar and the size of the data stored.

Speed

FFI::Raw provides a less complicated, more portable method for calling machine code from Perl. Any FFI (calling one language from an other) will inherently involve extra overhead, when you cross the language barrier. Because FFI::Raw is implemented itself in XS, you are taking a hit from both XS and libffi every time you call a function in your target library. Both XS and libffi are relatively fast, but XS will frequently be somewhat faster assuming you tune your XS correctly. The question that you should ask yourself: is the time spent debugging code, tuning XS and reading perlapi really worth it? If more time is being spent in the library you are calling than in crossing the language barrier, then you should definitely consider FFI::Raw and save yourself some programmer time at the expense of a comparably small amout of CPU time.

Perl Advent Calendar 2014