Making a Haskell Interface for the Rosie Pattern Language

I discovered the Rosie Pattern Language at Dr. Jamie Jennings’ talk about it at Strange Loop 2018. The Rosie Pattern Language (RPL) is a DSL for parsing strings which is more convenient and easier to work with than regular expressions. RPL is a Lua library with a C Foreign Function Interface (FFI). It also has Python and Go interfaces, but I noticed that there wasn’t a way to call it from Haskell.

I met Dr. Jennings after the talk and offered to add a Haskell interface for RPL, which I started working on after Strange Loop. I haven’t used Haskell’s C FFI before, so I thought I’d write a post about it. The documentation for the FFI is a little sparse, so I’m going to walk through how I built the interface step by step.

Making a FFI Project

In order to build a project using the Haskell FFI, you need to link to C .o files when you call ghc. I modified the Makefile used by the RPL Go interface to link to the .o when building a Rosie.hs module:

LUA_FILES=$(wildcard ../../../submodules/rosie-lpeg/src/*.o)
LUA_EXTRA_FILES=$(wildcard ../../../submodules/lua-cjson/*.o)
LIB_LUA_FILES=$(wildcard ../liblua/*.o)
Rosie: Rosie.hs ../binaries/$(ROSIE_OBJECT_FILE_NAME)
    ghc --make -main-is Rosie -o Rosie Rosie.hs ../binaries/$(ROSIE_OBJECT_FILE_NAME) $(LUA_FILES) $(LIB_LUA_FILES) $(LUA_EXTRA_FILES)

As you can see, all I had to do to call ghc with the FFI was to add the –make flag, specify a main module using -main-is (in this case Rosie) as well as listing all .o files I wanted to link to.

You can see the full Makefile I wrote here:

Building a Rosie Haskell module

The first thing you need to add when you use the C FFI is the Foreign Function Interface language extension:

{-# LANGUAGE ForeignFunctionInterface #-}

I made a module called Rosie which exports some RPL data types and all of the functions exported by the RPL C libary:

module Rosie (RosieStr(..),
              main) where

All of the functions starting with cRosie are direct calls to the RPL C interface.

rosieStringLength, rosieStringChars, and unsafeNewEmptyRosieString are helper functions I wrote to work with RosieString data in Haskell.

The RPL C functions are very low level, only supporting manual memory management so you need to remember to free the memory you allocated when you’re done with it. The Rosie module lets you call these unsafe allocations but I also added newRosieString and newRosieEngine in order to add automatic memory management to the Rosie module.

Managing foreign data structures in Haskell

Haskell can only refer to C data structures by pointer. In general you will have a handle to a foreign data structure which looks like this:

Ptr RosieEngine

In order to get the Ptr data type, you need to call:

import Foreign.Ptr

If you don’t need to access the internals of a C data structure, as in this RosieEngine example, you can simply specify an empty value constructor:

data RosieEngine = RosieEngine

The C struct engine contains references to Lua interface libaries. It’s difficult to determine the memory layout for the Lua interface. Instead of opening that can of worms, I just forbid access to the internals of the RosieEngine.

If you want internal access to the C data structures from Haskell, you need to specify the internal components using the record syntax:

data RosieStr = RosieStr
                { len :: Word32
                , ptr :: CString
                } deriving (Show, Eq)

CString is a C style string data structure you can access by calling:

import Foreign.C

and Word32 is a 32 bit unsigned integer which you can import using:

import Data.Word (Word32)

Note: CString is just a type alias to Ptr CChar.

In order to read and write from a RosieStr in the C code, I had to make RosieStr an instance of the Storable typeclass:

instance Storable RosieStr where
    alignment _ = 8
    sizeOf _    = 16
    peek ptr    = RosieStr
        <$> peekByteOff ptr 0
        <*> peekByteOff ptr 8
    poke ptr (RosieStr l p) = do
        pokeByteOff ptr 0 l
        pokeByteOff ptr 8 p

alignment is used to specify the alignment of each item in the record in memory. alignment is the least common multiple of the sizes of all data types in the record. In my case, I have a Word32, which is 4 bytes and a Ptr CChar, a pointer which on a 64 bit machine is 8 bytes in size; so alignment = 8, the lowest common multiple of 4 and 8.

peek and poke specify how a RosieStr is read from and written to memory. They use peekByteOff and pokeByteOff to read and write bytes from a pointer and offset in memory. Each element must be aligned to its size in memory. len is at offset 0, which is fine because 0 is a multiple of 4 (the size of Word32). ptr can’t be at offset 4, even though that would result in a tightly packed data structure because 4 isn’t a multiple of 8 (the size of Ptr), so we need to leave 4 bytes of empty space in the data structure and put ptr at an offset of 8 bytes.

In some cases, we may need to create a new RosieStr which is empty. I made a helper function for this which uses the new function from the Foreign.Marshall.Utils library:

import Foreign.Marshal.Utils

unsafeNewEmptyRosieString :: IO (Ptr RosieStr)
unsafeNewEmptyRosieString = new (RosieStr { len = 0, ptr = nullPtr })

I made a similar data structure for pattern matches called RosieMatch.

Calling foreign functions from Haskell

Now that we’ve covered how to make a foreign data structure, let’s look at how to call foreign functions. You can import a foreign function from a linked .o file using the foreign import ccall statement:

foreign import ccall "rosie_new_string_ptr" cRosieNewString :: CString -> Word32 -> IO (Ptr RosieStr)

This statement imports the C function rosie_new_string_ptr as cRosieNewString (which I modified to match Haskell’s function naming conventions). It has the type CString-> Word32 -> IO (Ptr RosieStr). You can call cRosieNewString with a CString and a Word32 length to create a new RosieStr on the heap.

Note: There is a function in the RPL C interface called rosie_new_string which returns a Rosie string by value. Haskell doesn’t support foreign functions which return foreign structs by value, so I couldn’t add that to the Haskell interface.

It’s very important for type safety that this function evaluates to a value in the IO monad! Foreign functions can have uncontrolled side-effects which must be captured or the guarantees provided by Haskell will be broken. The RPL interface functions can do unsafe things like leak memory and modify their inputs. The IO monad captures these side effects appropriately.

All I had to do to import all of the RPL functions is add a foreign import ccall statement for each of them.

At this point we have everything we need to call Rosie Pattern Language from Haskell!

Making a safer RPL interface

It’s still possible to leak memory with cRosieNewString, cRosieNew, and unsafeNewEmptyRosieString because the user has to manually call cRosieStringFree, cRosieFinalize, and free to free the memory they allocated for each.

Instead of relying on these manual memory allocation functions, I made newRosieString and newRosieEngine to allocate ForeignPtr managed memory pointers. Let’s look at newRosieString:

newRosieString :: String -> IO (ForeignPtr RosieStr)
newRosieString s = do
    let l = fromIntegral (length s)
    pRString <- withCString s (\cString -> cRosieNewString cString l)
    newForeignPtr ptrRosieFreeStringPtr pRString

newRosieString takes a Haskell string, and extracts its length using (length s). It then calls withCString with the Haskell string s and a lambda function. withCString converts the Haskell string s into a CString and passes it as the argument to the lambda function. The lambda function calls cRosieNewString to make a new Ptr RosieString. Finally, I create a new ForeignPtr RosieStr by calling:

newForeignPtr ptrRosieFreeStringPtr pRString

newForeignPtr takes a pointer to a function for freeing the Ptr RosieStr and the Ptr RosieStr itself and evaluates to a ForeignPtr RosieStr which will garbage collect itself when all references to it are unreachable.

In order to get a function pointer to free the Rosie string, I used:

foreign import ccall "&rosie_free_string_ptr" ptrRosieFreeStringPtr :: FunPtr (Ptr RosieStr -> IO ())

Importing a foreign function with a & prefix imports the function as a function pointer.

You need to call the following to import newForeignPtr and ForeignPtr:

import Foreign.ForeignPtr

Finally, let’s look at newRosieEngine:

foreign import ccall "&rosie_finalize" ptrRosieFinalize :: FunPtr (Ptr RosieEngine -> IO ())

newRosieEngine :: ForeignPtr RosieStr -> IO (ForeignPtr RosieEngine)
newRosieEngine messages = 
    withForeignPtr messages (\m -> do
        engine <- cRosieNew m
        newForeignPtr ptrRosieFinalize engine)

newRosieEngine uses withForeignPtr instead of withCString. It converts its first argument from a ForeignPtr RosieStr to a Ptr RosieStr and passes it as the argument to the lambda. The lambda constructs a new Ptr RosieEngine by calling cRosieNew and makes a memory managed ForeignPtr RosieEngine using newForeignPtr.

Right now, in order to call Rosie functions, you need to convert your Haskell data structures to Ptrs and ForieignPtrs using the with and withForeignPtr functions. My plan is to add some functional helper functions to make this process less verbose.

You can see the full Rosie module here:


Rosie Pattern Language

Rosie Pattern Language GitLab

Rosie Pattern Language IBM


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s