2016/04/05

rustlang dynamic library part 2

A previous post was about getting a simple rust dynamic library working. The next thing I wanted to do was try passing a more complex type from the dynamic library to the executable using it.

First, I changed dynamiclib to:

#[derive(Debug)]
pub struct Tuple (u32,);

#[derive(Debug)]
pub struct MyStruct {
    s1: &'static str,
    s2: String,
    t: Tuple,
}

#[no_mangle]
pub extern "C" fn test() -> MyStruct {
    MyStruct {
        s1: "hello",
        s2: "world".to_string(),
        t: Tuple(47),
    }
}
And dynamic to:
#[link(name="dynamiclib")]
extern {
    fn test() -> MyStruct;
}

fn main() {
    let retval = unsafe { test() };
    println!("Got: {:?}", retval);
}
And (somewhat surprisingly) it worked as expected. Although you get the following error:
warning: found struct without foreign-function-safe representation annotation in foreign module, consider adding a #[repr(C)] attribute to the type, #[warn(improper_ctypes)] on by default
At first I considered appeasing the warning, but if it's a library written in rust and called from rust, shouldn't the default
repr(Rust)
be safe/correct?

Next, I decided to try loading and re-loading the shared library using this more complex struct... and I started getting SIGSEGV the second time I attempted to use the library.

I came across libloading (crates.io) which purports to be safer wrapper for working with dyanmic libraries, and if nothing else has much better documentation.

A quick change to dynamic/src/main.rs (devoid of error-checking):

    let lib = Library::new("libdynamiclib.dylib").unwrap();
    let test: Symbol MyStruct> = unsafe {
        lib.get(b"test\0").unwrap()
    };
    println!("Got: {:?}", test());
And it works correctly even when re-loading the library. So, I was either using stale symbols, or something else. Still several things that I obviously don't fully understand. Hopefully pouring over Rustonomicon will reveal the secrets to me.

Xamarin Forms part 1

This is the first part of my Xamarin Forms dive.

Binding

Arguably the single most important thing to master as it connects your View (UI) to the ViewModel, it's only fitting I start out with Binding (1, 2 and 3). Since I’ve been using Forms I’ve found that binding is key to getting things working but various nuances are scattered about in numerous demos.

Bindings connect a "source" (either another UI element in the View or something in the ViewModel) to a "target" (some UI element in the View) that displays it.

To start with, you'll see that most projects create an XML namespace that refers to a .Net assembly generated by your project:

<!-- "xmlns:local" ties "local" XML namespace to the "luck" C# namespace in the "luck" assembly -->
<ContentPage
    xmlns="http://xamarin.com/schemas/2014/forms"
    xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
    xmlns:local="clr-namespace:luck;assembly=luck"
    x:Class="luck.MainPage"
    >
</ContentPage>

// "local:MainViewModel" in XAML thus refers to a C# class similar to:
namespace luck {
    public class MainViewModel {
        // Implementation
    }
}
You can then easily create static bindings, references, and resources:
<!-- Static -->
<Label Text="{x:Static local:MainViewModel.Text}" />

// C# ViewModel
public class MainViewModel {
    public static string Text = "text";
}

<!-- Reference -->
<Label Text="text" x:Name="MyText" />
<Label Text="{x:Reference MyText}" />

<!-- Resource -->
    <ContentPage.Resources>
        <ResourceDictionary>
            <local:LocalizationConverter
                x:Key="LocalizationConverter" />
        </ResourceDictionary>
    </ContentPage.Resources>
    <Label Text="{Binding SelectedItem.DisplayName,
        Converter={StaticResource LocalizationConverter}}" />
There's additional information about Resources and markup extensions here, and I'll come back to ResourceDictionary a bit later in this series when I get to Styles.

Alternatively, the Binding markup extension is the most common binding technique and more advanced as it supports several different properties. The easiest way of specifying a source is by setting the BindingContext for the entire Page since it is inherited by all children. This can be done either in your code-behind constructor:

public MainPage ()
{
     InitializeComponent ();
     BindingContext = new MainViewModel (); 
}
or via XAML:
    <ContentPage.BindingContext>
        <local:MainViewModel />
    </ContentPage.BindingContext>
You can then bind a simple value:
<Label Text="{Binding Text}" />

public class MainViewModel {
    public string Text { get; set; }
}
As already mentioned, Binding supports several different properties. Path specifies the property of the source to which we are binding and can be omitted if it's the first property. The following two are equivalent:
<Label Text="{Binding Text}" />
<Label Text="{Binding Path=Text}" />
The Path property is able to access sub-properties as well as the index operator. For example:
<Label Text="{Binding Values.Count}" />
<Label Text="{Binding Values[key]}" />

// In the ViewModel
public Dictionary Values { get; set; }
According to this, if you have large numbers of values to set you can simplify multiple indexed bindings by setting the binding context:
<StackLayout BindingContext="{Binding Values}" />
    <Label Text="{Binding [key]}" />
<StackLayout />
There is also a shorthand syntax to bind directly to the BindingContext itself:
<Label Text="{Binding .}" />
The Source property allows the binding to explicitly specify the "source" of a binding. It's analogous to the BindingContext (where the target specifies the source) and the following are equivalent:
<Label BindingContext="{x:Reference items}" Text="{Binding SelectedItem, StringFormat='{0}'}" />
<Label Text="{Binding SelectedItem, Source={x:Reference items}, StringFormat='{0}'}" />
The biggest difference being using BindingContext will be inherited by all sub-children of the UI element. At one point I found myself repeatedly specifying BindingContext and Source for a number of nested elements and then running into problems where I could no longer reference the original BindingContext, etc. In this case I had to carefully consider what I was binding where and reorganise the data provided by my ViewModel to cut back on the excessive use of Source.

StringFormat can be used to apply string conversion and formatting:

<Label Text="{Binding Value, StringFormat='Formatted {0}'}" />
Internally it uses String.Format method so it accepts all the same forms. Note that because both .Net formatting and markup extensions use curly braces, StringFormat must be enclosed in single quotes.

The Converter property is more generalised than StringFormat and also accepts an optional ConverterParameter value:

 <Label Text="{Binding SomeInt, Converter={StaticResource IntConverter}, ConverterParameter=10}" />

 public class IntConverter : IValueConverter
 {
  public object Convert(object value, System.Type targetType, object parameter, System.Globalization.CultureInfo culture)
  {
   int intparam;
   if (!int.TryParse(parameter as string, out intparam))
    intparam = 1;

   return (int)value * intparam;
  }

  public object ConvertBack(object value, System.Type targetType, object parameter, System.Globalization.CultureInfo culture)
  {
   int intparam;
   if (!int.TryParse(parameter as string, out intparam))
    intparam = 1;

   return ((int)value) / intparam;
  }
 }
Mode The final property Mode is available to solve some specific binding problems. I've not done anything with it just yet, so I'll refer to two relevant documents from Xamarin (1, 2) and revisit it later when I understand it better.

Dynamic Bindings

For values that can be changed at runtime, either the property you're binding to or the entire ViewModel needs to implement INotifyPropertyChanged interface. For example, using System.Collections.ObjectModel.ObservableCollection:
public static ObservableCollection Chats = new ObservableCollection();


You can even define your own:
    public class ObservableString : System.ComponentModel.INotifyPropertyChanged
    {
        public event System.ComponentModel.PropertyChangedEventHandler PropertyChanged;

        string val;
        public string Value
        {
            get { return val; }
            set {
                val = value;
                OnPropertyChanged ("Value");
            }
        }

        void OnPropertyChanged(string propertyName = null)
        {
            if (PropertyChanged != null)
                PropertyChanged (this, new System.ComponentModel.PropertyChangedEventArgs (propertyName));
        }
    }
But this only seems to work with static bindings. The more standard approach is to implement INotifyPropertyChanged for your entire ViewModel:
    public class LoginViewModel : INotifyPropertyChanged
    {
        bool m_isBusy = false;
        public bool IsBusy {
            get {
                return m_isBusy;
            }
            set {
                if (value != m_isBusy) {
                    m_isBusy = value;
                    OnPropertyChanged ("IsBusy");
                }
            }
        }

        #region INotifyPropertyChanged

        public event PropertyChangedEventHandler PropertyChanged;

        protected virtual void OnPropertyChanged (string propertyName)
        {
            var changed = PropertyChanged;
            if (changed != null) {
                PropertyChanged (this, new PropertyChangedEventArgs (propertyName));
            }
        }
        #endregion
    }
With .Net 4.5 OnPropertyChanged can be simplified:
        public bool IsBusy {
            get {
                return m_isBusy;
            }
            set {
                if (value != m_isBusy) {
                    m_isBusy = value;
                    OnPropertyChanged ();
                }
            }
        }

        protected virtual void OnPropertyChanged ([System.Runtime.CompilerServices.CallerMemberName] string propertyName = "")
        { /* same as above */ }
A minor improvement, but now the compiler guarantees that the property name always matches the "magic string". There’s further improvements to reduce all the boilerplate needed to implement properties (specifically this, and more generally like this and this).

This ran a bit long, but it should wrap up the first dump of my notes on Xamarin Forms. Next up: Callbacks, Commands, and Triggers... oh my!

2016/03/25

Xamarin Forms

While never a UI programmer or UX/UI designer, I’ve used or been exposed to several UI/layout/formatting systems over the years: LATEX, Tkinter, MFC, Scaleform, HTML/stylesheets, several in-house systems, EZGUI and uGUI in Unity, etc. And no matter what, it seems like the UI of every project ends up horribly out of control. Occasionally (often?) no attempt is made to embrace a sensible workflow or sanely design/organise the UI. But, even when MVC or some other established design pattern is attempted it usually fails.

More often than not it’s due to lack of really understanding the model, developers taking shortcuts to "save time" (this is really a production failure), or a disconnect with the pattern in the heat of implementation. But, I think a framework or environment (i.e. lack of process) that permits such transgressions is just as much to blame.

One of the things that initially attracted me to Xamarin was the ability to develop multi-platform native apps in a language we were already using for other things- C#. Additionally, via Xamarin Forms their embrace of MVVM (wikipedia) similar to WPF. While I don't believe in silver bullets, I do like investigating the approaches taken by different frameworks.

Invariably, any UI-oriented project needs superfluous animation to give it flair. Spurned by this post I decided this was a good opportunity to investigate ways of translating design requirements into concrete implementation using Xamarin’s framework targeting MVVM. Particularly looking at how it enables for creating a mostly data-driven UI, and permitting UI prototyping or UX/UI iteration de-coupled from writing code (an insurmountable obstacle to most non-programmers for some reason- although that’s a post for another day).

There’s lots of basic tutorials on getting a button on screen etc. so I’m not really covering that and assume you already have a basic understanding of Xamarin Forms. I want to reference some of the various other sources I’ve looked at, how and why I was trying to compose things, and the problems I ran into. For reference, this is all in the context of a companion app prototype I'm writing for our in-development PC/XboxOne MMOFPS. Because it's still under-wraps I obviously can't post screenshots, but I'll try to present (carefully scrubbed) code.

This series is broken into three parts: basics, intermediate, and finally visual aspects where I tie everything together for my fabulous programmer UX.

2016/03/21

rustlang dynamic library

Was wanting to figure out how to do a rust dynamic library because I've been thinking about how I could apply software updates to a running system. I came across dynamic_lib which is apparently deprecated and replaced with dylib (with the same api), but there's still fairly little documentation. Pretty much the only simple working example I was able to find was this StackOverflow question. But it's from over a year ago and predates even rust 1.0 (I'm currently using 1.7).

To start off, use cargo to setup two projects; one for the library itself, another for the executable that uses it:

cargo new dynamiclib
cargo new --bin dynamic
cd dynamiclib
Add the following to dynamiclib's Cargo.toml:
[lib]
crate-type = ["dylib"]
Setting the crate-type to "dylib" means a dynamic library will be built (rather than an executable or static library). Save the following as src/lib.rs:
#[no_mangle]
pub extern "C" fn test() -> u32 {
    47
}
And build the library by running:
cargo build
Next, we need to create the executable that loads and uses the library we just created. Add the following to dynamic/Cargo.toml:
[dependencies]
dylib = "0.0.2"
And save the following as dynamic/src/main.rs:
extern crate dylib;
use dylib::DynamicLibrary;
use std::path::Path;

fn main() {
    // I'm on OSX, obviously
    match DynamicLibrary::open(Some(Path::new("libdynamiclib.dylib"))) {
        Ok(lib) => {
            let test = unsafe {
                let ptr = lib.symbol::("test").unwrap();
                println!("Found it: {:?}", ptr);
                std::mem::transmute::<_, fn() -> u32>(ptr)
            };
            println!("Got: {}", test());
        },
        Err(e) => println!("Failed: {}", e)
    }
}
I eventually came across this post that's about calling rust from Node.js, and it made me realise that dylib is just a wrapper around Foreign Function Interface (FFI) (as opposed to some special/magical functionality provided by the runtime or a core library). This is easily confirmed if you check the dylib source; it just uses libc to call dlopen and kin. That means that rather than using prepend_search_path to ensure the dynamic library can be found, you can set LD_LIBRARY_PATH to the output path of dynamiclib (e.g. $HOME/dynamiclib/target/release). Allowing you to build and run the executable with:
LD_LIBRARY_PATH=$HOME/dynamiclib/target/release crate run
I'm admittedly a little bit disappointed that dynamic libraries are so painfully close to native interop preventing idiomatic implementations that transparently work like native code. Something a bit more like the magic in .Net- but that would bring the runtime nastiness that rust tries so hard to avoid.

It's worth noting that if you have no intention of re-loading the dynamic library and you simply want to link with it, I could have changed dynamic/src/main.rs to:

#[link(name="dynamiclib")]
extern {
    fn test() -> u32;
}

fn main() {
    println!("Got: {}", unsafe { test() });
}
Now dynamiclib needs to be in one of the locations that rust searches for libraries (e.g. dynamic/target/debug) at build time.

2016/03/05

The Rising Tide of Alt Languages

We're in some kind of renaissance of programming languages. When I was first starting out I'd used a few languages (Basic, Pascal, C, a porpourri of assembly dialects), but the lack of Internet prevented me from knowing about all the miscellaneous toy "research" languages and I was only vaguely aware of Fortran/Cobol and so on. In Uni I was exposed to Java, VHDL, Scheme, Python, and a few more assembly dialects.

But upon graduating, for years and years I regarded myself as a snooty, elitist c++ programmer. Besides occasional forays into Python or (eventually) c# for doing "scripting" or maybe making tools, I accepted that all real work was done with c++. Other languages were relegated to fooling around, teaching children, or "web programming"- and a select few (Perl) seemed to be specifically designed to torment people. After years of in-house engines, the horrors of UE3/CryEngine, etc. it took a few prototypes with Unity3D to make me see the errors of my ways. My university professors were correct; "software engineering" was about a lot more than banging out code and sweet macro shenanigans.

More recently, JavaScript- which in my mind is still closely associated 90's-era "mouseover" events- came back on the radar on account of the Node.js juggernaut. Only this time it wasn't just for throw away web clients, it was for servers too. And, while I've still yet to look into it, I am aware that it is a thing. And a big one at that.

As part of Microsoft's increasingly virile affair with open source and multi-platform love-ins they released Visual Studio Code. After a bit of rummaging around, I was shocked to find out it too was made in JavaScript with a framework called Electron. Poking around their website uncovered a number of interesting platforms for developing apps/games like pixate, ionic, fireball, and superpowers. All of which I think of as the "next-generation" app/game development tools- similar to MaxPlay (which specifically mentions Electron in a Software Engineer job posting).

I've been infatuated with F# for a while now. It had this sudden-outbreak-of-common-sense freshness about it: options, statically typed with powerful type inference, interoperability with c#, asynchronous computation expressions (monads), and other cornerstones of the functional world. But maybe I'd just grown tired of OO programming; it was getting a bit old and fat and FP was new hotness. Or, the old is new again, rather. Used it for a pair of backend prototypes but nothing in production. Somehow I can't justify subjecting other people to the OCaml/ML universe just because it amuses me.

For a while we were evaluating golang for constructing our new backend. Partially swept up by it's apparently burgeoning popularity in China, we were specifically motivated by the experiences of Qihoo et al. I was also intrigued by the kind of performance numbers they were claiming (TODO: add disclaimer regarding benchmarks and contrived tests and real-world performance blah blah blah). I was pleased with their sensible approach to errors and how easy it was to get a working program, but more than a little disappointed with their insistence on flip-flopping between (IMHO) overly opinionated design decisions and embracing horrific mistakes like void* and the null.

I've been spending some of my free time toying around with Rust. I've probably started reading The Book or futzed with Rust By Example 2 or 3 times now. But, I get busy or distracted and put it aside for a month or two such that I never really get it and am still sufficiently noob that I spend most of my time fighting with the compiler. Maybe it's too strict... or maybe I need to stop trying to write c++ with it.

While we're on the topic of c++ "killers", let's not write out Java just yet. Despite my expectations that Oracle would succeed in smothering it, it's instead staged a comeback and clawed itself to the top of the Tiobe index. It's also being used (along with Akka) for the backend being developed in parallel by one of our other teams.

As a final note, it's difficult to bring up f#, golang, and rust without giving a shoutout to the numerous other languages that are often mentioned in the same breath: julia, kotlin, scala, nimrod, and numerous others. There really isn't enough time in a day to investigate them all to a satisfactory level.

Exciting stuff.

2016/03/02

C# Wrapper for AllJoyn via SWIG

I was original interested in AllJoyn because it looked like a good way to build local multiplayer mobile games. At one point a few years back they even provided a Unity3D library. I got a simple "hello world" app working on PC/iOS/Android, but then got busy with other things and the project was eventually cancelled.

I've been thinking about it again and looking at ways of getting it into .Net applications. Unfortunately they no longer provide the Unity library which would have given me a good start.

Ran into some issues with their Linux build instructions. apt-get didn't want to install ia32-libs since I already had ncurses installed. Next, the src zip directory structure didn't quite match what's in the documentation (I guess they want you to move directories around). But you can just run scons from the root of the decompressed source code. I got a compile error about not being able to find sys/capability.h. apt-get had given up ia32-libs and didn't install libcap-dev. Once I got that everything seemed to work.

Basically, I ended up doing:

sudo apt-get install build-essential libgtk2.0-dev libssl-dev xsltproc libxml2-dev libcap-dev python scons libssl-dev
scons BINDINGS=cpp WS=off BT=off ICE=off SERVICES="about"

You can verify everything works by running the sample app. On a 64-bit system scons seems to build for x86_64 by default.

In one shell:

  1. export LD_LIBRARY_PATH=`pwd`/build/linux/x86_64/debug/dist/cpp/lib:$LD_LIBRARY_PATH
  2. build/linux/x86_64/debug/dist/cpp/bin/samples/basic_service
And in another:
  1. export LD_LIBRARY_PATH=`pwd`/build/linux/x86_64/debug/dist/cpp/lib:$LD_LIBRARY_PATH
  2. build/linux/x86_64/debug/dist/cpp/bin/samples/basic_client
Now I've got the native AllJoyn library, but I still need to get it working with .Net. SWIG is my first thought to do this. PInvoke Interop Assistant is potentially another option, but it looks like it hasn't been developed for some time. Following the SWIG tutorial I created the interface file alljoyn.swig specifying header files included by samples/basic/basic_client.cc:
%module alljoyn
%{
/* Includes the header in the wrapper code */
#include "alljoyn_core/inc/alljoyn/AllJoynStd.h"
#include "alljoyn_core/inc/alljoyn/BusAttachment.h"
#include "build/linux/x86_64/debug/dist/cpp/inc/alljoyn/Status.h"
#include "alljoyn_core/inc/alljoyn/Init.h"
#include "alljoyn_core/inc/alljoyn/version.h"

// Needed for alljoyn_wrap.cxx to compile
using namespace qcc;
using namespace ajn;
%}

/* Parse the header file to generate wrappers */
%include "alljoyn_core/inc/alljoyn/AllJoynStd.h"
%include "alljoyn_core/inc/alljoyn/BusAttachment.h"
%include "build/linux/x86_64/debug/dist/cpp/inc/alljoyn/Status.h"
%include "alljoyn_core/inc/alljoyn/Init.h"
%include "alljoyn_core/inc/alljoyn/version.h"
Then build everything:
mkdir alljoyn_cs
cd alljoyn_cs
# Create alljoyn.swig here

# Generate SWIG wrapper code
swig -c++ -csharp -cpperraswarn alljoyn.swig

# Build shared lib
g++ -fPIC -c -DQCC_OS_GROUP_POSIX -I../build/linux/x86_64/debug/dist/cpp/inc -I../alljoyn_core/inc -I../common/inc alljoyn_wrap.cxx
g++ -shared -L../build/linux/x86_64/debug/dist/cpp/lib -o liballjoyn.so alljoyn_wrap.o -lajrouter -Wl,-Bstatic -lalljoyn -Wl,-Bdynamic -lssl -lcrypto

# Update so mono can find liballjoyn.so
export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH
I included all the swig generated csharp source files into a MonoDevelop project, and you can now call simple AllJoyn functions from C#:
if (alljoyn.AllJoynInit () != QStatus.ER_OK)
 return;

Console.WriteLine ("AllJoyn Library version: {0}", alljoyn.GetVersion ());
Console.WriteLine ("AllJoyn Library build info: {0}", alljoyn.GetBuildInfo ());

var status = QStatus.ER_OK;

var msgBus = new BusAttachment ("myApp", true);
if (msgBus == null)
 status = QStatus.ER_OUT_OF_MEMORY;
That's a good start, but I've run into a problem. As expected, swig doesn't always generate "natural" c# code. BusAttachment.CreateInterface() has the following signature:
public QStatus CreateInterface(string name, SWIGTYPE_p_p_InterfaceDescription iface)
Where SWIGTYPE_p_p_InterfaceDescription is some pointer wrapper for the managed/unmanaged magic that swig generated for us rather than something like "out InterfaceDescription". It looks like swig provides typemaps to solve these sorts of problems, but in any case I'm not quite done yet. Misc other docs that were helpful:

2016/02/03

Flashbacks...

I completely forgot this even existed.

I just rummaged through drafts I'd started over half a decade ago. Most only contained a few notes on things I was working on back when "blogging" was still a thing.

Fascinating.