Haxe extern keyword and modular applications

19.12.2012 4334 0

What does "extern" mean ?

Extern classes are kind of classes without any variable initialization neither any fonction's body (look like intrinsic AS2 classes...)
They just contain the variables visiblitity, their types and the functions' signatures too (function's arguments types and its return type).
This kind of classes are used in order to get the compiler completion and the comile-time type checking for classes which will exist at run-time but will not be a part of the current building project.
So, some years ago when I had to work on modular applications, I've experimented the Haxe's extern keyword and I've seen that Haxe didn't complain when a class tagged as extern contains variables initializations and functions' bodies, but just takes it as is for compile-time type checking and exclude it at code generation.
That permitted me doing interesting things quickly and without having an process of generating externs using Haxe compiler and maintain the duplicated code...which was the only other way of working with modular applications with Haxe :

--gen-hx-classes

My scenario

Targeting the Flash plateform, I have a main application that holds some core classes and loads the modules.
When I write a module, I sometimes have to use these core classes, so I used to do something like that in order to prevent code duplication :

package core;
import flash.Lib;
#if !main extern #end
class MyClass {
	public static inline var SIZE	= 7; 	// It's quite a good size...
	public static var myCollection = new Hash();
	public function new() : Void {
		Lib.current.addChild( new core.assets.MyAsset() );
	}
}
}

Notes :

  • This post is about extern semantics changes in Haxe 3 about variable initializations, so I added "critical" static non constant and static inline ones
  • In Haxe 2, extern class variables and types must be strictly typed (no inferance). But the constructor new must return the strange Void type...
  • IMPORTANT : I used to embed in my main app some assets that of course are available for the modules at run-time...That's why I've added the core.assets.MyAsset exemple.
    Note : In Haxe 2, you can use the full classptah in extern class' methods' bodies. Haxe just skip what happen inside the function's body. But you cannot use a top module import since Haxe will complain and look for this class.

Then I could use this class as is or as an extern one, depending on a compilation directive #main.
Build script for the main app :

-cp src
-main core.Main
-swf-lib assets/swf/assets.swf
-D main
-swf bin/main.swf

Of course, when building a module, I won't add this directive so the class will exist as is, available for the compiler auto completion and for the type checking on comile-time, but will not be generated to the output.
Build script for the module :

-cp src
-main modules.Module
-swf9 bin/module.swf

This all compiles well in Haxe 2.10

Changes in Haxe 3

As said in the intro, Haxe 3 brings new semantics for extern classes that can be seen on this picture :

On the picture, it looks great But the example above doesn't compile anymore, it fails saying : "Extern variables initializations must be a constant".
Ok, it means that there is from now, a different treatement for an extern class than before : Instead of just taking the class as is for compile-time and exclude it from the output, it asks for some new knowledge now on new Haxe extern semantics...(enough ?! )
Haxe, growing up on his 3rd major version, these changes must have reasons...
But, this time, and in my case, having for years now, a lot of projects built on this kind of modular system using the extern keyword, it brought me additional work and some new experimentations...
These changes on extern come "in order to clean things up and not allow invalid syntax." acording to Nicolas Cannasse, the Haxe creator.
On my side, I just can't see what was invalid in all that ?
I think a valid language sythax should be the same on a normal or an extern class used in this same language but anyway it's my own opinion and time now to see how we can get the same behaviour in Haxe 3 as before in Haxe 2.10 or less (or how to bypass the new Haxe extern semantics !)

Haxe 2 extern concept simulated in Haxe 3

Since the release 2.06, Haxe has got macros. And in the haxe.macro.Compiler class, there is a exclude method, which as its name tells, exclude a class or an entire package from compilation, and that we can use in the build script :

--macro exclude
// What is an alias for :
--macro haxe.macro.Compiler.exclude

It seemed to be a good solution for my extern problematics and doing this way, there should no more be needed to use a compilation directive. This gives me the raw core class like that :

package core;
import flash.Lib;
class MyClass {
	public static inline var SIZE	= 7; 	// It's quite a good size...
	public static var myCollection = new Hash();
	public function new() : Void {
		Lib.current.addChild( new core.assets.MyAsset() );
	}
}

And this build script for the module :

-cp src
-main modules.Module
-D haxe3
-swf9 bin/module.swf
--macro exclude( 'core.MyClass' ) // here I am 

Let's test it :

characters 24-49 : Class not found : core.assets.MyAsset

Hmm, Fail !
Why ? It seems that here Haxe omits the variables initializations, so it works fine until there. But what really happen ?
Since no extern keyword is used so it seems normal. Haxe just follows the normal class semantics not the extern one.
But then, the compiler checks the whole class too even if it excludes it then...strange way and because of that, it isn't really a good solution here
No problem, there are still other ways to do that !
It seems, we haven't many choices but control the build process of the class and something like map the normal class semantics into the extern ones

Control the build process using macros

We can use the @:build meta on the class for that. This way you should add this meta on all the classes like that :

package core;
import flash.Lib;
#if !main @:build( MyMacro.makeExtern() ) #end
class MyClass {
	public static inline var SIZE	= 7; 	// It's quite a good size...
	public static var myCollection 	= new Hash();
	public function new() : Void {
		Lib.current.addChild( new core.assets.MyAsset() );
	}
}

Here, we call our custom macro method that will patch the fields of the building class.
This macros method looks like that :

#if macro
import haxe.macro.Context;
import haxe.macro.Expr;
#end
@:macro class MyMacro {
    // Makes a normal class as an extern one (bypass Haxe extern var initializations semantics)
    public static function makeExtern() : Array {
        var fields	    = Context.getBuildFields();
        var hasSuperClass   = Context.getLocalClass().get().superClass != null;
	var isInline	    = function( fieldIn : Field ) {
            for ( access in fieldIn.access ) {
	        switch( access ) {
	            case AInline   : 
		        return true;
		    default        :
	        }
	    }
	    return false;
	}
	for ( field in fields ) {
	    switch( field.kind ) {
		case FVar( t, _ )	    :		
	            if ( isInline( field ) )
			continue;
		    field.kind = FVar( t, null );
		case FProp( g, s, t, _ )    :
		    if ( isInline( field ) )
			continue;
		    field.kind = FProp( g, s, t, null );
		case FFun( f )    :
		    var exprs = [];
		    // Constructor super call
		    if ( field.name == 'new' && hasSuperClass ){
			switch( f.expr.expr ) {
		            case EBlock( exprsIn )	:
				for ( expr in exprsIn ) {
				    switch( expr.expr ) {
					case ECall(e, p)	:
					    switch( e.expr ) {
						case EConst( c )	:
						    switch( c ) {
							case CIdent( s )	:
							    if ( s == 'super' ) {
								exprs.push( { expr : ECall( e, p ), pos : Context.currentPos() } );
							    }
							default:
						    }
						default:
					    }
					default:
				    }
				}
			    default:
			}
		    }
		    if ( f.ret != null ) {
			switch( f.ret ) {
			    case TPath( p )	:
				switch( p.name ) {
				    // Flash9 Float, Int, Bool can't be null
				    case "Float" :
					exprs.push( { expr : EReturn( { expr : EConst( CFloat( "0" ) ), pos : Context.currentPos() } ), pos : Context.currentPos() } );
				    case "Int" :
					exprs.push( { expr : EReturn( { expr : EConst( CInt( "0" ) ), pos : Context.currentPos() } ), pos : Context.currentPos() } );	
				    case "Bool" :
					exprs.push( { expr : EReturn( { expr : EConst( CIdent( "false" ) ), pos : Context.currentPos() } ), pos : Context.currentPos() } );	
				    case "Void"	:
										
				    default	:
					exprs.push( { expr : EReturn( { expr : EConst( CIdent( "null" ) ), pos : Context.currentPos() }  ), pos : Context.currentPos() } );	
				}
			    default:
			}
		    }
		    f.expr.expr = EBlock( exprs );
	     }  
	}
        //Context.getLocalClass().get().exclude();	// It makes the class as tagged with extern keyword (isExtern = true)
        Context.getLocalClass().get().isExtern = true;	// The real code generation exclude process
	return fields;
    }
}

Notes :

  • We just empty all variables initializations and all fonctions' bodies
  • Becareful about the static inline that has to be initialized immediately...
  • Becareful also about methods' return values.On Flash9 plateform i.e, Float, Int and Bool can't be null...
    A chance is that, static inline variable's initialization is the same in both the normal and the extern semantics, so we can let it as is
  • This macro works on all my projects using Haxe SVN r5691. If other changes come later, this example shows us how to proceed in order to control the building process of a type as a class so it wouldn't be hard to adjust this function later.

Update :

Some changes has occured since I wrote this post, so the new macro function looks like that now :

#if macro
import haxe.macro.Context;
import haxe.macro.Expr;
#end
@:macro class MyMacro {
    // Makes a normal class as an extern one (bypass Haxe extern var initializations semantics)
    public static function makeExtern() : Array {
        var fields	    = Context.getBuildFields();
        var isInline	    = function( fieldIn : Field ) {
            for ( access in fieldIn.access ) {
	        switch( access ) {
	            case AInline   : 
		        return true;
		    default        :
	        }
	    }
	    return false;
	}
	for ( field in fields ) {
	    switch( field.kind ) {
		case FVar( t, _ )	    :		
	            if ( isInline( field ) )
			continue;
		    field.kind = FVar( t, null );
		case FProp( g, s, t, _ )    :
		    if ( isInline( field ) )
			continue;
		    field.kind = FProp( g, s, t, null );
		default :
	     }  
	}
        Context.getLocalClass().get().exclude();	// Behaviour has changed
        return fields;
    }
}

The final solution

To finish, another more elegant way to do the same thing is to use the @:autoBuild meta on an empty interface, that will be a kind of magic interface.
Then we have to make all the classes we want to be "externables" implement this magic interface.
This interface calls the same macro method on all types that implement it during the building process (Doesn't that remember the magic haxe.xml.Proxy class, haxe.rtti.Generic interface or things like that ? )
Anyway, all that looks like that :
Interface :

@:autoBuild( MyMacro.makeExtern() ) extern interface IExtern {}

The "externable" class :

package core;
import flash.Lib;
class MyClass #if !main implements IExtern #end {
	public static inline var SIZE	= 7; 	// It's quite a good size...
	public static var myCollection 	= new Hash();
	public function new() : Void {
		Lib.current.addChild( new core.assets.MyAsset() );
	}
}

Compilation time

I've done some tests on projects using 100 extern classes and compared the compiler processing behaviour (-v) and the compilation time (--times) using both Haxe 2 classic extern keyword and the new Haxe 3 macro process (and mixed also...)
Here are the results :
Haxe 2.10 - classic extern keyword :

Total time : 0.248s
------------------------------------
filters : 0.005s, 2%
generate swf : 0.003s, 1%
other : 0.049s, 20%
parsing : 0.018s, 7%
typing : 0.167s, 67%
write swf : 0.006s, 3%

Haxe 2.10 - Macro process :
Total time :  1.432s
------------------------------------
filters : 0.005s, 0%
generate swf : 0.003s, 0%
macro execution : 1.104s, 77%
other : 0.063s, 4%
parsing : 0.058s, 4%
typing : 0.193s, 13%
write swf : 0.006s, 0%

Haxe 3 - Macro process :
Total time : 0.496s
------------------------------------
filters : 0.006s, 1%
generate swf : 0.005s, 1%
macro execution : 0.166s, 33%
other : 0.062s, 12%
parsing : 0.056s, 11%
typing : 0.190s, 38%
write swf : 0.010s, 2%

Note : Macros process time has been really improved since Haxe 2.10, but it's still 2x longer than Haxe 2 classic extern behaviour.

Sources and downloads

You can download the macro method and the examples sources here.

A last thing...

I've experimented many other ways to achieve my goal, that I haven't wrote in this final post. But during these experimentations, I've seen many things in details, what's going on behind the Haxe compiler scene, about the build process, the different macros steps (build script and Context.onGenerate, inline macro call or @:build and @:autoBuild...)
The errors that I encoured when doing these tests also permit me to see some of the huge work done in Haxe 3 and especially to understand the real reasons and impacts on this Haxe extern keyword story...
But I have a strange feeling that many of Haxe 3 changes are linked to the Haxe macros system, as "opened to the public".
And it seems that many features are kind of "translated" into macros, which is still a long process, even if improved and using the server compilation...The compiler completion and the compilation time are extended
Haxe macros seems also to be a bit weak because incompatible with some ohter features (macros in macros, some generic manipulations...)
Will we let macros kill the legendary Haxe compiler speed ?!
I hope you enjoyed reading this post as I really enjoyed doing these experimentations !

Commentaires

Laisser un commentaire

http://
×