Coding – Developing Software

Posted Friday, April 21st, 2017

tl;dr Firefox Nightly on Linux supports running SlimerJS headlessly. More platforms and full headless Firefox are coming soon.

Over the last couple of years, I’ve worked on a few big web projects like PDF.js and PluotSorbet where I’ve wanted a better way to run automated tests in Firefox. My usual workflow would either involve opening a test page in Firefox manually, using a homegrown automation script that would pop open Firefox and steal focus as it ran tests, or fighting with XVFB to run Firefox in the background. I missed the days of running C++ or Node unit tests directly from the command line and not having some new window open. If only Firefox supported a headless mode…

Headless Firefox had been in the back of my mind for awhile, so when January rolled around this year and my previous project was winding down, I was happy to hear I’d be working on headless Firefox. It also turns out, I wasn’t the only person who wanted a headless mode for Firefox. We heard from multiple organizations and developers that they wanted an easier way to test Firefox and there was even a feature request nine years ago with quite a bit of following. For the first phase of headless browsing we decided to target SlimerJS, since it is a simpler application based on Firefox, it comes with an API for controlling the browser, and developers are already using it for testing. Fast forward a few months, and I’m happy to announce the I’ve finished the first big milestone by getting SlimerJS to run completely headlessly with Firefox Nightly on Linux.

If you’re on Linux and want to try it out:

Download and install Firefox Nightly.
Get the SlimerJS source. Note: once the changes land in a stable release you can just follow the regular download instructions for SlimerJS.

Shell

git clone https://github.com/laurentj/slimerjs.git cd slimerjs

1
2

git clone https://github.com/laurentj/slimerjs.git
cd slimerjs

Modify src/application.ini to bump the version of Firefox supported.

--- a/src/application.ini
+++ b/src/application.ini
@@ -8,7 +8,7 @@ Copyright=Copyright 2012-2017 Laurent Jouanneau & Innophi
 
 [Gecko]
 MinVersion=38.0.0
-MaxVersion=52.*
+MaxVersion=55.*
 
 [XRE]
 EnableExtensionManager=1

1

2

3

4

5

6

7

8

9

10

11

--- a/src/application.ini

+++ b/src/application.ini

[Gecko]

MinVersion=38.0.0

-MaxVersion=52.*

+MaxVersion=55.*

[XRE]

EnableExtensionManager=1

Create a SlimerJS test script. For example snapshot.js:

var webpage = require('webpage').create();
webpage
  .open('http://mozilla.com')
  .then(function () {
    // store a screenshot of the page
    webpage.viewportSize = { width: 650, height: 320 };
    webpage.render('page_snapshot.png',
                   { onlyViewport: true });
    slimer.exit();
  });

1

2

3

4

5

6

7

8

9

10

var webpage = require('webpage').create();

webpage

.open('http://mozilla.com')

.then(function () {

// store a screenshot of the page

webpage.viewportSize = { width: 650, height: 320 };

webpage.render('page_snapshot.png',

{ onlyViewport: true });

slimer.exit();

});

Test it out. Make sure to set the MOZ_HEADLESS environment variable and change the SLIMERJSLAUNCHER path to point to wherever you installed Firefox nightly

Shell

MOZ_HEADLESS=1 SLIMERJSLAUNCHER=/home/bdahl/nightly/dist/bin/firefox ./src/slimerjs snapshot.js

1

MOZ_HEADLESS=1 SLIMERJSLAUNCHER=/home/bdahl/nightly/dist/bin/firefox ./src/slimerjs snapshot.js

Next up, I’ll be working on getting Firefox to run headlessly without SlimerJS and instead being controlled by either WebDriver (Selenium) or maybe a devtools protocol (to be decided) and on getting headless working on MacOS and Windows. For more updates, follow along in the headless browsing meta bug and in SlimerJS’s bug.

Posted Tuesday, December 7th, 2010

About
With all the recent security problems in Adobe Acrobat and after reading about the new features of HTML5, I thought it would be interesting to build a PDF reader entirely with JavaScript and HTML5. Trapeze is still in its infancy and lacks full support for the PDF spec, but the majority of simple PDF’s render correctly using Firefox and Chrome. Although some will shudder at the thought of a JavaScript PDF reader, render speed for smaller PDF’s is relatively quick with today’s high performance JavaScript engines. Try out some sample files or use your own PDF’s (In testing Chrome is the stablest and quickest).

HTML5
Trapeze uses many new features available in HTML5 including: canvas, web workers, drag/drop files, and some of the new file API’s. The new canvas element is the biggest piece that enables all the 2D drawing commands that a PDF needs to render correctly. Web workers are used to parse the PDF files in a another thread so the browser doesn’t lock up. Web workers also draw the PDF to what I call a FauxCanvas which basically just stores all the operations to the a fake canvas object and then sends them back to the main thread to apply them to the real canvas when its ready. Hopefully, in the future this will not be needed if an off screen canvas is made available to web workers. The drag and drop file opening was added just to see how it works.

Some of the limitations:

No ability to open a PDF from a URL. This is because cross site AJAX requests are not allowed by the browser. To get around this a proxy can be used, but I don’t currently have the funds to pay for the bandwidth a proxy pdf server would require.
Only tested in Firefox and Chrome.
Missing support for portions of the PDF spec (full color space support, pdf functions, patterns, certain image types, …).

Credits
This project would have been a lot more work without the help of some previous PDF reader programs such as pdf-renderer and PDFBox. Also, thanks to the developers of JQuery, Binary Functions for JavaScript, and several other external JavaScript libraries I used that can be found in the external folder.

Source
The project source is available at https://github.com/brendandahl/trapeze-reader/

Posted Friday, August 13th, 2010

After learning that many lotteries now use software and hardware random number generators(RNG) and reading a story about Daniel Corriveau, I decided it would be fun to explore trying to figure the state of a RNG and predict values.

One of the most commonly used RNG algorithms today is the Mersenee Twister(MT) algorithm. On the wikipedia article on MT it mentions after observing 624 generated numbers it is possible to predict all future iterates. At first glance I didn’t see how this was possible because of the algorithm’s tempering function, but after reading comments on wikipedia the tempering function is bijective(reversible). If you’re looking for a good puzzle to solve, try writing your own reverse tempering function. The tempering function is defined as:

k = mt[state->mti];
k ^= (k >> 11); // Temper shift U
k ^= (k << 7) & 0x9d2c5680UL; // Temper shift S & TemperingMaskB
k ^= (k <> 18); // Temper shift T

1

2

3

4

k = mt[state->mti];

k ^= (k >> 11); // Temper shift U

k ^= (k << 7) & 0x9d2c5680UL; // Temper shift S & TemperingMaskB

k ^= (k <> 18); // Temper shift T

My solution is broken up in to the different parts and is definitely not optimized but it works.

class MersenneReverser
{
  private const UInt32 TemperingMaskB = 0x9d2c5680;
  private const UInt32 TemperingMaskC = 0xefc60000;

  static UInt32 undoTemper(UInt32 y)
  {
    y = undoTemperShiftL(y);
    y = undoTemperShiftT(y);
    y = undoTemperShiftS(y);
    y = undoTemperShiftU(y);
    return y;
  }
  static UInt32 undoTemperShiftL(UInt32 y)
  {
    UInt32 last14 = y >> 18;
    UInt32 final = y ^ last14;
    return final;
  }
  static UInt32 undoTemperShiftT(UInt32 y)
  {
    UInt32 first17 = y << 15;
    UInt32 final = y ^ (first17 & TemperingMaskC);
    return final;
  }
  static UInt32 undoTemperShiftS(UInt32 y)
  {
    /*
     * This one also sucked to figure out, but now I think i could write
     * a general one.  This basically waterfalls and keeps restoring original
     * bits then shifts the values down and xors again to restore more bits
     * and keeps on doing it.
     */
    UInt32 a = y << 7;
    UInt32 b = y ^ (a & TemperingMaskB);
    UInt32 c = b << 7;
    UInt32 d = y ^ (c & TemperingMaskB); // now we have 14 of the original
    UInt32 e = d << 7;
    UInt32 f = y ^ (e & TemperingMaskB); // now we have 21 of the original
    UInt32 g = f << 7;
    UInt32 h = y ^ (g & TemperingMaskB); // now we have 28 of the original
    UInt32 i = h <> 11;
    UInt32 b = y ^ a;
    UInt32 c = b >> 11;
    UInt32 final = y ^ c;
    return final;
  }
}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

class MersenneReverser

{

private const UInt32 TemperingMaskB = 0x9d2c5680;

private const UInt32 TemperingMaskC = 0xefc60000;

static UInt32 undoTemper(UInt32 y)

{

y = undoTemperShiftL(y);

y = undoTemperShiftT(y);

y = undoTemperShiftS(y);

y = undoTemperShiftU(y);

return y;

}

static UInt32 undoTemperShiftL(UInt32 y)

{

UInt32 last14 = y >> 18;

UInt32 final = y ^ last14;

return final;

}

static UInt32 undoTemperShiftT(UInt32 y)

{

UInt32 first17 = y << 15;

UInt32 final = y ^ (first17 & TemperingMaskC);

return final;

}

static UInt32 undoTemperShiftS(UInt32 y)

{

/*

* This one also sucked to figure out, but now I think i could write

* a general one. This basically waterfalls and keeps restoring original

* bits then shifts the values down and xors again to restore more bits

* and keeps on doing it.

*/

UInt32 a = y << 7;

UInt32 b = y ^ (a & TemperingMaskB);

UInt32 c = b << 7;

UInt32 d = y ^ (c & TemperingMaskB); // now we have 14 of the original

UInt32 e = d << 7;

UInt32 f = y ^ (e & TemperingMaskB); // now we have 21 of the original

UInt32 g = f << 7;

UInt32 h = y ^ (g & TemperingMaskB); // now we have 28 of the original

UInt32 i = h <> 11;

UInt32 b = y ^ a;

UInt32 c = b >> 11;

UInt32 final = y ^ c;

return final;

}

So then after observing 624 values and using the reverse temper function it is possible fill in the 624 internal state values of the mersenne twister algorithm and predict all the future values. Unfortunately this doesn’t help in many ways for practical purposes, since you need all 624 values unaltered. Take for instance the lottery. First, you don’t actually know the random number. Second, most likely the random number was used to shuffle the numbers in a certain way and only a portion of the random number was used. Third, if the algorithm gets reseeded each time you don’t know your 624 consecutive values. Also, some of the newer versions of the MT generate two values and combine them to create a better random double.

Posted Tuesday, October 20th, 2009

ERROR 1126 (HY000): Can't open shared library

1	ERROR 1126 (HY000): Can't open shared library

If you receive the error from MySQL while installing a plugin or a user defined function(UDF) there can be several causes:

You mistyped your dll file name, double check the name
MySQL can’t find the dll file in the plugin folder
1. Run:
  
  mysql> show variables like 'plugin_dir';
  
  1
  
  mysql> show variables like 'plugin_dir';
2. And make sure that path matches the path where your dll is stored.
A dependent dll can’t be found
1. Download Dependency Walker
2. Open your dll on the machine you’re installing the plugin. It should tell you if there was a missing dependency

In my case it turned out the be #3.

Dependency walker said I was missing MSVCR90D.dll, which I learned from Stack Overflow is actually the debug build of MSVCR90 (Notice: the “D” on the end). So I went back to Visual Studio Express 2008 and made sure my configuration was set to Release and rebuilt the build and copied it over and everything worked.

Posted Saturday, October 17th, 2009

If you want to execute a native SQL query and don’t care about the result, just grab the ADO.NET database connection from the NHibernate session and use that to run the query.

Here’s the code:

// Get the NHibernate Session (this line will change depending how you do it)
NHibernate.ISession session = SessionManager.GetInstance();
// Create a SQL Command
System.Data.IDbCommand command = session.Connection.CreateCommand();
// Set the query you're going to run
command.CommandText = "INSERT INTO Users ('Peter', 'Piper')";
// Run the query
command.ExecuteNonQuery();

1

2

3

4

5

6

7

8

// Get the NHibernate Session (this line will change depending how you do it)

NHibernate.ISession session = SessionManager.GetInstance();

// Create a SQL Command

System.Data.IDbCommand command = session.Connection.CreateCommand();

// Set the query you're going to run

command.CommandText = "INSERT INTO Users ('Peter', 'Piper')";

// Run the query

command.ExecuteNonQuery();

Headless SlimerJS with Firefox

Introducing Trapeze the Open-Source PDF Reader in JavaScript and HTML5

Reversing the Mersenne Twister RNG Temper Function

MySQL Error 1126

Raw SQL Queries with NHibernate