Scritping the guest VM and OS in VirtualBox from python

I like Innotek/Sun/Oracle VirtualBox. I liked the GUI of it. I liked the VBoxManage command line utility to maintain and manage the guest machines on my host. But later, I wanted to make the guest OS what I want - wanted to script them!

I was looking for a usable solution how could I script the guest OS inside of the vm in VirtualBox and accidentally I found the automatically installed /usr/lib/virtualbox/vboxshell.py python script, which is also a pretty good Cisco IOS fashioned shell. Its power is the usability and the open API. You can find similar possibility in VBoxManage command, but this is python, and rather a shell.

Let's see an example how good is it.

$ /usr/lib/virtualbox/vboxshell.py
Running VirtualBox version 3.2.10
vbox> list
    Machine 'OpenSolaris' [e2efac1d-4cd6-40ec-c388-d33fbd3c93b2], machineState=Saved, sessionState=Closed
    Machine 'OpenIndiana' [e6434c9f-eef5-43fd-9e63-cac8bf23d1f9], machineState=PoweredOff, sessionState=Closed
    Machine 'ubuntu64' [ffdf6bd9-81b5-433b-be0d-b4b342e9fa76], machineState=PoweredOff, sessionState=Closed
vbox> start ubuntu64

And voilà the guest ubuntu just started after listing the virtual machines. Now, from version 3.2 VirtualBox it has a great feature (note, the help command always helps):

vbox> help typeGuest
    typeGuest: Type arbitrary text in guest: typeGuest Linux "^lls\n&UP;&BKSP;ess /etc/hosts\nq^c" 0.7 

So let's use it, and type the command of "date" here:

vbox> typeGuest ubuntu64 "date\n"

And you can check the started VM's screen, it executed the keystrokes. Now it's time to make a screenshot about this output:

vbox> help screenshot
    screenshot: Take VM screenshot to a file: screenshot Win /tmp/win.png 1024 768
vbox> screenshot ubuntu64 /tmp/bubuntu.png
Saving screenshot (720 x 400) screen 0 in /tmp/bubuntu.png...
vbox>

and here we go, we got a capture about our guest VM.

Now let's play with it, create a snapshot of the guest's state, then change it by a "date" command, try to restore the snapshot of a running machine (you got the error message), turn off (powedown) the vm and restore to the previous snapshot and start the vm:

vbox> snapshot help
Take snapshot:    snapshot vm take name
Restore snapshot: snapshot vm restore name
Merge snapshot:   snapshot vm merge name
vbox> snapshot ubuntu64 take 'Snapshot 3' 'blahblah'
vbox> typeGuest ubuntu64 "date\n"
vbox> snapshot ubuntu64 restore 'Snapshot 3'
0x80bb0002 (Cannot delete the current state of the running machine (machine state: Running))
vbox> powerdown ubuntu64
vbox> snapshot ubuntu64 restore 'Snapshot 3'
vbox> start ubuntu64
vbox>

Now you can see, you even can reach the shooting starson the sky. :) Why are these good for us? Because it's python, and you can use this shell as a primary, easy-going API, to script the VirtualBox. How? On the host, you will need:

toma@clamshell:~$ mkdir pybox
toma@clamshell:~$ cd pybox
toma@clamshell:~/pybox$ mkdir vboxshell
toma@clamshell:~/pybox$ cp /usr/lib/virtualbox/vboxshell.py vboxshell/__init__.py #making a library from python script
toma@clamshell:~/pybox$ cat - > tst.py
#!/usr/bin/env python

machine_name = 'ubuntu64'
snapshot_name = 'Snapshot 2'
type = 'gui' #'headless' | 'gui'
delay = 0.0001

from vboxshell import * #using our newborn library
from vboxapi import VirtualBoxManager
from time import sleep

style = None
g_virtualBoxManager = VirtualBoxManager(style, None)
ctx = {'global':g_virtualBoxManager,
       'mgr':g_virtualBoxManager.mgr,
       'vb':g_virtualBoxManager.vbox,
       'const':g_virtualBoxManager.constants,
       'remote':g_virtualBoxManager.remote,
       'type':g_virtualBoxManager.type,
       'run': lambda cmd,args: runCommandCb(ctx, cmd, args),
       'guestlambda': lambda id,guestLambda,args: runGuestCommandCb(ctx, id, guestLambda, args),
       'machById': lambda id: machById(ctx,id),
       'argsToMach': lambda args: argsToMach(ctx,args),
       'progressBar': lambda p: progressBar(ctx,p),
       'typeInGuest': typeInGuest,
       '_machlist': None,
       'prompt': g_prompt
       }
ctx['perf'] = ctx['global'].getPerfCollector(ctx['vb'])

#doing the things with guest vm:
snapshotCmd(ctx, ['snapshot', machine_name, 'restore', snapshot_name])
#if it's already started, let's move on:
try:
    startCmd(ctx, ['start', machine_name, type])
except Exception as e:
    print e
typeGuestCmd(ctx, ['typeGuest', machine_name, '\n', delay])
typeGuestCmd(ctx, ['typeGuest', machine_name, 'iptables -|p |i|n|p|u|t |a|c|c|e|p|t\n', delay])
typeGuestCmd(ctx, ['typeGuest', machine_name, 'iptables -|p |o|u|t|p|u|t |a|c|c|e|p|t\n', delay])
typeGuestCmd(ctx, ['typeGuest', machine_name, 'iptables -|f\n', delay])
typeGuestCmd(ctx, ['typeGuest', machine_name, 'iptables -|x\n', delay])
sleep(3)
screenshotCmd(ctx, ['screenshot', machine_name, '/tmp/vm.png'])
powerdownCmd(ctx, ['powerdown', machine_name])
toma@clamshell:~/pybox$ chmod 755 tst.py
toma@clamshell:~/pybox$

 A few comments, the machine_name is trivial - it's the guest VM's name. The snapshot_name, what we'd like to restore all the time, it's already existing for me in this example. The type how we start the VM, it can be either gui or headless, where the last one can be useful if we'd like to have no graphics output of the running guest. And the last parameter, the delay, is the typing speed. It's pretty low latency. And what is that "|f|o|o"? read as shifted: "FOO". The API is using capitals for special keys such as "SHIFT" so we need to use this kind of escaping.
The g_virtualBoxManager and ctx responsible to get the VirtualBox' context, it's borrowed from the main function of the vboxshell to get the things working.

Now let's run it and see what it does:

$ ./tst.py
0x80bb0002 (Cannot delete the current state of the running machine (machine state: Running))
0x80bb0007 (A session for the machine 'ubuntu64' is currently open (or being opened or closed))
Saving screenshot (720 x 400) screen 0 in /tmp/vm.png...

It is its output, which is not much, but in the background it did:

• snapshotCmd(ctx, ['snapshot', machine_name, 'restore', snapshot_name]) # restoring the snapshot
  
• startCmd(ctx, ['start', machine_name, type]) # starting the machine, but if we got exception (that's why we have try/except) if the machine is already running, just go on
• typeGuestCmd(ctx, ['typeGuest', machine_name, '\n', delay]) # press enter
  typeGuestCmd(ctx, ['typeGuest', machine_name, 'iptables -|p |i|n|p|u|t |a|c|c|e|p|t\n', delay]) #type "iptables -P INPUT ACCEPT"
  [...]
• sleep(3) #gues what, 3 secs
• screenshotCmd(ctx, ['screenshot', machine_name, '/tmp/vm.png']) #making a screen capture
• powerdownCmd(ctx, ['powerdown', machine_name]) #turning off the guest vm

So that's so easy to script VirtualBox 3.2 from python, I hope you found this article handy. There are so many possibility (including moving the mouse, click, so I recommend to check the VirtualBox SDK document for further information.
Best luck!

Running PHP (eg. Drupal) under Tomcat - with Caucho's Quercus PHP engine

Hello,

As the title is self descriptive, I'm trying to guide you how to run eg. drupal or wordpress on a java servlet container, for now, on Apache Tomcat 6; with help the Caucho's Quercus Engine. The wordpress is pretty much the same - but no urlrewrite needed at all.

As you may know, quercus is a reimplementation of php5 and its core libraries in java natively, which means, that no CGI trick has been adapted.

I did it on ubuntu, with the easiest way to have the necessary tools:

sudo apt-get install tomcat6 mysql

then

mysql -u root -p
create database drupal default character set utf8;
grant all privileges on drupal.* to 'drupal'@'localhost' identified by 'drupal';

which creates the database for drupal, then creates the drupal user with drupal password and grant all privileges.

now, it's time to configure tomcat, /etc/tomcat6/server.xml:

<!-- MySQL database resource -->
<Resource name="jdbc/drupal"
type="javax.sql.DataSource"
auth="container"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/drupal"
username="drupal"
password="drupal"
maxActive="10"
maxIdle="0"
maxWait="-1"
description="Drupal MySQL Data Source"/>

because quercus when deployed as a web app, it needs a JNDI lookup name for database connection, defined here.

created a pretty simple /var/lib/tomcat6/webapps/drupal
which is exploded from http://quercus.caucho.com/download/quercus-3.1.2.war:

META-INF/
META-INF/MANIFEST.MF
WEB-INF/WEB-INF/lib/
WEB-INF/lib/quercus.jar
WEB-INF/lib/resin-util.jar
WEB-INF/lib/script-10.jar
WEB-INF/web.xml

and modifying the web.xml to let quercus php engine to use our prepared JNDI database:

<!--
Tells Quercus to use the following JDBC database and to ignore the
arguments of mysql_connect().
-->
<!--
<init-param>
<param-name>database</param-name>
<param-value>jdbc/drupal</param-value>
</init-param>
-->

Now let's download the mysql's jdbc connector to /var/lib/tomcat6/webapps/drupal/WEB-INF/lib/ or, as I did,

sudo apt-get install libmysql-java
ln -s /usr/share/java/mysql.jar /var/lib/tomcat6/webapps/drupal/WEB-INF/lib/mysql.jar

only symbolic linking mysql connector to the webapp. I never put anything into tomcat's lib dir (/var/lib/tomcat6/lib/) which is not organic part of the tomcat, mysql, as in this example, part of the webapp, not the container!

the others pretty much the same as in the original quercus webapp .war file.

now, the an important, to download the urlrewrite module for servet container, to let drupal run properly, because installed drupal has this .htaccess file (used by Apache HTTPD), the essentials:

<ifmodule mod_rewrite.c="">
RewriteEngine on
[..]
# Rewrite URLs of the form 'x' to the form 'index.php?q=x'.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !=/favicon.ico
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]
</ifmodule>

download it and put it to WEB-INF/lib directory, ie. to /var/lib/tomcat6/webapps/drupal/WEB-INF/lib/urlrewrite-3.2.0.jar.

So we need to implement this under Tomcat. Since urlrewrite (ATM version 3.2) does not implements the REQUEST_FILENAME condition, we have two choices, first one to implement a custom class, as can be seen on http://www.brianshowalter.com/blog/running_drupal_on_quercus page, it's not so dynamic, since the install URI should be changed, so I decided to workaround, so I've put

<filter>
<filter-name>UrlRewriteFilter</filter-name>
<filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>
<init-param>
<param-name>confReloadCheckInterval</param-name> <param-value>0</param-value> <param-name>confPath</param-name> <param-value>/WEB-INF/urlrewrite.xml</param-value>
<param-name>logLevel</param-name> <param-value>TRACE</param-value> </init-param></filter>

into the WEB-INF/web.xml to allow urlrewrite run and check it at every single hit, later the values recommended to change.

I placed the following config in WEB-INF/urlrewrite.xml (the whole file):

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE urlrewrite PUBLIC "-//tuckey.org//DTD UrlRewrite 3.2//EN"
"http://tuckey.org/res/dtds/urlrewrite3.2.dtd">

<urlrewrite>
<!--
<class-rule class="com.brianshowalter.drupalrewrite.DrupalRule" />
-->
<rule>
<from>\.(php|gif|css|jpg|png|ico|js|html|htm|txt)</from>
<to last="true">.$1</to>
</rule>

<rule>
<from>^/(.*)</from>
<to>/index.php?q=$1</to>
</rule>
</urlrewrite>

which have been taken from http://wiki.caucho.com/Quercus:_Drupal page, and does pretty much the same, not on Caucho's Resin, but on Apache's Tomcat with urlrewrite, the same thing is the Quercus PHP engine.

If the the URL ended with .php or .gif etc., no action is taken, that's why it is the "last", on other cases, it is rewritten to forward the request to index.php.

and for the sake of understanding, the whole WEB-INF/web.xml file:

PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.2//EN"
"http://java.sun.com/j2ee/dtds/web-app_2_2.dtd"&gt;

<web-app>
<description>Caucho Technology's PHP Implementation</description>

<filter>
<filter-name>UrlRewriteFilter</filter-name>
<filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>
<init-param>
<param-name>confReloadCheckInterval</param-name> <param-value>0</param-value> <param-name>confPath</param-name> <param-value>/WEB-INF/urlrewrite.xml</param-value>
<param-name>logLevel</param-name> <param-value>TRACE</param-value> </init-param></filter>

<filter-mapping>
<filter-name>UrlRewriteFilter</filter-name>
<url-pattern>/*</url-pattern></filter-mapping>

<servlet>
<servlet-name>Quercus Servlet</servlet-name>
<servlet-class>com.caucho.quercus.servlet.QuercusServlet</servlet-class>
<init-param>
<param-name>license-directory</param-name> <param-value>WEB-INF/licenses</param-value> </init-param>
</servlet>

<servlet-mapping>
<servlet-name>Quercus Servlet</servlet-name>
<url-pattern>*.php</url-pattern>
</servlet-mapping>

<welcome-file-list>
<welcome-file>index.php</welcome-file>
</welcome-file-list></web-app>

let's see what we got here:

find /var/lib/tomcat6/webapps/drupal/WEB-INF/ -ls
6826814 4 drwxr-xr-x 4 tomcat6 tomcat6 4096 Jul 30 17:40 /var/lib/tomcat6/webapps/drupal/WEB-INF/
6826870 4 drwxr-xr-x 2 tomcat6 tomcat6 4096 Jul 30 17:04 /var/lib/tomcat6/webapps/drupal/WEB-INF/lib
3064049 0 lrwxrwxrwx 1 tomcat6 tomcat6 25 Jul 21 11:38 /var/lib/tomcat6/webapps/drupal/WEB-INF/lib/mysql.jar -> /usr/share/java/mysql.jar
3065516 10296 -rw-r--r-- 1 tomcat6 tomcat6 10524501 Jul 1 16:47 /var/lib/tomcat6/webapps/drupal/WEB-INF/lib/resin.jar
6733981 368 -rw-r--r-- 1 tomcat6 tomcat6 371264 Jul 1 16:47 /var/lib/tomcat6/webapps/drupal/WEB-INF/lib/javamail-141.jar
6733982 96 -rw-r--r-- 1 tomcat6 tomcat6 91794 Jul 1 16:47 /var/lib/tomcat6/webapps/drupal/WEB-INF/lib/inject-16.jar
6733983 200 -rw-r--r-- 1 root root 198436 Dec 4 2008 /var/lib/tomcat6/webapps/drupal/WEB-INF/lib/urlrewrite-3.2.0.jar
6827312 4 -rw-r--r-- 1 tomcat6 tomcat6 2029 Jul 30 17:24 /var/lib/tomcat6/webapps/drupal/WEB-INF/web.xml
6826871 4 drwxr-xr-x 2 tomcat6 tomcat6 4096 Jul 1 16:47 /var/lib/tomcat6/webapps/drupal/WEB-INF/licenses
6827313 4 -rw-r--r-- 1 root root 454 Jul 30 17:40 /var/lib/tomcat6/webapps/drupal/WEB-INF/urlrewrite.xml

Make sure, that every file is under the permission of tomcat6:

sudo chown tomcat6:tomcat6 -R /var/lib/tomcat6/webapps/drupal/

and I'd recommend for now, NOT IN PRODUCTION to turn off the tomcat's java security in /etc/default/tomcat6 by:

# Use the Java security manager? (yes/no, default: yes)
# WARNING: Do not disable the security manager unless you understand
# the consequences!
TOMCAT6_SECURITY=no

and later to set up more granular settings in /etc/tomcat6/policy.d/*.policy

And at last but not least, the best news, that if we move /var/lib/tomcat6/webapps/drupal to another (eg. /var/lib/tomcat6/webapps/site or /var/lib/tomcat6/webapps/ROOT - which is the / on the site), it will work as well, and if you upgrade, you just need to update the urlrewrite configuration - no need to recompile your java classes!

Best luck for it!

1848

I believe this is the one of the worst marketing actions ever: Putting a cockade, the symbol of Hungarian Revolution of 1848 to the Kaiser salami. Imagine a Julius Jacob von Haynau salami, it would be even worse! Not mentioning that the Hungarian cockade is different, it has the green outside, and the red inside, but I have never seen a correct one at any Hungarian sho. :) Nice one, but next time, think twice! Anyway please read more about probably the biggest Hungarian Celebration's background at: http://en.wikipedia.org/wiki/Hungarian_Revolution_of_1848