Howto: Set up a jekyll-based gh-pages site

I’m part of the team developing WebVirt, a web-based graphical libvirt aggregator. We decided to take advantage of Github’s Pages support of a popular Ruby based website generator called Jekyll. Here is (roughly) how the process went:

Step 1: Create your directory structure

Jekyll, when called, will crawl the directory structure you specify, and generate a web site based on it. By creating subfolders that correspond with “categories” of articles, a clearer picture begins to emerge:

.
├── _layouts
├── _includes
|
├── js
├── css
|
├── architecture
│   └── _posts
│       
├── install
│   └── _posts
|
├── reference
│   ├── managerapi
│   │   └── _posts
│   ├── managerconfig
│   │   └── _posts
│   ├── nodeapi
│   │   └── _posts
│   └── nodeconfig
│       └── _posts
├── requirements
│   └── _posts
└── userguide
    └── _posts

All of the categories I intend to generate data about have a subfolder called _posts that will store the copy (think print-copy) that I will be displaying. The _layouts folder on line 2 holds repeated page structures, while _includes holds code snippets for reuse.

Step 2: Set up YAML metadata

Using the Liquid Templating System, an HTML shell can be created as a layout, using special Liquid syntax to indicate where content goes:

<!DOCTYPE html>
<html>
  <head>
    <title>WebVirt Documentation - {{page.title}}</title>
  </head>
  <body data-spy="scroll"  data-offset="25" data-target=".sidebar-nav">
    <div id="virshmanagerapp">
      <div id="main" class="container">
        <div class="row">

            {{ content }}

          <!-- Footer -->
          <footer>
            <div class="row-fluid">
              <div class="span12">
                <footer>
                  <hr>
                  <p class="muted">Designed and Maintained by <abbr title="Centre for the Development of Open Technology @ Seneca College">CDOT</abbr> | Built on {Bootstrap.js Jekyll}</p>
                </footer>
              </div>
            </div>
          </footer>
          <script src="/js/plugins/jquery-1.9.1.min.js"></script>
          <script src="/js/plugins/underscore.js"></script>
          <script src="/js/plugins/toastr.min.js"></script>
          <script src="/js/plugins/bootstrap.min.js"></script>
        </div>
      </div>
    </div>
  </body>
</html>

In addition to a _posts folder, each content category contains an “index.html” file that is loaded when the directory is accessed. This index.html folder uses YAML metadata to indicate a few things to Jekyll – mainly which template to use:

---
layout: default
title: Node API
---

{% include topnav.html %}

<!-- HTML HERE -->

Line 2 indicates that all content below line 4 (‘—‘) will replace Line 10 in the last example ( {{ content }} ). Line 6 is an example of using an html snippet, and makes maintaining common code laughably easy!

Step 3: Set up posts

In order to generate the data properly, the markdown files must be saved with a specific filename format:

XXXX-XX-XX-THIS-IS-THE-TITLE.markdown

Where XXXX is the year, followed by the month then the day. After creating these files in the appropriate _posts folder, we can add metadata for the Liquid Templating system in our HTML files:

---
title: example1
---

## Markdown goes here

Step 4: Use Liquid to display content

On the index.html pages for each category, we can use Liquid’s tag syntax to do basic operations on template variables that jekyll uses to store data programatically:

---
layout: default
title: Node API
---
<div class="row">
  <div class="span2">
    <!-- Navigation -->
    <div class="well sidebar-nav affix">
      <ul class="nav nav-list"> 
        {% for post in site.categories.nodeapi %}
          <li><a href="#{{ post.title }}">{{ post.title }}</a></li>
        {% endfor %}
      </ul>
    </div>
  </div>
  <div class="span10">
    <div id="heading"></div>
    <div class="row">
      <div data-offset="25" class="span10">
	{% for post in site.categories.nodeapi %}
	  <article class="well post" id="{{ post.title }}">
	    {{ post.content }}
	  </article>
	{% endfor %}
      </div>
    </div>
    <div id="pagination" class="pagination pagination-right"></div>
  </div>

By iterating through each post, we can display information in a variety of ways. It becomes very simple to create an API reference. For example, we are using this markdown template:

---
title: example1
---

## API Call `call\format` ##

### Purpose ###

### Usage ###

### Return ###

### Snippets ###

All we have to do is fill in the information, save the files with the appropriate name in the appropriate place and the site will generate itself!

Brilliant. Next post will be an accounting of some common bugs I ran into!

Advertisements

WebVirsh: Migrating Virtual Machines using Libvirt

(See end of post for sources)

Libvirt, and some hypervisors, have the ability to migrate virtual machines from one host to another. I find this extremely impressive, so I’d like to go over the basic concepts of this process. Finally, I will conclude with a description of Libvirt’s migration capabilities in some detail.

Migration

Migration is a three-step process.

FIRST, the virtual machine is suspended, at which point the state of all of its running applications and processes is saved to a file. These files can be referred to as snapshots, since they store information about a VM’s activities at a particular point in time.

SECOND, the snapshot is transferred to the destination host, along with the VM details (usually in the form of an XML file). These details provide information necessary to properly emulate the VM, like the kind of hardware being emulated.

THIRD, the virtual machine is resumed on the destination machine, which constructs an emulation of the hardware from VM details and loads the snapshot into memory. At this point, all network connections will be updated to reflect the new MAC address associated with the VM’s virtual network interfaces and their associated IPs.

Live Migration

Live migration is the holy grain of virtualization technology, and solves the problem of High Availability in many cases. When live migration is possible, it means that an administrator can change which physical machine is hosting a VM, without interrupting the VM’s operations for more than 60-100 milliseconds. Quite a feat! And very useful for balancing load and energy costs without affecting whatever service the VM was providing.

The steps are similar to static migration, but involve some serious trickery when migrating four key components:

  1. CPU state: Required for process migration, so as not to interrupt/corrupt execution.
  2. Storage content: Required for persistent storage access.
  3. Network connections: Required for network activity migration, so as not to interrupt the transport layer of the VM’s network stack.
  4. Memory content: Required for RAM migration. The trickiest part of them all, because the VM is likely to continue to modify its memory even as the migration is occurring.

CPU

As fascinated as I am by the idea, this is too complex a topic to research and explain here.

Storage

Because transferring a virtual hard disk can be time consuming (50-120 seconds, possibly more), cloud providers have side-stepped the problem by having all hosts use a shared storage pool mounted on each host in an identical manner. This way, the transfer can be skipped completely and all machines connected to the pool become (from a storage standpoint) potential hosts.

Network

If the source and destination nodes are on the same subnet, it is as easy as updating the MAC address of the IP associated with the VM’s virtual interface and sending an ARP broadcast to ensure all other machines on the network are aware of the change. If the machines are on separate subnets, there is no way to accomplish this without severely degrading performance.

RAM

For a live migration, the entirety of the VM’s memory (in the form of multiple memory ‘pages’) is copied to the destination host. Memory pages will be repeatedly copied as the VM makes changes to pages on the source host. When the rate of the copying matches the rate of the dirtying, leaving a small number of dirty pages left to be copied, the VM is suspended on the source host. Immediately after suspension, the final “dirty” pages are copied to the destination and the VM is resumed on the new machine. The time between suspension on the source machine, and resumption on the destination machine is trivial with this method – from a few milliseconds to a second or two depending on the size of the paging files. Though the entire process may take longer, the downtime is measured in between these two specific events.

Libvirt Migration

In the same way that Libvirt provides a standard method of sending commands to VMs hosted on a range of hypervisors, there is a standard Libvirt command for migrating a VM from one host to another:

# The most basic form
$> virsh migrate (flags) instanceName destinationURI

In this case, the machine running the command is considered the client, the machine hosting the VM is considered the source and the machine being migrated into is considered the target. In this command, because only the destination URI is specified, Libvirt assumes the client is also the source.

Prerequisites

Migration, live or otherwise, has basic requirements for the source and target machines. Without meeting these, migration will simply fail. Broadly speaking, both machines must have hardware supporting the VM’s emulated environment. Usually this means a similar CPU and a compatible motherboard. The other major requirement is identical storage & networking setups on both machines. On the storage side, the path of the VM’s image on both hosts must be the same. On the network side, all bridged connections to hardware interfaces must be named identically. Ideally, both machines should have identical components, but the networking, storage and basic hardware should be the same.

Options

In order to ensure security, Libvirt can use its daemons on each host to tunnel the transfer of all data during the migration. This is not the default method of migration. A tunnelled connection also requires an explicit peer-to-peer declaration – that is, a tunneled connection must also have the P2P flag enabled, making the command look like this:

# P2P, Tunneling enabled
$> virsh migrate --p2p --tunneled instanceName destinationURI

Conclusion

All in all, this is one impressive piece of technology. Libvirt makes the process quite easy, adding options for letting the Hypervisor manage the process instead of Libvirt, as well as more. See the man page (linked below) for details.

Sources:
Pradeep Padala’s Blog
Virsh MAN Page

VNM Web App with Backbone.js Continued: Changing tactics

In my last post, I talked about the options we had when developing our app with Backbone.js. Because we wanted to avoid persistent storage and learning unnecessary new libraries, we opted to store information about virtual machines (or instances) running on a host in that host’s Host model object.

The hope was to do less work for the same result, but the further we developed the app, the more apparent it became that this approach was inefficient. One of the strengths of Backbone.js is to automatically update the View (and therefore, the DOM) attached to any Model that has changed. Unfortunately, because we didn’t have Models for the instances we wanted to display information about, accessing their information from a Host model, or reacting to changes, required extra code.

As an example, lets assume we had used the two-model architecture we were considering: if an Instance model were to change, only one line of code is required in the appropriate View to ensure that it updates itself automatically.

var InstanceView = Backbone.View.extend({
  initialize: function() {
    ...
    // Instructs the view to re-render the view each time
    // a change event is fired on the model.
    this.listenTo(this.model, "change", this.render); 
  },

  ...  
 
});

// Step 1: Create an instance model object
var instance = new InstanceModel(dataGoesHere);

// Step 2: Create an InstanceView by passing the new
// model object
var exampleView = new InstanceView(instance);

// Step 3: Update a value in the instance model, 
// firing a change event on it and 
// automatically updating the view.
instance.set("dataKey", "newValue");

You can see that the steps are a simple logical mapping of Backbone.js functionality to the MVC paradigm.

This works so well because of how Backbone.js defines Models and Views. Usually, there is a single Model for each View, and the two are built for easy monitoring. This assumes that the definition of the Model being used is (in terms of Object Oriented design goals) concise and simple, with only data that logically fits what the Model represents being contained within it.

The problem

Our application on the other hand, uses the architecture without Instance models. The Host model was built to contain information that doesn’t directly relate to what it is modelling, in the form of data about instances running on that host.

The relationship is a logical one (the actual hosts are hosting the instances their models contain information about), but if we are forced to treat the Host objects like they are also Instance objects, we add complexity when trying to relate other Backbone.js components to them. This complexity starts with the Host model object requiring a suite of methods to enable CRUD operations for the instances contained in the model.

A small pseudocoded example:

var Host = Backbone.Model.extend({
  initialize: function() {
    // Call custom method of the Host 
    // Object to populate this model
    // with Instance data 
    this.syncInstances();
  },

  ...  
 
  syncInstances: function() {
    // Pseudocall, intended to return an array of 
    // JSON objects representing instances
    var instances = API.callServer("instanceInfoRoute", this.get("ip"));

    // Iterates through the array, 
    // assigning each instance name to
    // a key in the Host model, with 
    // the data about that instance as 
    // its value.
    _.each(instances, function(element) {
      // this.set(attributeKey, attributeData)
      this.set("vm-" + element.name, element.data); 
    }); 
  }
});

Also, Read and Update operations require more resources! In a proper Instance model, data members could be easily and efficiently updated:

var i = new Instance({hostIp: 10.0.0.0});

i.get("status"); // "running"
i.set("status", "shutdown"); // Updates VM status
i.get("status"); // "shutdown"

but with our current architecture, since the instance-data exists as a key of its associated Host object, every piece of data about an instance will be retrieved, copied and returned when the get method is used:

var i = new Host({ip: 10.0.0.0});

// First, copy to a local variable
var instance = i.get("vm-InstanceName") // Returns: {status: "running", id: "1"}

// Then, update required attributes
instance.status = "shutdown";

// Finally, copy entire object back into the model
i.set("vm-InstanceName", instance);     // Copies: {status: "shutdown", id: "1"}

Further, this snippet didn’t touch on the logic needed to easily retrieve an Instance by name! Currently, a fixed prefix is prepended to the instance name, which then becomes the key for the instance data inside the Host model.

This is the only way to logically group the instances within the model, but requires yet MORE logic to remove or add the fixed prefix, depending on the action being taken. A more realistic pseudocode example would look like this:

var i = new Host({ip: 10.0.0.0});

// Function to get instance data when passed an
// instance name and a model
function getInstanceData (instanceName, model) {
  var key = "vm-" + instanceName;

  return model.get(key);
}

// Function to set instance data
function sendInstanceData (instanceName, model, data) {
  var key = "vm-" + instanceName;

  i.set(key, data);
}

// First, copy to a local variable
var instance = getInstanceData("instance-0000001", i); // Returns: {status: "running", id: "1"}

// Then, update required attributes
instance.status = "shutdown";

// Finally, copy entire object back into the model
sendInstanceData("instance-0000001", i, instance);    // Copies: {status: "shutdown", id: "1"}

With this all in mind, here is how we force a View to update itself without an Instance model object. It retains most of the simplicity of the first example that had two model types, but at the cost of overly complex and needlessly messy model logic:

var InstanceView = Backbone.View.extend({
  model: host,

  initialize: function() {

    ...

    // Instructs the view to re-render the view each time
    // a change event is fired on the model.
    this.listenTo(this.model, "change:" + this.options.instanceName, this.render); 
  },

  ...  
 
});

// Create an instance model object
var host = new HostModel(dataGoesHere);

// Create an InstanceView by passing the new
// model object
var exampleView = new InstanceView(host, {instanceName: name});

// Updates a value in the instance attribute
// of the host model, causing a "change:instanceName"
// event, updating the view
host.setInstance(instanceName, {data: newValue));

The Solution

We decided to refactor the code to use two models, now knowing that it would be much less work to co-ordinate the relationship between the models manually. Even without the assistance of Backbone.relational, or a similar framework for managing those relationships, the code ended up cleaner and more concise.

VirshNodeManager Web App Development with Backbone.js

This has been the week of javascript for me, and it came together in my first few attempts to create an MVC client-side application using the Backbone.js library.

Defining the Models

I wrote a post about the MVC architecture, and if you aren’t familiar with the concepts on a basic level, go read it. I’ll be right here! Assuming you’re up to speed, my task was trying to decide how to represent the data our app needs to work with in models.

Our application uses data about two things, broadly speaking:

  1. Machines hosting libvirt
  2. The VMs being managed by each libvirt host

In light of this, my first thought was to create a model for each:

var Host = Backbone.Model.extend({
  ...
});

// &&

var Instance = Backbone.Model.extend({
  ...
});

Then all I would have to do is create a collection (like an array) of Instance models, and associate it with the Host model that represents the physical machine managing the actual VMs the collection represents. But how to associate the two? It would be clearly beneficial for there to be a programmatic relationship between each collection of Instance models and the appropriate Host model, but my options were limited by Backbone.js itself.

Backbone.js does not natively support relationships between models and models (like Instance and Host) or models and collections of a different model type (like a Host model and collection of Instance models). I did find a library to support this (backbone.relational), but we have deadlines to hit and I couldn’t afford to spend another half a day learning yet another library.

I pseudocoded the Instance and Host models to duplicate the information that binds them conceptually – the IP of the host machine. As you read it, keep in mind that the extend() method of Backbone.Model accepts a JSON object (a series of key/value pairs) to set features of the model being created:

var Host = Backbone.Model.extend({
  initialize: function () {
    if (!this.ip) { this.ip = 0; } // Safe state
  }
});

// &&

var Instance = Backbone.Model.extend({
  initialize: function () {
    if (!this.hostIp) { this.hostIp = 0; } // Safe state
  }
});

But, on thinking about it, I saw that to take this approach would effectively neuter Backbone.js – isn’t the point of a framework to keep me from having to manually manage relationships like this?

So I pseudocoded another possibility, where each Host model object contains a key for each VM on the machine it represents:

var Host = Backbone.Model.extend({
  initialize: function () {
    if (!this.ip) { this.ip = 0; } // Safe state

    (this.getInstances = function() {
      // AJAX API call to our interface-server to return VM data for the provided ip
      var result = $.ajax("/list/vms/this.ip", ... ); 

      // A fictional method that would parse VM data from the AJAX call
      // and create keys within the model for each VM detailed
      this.setInstanceData(result); 
    };) () // Immediate invocation after creating the method 
  }
});

This option appeared to be the lesser of the two evils. Strictly speaking, it would also break Backbone.js’s usefulness by ignoring an obvious candidate for a model (Virtual Machines), but also would prevent the user from having to keep track of which Instance collection was paired with which Host model without the relevant library.

In the end, we decided that I was right to forgo learning another library, and that my second approach would be the most reusable.

Host Model Implementation

Backbone.js provides a constructor for each model, where the user can pass key/value pairs and they will be assigned to the new model object as it is invoked. Then, after the constructor logic completes, Backbone.js looks to see if an “initialize” key was defined (like in my examples above) and runs that logic.

For the Host model, the initialize method needed to do three things:

  1. Ensure an IP was passed, and set a safe state if not
  2. Pull vital information about the host machine (cpu/memory usage etc.), assigning them as keys
  3. Pull information on all instances being managed by Libvirt on that host, and then assign them as keys for the model representing that host

AJAX & jQuery

At this point, setting a safe-state IP address was a cakewalk – as it should be. The other two required me to learn how to use jQuery’s AJAX method.

The jQuery library is a dependency for Backbone.js, and is available as a global object throughout an app using Backbone.js. To leverage this, I read the documentation for the AJAX method, and created the following pseudostructure for my AJAX calls:

$.ajax({
  url: "apiCallGoesHere",
  datatype: "json", // Defines the return datatype expected
  cache: false,     // Just in case
  success: function() { successCaseLogicGoesHere },
  error: function(textStatus) { 
    switch (textStatus) { // "textStatus" is the error code passed to this function
      case "null":
      case "timeout":
      case "error":
      case "abort":
      case "parseerror":
      default:
        // Error logic will populate these cases
        console.log("XX On: " + this.url + " XX");
        console.log("XX Error, connection to interface-server refused XX");
        break;
     } // END SWITCH
  } // END ERROR LOGIC
}); // END AJAX CALL

By bumbling around with my now intermediate understanding of JavaScript, I threw together a test version of the model, and ran into a few challenges:

The context of “this”

In JavaScript, nested functions lose access to the this attribute of their parent. This is a noted design flaw in the language, and made my code do some funny things until I tracked it down as the root of the problem. The solution was to define a that variable that contained a reference to the parent function, e.g.

var that = this;

…which would be accessible to all subfunctions as an alias for the original context of this.

Get/Set methods

Backbone.js model objects aren’t just key/values that the user defines – they inherit from the Backbone.Model object, which contains the constructor logic and more. Trying to define and retrieve attributes directly failed badly because the user-defined keys were actually stored as a JSON object under a key called attributes:

var host1 = new Host("0.0.0.0"); // Create new model

host1.ip = "2.3.4.5"; // works, but creates a new key because...

alert( host1.attributes.ip ); // 0.0.0.0
alert( host1.ip ); // 2.3.4.5

In order to take advantage of Backbone.js’s ability to use user-defined validation, the user has to use the provided set/get methods for the most secure, cohesive implementation of a Backbone.js powered app:

var host1 = new Host("0.0.0.0"); // Create new model

host1.set("ip", "2.3.4.5"); // BINGO
alert( host1.get("ip") ); // 2.3.4.5

The other benefit of using the get/set methods is that using them fires a “change” event, which can be used to trigger actions (like a re-rendering of the webpage to show the new information).

Redundant code

I repeated the same basic AJAX call structure at least four times before I decided to factor it out into a helper object called API. This way, each component of Backbone.js could use a standard method of making a call without duplicate code:

var API = {
  // AJAX wrapper
  callServer:  function(call, success, error) {
    $.ajax({
      url: "/" + call,
      datatype: "json",
      cache: false,
      success: success,
      error: function(textStatus) {
        // INTERFACE-SERVER ERROR HANDLING
        switch (textStatus) {
          case "null":
          case "timeout":
          case "error":
          case "abort":
          case "parsererror":
          default:
            console.log("XX On: " + this.url + " XX");
            console.log("XX Error, connection to interface-server refused XX");
            error();  
            break;
        } // END-Switch
      } // END-Error
    }); // End ajax call
  }, // END callServer function
} // END API object

Now it was as easy as:

API.serverCall("apiCallPath", function() { successCallbackLogic }, function() { errorCallbackLogic});

Defining a collection

The next challenge was to define a collection for Host model objects. To over-simplify it, a collection object is an array of a specified type of model object, along with methods for manipulating that array. In this case, we needed it to make an API call to find all the IPs of hosts running Libvirt, and then create a Host model for each one.

The logic was very simple, since collection objects support the initialize key in a similar fashion to the model objects.

Making it run

By strategically adding console.log() calls, we were able to watch the app run – headfirst into a wall. Diogo wrote a post about that particular issue, and as we resolved it, we reflected on how inelegantly our Backbone.js app handled the error.

In response, we standardized the format of our API into a JSON object with the following attributes:

{
  err: "errorCode",
  data: {
    dataKey: value,
    ... : ...,
    ...
  }
}

…which simplified the implementation of error handling within our application.

Our next try still didn’t work, but showed us how beautifully the error-handling resolved.

Our console log:

[creating hosts] VNMapp.js:200
-begin collection calls VNMapp.js:182
Failed to load resource: the server responded with a status of 404 (Not Found) http://192.168.100.2/list/daemons/?_=1360940232244
XX On: /list/daemons/?_=1360940232244 XX VNMapp.js:23
XX Error, connection to interface-server refused XX VNMapp.js:24
XX Cannot find daemon-hosts! XX 

Our next next try was our last – we had proof the app was working.

Our console log:

[creating hosts] VNMapp.js:210
-begin collection calls VNMapp.js:193
--Add Model | ip: 10.0.0.4 VNMapp.js:199
New Host!
IP:10.0.0.4 VNMapp.js:189

Next Steps

Next is developing a View object for each part of the app’s UI that will be dynamically updated, and splitting the static parts of the page into templates to make duplication and updating easier.

This will be covered in a further post, but so far we are happy with our work finally coming together!

Example: JavaScript Closure

I had the opportunity to suffer through learn JavaScript these past two weeks, and now that I have I’m glad I did. That said, there were a few things that took me some time to wrap my brain around, primarily the concept of closure.

JavaScript is not a class-based language though, as a professor at my college puts it, “people like to treat it as one”. If you’re using the language honestly however, you’ll find that the native method of setting private anythings is slightly confusing – especially if you come from a more classical (pun intended) background when it comes to programming languages. In fact, slightly confusing might be an understatement.

Now that I (roughly) understand the concept of closure, I’d like to share my thoughts on it.

Closure is:

The ability to access the attributes and parameters of a function that has finished executing, through a function that it returns OR through one of its function properties.

I’ll use a simple example:

function imTheParent (foo) {
  var animal = foo + "are awesome";

  return function() {
    return animal;
  };
}

var imTheChild = imTheParent("kittens - deal with it - ");
// imTheChild contains a reference to the function imTheParent returned

alert( imTheChild() );
// Returns "kittens - deal with it - are awesome" to the alert function,
// despite the data (animal) being in a scope that has already passed.

This example demonstrates how a JavaScript programmer can protect data members from unauthorized access. The variable animal, as well as the parameter foo, can be referenced through the secondary function – if it exposes them.

I could also have written the code without the return of the function:

function imTheParent (foo) {
  var animal = foo + "are awesome";

  this.getAnimal = function() {
    return animal;
  };
}

var imTheChild = new imTheParent("kittens - deal with it - ");
// imTheChild is built as an object of type imTheParent

alert( imTheChild.getAnimal() );
// Returns "kittens - deal with it - are awesome" to the alert function,
// despite the data (animal) being in a private scope.

And I could go on, but there is already a wealth of information on the topic.

Here are some choice examples:

1) David Crockford’s techniques for creating private attributes and methods

2) Mozilla’s Developer Network documentation on the topic

Overview: MVC Software Architecture Model

As part of my role on the team, I’m developing the client-side code and UI for our new VirshManager application. Before I got started, Diogo suggested I think about how I’m going to organize the code – both in terms of structure and concept. After looking into it, and asking him some pointed questions, I discovered that there’s no way around applying a programming architecture to the problem.

Diogo suggested I look into Backbone.JS, a client-side MVC framework.

Wait. What?

Frameworks

A framework, simply put, is a pre-built conceptual and literal structure for the code of an application. It’s purpose is to separate components to accomplish all of the goals of good object-oriented programming; things like encapsulation, de-coupling and elimination of repeated code are all hallmarks of a framework.

However, these frameworks use specific architecture types out of the box. This is to say that each framework is built around a conceptual architecture that helps to accomplish the object-oriented goals listed above. Then, the framework takes charge of implementing a literal architecture that a programmer (or team) can use to organize their project.

M.V.C.

It appears that the most common conceptual architecture underlying most frameworks is the MVC architecture, AKA “Model, View, Controller”. Three hours ago, I’d heard the term but never been taught precisely what it meant. Now, after some research (required to understand what the heck is going on in Backbone.JS), I can explain the concept quite succinctly using a web site as an example:

bsw

In this (simplified) representation of how a web site works, the user sends a request for a WEBPAGE from the BROWSER, to the SERVER. By its own logic, the server finds the WEBPAGE being requested, and sends it back to the BROWSER the user is using. This is a standard chain of events, separate from what the WEBPAGE actually does or displays.

Let’s pretend that this WEBPAGE has some dynamic functions, like being able to switch between multiple full size “pages” of information without changing the URL. In other words, let’s pretend that this web page contains Javascript code to accomplish its dynamic function.

If this function is complicated, the single WEBPAGE will contain code for a number of different sub-functions:

1) Process the user’s interactions with the page
2) Process information based on user’s interactions with the page
3) Display specific information in a specific way based on user’s interaction with the page

All three of these sub-functions could get very complicated. Looking at Gmail as an example, even if we take out the parts of the functionality that are only possible by retrieving information from a web-server, the amount of client-side code and function left over is staggering! How on earth do they manage, organize and maintain that code? Lots of impressive stuff happens in the Gmail web application without any server calls. What do they do to manage object-oriented design goals?

The answer ties back to my previous example. But instead of one WEBPAGE performing the three sub-functions I listed above, we will conceptually separate the webpage into three objects (or classes) to encapsulate those functions:

Overview

In this (simplified) diagram, you can see that what was a WEBPAGE is now a CONTROLLER, a MODEL and a VIEW, with a MySQL database in case advanced functions are needed.  What?

Overview

At its most basic, the MVC paradigm says that there will be an object whose job is to run the application, the CONTROLLER, another object to de-couple data storage from its representation in the program, the MODEL, and a final object to de-couple information from how that information is displayed (the VIEW). Let’s go back to my example:

CONTROLLER

The CONTROLLER is the application logic – it receives user commands from the BROWSER, retrieves information necessary to the command from the MODEL, and then sends the final form of that data to the VIEW to be processed for display.

MODEL

In conceptual terms, the model is a standard (to the application) object that gives access to some kind of data requiring storage. This way, if the source of the data changes (from one database type to another for example), the function of the CONTROLLER and the VIEW won’t be affected, because the way they interact with the MODEL will be through the same object methods. Obviously the MODEL logic will need to be rewritten, but the MODEL objects will appear to behave the same way to the other two components, and will give them data in the same form as before.

VIEW

The VIEW object is conceptually similar to the MODEL in the sense that it presents a standard set of methods for the CONTROLLER that are separate from how those methods operate. The logic of the VIEW methods will depend on what technology is being used to represent the application’s data. In the case of a webpage, the technology being used is HTML/CSS and possibly client-side Javascript. In other words, the VIEW will take data from the CONTROLLER, and will generate a representation of it based on what VIEW method the CONTROLLER called in order to complete the user’s command.

Finally, this component will pass the representation of the data to the user interface for display.

Use Case

Let’s put it together with a “saving a new contact in gmail” example use case. This blurs the line between client-side and server-side, but the point is the same:

1) User enters information on an HTML form, clicks “save contact”, which is submitted to the SERVER

Step1
2) “Save contact” command + relevent information from the HTML form (contact name, number etc.) is forwarded to the CONTROLLER object

step 2
3) The CONTROLLER carries out the “save contact” command by sending this contact information to the MODEL object through a MODEL object method. (Likely one just for creating and storing a new contact)

step3
4) The MODEL stores the data in whatever backend is being used (MySQL in this example)

step4
5) The MODEL returns a confirmation that the contact has been created

Step5
6) The CONTROLLER asks the MODEL for a list of existing contacts to display through the MODEL method built for that purpose

step3
7) The MODEL retrieves data from the data storage back-end

Step6-7
8) The MODEL sends this data to the CONTROLLER

Step5
9) The CONTROLLER sends this data to the VIEW through a VIEW object method for displaying a list of contacts

Step9
10) The VIEW object uses this data to generate client-side HTML/CSS/JAVASCRIPT and returns this code to the browser, through the SERVER

Step 10
11) The user now sees an updated contact list.

Conclusion

As with anything that is entirely abstract, this design architecture is difficult to describe without a specific example.  The examples I used contradict each other slightly, because I was originally describing client-side MVC systems and then proceeded to use a use-case that described a server-side MVC system.

For this, I apologize!  I hope it proves helpful, because it certainly helped me figure out how an MVC based framework might operate.

Onward!

CDOT’s VirshManager API: Application Architecture

No, Watson, this was not done by accident, but by design.
— Sherlock Holmes

It’s no small task to create a plan for something entirely consistent with itself.  This really is the definition of good design: consistency.  So, with that in mind, my research parter and I began planning how we were going to take a problem – “We need an API for interacting with Libvirt on multiple hosts!” – and come up with a solution.

First: Break it down

We needed a way to:

  • Easily detect libvirt-hosts on a network, and store their IP addresses
  • Directly communicate with those libvirt instances to check on the health and status of their guest VMs
  • Separate this API from other applications and platforms running concurrently to ours

Identifying the LIbvirt Hosts

Our first thought was to create a daemon that would be installed on each libvirt host in the cluster.  This way, RESTful calls could be made to the daemon, which would then run the relevant virsh command.  This presented the problem of access though, meaning one would have to have direct access to the local network of the libvirt-hosts in order to use the API.

To solve this problem we decided we would have write a web-server program to act as the host of whatever interface the client wanted for the API. So long as the interface host was on the same network as the libvirt hosts, it would have a path to them. Also, if the user chose to connect the interface host to a public network as well, an admin could use the API even if they were woken up in the middle of the night by an emergency VM failure.

On a practical level, this led to another problem: How would the web-server know which IPs on the network were libvirt hosts? We had three options:

  1. Manually enter each libvirt host IP into the interface’s configuration
  2. Manually enter the interface-server’s IP address in the configuration file of the daemon on each libvirt host
  3. Automate the process in some way

We chose number 3, and here’s how we did it:

The Daemon (Or, a Story of Nmap-ing)

First, we had to make it so that the daemon would be listening for API calls through a TCP port that wasn’t used by any other application on any of the machines in the cluster.  This could be set to an obscure default and/or defined at install-time.

Second, we theorized that all the interface-server would have to do is test each IP on the cluster network to see if that particular port was open on whichever IP it was testing – if it was, the daemon was installed and it was hosting virtual machines.  If not, it was an active host on the cluster that had another function.  Either way, this would automate the discovery of the nodes our API was to manage.

With this approach, no special configuration was needed for the daemons other than to specify which port to keep open, and even then only if the default was unavailable.

The Crawler

We decided to write a program, the Crawler, that would use the extremely powerful NMAP linux utility to quickly test a client-specified CIDR range in the manner described above.  But in a huge cluster of possibly hundreds (or more) machines, could this cause network congestion?  We weren’t sure, but just in case we split the Crawler’s function into four:

  1. Scan the entire network, recording IPs of hosts running libvirt, and hosts that were not.
  2. Scan the remainder of the CIDR range that was not originally found to contain any active IPs for new hosts of either kind
  3. Scan the hosts that were not running libvirt to see if there was a change
  4. Periodically probe the libvirt-hosts the Crawler had discovered to ensure there was an active connection with the daemon

Though we may have been overthinking it, we figured that this would cover every possible problem arising from this automated system.

The Agent

The interface-server would need a way to transmit calls to the daemons of specific hosts, and receive data back from those calls.  For this, we conceived of the Agent, which acts as a proxy for calls from the user interface to the daemons, and results from the daemons to the user interface.

The Interface

The final piece of the conceptual puzzle was a way for the client to actually use this RESTful API in a meaningful and streamlined fashion.  Seeing as we already had a interface-server to allow for external access to the interface, a web-based application seemed to make the most sense.

I began the design process yesterday, and began developing the interface this afternoon.  You can read about it here.

The Details

We now needed:

  1. A way for the web-server to access and store vital information without unreasonably increasing the resource footprint of the application
  2. A way to encapsulate a web-server and the scripts that make it do what we need it to do

For the first point, Diogo discovered Redis – a noSQL database that uses RAM to store its data and calculations while the computer is on.  He’s written a post about it here.

For the second, we decided to use Node.JS to write a robust, efficient and portable web server.  This will likely be detailed in a later post.

Conclusion

Now the work begins! Diogo has been furiously writing the logic for our server (using Node.JS) while I’ve been busy developing the user interface.

Expect more to come!