Loomiere 2.0 teaser

[Update]
The new high is now ~2600 active streams, summing ~1500 Mbps of bandwidth. Some graphs:

• Network traffic (cumulated for all interfaces): https://www.bijk.com/s/7c52.png


• Active connections: https://www.bijk.com/s/0F48.png

---

This article is a quick peek into how Loomiere 2.0 is (currently) able to serve ~1700 active streams from a single server in production, summing up to more than 1100 Mbps of bandwidth.

All these statistics represent a single machine, in real time, over a period of 12 hours, equipped as follows:

• 2 x Xeon Dual Core (with HyperThreading) at 3.2GHz


• 16 GB RAM


• 15 x 1TB HDD (7200rpm)


• 4 x 1GB NIC's


• This system runs on Ubuntu 10.04.1 LTS.

OK so here are the graphs:

• Cpu load: https://www.bijk.com/e/812e


• Network traffic (cumulated for all interfaces): https://www.bijk.com/e/8423


• Active connections: https://www.bijk.com/e/AF6E


• Memory consumption: https://www.bijk.com/e/E379

The graphs are not quite stable at the moment since I am currently working hard to test this setup and Loomiere itself, but I think soon I will have a more stable configuration and also some time to comment on these graphs.

Enjoy your weekend!

P.S.: I love graphs :).

Case study: fixed number of iterations with LPeg

Words fail me to describe just how awesome LPeg is. Designed as a Lua implementation of the PEG concept, it is a true programming gem! Please, if you dont't know what it is, take some time to familiarize yourself with it! It's not the easiest thing to grasp, but you will *not* regret it! It is certainly one of the most worthwhile learning efforts you can make in generic programming.

One great feature of LPeg is that it's binary-safe, meaning that (unlike regular expressions) it can be safely used to parse binary data! This makes it an excellent tool for parsing binary protocols, especially network communication protocols, such as the Action Message Format (used by Adobe Flash for making remote calls and even in FLV movie files). I'll leave it to you to explore the possibilities...

Beware that from here on, I assume that you know your way around Lua, LPeg and how they work.

The problem

That being said, this article is actually about an unusual roadblock I hit while using LPeg to build a Lua-based AMF parser, and the various solutions I found and/or came up with to overcome it (you didn't think that I mentioned AMF before by accident, did you?).

The issue is LPeg's implementation of repetitive patterns: in particular, its inability to match (or capture) a fixed number of occurrences of a certain pattern, although it can match a minimum or a maximum number of such occurrences, which is perfect for stream-oriented parsing (such as parsing programming languages) but insufficient for binary data.

Just to clarify, here's a small list of LPeg patterns which correspond to the typical PCRE repetitive constructs (in each case we're trying to match the string 'cloth'):

Nr.		Matching occurrences of 'cloth'		PCRE pattern		LPeg pattern
1		0 or more (at least 0)		[cci_text]/(cloth)*/[/cci_text]		[cci_lua]lpeg.P'cloth'^0[/cci_lua]
2		1 or more (at least 1)		[cci_text]/(cloth)+/[/cci_text]		[cci_lua]lpeg.P'cloth'^1[/cci_lua]
3		X or more (at least X)		[cci_text]/(cloth){X,}/[/cci_text]		[cci_lua]lpeg.P'cloth'^X[/cci_lua]
4		1 or less (at most 1)		[cci_text]/(cloth)?/[/cci_text]		[cci_lua]lpeg.P'cloth'^-1[/cci_lua]
5		X or less (at most X)		[cci_text]/(cloth){,X}/[/cci_text]		[cci_lua]lpeg.P'cloth'^-X[/cci_lua]
6		precisely X (no more, no less)		[cci_text]/(cloth){X,X}/[/cci_text]		[cci_lua]-- not implemented --[/cci_lua]
7		anywhere between X and Y		[cci_text]/(cloth){X,Y}/[/cci_text]		[cci_lua]-- not implemented --[/cci_lua]

For cases 6 and 7, LPeg does not offer any simple constructs so we have to find a complex one. But let's put case 7 aside for a while, and try to tackle case 6, then we'll see...

Lua2C, an updated version

I know I have been "missing in action" lately but I am working furiously, and I seem to have too little time for my blog (very sad face). But, just for a breath of fresh air, I thought I'd share something with the world.

Entering lua2c.lua

Lately I became quite interested in Lua (a lot actually). It has phenomenal speed, exceptional interfacing with C and some features and libraries that just make my day (i.e. coroutines, lpeg, lua-ev and others), and since I needed to embed some Lua scripts (entirely) in a C project I'm currently working on, I ended up adapting Mike Edgar's "bin2c.lua" script (which takes a Lua script and turns it into a C header file) to suit my needs.

Basic functionality

Specifically, this adaptation generates a function that takes a Lua state as the only argument and then runs the embedded Lua code in the given state after which it returns the status (as opposed to putting the code straight in the top-level scope of the generated file). This makes it easier to embed code in C and then invoke it, and also to apply the same code onto multiple Lua states (e.g. multiple threads).

Check the end of the post for a usage sample.

Loomiere/Stream: Revived!

OK folks, things have settled down and we are good to go. After some considerations the legal concerns for Loomiere/Stream are now cleared and gone. The source is now released and will be available indefinitely. Again, this streamer (minimally customized) has already been serving all the video content at http://peteava.ro for 3 full months (for those seeking a demo).

Source code: loomiere-0.2.1-tar.gz

Warning:
Any software downloaded from this website is *never* to be associated in any form with pornographic or erotic content! I.E. "Loomiere/Stream" must *never* be used to stream pornographic or erotic videos! There are plenty alternatives if you can't help it.

Just a teaser:
Fighting the video-streaming problem has taught me very many things which, in turn, led me to realize that I might be able to use a substantially different streaming approach to achieve a massive amount of optimization in this field. So quite soon, I think I'll have ready a brand new (and far more powerful) streamer that aims at making it possible for a single server to serve many thousands (I am still testing this) of streams simultaneously using commodity hardware. I am not yet settled on whether this project will be commercial, open-sourced or both but I hope to clear this aspect soon as well. Real-world streaming (i.e. before and after) statistics will be made available on release.

Stay tuned! :)

Seven essential WordPress plugins

Here are the plugins I recommend for use on a WordPress blog.

Most (not all) of these plugins are generic, meaning that they are not bound to a specific type of blog and may (or should) be used regardless of what you write.

All of these plugins are top quality (IMHO) and provide an invaluable service for any blogger. I wholeheartedly recommend that you use and support them. I think that all of these should be apart of a standard WordPress installation.

@font-face for a programmer's blog?

[caption id="attachment_605" align="alignright" width="175" caption="See Consolas in action"][/caption]I'm sure that many of you are well aware of the exceptional @font-face CSS3 rule, which made life much easier for many designers, I would think. A few days ago I started wondering if it would be appropriate to use such an instrument on my blog... the site you're on! Yoopie!

And, why not!? I love the idea of making my source codes more readable by using some custom font (Consolas anyone?). Anyway, I started looking around for some fonts and was not pleased with any of them (well to be fair I did like DejaVu Sans Mono but not as much as Consolas, unfortunately).

URI parsing using Bash built-in features

A bit of background

A while ago I posted an article describing how one could parse complete URIs in Bash using the [cci_bash]sed[/cci_bash] program. Since then, I have realized that there is a better way to do it, a much better way: via Bash built-in pattern matching! 
Here are some benefits of this improvement:

• It no longer executes external programs (i.e. [cci_bash]sed[/cci_bash]) for pattern matching. This translates to higher speed and lower memory and CPU usages, which means that you could use this parser for much more intense URI crunching.


• The new regular expressions are drastically simplified thanks to the [cci_bash]${BASH_REMATCH[*]}[/cci_bash] array that is able to hold more than 9 matched sub-expressions, unlike [cci_bash]sed[/cci_bash] that can only work with single-digit escapes: [cci_bash]\1-\9[/cci_bash] (yuck!).


• The parsing algorithm is contained in a single Bash function, so no external file is needed to hold the regular expressions. This also means, obviously, that the pattern file is no longer loaded from disk on every execution (so HDD is saved as well).


• The generated variables are named identically to the first version, so you should be able to upgrade your scripts to this version with absolutely minimal effort.


• [Edit]
  No eval instruction is needed (unlike in the first version), further improving performance.