Loomiere 2.0 teaser

[Update]
The new high is now ~2600 active streams, summing ~1500 Mbps of bandwidth. Some graphs:

• Network traffic (cumulated for all interfaces): https://www.bijk.com/s/7c52.png


• Active connections: https://www.bijk.com/s/0F48.png

---

This article is a quick peek into how Loomiere 2.0 is (currently) able to serve ~1700 active streams from a single server in production, summing up to more than 1100 Mbps of bandwidth.

All these statistics represent a single machine, in real time, over a period of 12 hours, equipped as follows:

• 2 x Xeon Dual Core (with HyperThreading) at 3.2GHz


• 16 GB RAM


• 15 x 1TB HDD (7200rpm)


• 4 x 1GB NIC's


• This system runs on Ubuntu 10.04.1 LTS.

OK so here are the graphs:

• Cpu load: https://www.bijk.com/e/812e


• Network traffic (cumulated for all interfaces): https://www.bijk.com/e/8423


• Active connections: https://www.bijk.com/e/AF6E


• Memory consumption: https://www.bijk.com/e/E379

The graphs are not quite stable at the moment since I am currently working hard to test this setup and Loomiere itself, but I think soon I will have a more stable configuration and also some time to comment on these graphs.

Enjoy your weekend!

P.S.: I love graphs :).

Case study: fixed number of iterations with LPeg

Words fail me to describe just how awesome LPeg is. Designed as a Lua implementation of the PEG concept, it is a true programming gem! Please, if you dont't know what it is, take some time to familiarize yourself with it! It's not the easiest thing to grasp, but you will *not* regret it! It is certainly one of the most worthwhile learning efforts you can make in generic programming.

One great feature of LPeg is that it's binary-safe, meaning that (unlike regular expressions) it can be safely used to parse binary data! This makes it an excellent tool for parsing binary protocols, especially network communication protocols, such as the Action Message Format (used by Adobe Flash for making remote calls and even in FLV movie files). I'll leave it to you to explore the possibilities...

Beware that from here on, I assume that you know your way around Lua, LPeg and how they work.

The problem

That being said, this article is actually about an unusual roadblock I hit while using LPeg to build a Lua-based AMF parser, and the various solutions I found and/or came up with to overcome it (you didn't think that I mentioned AMF before by accident, did you?).

The issue is LPeg's implementation of repetitive patterns: in particular, its inability to match (or capture) a fixed number of occurrences of a certain pattern, although it can match a minimum or a maximum number of such occurrences, which is perfect for stream-oriented parsing (such as parsing programming languages) but insufficient for binary data.

Just to clarify, here's a small list of LPeg patterns which correspond to the typical PCRE repetitive constructs (in each case we're trying to match the string 'cloth'):

Nr.		Matching occurrences of 'cloth'		PCRE pattern		LPeg pattern
1		0 or more (at least 0)		[cci_text]/(cloth)*/[/cci_text]		[cci_lua]lpeg.P'cloth'^0[/cci_lua]
2		1 or more (at least 1)		[cci_text]/(cloth)+/[/cci_text]		[cci_lua]lpeg.P'cloth'^1[/cci_lua]
3		X or more (at least X)		[cci_text]/(cloth){X,}/[/cci_text]		[cci_lua]lpeg.P'cloth'^X[/cci_lua]
4		1 or less (at most 1)		[cci_text]/(cloth)?/[/cci_text]		[cci_lua]lpeg.P'cloth'^-1[/cci_lua]
5		X or less (at most X)		[cci_text]/(cloth){,X}/[/cci_text]		[cci_lua]lpeg.P'cloth'^-X[/cci_lua]
6		precisely X (no more, no less)		[cci_text]/(cloth){X,X}/[/cci_text]		[cci_lua]-- not implemented --[/cci_lua]
7		anywhere between X and Y		[cci_text]/(cloth){X,Y}/[/cci_text]		[cci_lua]-- not implemented --[/cci_lua]

For cases 6 and 7, LPeg does not offer any simple constructs so we have to find a complex one. But let's put case 7 aside for a while, and try to tackle case 6, then we'll see...

Lua2C, an updated version

I know I have been "missing in action" lately but I am working furiously, and I seem to have too little time for my blog (very sad face). But, just for a breath of fresh air, I thought I'd share something with the world.

Entering lua2c.lua

Lately I became quite interested in Lua (a lot actually). It has phenomenal speed, exceptional interfacing with C and some features and libraries that just make my day (i.e. coroutines, lpeg, lua-ev and others), and since I needed to embed some Lua scripts (entirely) in a C project I'm currently working on, I ended up adapting Mike Edgar's "bin2c.lua" script (which takes a Lua script and turns it into a C header file) to suit my needs.

Basic functionality

Specifically, this adaptation generates a function that takes a Lua state as the only argument and then runs the embedded Lua code in the given state after which it returns the status (as opposed to putting the code straight in the top-level scope of the generated file). This makes it easier to embed code in C and then invoke it, and also to apply the same code onto multiple Lua states (e.g. multiple threads).

Check the end of the post for a usage sample.

Loomiere/Stream: Revived!

OK folks, things have settled down and we are good to go. After some considerations the legal concerns for Loomiere/Stream are now cleared and gone. The source is now released and will be available indefinitely. Again, this streamer (minimally customized) has already been serving all the video content at http://peteava.ro for 3 full months (for those seeking a demo).

Source code: loomiere-0.2.1-tar.gz

Warning:
Any software downloaded from this website is *never* to be associated in any form with pornographic or erotic content! I.E. "Loomiere/Stream" must *never* be used to stream pornographic or erotic videos! There are plenty alternatives if you can't help it.

Just a teaser:
Fighting the video-streaming problem has taught me very many things which, in turn, led me to realize that I might be able to use a substantially different streaming approach to achieve a massive amount of optimization in this field. So quite soon, I think I'll have ready a brand new (and far more powerful) streamer that aims at making it possible for a single server to serve many thousands (I am still testing this) of streams simultaneously using commodity hardware. I am not yet settled on whether this project will be commercial, open-sourced or both but I hope to clear this aspect soon as well. Real-world streaming (i.e. before and after) statistics will be made available on release.

Stay tuned! :)

Seven essential WordPress plugins

Here are the plugins I recommend for use on a WordPress blog.

Most (not all) of these plugins are generic, meaning that they are not bound to a specific type of blog and may (or should) be used regardless of what you write.

All of these plugins are top quality (IMHO) and provide an invaluable service for any blogger. I wholeheartedly recommend that you use and support them. I think that all of these should be apart of a standard WordPress installation.

@font-face for a programmer's blog?

[caption id="attachment_605" align="alignright" width="175" caption="See Consolas in action"][/caption]I'm sure that many of you are well aware of the exceptional @font-face CSS3 rule, which made life much easier for many designers, I would think. A few days ago I started wondering if it would be appropriate to use such an instrument on my blog... the site you're on! Yoopie!

And, why not!? I love the idea of making my source codes more readable by using some custom font (Consolas anyone?). Anyway, I started looking around for some fonts and was not pleased with any of them (well to be fair I did like DejaVu Sans Mono but not as much as Consolas, unfortunately).

URI parsing using Bash built-in features

A bit of background

A while ago I posted an article describing how one could parse complete URIs in Bash using the [cci_bash]sed[/cci_bash] program. Since then, I have realized that there is a better way to do it, a much better way: via Bash built-in pattern matching! 
Here are some benefits of this improvement:

• It no longer executes external programs (i.e. [cci_bash]sed[/cci_bash]) for pattern matching. This translates to higher speed and lower memory and CPU usages, which means that you could use this parser for much more intense URI crunching.


• The new regular expressions are drastically simplified thanks to the [cci_bash]${BASH_REMATCH[*]}[/cci_bash] array that is able to hold more than 9 matched sub-expressions, unlike [cci_bash]sed[/cci_bash] that can only work with single-digit escapes: [cci_bash]\1-\9[/cci_bash] (yuck!).


• The parsing algorithm is contained in a single Bash function, so no external file is needed to hold the regular expressions. This also means, obviously, that the pattern file is no longer loaded from disk on every execution (so HDD is saved as well).


• The generated variables are named identically to the first version, so you should be able to upgrade your scripts to this version with absolutely minimal effort.


• [Edit]
  No eval instruction is needed (unlike in the first version), further improving performance.

Quality Coding Practices

This is a set of guidelines that I put together starting a while back, which I try to respect when I write code. It is mostly (at least for now) oriented towards programming in C, but any systems programming language could benefit. Be aware that these do not apply to everyone or every scenario, adaptation and clever compromises are essential!

Now, this is obviously only a humble start, so I would like to ask the community (yes, you!) to pitch in and help grow (and correct) this list of useful principles, so that, in time, it may actually become something used by folks around the web to better their coding practices. C'mon, this will be fun, fun, fun! :D

First impressions on Loomiere/Stream performance

As promised, here are some of the first monitoring statistics of Loomiere/Stream in a production environment after moving away from psstream. Only one server is considered, a Quad-core Xeon with 8GB RAM (not that they are used anymore).

This shows the memory usage over one week (the switch was made on the 29th as is obvious).

Loomiere/Stream - A high performance streaming server

UPDATE: The code is now freely available under GPLv3. Please see this post for an update.

ATTENTION: Due to the fact that myself and my company are not yet satisfied with some issues regarding the legal terms of Loomiere/Stream, I think it is wise to take it (temporarily) offline until we resolve our concerns! Also some of the technical characteristics have been retracted.

Are you killing psstream?

Well, yes! I am sure that many of you already know about psstream (the PHP streaming extension I made a while back). Well, many things happened since then and I came to realize I could do better; a whole lot better actually. As of now the 'psstream' project is officially no longer developed (see below). It will remain on the website for some time to come for archiving purposes but that is it.

But wait, why did you do it?

For some time now I have be looking into improving the streaming mechanism for a large video-sharing project run by my company. PSStream was a first effort, and it did the job but soon ran into problems. None of our servers was able to properly stream more than 150 clients simultaneously and the resources were grossly wasted, hence, Loomiere/Stream.