Living the Stream Dream

By Ryan Ramage / @ryan_ramage_

INTERACTIVE LIVE ACTION DEMO

Just remember:

PEOPLE == DATA

Typical Search Pattern

Result: ONE BIG TABLE

Better Search Pattern

What we learned

• Streaming decrease user idle time. [time]
• Streaming can lower memory requirements [space]

GTSTTUASAP principle

Get the shituff to the user as fast as possible

Because Speed is King in applications

Streams Are Good Abstractions

Garden hose

Dealing with disk/network/io/processor variability

Keep memory spikes down

What is the stream dream?

Streaming support across your stack

• Streaming Transport Level
• Streaming Browser Libs
• Streaming Middleware
• Streaming DB
• Streaming Build System (optional)

Streaming Transport Level

TCP

The internet was built for streams.

The Stream Dream?

Streaming support across your stack

• Streaming Transport Level
• Streaming Browser Libs
• Streaming Middleware
• Streaming DB
• Streaming Build System (optional)

Oboe.js

Wrapping xhr with a progressive streaming interface
A JSON parser that sits somewhere between SAX and DOM.

Oboe.js

Dealing with failure mid request/response

Oboe.js

Incremental loading, and browser caching

Oboe.js

Easy JSON slice and dice

GET /myapp/things

{
   "foods": [
      {"name":"aubergine",    "colour":"purple"},
      {"name":"apple",        "colour":"red"},
      {"name":"nuts",         "colour":"brown"}
   ],
   "badThings": [
      {"name":"poison",       "colour":"pink"},
      {"name":"broken_glass", "colour":"green"}
   ]
}

MyApp.showSpinner('#foods');

oboe('/myapp/things')
   .node('foods.*', function( foodThing ){
      $.templateThing.append( foodThing.name + ' is ' + foodThing.colour );
   })
   .node('badThings.*', function( badThing ){
      console.log( 'Danger! stay away from ' + badThings.name );
   })
   .done( function(things){
	   	MyApp.hideSpinner('#foods');
      console.log( 'there are ' + things.foods.length + ' things you can eat ' +
                   'and ' + things.nonFoods.length + ' that you shouldn\'t.' );
   });

Duck Typing

oboe('/myapp/things')
   .node('{name colour}', function( foodObject ) {
      // I'll get called for every object found that
      // has both a name and a colour
   };

Shoe

Streaming between node and the browser

The Stream Dream?

Streaming support across your stack

• Streaming Transport Level
• Streaming Browser Libs
• Streaming Middleware
• Streaming DB
• Streaming Build System (optional)

nodejs

var http = require('http');
var fs = require('fs');

var server = http.createServer(function (req, res) {
    fs.readFile(__dirname + '/movie.mp4', function (err, data) {
        res.end(data);
    });
});
server.listen(8000);

BAD

var http = require('http');
var fs = require('fs');

var server = http.createServer(function (req, res) {
    var stream = fs.createReadStream(__dirname + '/movie.mp4');
    stream.pipe(res);
});
server.listen(8000);

Good

Copy A file

var fs = require("fs");

// Read File
fs.createReadStream("input/people.json")
    // Write File
    .pipe(fs.createWriteStream("output/people.json"));

Un-Gzipping a File

var fs = require("fs");
var zlib = require("zlib");

// Read File
fs.createReadStream("input/people.csv.gz")
    // Un-Gzip
    .pipe(zlib.createGunzip())
    // Write File
    .pipe(fs.createWriteStream("output/people.csv"));

Node Modules For Streaming

Advice: Watch for > 1 year inactivity.
Streams have changed a lot

Node Modules For Streaming

	request('http://google.com/doodle.png').pipe(fs.createWriteStream('doodle.png'))
	

Node Modules For Streaming

The EventStream functions resemble the array functions, because Streams are like Arrays, but laid out in time, rather than in memory.

Example of event stream

var inspect = require('util').inspect
var es = require('event-stream')    //load event-stream

es.pipe(                            //pipe joins streams together
  process.openStdin(),              //open stdin
  es.split(),                       //split stream to break on newlines
  es.map(function (data, callback) {//turn this async function into a stream
    var j
    try {
      j = JSON.parse(data)          //try to parse input into json
    } catch (err) {
      return callback(null, data)   //if it fails just pass it anyway
    }
    callback(null, inspect(j))      //render it nicely
  }),
  process.stdout                    // pipe it to stdout !
  )


// curl -sS registry.npmjs.org/event-stream | node pretty.js
//

Stream Adventure

> npm install -g stream-adventure
> stream-adventure

The Stream Dream?

Streaming support across your stack

• Streaming Transport Level
• Streaming Browser Libs
• Streaming Middleware
• Streaming DB
• Streaming Build System (optional)

CouchDB

of course

Streamable HTTP

    request('http://localhost:5984/db/_design/app/_view/things_by_date').pipe(resp);

Filter with node!

  var filter = function(data, emit) {
      data.forEach(function(db){
          if (db.indexOf('dumb_user-') === 0) {
              emit('"' +  strip_prefix(db) + '"');
          }
      });
  }
  var filter_through = new FilterThrough(filter);
  request('http://localhost:5984/_all_dbs')
	  .pipe(filter_through)
	  .pipe(resp);

Continuous Changes

00:00: > curl -X GET "http://localhost:5984/db/_changes?feed=continuous&since=3&filter=app/important"
{"seq":4,"id":"test4","changes":[{"rev":"1-02c6b758b08360abefc383d74ed5973d"}]}
{"seq":5,"id":"test5","changes":[{"rev":"1-02c6b758b08360abefc383d74ed5973d"}]}

rvagg/node-levelup

rvagg/node-levelup

Everything streams

var levelup = require('levelup');
var srcdb = levelup('./srcdb');
var dstdb = levelup('./destdb');

srcdb.put('name', 'LevelUP');

srcdb.createReadStream().pipe(dstdb.createWriteStream()).on('close', onDone)

rvagg/node-levelup

Start/End keys

srcdb.createReadStream({
	start: 'n',
	end: 'k'
}).pipe(resp)

The Stream Dream?

Streaming support across your stack

• Streaming Transport Level
• Streaming Browser Libs
• Streaming Middleware
• Streaming DB
• Streaming Build System (optional)

Gulp - Streaming builds

Sample Gulpfile

								

var scriptFiles = './src/**/*.js';

gulp.task('compile', function(){
  // concat all scripts, minify, and output
  gulp.src(scriptFiles)
    .pipe(concat({fileName: pkg.name+".js"})
    .pipe(minify())
    .pipe(gulp.dest('./dist/'));
});

gulp.task('test', function(){
  // lint our scripts
  gulp.src(scriptFiles).pipe(jshint());

  // run our tests
  spawn('npm', ['test'], {stdio: 'inherit'});
});

gulp.task('default', function(){
  gulp.run('test', 'compile');
  gulp.watch(scriptFiles, function(){
    gulp.run('test', 'compile');
  });
});
								
							

Cons

Because nothing is free

Might not play well with your 'Framework'

MVC-rap routes and functions

								
// please provide
class People extends MVCrap {
	getAll()
	getByX()
}
								
							

much confuse

stream-to-pull-stream: Convert a classic-stream, or a new-stream into a pull-stream

invert-stream: Create a pair of streams (A, B) such that A.write(X) -> B.emit('data', X) and B.write(X) -> A.emit('data', X)

Backpressure and Unbounded Concurrency

Backpressure and Unbounded Concurrency

But the dream lives on...

Mix and match:

• Oboe.js
• Node Streams
• CouchDB
• node-LevelDB
• gulp

Try them out today!

THANKS!

Living the Stream Dream

By Ryan Ramage / @ryan_ramage_