Everything You Need To Know About Async & Meteor

Meteor runs on top of Node.js. This means I/O activities such as reading a file or sending a request over the network won’t block the whole program. Instead, we provide callbacks that will be executed in the Event Loop later when those activities finish. Ok, I agree, that may not make a lot of sense. How about some cartoons!

Let’s say our task is to read an encrypted file, then decrypt it and get the secret content:

var aes = Meteor.require('aes-helper')
    , fs = Meteor.require('fs');

var getSecretData = function(key) {
	fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
		if (err) console.log(err);
		else console.log( 'Secret: ' + aes.decrypt(res, key) );
	}
};

getSecretData('my-secret-key');

var aes = Meteor.require('aes-helper')

, fs = Meteor.require('fs');

var getSecretData = function(key) {

fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {

if (err) console.log(err);

else console.log( 'Secret: ' + aes.decrypt(res, key) );

}

};

getSecretData('my-secret-key');

Here is what a generic, garden-variety Event Loop looks like:

The Event Loop is just a queue of functions waiting to be executed. Every time we call a function, it is put onto the Event Loop

When we execute getSecretData to decrypt and print out the secret, the function readFile get called and appear on the Event Loop:

That readFile guy doesn’t care about what happens later at all, he just tells the OS to send the file and then go away!

Some moment later, the readFile operation is completed. A guy with the name ‘callback’ will jump into the Event Loop:

Later, when the file is received, our hero appears and finishes off the job

That’s quite nice and get the job done. But what if our task is more sophisticated and require many level of async operations? We might end up with something like this:

The problem of async control flow is it makes the code more difficult to read and maintain. It would be nicer if we can have getSecretData return the secret content and print it out synchronously, like this:

/* This code looks nicer, but sadly it doesn't work */

getSecretData = function(key) {
	var decryptedSecret;
	
	fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
		if (err) console.log(err);
		else decryptedSecret = aes.decrypt(res, key);
	}
	
	return decryptedSecret;  // undefined  <-- oops!
};

// So sweet. We have getSecretData return the value, then print it out, all synchronously.
// If only life were that simple...
var result = getSecretData('my-secret-key'); // undefined
console.log(result); // undefined

/* This code looks nicer, but sadly it doesn't work */

getSecretData = function(key) {

var decryptedSecret;

fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {

if (err) console.log(err);

else decryptedSecret = aes.decrypt(res, key);

}

return decryptedSecret; // undefined <-- oops!

};

// So sweet. We have getSecretData return the value, then print it out, all synchronously.

// If only life were that simple...

var result = getSecretData('my-secret-key'); // undefined

console.log(result); // undefined

This code does not work, because getSecretData will not wait for the readFile operation to finish. It just goes on and return decryptedSecret as an undefined value. To solve this problem, we need a new hero. Here comes Fiber!

Meet Fiber, the big guy who can carry many functions inside him

A Fiber is a special container function. He can be put into the Event Loop like other normal functions. But Fiber has a special power: He can halt at any point in the middle of his execution, get out of the Event Loop to take a rest, then come back at any time, all at his will (or in fact, the developer’s will). When a Fiber halts, control will be passed to the next function in the Event Loop (which may be a normal function or yet another Fiber).

You properly already see the advantage here: If our Fiber contains a function that performs a time-consuming I/O activity, he can just get out of the Event Loop and wait for the result. In the mean time, we can go on and run the next functions waiting in the queue. Life is short and time is precious! When the I/O activity finishes, our Fiber can jump back in again and resume what he was doing the last time.

Here is our code, Fiber-powered:

var Fiber = Npm.require('fibers');

// Our Fiber-powered getSecretData function
getSecretData = function(key) {
	var fiber = Fiber.current; // get the currently-running Fiber
	
	fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
		if (err) console.log(err);
		else fiber.run( aes.decrypt(res, key) ); // resume execution of this fiber. The argument passed 
                                                 // to fiber.run (i.e. the secret data) will become
		                                         // the value returned by Fiber.yield below
	}
	
    // halt this Fiber for now. When the execution is resumed later, return whatever passed to fiber.run
	var result = Fiber.yield();
	return result;
};

// We wrap our code in a Fiber, then run it
Fiber(function() {
	var result = getSecretData('my-secret-key');
	console.log(result);  // the decrypted secret
}).run();

var Fiber = Npm.require('fibers');

// Our Fiber-powered getSecretData function

getSecretData = function(key) {

var fiber = Fiber.current; // get the currently-running Fiber

fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {

if (err) console.log(err);

else fiber.run( aes.decrypt(res, key) ); // resume execution of this fiber. The argument passed

// to fiber.run (i.e. the secret data) will become

// the value returned by Fiber.yield below

}

// halt this Fiber for now. When the execution is resumed later, return whatever passed to fiber.run

var result = Fiber.yield();

return result;

};

// We wrap our code in a Fiber, then run it

Fiber(function() {

var result = getSecretData('my-secret-key');

console.log(result); // the decrypted secret

}).run();

Alright, that may not make a lot of sense. Here are your cartoons:

When Fiber encounters a yield, he knows it’s time to take a rest!

Calling run() will signal the resuming of execution for this Fiber. Whatever passed to run() will become the value returned by yield()

I hear you saying: ‘Ok, that’s looks good. But the yield and run stuffs still sound weird to me’.
I got you. We’ll see something even nicer than a Fiber. It’s a Future!

You can see Future as an abstraction layer on top of Fiber. This gives us the power of Fiber with a nicer API. Like a well-groomed Fiber.

var Future = Npm.require('fibers/future');

// Our Future-powered getSecretData function
getSecretData = function(key) {
	var future = new Future; // create a new, bright future
	
	fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
		if (err) console.log(err);
		else future.return( aes.decrypt(res, key) );  // signal that the future has finished (resolved) 
		                                              // the passed argument  (the decrypted secret) 
                                                      // will become the value returned by wait() below
	}
	
	return future;  // we return the future instance so other code can wait()  for this future
};


// The future method is added to the prototype object of every function
// Calling future() on a function will return a Fiber-wrapped version of it
(function() {
    // we wait for the future to finish. While we're waiting, control will be yielded
    // when this future finishes, wait() will return the value passed to future.return()
	var result = getSecretData('my-secret-key').wait();  
	                                                                                            
	console.log(result); 
}.future()) ();

var Future = Npm.require('fibers/future');

// Our Future-powered getSecretData function

getSecretData = function(key) {

var future = new Future; // create a new, bright future

fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {

if (err) console.log(err);

else future.return( aes.decrypt(res, key) ); // signal that the future has finished (resolved)

// the passed argument (the decrypted secret)

// will become the value returned by wait() below

}

return future; // we return the future instance so other code can wait() for this future

};

// The future method is added to the prototype object of every function

// Calling future() on a function will return a Fiber-wrapped version of it

(function() {

// we wait for the future to finish. While we're waiting, control will be yielded

// when this future finishes, wait() will return the value passed to future.return()

var result = getSecretData('my-secret-key').wait();

console.log(result);

}.future()) ();

Wait! In the examples above, we have freely modified our getSecretData function. What if you come across an async function that you can’t modify (like functions from external APIs)? No worry, instead of modifying it, we can wrap it up!

// A native, garden-variety async function
getSecretData = function(key, callback) {
	fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
		if (err) throw new Error(err.message);
		else callback && callback( null, aes.decrypt(res, key)  );
	}
};

// Let's wrap it up
// What returned here is actually a future instance. When the async operation completes,
// this future instance will be marked as finished and the result of the async operation
// will become the value returned by wait()
var getSecretDataSynchronously =  Future.wrap(getSecretData); 

(function() {
    // we call wait() on the returned future instance to, well, wait for this future to finish
    // and get the result later when it does
	var result = getSecretDataSynchronously ('my-secret-key').wait();   
	                                                                                                                        
	console.log(result); 
}.future()) ();

// A native, garden-variety async function

getSecretData = function(key, callback) {

fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {

if (err) throw new Error(err.message);

else callback && callback( null, aes.decrypt(res, key) );

}

};

// Let's wrap it up

// What returned here is actually a future instance. When the async operation completes,

// this future instance will be marked as finished and the result of the async operation

// will become the value returned by wait()

var getSecretDataSynchronously = Future.wrap(getSecretData);

(function() {

// we call wait() on the returned future instance to, well, wait for this future to finish

// and get the result later when it does

var result = getSecretDataSynchronously ('my-secret-key').wait();

console.log(result);

}.future()) ();

Hmm, looks like we’ll need to remember to call wait() every time. What a hassle!
Fortunately, it’s even simpler if we use Meteor.wrapAsync:

getSecretData = function(key, callback) {
	fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {
		if (err) throw new Error(err.message);
		else callback && callback( null, aes.decrypt(res, key)  );
	}
};

var getSecretDataSynchronously =  Meteor.wrapAsync(getSecretData); 
var result = getSecretDataSynchronously(key);  // <-- no wait() here!
return result;

getSecretData = function(key, callback) {

fs.readFile('/secret_path/encrypted_secret.txt', 'utf8', function(err, res) {

if (err) throw new Error(err.message);

else callback && callback( null, aes.decrypt(res, key) );

}

};

var getSecretDataSynchronously = Meteor.wrapAsync(getSecretData);

var result = getSecretDataSynchronously(key); // <-- no wait() here!

return result;

Actually there’s more to async than meets the eye. Other useful things that are worth mentioning:

– – –

Future.wrap and Meteor.wrapAsync are very selective

They only do business with native, pure async functions. That is, functions that expect a callback with error and result as arguments. Also, they only work on the server-side (since yielding is not possible on the client – there’re no Fibers living there).

– – –

Meteor.wrapAsync will turn your innocent function into Two-Face !!!

Fortunately, two-faced functions are not as destructive as Harvey Dent. In fact, they’re pretty useful: They can be called synchronously (like what we were doing above) or asynchronously (with a callback passed to them).

On server-side, methods such as HTTP.call and collection.insert/update/remove are all already pre-wrapped this way. Take HTTP.call for example: If you call it without a callback, the method will block until the response is received. If called with a callback, HTTP.call returns immediately, and will later excute the provided callback when network response has arrived.

On client-side, since blocking/yielding is not possible, we always have to provide a callback to these methods.

– – –

Fiber hiccups

By default, method calls from a client are run inside a single Fiber – they’re run one at a time. This Fiber gets access to a set of environment variables that are specific to the currently connected client (e.g. Meteor.userId()). This may result in two common problems:

1) On server-side, calling methods like HTTP.call synchronously will block other subsequent method calls from the current client. This may not be a good thing. If subsequent methods are independent from the current running method, we can save time by using this.unblock(), which will allow other method calls to be run in a new Fiber:

Meteor.methods({
	requestSecret: function() {
		this.unblock();
		return HTTP.call('GET', 'http://www.nsa.gov/top-secrets');
	}
});

Meteor.methods({

requestSecret: function() {

this.unblock();

return HTTP.call('GET', 'http://www.nsa.gov/top-secrets');

}

});

2) “Meteor code must always run within a Fiber”

Looks familiar? This error often occurs when you try to call a third-party API with async callback. You’re not allowed to do this, since the callback function would be executed outside Fiber, without access to necessary environment variables. One way to solve this is wrapping the callback function with Meteor.bindEnvironment, which will return a Fiber-wrapped and environment-packed version of the function. The second way is using Meteor.wrapAsync like what we were doing above (actually wrapAsync already called bindEnvironment internally for us!).

I hope you’ve learned something useful about async and Meteor in this article. Happy coding!

P.S. For anyone wanting to learn more about this topic, here are two interesting classes from EventedMind:

Meteor Fibers and Dynamics

The JavaScript Runtime

30 Comments → Everything You Need To Know About
Async & Meteor

Rainteller November 19, 2014 at 2:51 pm

Fantastic article. Thank you very much! (got linked in Meteor Daily) The lack of SSL hurts, though

Reply ↓
1. Phuc Nguyen November 22, 2014 at 2:34 am
  
  Hi,
  
  What do you mean by ‘the lack of SSL hurts’?
  
  Reply ↓
  1. Rainteller January 10, 2015 at 11:01 pm
    
    Hi,
    That your site doesn’t use SSL. Sorry for late reply.
    
    Reply ↓
Toby November 19, 2014 at 10:27 pm

Great article, thanks for sharing! Loved the cartoons

Reply ↓
Praney November 21, 2014 at 3:30 am

Awesome article, loved the way you explained and the cartoons were sweet

Reply ↓
Phuc Nguyen November 22, 2014 at 2:29 am

I’m very glad to hear that the article is helpful!

Reply ↓
Thanish November 24, 2014 at 10:06 am

Nice cartoons

Reply ↓
Rodrigo December 24, 2014 at 8:09 pm

Really good article! It’s a difficult topic to understand but your explanation shines. This video complements the post: https://www.eventedmind.com/feed/meteor-meteor-wrapasync

Reply ↓
aaron March 23, 2015 at 10:47 pm

fantastic! this post just saved me hours of time!

This one line: “They can be called synchronously (like what we were doing above) or asynchronously (with a callback passed to them).”

Is what clear up the confusion for me!

Thanks again!

Reply ↓
Pingback: Everything You Need To Know About Async & Meteor | Dinesh Ram Kali.
Pingback: Best Learning Resources for Meteor.js | 懒得折腾
Pingback: Meteor-Learning | 懒得折腾
Chetan May 7, 2015 at 12:52 pm

Fantastic article.Love the way you explain with those cartoons.:)

Reply ↓
dddbseyhseuuhb June 30, 2015 at 3:15 pm

są pierwsze wyimaginować sumę stwórz matrymonialnych, wespół z poufnymi, skoro mają że awansowanie się na nie
da im pewną przewaga. Istnieją niewyobrażalnie perfidne
natomiast skoro faktycznie toż potrafię zachwycić – bezwyjątkowe.
W 4 casusach na 5 swobodnie porwane użyciem podkreślaj do skoncentrowanego posagu.

Reply ↓
Pingback: Meteor.js异步全解 | Meteor中国
MaikelH July 25, 2015 at 5:51 pm

phucnguyen.info has potential, you can make your blog go viral easily using one tricky method. Just search in google:
Sulingi’s Method To Go Viral

Reply ↓
David Patte September 16, 2015 at 12:18 am

Clear, concise, well-thought-out article. I especially like that you included all the intermediate steps showing how Futures are actually fibers underneath.
Canada

Reply ↓
wisarut November 13, 2015 at 5:30 pm

Good Article

Reply ↓
Pingback: List of resources to learn to build amazing apps with JavaScript – Meteor |
Pingback: Synchronous call using wrapAsync - HTML CODE
King of Thieves Hack January 23, 2016 at 6:33 am

Awesome things here. I am very happy to look your post.
Thanks a lot and I’m taking a look ahead to touch you. Will you kindly drop me a mail?

Reply ↓
Pingback: How to learn meteor | About Resarch
home November 29, 2016 at 5:22 pm

ifqmyidmpvjuanqfvfxfbqhrfwqbeu
home https://bluefintrading.co.uk/

Reply ↓
Carl Jakobsson December 6, 2016 at 4:59 pm

Just wanted to say thank you for showing how to create a synchronous function from a custom made asynchronous function. Thank you so much, it has really helped me!!

Reply ↓
1. Jobeth December 26, 2016 at 8:26 pm
  
  HiI have connected my Mac to a Linux box directly using a cable (no home router involved).In Mac, I am setting a static IP address to 10.10.11.13 with subnet mask 255.255.255.0 and in Linux 10.10.11.16 with same mask. My Mac has two NICs (2nd one is connected to router and WWW). How can I add a route in Mac so all the traffic intended for the Linux box goes through first NIC? I am interested in waking the Linux machine up using something like:$ wakeonlan -i 10.10.11.16 00BA::AB:XX:XXIs this routing possible?
  
  Reply ↓
2. sofort barkredit ohne January 29, 2017 at 1:36 am
  
  Robert, America only has one r in it.Slow down.That’s like a spinoff of the Russian spelling.reply from Robert: Thanks Clay. Didn’t realize I had done that. It’s fixed now.
  
  Reply ↓
3. autoversicherungen berlin February 2, 2017 at 10:43 pm
  
  Excelent blog Radule.Un alt amanunt, nu numai botanistul casei regale dar chiar Printul Charles este "atasat" de flora din Romania, in particular de pe langa Saschiz.De cate ori ma intorc in vizita, drumul Brasov – Sighisoara este in itinerariul nostru.Liviu,Un Brasovean in Canada
  
  Reply ↓
4. kündigungsschreiben kfz versicherung vordruck lebenslauf February 2, 2017 at 11:21 pm
  
  Beyond lovely! The print and your Lydia mods are just right, Nettie. I’m so very glad to hear the making and wearing of this gorgeous dress was a source of cheer up for you. What a nod to sewing as therapy!
  
  Reply ↓
Sarah Carlson September 1, 2017 at 4:59 pm

I came to your Everything You Need To Know About Async & Meteor | Phuc Nguyen’s Blog page and noticed you could have a lot more traffic. I have found that the key to running a website is making sure the visitors you are getting are interested in your subject matter. We can send you targeted traffic and we let you try it for free. Get over 1,000 targeted visitors per day to your website. Check it out here: http://pcgroup.com.uy/15 Unsubscribe here: http://pcgroup.com.uy/2a

Reply ↓
tapicer warszawa November 8, 2017 at 7:07 pm

Nа indywidualne życzenie Klienta odpowiednie materiałydo
stworzenia wymarzonej tapicerki sprowadzam z różnych stron świata.
Ɗo renowawcji zawsze używam ekologicznych,
naturalnych skór. Proponuję kompleksowy zakres սsług
tapicerskich dla zastosowań motoryzacyjnych – oferuję tapicerowanie dla
aut osobowych, ϲiężarowych, przyczep kempingowych,
autobusóᴡ,…

Reply ↓

Grow

. . . (hopefully) useful stuff for web development and life

Everything You Need To Know About
Async & Meteor

30 Comments → Everything You Need To Know About
Async & Meteor

Leave a Reply Cancel reply

30 Comments → Everything You Need To Know About Async & Meteor

Leave a Reply Cancel reply

30 Comments → Everything You Need To Know About
Async & Meteor