The new HTML5/ECMA6 Promise API is now available! In reading about it here, I knew I had to apply it. Check it out…
Relevant quotes:
“When you return something from a “then” callback, it’s a bit magic. If you return a value, the next “then” is called with that value. However, if you return something promise-like, the next “then” waits on it“
“Parallelism and sequencing – Getting the best of both”
Parallelism
Why does iQ use it?
- If a user queries six iXBRL aspects; (date, name, member, etc)
- and each aspect query takes one second;
- parallelism means results are ready to be “and/or”ed in one second, instead of six seconds
- and as far as I’m concerned, faster performance is a first-class feature
Sequencing
How does iQ use it?
- order of operations for “and/or”ing results matters
- at first I’ll evaluate everything left-to-right
- “green and blue or gold” is not the same as “green or gold and blue”;
- Likewise, .name(‘cash’).date(‘<2012’).or().member(‘subsidiary’) must be evaluated in that order!
- precedence of “and” vs “or” will eventually matter;
- iQ will evaluate “and” before “or”‘s; this means we can eventually bend the order-of-operationsrule;
- “red and black or blue and gold” is the same as “blue and gold or red and black”
- iQ will eventually provide a mechanism for “not” (.not()) and “parentheses” (.and(iQ.name(”).or()…))
- iQ will evaluate “and” before “or”‘s; this means we can eventually bend the order-of-operationsrule;
The article emphasizes the difference between two goals:
- Work that should be parallelized because order isn’t relevant yet; and if the user doesn’t need to wait; don’t make them wait.
- Parallel work doesn’t block any other processing
- Examples include
- retrieval (i.e. GET request),
- or processing (i.e. rendering an image),
- or querying (i.e. iQ), while the main script is reserved for the UI
- Note:Promises don’t parallelize for you; you need a mechanism to do it;
- i.e. get requests (use an XHR and event listeners, browsers can natively parallelize them);
- i.e. processing (use Workers, they’re made for parallelizing)
- Work that must be done in sequence because the user cares about order; if it’s out of order it’s wrong; don’t give the user wrong results.
- Sequence work must block other processing, else you can’t be sure your script executes in the right order
- Examples include:
- Chapters that must be read in order,
- queries must be “and/or”ed in order
The author makes three attempts to use Promises to get the best of parallelizing and sequencing.
First Attempt
The first is essentially like this:
(workThatIsParallelized).then(workOnResultInSequence) //1.then(workThatIsParallelized) //2.then(workOnResultInSequence) //etc
The problem is that sequence work (//1) blocks the next parallelized work (//2), because they’re connected with a .then.
The result: the user gets results in the right order, but must wait on retrieval that could’ve been parallelized;
That’s why his network panel starts to resemble a staircase; one-at-a-time.
In other words, (per comment above) it takes six seconds where it should only need one. Don’t make the user wait.
+ Sequence
– Parallel
My Second Attempt
I personally contemplated a second attempt that looked like this:
(workThatIsParallelized) //1.then(workOnResultInSequence) (workThatIsParallelized) //2.then(workOnResultInSequence)
Now all the work is parallelized; it will only take one second; since they aren’t connected with a .then, parallelized work starts immediately and does not wait on sequence work.
The problem with this attempt is that the parallelized work at (//2) might be finished before the work at //1;
Therefore the sequence work would begin out-of-order. It’s the opposite problem.
The user gets a gold-and-blue sweater, but they wanted gold-and-green, or blue. Don’t give the user wrong results.
– Sequence
+ Parallel
Second Attempt
In fact, the author’s second attempt is essentially like this:
(waitOnAllWorkThatCouldBeParallelized).then(workOnEachResultsInSequence)
Alright! This means parallelization works and sequencing works; so his network panel looks like many threads at once, and the user sees results in the correct order.
+ Sequence
+ Parallel
But we can do even better; waitOnAllWork means that we won’t do sequence work on chapter //1 until we’ve received chapter //2, and //3, and all the rest of the chapters… even if we’ve already received chapter 1 and we’re ready to do some sequence work in the right order.
How did he improve it?
Third Attempt
The author’s third attempt is awesome:
(allWorkThatCouldBeParallelized).then(workOnFirstResultInSequence) //1.then(workOnSecondResultInSequence) //2 ...etc
This is truly the best of both worlds– it will initiate all parallelized work at once, but it won’t wait on all of it; it starts processing the first result as-soon-as-possible.
This is the ultimate expression of our goal:
- Parallelize all work that can be parallelized,
- and sequence only what must be sequenced.
- It achieves a third hidden goal; start sequencing as soon as possible.
Now, with the exception of the .then method, I used pseudocode and made-up concepts(like “workOnSecondResultInSequence”) to express the solutions. This made it easy for me to understand the problems with each solution.
If you want to learn the clever Javascript techniques he used (including .reduce, .map, .all, .resolve, and more), I encourage you to read the original article.
Or check out my iQ github commit (forthcoming) where I apply these techniques.
Cheers,
Edit
This MapReduce idea sounds very similar to the current iQ architecture;
I recognize that iQ’s use of a “parallel” architecture to process only 567 values across, to-date, only 3 aspects (aka, “fields” — name, date, unit), seems excessive.
Parallel processing is better with more workers; I know a client/browser doesn’t have many workers (three cores?) compared to a server/cluster (hundreds or thousands?).
Even if iQ’s parallel architecture only boosts performance by a fraction, I’ll say it’s worthwhile;
And even if it doesn’t improve performance at all, I think the organization of code into separate files is a nice benefit.
Nate's XBRL Blog 
