Apex FlexQueue muscles Apex Batch Processing

Batch Apex is salvation for developers. It allows you to serve large number of dataset(upto 50 million). I always employ Batch Apex to build complex, long-running processes on the Force.com platform. Although I can process over millions of records with Batch Apex, it has some platform limits. There can be only five active or queued jobs.

Why is this limit like this in the first place?

Salesforce always put limit in the first place because of its multi-tenant architecture. All asynchronous jobs are end up with the message queue. These message queues are having some sort of concurrency control mechanism just to make sure that one org does not monopolize.

A batch process can iterate over millions of records, and can take hours to run.  If your org had five of these long-running batch jobs in flight, anything you enqueued after that would be busy-waiting, due to concurrency controller. As such, the limit was created to prevent you from adding a sixth job while the five long-running jobs were consuming resources.

FlexQueue muscles Apex Batch Processing

In this Summer’14 release salesforce introduce new feature called “FlexQueue“. FlexQueue allows you to submit batch jobs beyond the limit of five queued or active jobs and give access to those jobs which are waiting to run. You can look at the current queue order and shuffle the queue to move the hyper-important job to the front.

shuffle queue

How it works?

In the current architecture, your batch job is serialized and included directly in the MQ message. For FlexQueue, your job is serialized to a “holding pen”, and a “token” is enqueued in the message queue. While tokens are in the queue waiting for resources, the holding jobs can be reordered. When a token reaches the front for processing, it will take the first job in the holding area.

To submit a batch job, call Database.executeBatch. The resulting outcome of Database.executeBatch depends on whether your organization has reached the five queued or active batch job limit.

  • If system resources are available for processing jobs, the batch job is queued for execution and its status is Queued.
  • If no system resources are available, the batch job is placed in the Apex flex queue and its status is Holding.
  • If the Apex flex queue has the maximum number(100) of jobs, this method returns an error and doesn’t place the job in the queue.

Currently in Summer’14 this feature is in pilot, but in Winter’15 it will generally available. (if everything goes well in pilot)

FlexQueue is like Cherry on the Cake. Salesforce not only allow me to process millions of data but also allows to manage my own org’s order-of-operations. I am just impress with this new feature. Yes, actually it muscles batch processing. 🙂