How to handle multiple api calls in a time-constrained scenario?

Question

I have to make multiple API calls in as short a time as possible. The need to make multiple calls arises from me having to populate a wide range of conditional data sets. Say I have n metrics, each to be filtered with m possible filters. I would want to get the results.totalsForAllResults for the n*m queries I generate.

While I faced a lot of hiccups initially, knowing about closures solved many issues about my trouble with sending async API calls. I was even able to handle results well, in the proper order. Now I'm facing an issue where the maximum number of API requests per second poses an issue for me. Google Core Reporting API v3 allows a maximum of 10 API requests in a second, and I'm well past this limit.

Here is how I tried to make the API calls and process the responses. I have little freedom with the structure:

function getMetrics() {
    initResultsArray();
    for (mI = 0; mI < metricsArray.length; mI++) {
        (function (mI) {    //Closure. Holds a local value of 'mI' within the scope
            for (fI = 0; fI < filtersArray.length; fI++) { 
                (function (fI) {   //Closure. Holds a local value of 'fI' within the scope
                    gapi.client.analytics.data.ga.get({
                        'ids': tableID,
                        'start-date': startDate,
                        'end-date': endDate,
                        'metrics': metricsArray[mI],
                        'filters': filtersArray[fI],
                        'samplingLevel': 'HIGHER_PRECISION',
                    }).execute(function putToVar(results) {   //this fn is defined inline to get access to fI
                        console.log(results.totalsForAllResults);
                        resultsArray[mI][fI] = parseInt(results.totalsForAllResults[metricsArray[mI]]);
                    });
                })(fI); //This ends the closure for fI
            }
        })(mI); //This ends the closure for mI
    }
}
//Print results to console, called when I believe all results have been populated.
function logResults() {
    console.log(resultsArray);
}

I need to be able to find out if I have made 10 queries in the last second and wait to send the remaining queries, because as soon as I exceed 10 queries per second I get null objects as response for my API calls and it ruins my ability to retrieve values into arrays. How can this be done? I do not know how to use wait() and people say the browser becomes unresponsive if you use wait(), and I don't know how setTimeOut() can be applied to my problem.

mI and fI are global iterators for metrics and filters and metricsArray and filtersArray are arrays of strings representing metrics and filters in the way GA API would expect them, I just need to iterate through them to obtain a lot of results.TotalsForAllResults. There is no problem with the execution of the API calls and responses. My only issue is exceeding the 10 queries per second limit and not getting further responses.

I've edited my answer again. I forgot to update the current variable in the while loop. — RobH
– RobH, Commented Jan 15, 2014 at 13:38

RobH · Accepted Answer · 2014-01-15 14:48:22Z

1

You could solve this by first creating a list of the calls you need to make, then making them 10 at a time. This is all off the cuff, so no guarantees that it actually works but hopefully you can apply it to your situation.

The general idea is to create a simple Scheduler constructor function that takes in an array of stuff to process. Naming stuff more descriptively would be better :). The created object has a single function - start.

var Scheduler = function (stuffToProcess) {
    var started,
        executeFunction;

    getExecuteFunction = function (current) {
        return function (results) {
            console.log(results.totalsForAllResults);
            resultsArray[current.mI][current.fI] = parseInt(results.totalsForAllResults[metricsArray[current.mI]], 10);
        };
    }

    var processNext = function () {
        var current = stuffToProcess.shift(),
            counter = 0;

        while (current && ++counter <= 10) {
            gapi.client.analytics.data.ga
                .get(current.gaBit)
                .execute(getExecuteFunction(current));
            if (counter !== 10) {
                current = stuffToProcess.shift(); // <- EDIT: Forgot this in original answer.
            }
        }
        if (stuffToProcess.length > 0) {
            window.setTimeout(function () {
                processNext();
            }, 1000);
        }
    };

    this.start = function () {
        if (!started) {
            started = true;
            processNext();
        }
    };    
};

Then in your getMetrics function, instead of calling ga directly, you build an array of the calls you want to make, then create a scheduler instance and start it.

function getMetrics() {
    initResultsArray();
    var listOfCalls = [];
    for (mI = 0; mI < metricsArray.length; mI++) {
        for (fI = 0; fI < filtersArray.length; fI++) { 
            listOfCalls.push({
                gaBit: {
                    'ids': tableID,
                    'start-date': startDate,
                    'end-date': endDate,
                    'metrics': metricsArray[mI],
                    'filters': filtersArray[fI],
                    'samplingLevel': 'HIGHER_PRECISION'
                },
                mI: mI,
                fI: fI
            });
        }
    }
    var s = new Scheduler(listOfCalls);
    s.start();
}

EDIT: Modified code to use getExecuteFunction instead.

edited Jan 15, 2014 at 14:48

answered Jan 13, 2014 at 11:46

RobH

3,6101 gold badge25 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

artfuldev Over a year ago

I thought I had provided enough explanation for mI and fI - they stood for metricsIterator and filtersIterator and were shortened for byte-size optimisation. :D Thanks, will try your solution out and post a reply. :)

RobH Over a year ago

Can you console.log(current.gaBit)? It should be the same as the object you were passing in before...

artfuldev Over a year ago

I can say, the code works. I mean, what you intended to convey to me, how to do it, is done. I understand what a scheduler does and what you have done here, but I have an issue. If I do push a similar object as the gaBit you have constructed to a testVar (I didn't add the filters thing) and write

gapi.client.analytics.data.ga.get(testVar[0].gaBit).execute(function tempHandler(results){console.log(results.totalsForAllResults[visits]);});

this works, I get the number I want. But somehow the return function(results) doesn't seem to work. Clarifying. Object is being passed as a valid arg.

artfuldev Over a year ago

Yes it is the same as before, but before, I used to be appending a trailing comma, in case you didn't notice. But I constructed a similar object and passed it as an argument to gapi.client.analytics.data.ga.get() and it does work. But the statements inside return function(results){...} don't seem to work.

RobH Over a year ago

That's my bad. One sec, I'll edit. Update: I've changed the way the execute function is returned now.

|

Collectives™ on Stack Overflow

How to handle multiple api calls in a time-constrained scenario?

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related