Have you ever had a script activity that just kept running over 30 minutes, but it wouldn’t break into individual pieces well? Its a struggle to find a solution because it’s not like you can easily break out of your loops after a set amount of time, right? Wrong.

There are a couple different tricks you can do in SSJS to ensure you don’t time out and also that you can keep running the same script in multiple steps in the same automation to finish it out.

So first thing first, how are we going to keep track of elapsed time? Some handy dandy Javascript will take care of that.

First you need to grab the start time:

var now = new Date()
var start = now.getTime();

This will give you the time you started the automation at (start). Now armed with this number, you can use it to gather a run time periodically.

Something like:

(new Date().getTime() - start)

This will give you the time in milliseconds since you began. Keep in mind though, that this will need to be run multiple times throughout your Script in order to accurately get a current run time.

What I do to account for this is to put the whole script (or run time heavy function) inside of a Do/While Loop. This way it will continue running while within your time parameters.

First, you would need to set a timeout amount in milliseconds (I usually use a variable). For this sample, lets say your run time limit is 20 minutes (to leave wiggle room for other functions and processing). So in this case, that would be 1,200,000 milliseconds.

Below is a quick sample of our Do/While loop so far:

var now = new Date()
var start = now.getTime();
var timeout = 1200000;
do {
    ...
} while((new Date().getTime() - start) < timeOut)

This will run whatever you contain inside of it for as long as the current time (Date().getTime()) minus the start time is less than your timeOut value. Do note, that this will only be taken into account during the beginning of each run, and not during – which is why we needed to set the timeout to 20 minutes, instead of 30.

So now super dooper! We can have it stop without erroring (timing out) – but how does that help since the script wouldn’t finish everything it was supposed to do?

Well, now comes the cool part. You can set up a Data Extension to contain last state of your current script and save it to the DE. This way, the next time the script runs, it can pick up right where it left off.

For our example, let’s say you are doing a REST API call to an external endpoint to collect data and push it into a Data Extension. Due to the limitations of the endpoint, you can only collect 2,000 per call and need to collect 100,000 records from it.

After a bit of exploration, you find each retrieve call takes around 5 minutes to execute, retrieve and insert into the DE. So, armed with that knowledge, we can gather that 5 runs (10,000) would put us at around 25 minutes. Giving us a buffer of 5 minutes prior to timeout.

So now that we know the records per script we can bring in, we can figure out that you would need your automation to run this script 10 times to retrieve all 100,000 records.

Ok cool, now we run into how do we carry the current status over between scripts so that it doesn’t just keep collecting the first 5 pages of records each time? Pagination variable through a DE value. Basically you would have, at the end of your script, the iterative value (page to pull) pushed into a Data Extension. This is then what is pulled in at the beginning of the next script – continuing the pagination from where the previous script left off.

Now for our sample, let’s say that this will be run once a day only, so we want to make sure that if the day is different from the last record in the DE, that we want to start over from 1.

This would be the way to set the iterative (i) to the most recent run, or to 1 if it falls outside of the set parameters.

var lastRunDate = Platform.Function.Lookup(dePage,'Date',['AutoType'],['ContactDE']);
var today = new Date(now.getFullYear(),now.getMonth(),now.getDate())
if (lastRunDate) {
	lastRunDate = new Date(lastRunDate.getFullYear(),lastRunDate.getMonth(),lastRunDate.getDate())
	if (Stringify(lastRunDate) === Stringify(today)) {
		var i = Platform.Function.Lookup(dePage,'Page',['AutoType'],['ContactDE']);
	} else {
		var i = 1;
	}
} else {
	var i = 1;
}

ok. Now that we have an iterative that continues on, but are not doing a for loop, how do we control when the loop is done, so it doesn’t just continue on forever?

Simple solution I use is to check the ‘count’ returned (most APIs include a count of returned rows in their response) and if it is below the max value (in this case 2000) then that is the last call as there were not enough remaining items to fill the page. At that point you can end the loop and reset your iterative back to 1.

Example of a Do/While loop to verify count:

	do { //This is the count Do/While
		do { //This is the time based Do/While
		  var url = externalDomain + '/coolData/?$Page=' + i + '&$PageSize=2000';
		  var contentType = 'application/json'
		  var payload = ''
		  var res = HTTP.Post(url, 'application/json',payload,["Authorization"],[authToken]);
		  var response = Platform.Function.ParseJSON(res.Response[0]);
                   //Gathers the count from the response for verification
		  var count = response.count;
		  var items = response.items;
		  i++;
		} while((new Date().getTime() - start) < timeOut)
	} while(count == 2000) 
       //Takes the count set in the loop and compares it. If it does not match the page limit (2000) then it breaks the loop
	if (count < 2000 && count > 0) {
		i = 1
	} 
        //Resets i to 1 if loop is broken due to count

You now can take the script, add in your upsert to push the i value into the DE and you are all set to go.

What I recommend for a DE is to have 3 fields in it AutoType | Date | Page with AutoType being the primary key. I would use this as the Unique ID for each automation, this way you can reuse this DE for any other automations that may require this process. For example in ours above, I would have it be

AutoType  |  Date       |  Page
ContactDE | 11/10/2019  |  5

You then just add in an upsert at the bottom of your script to push the data into this DE after all has been completed.

	var rows = Platform.Function.UpsertData(dePage, ['AutoType'],['ContactDE'],['Page','Date'],[i,new Date()])

All of it would come together and look something like this: (using the 25 mins instead of the 20 as we showed up top)

var dePage = 'myAutoPageDE' //Name of DE that holds page info
var now = new Date()
var start = now.getTime();
var timeout = 1500000; //This is 25 minutes
var lastRunDate = Platform.Function.Lookup(dePage,'Date',['AutoType'],['ContactDE']);
var today = new Date(now.getFullYear(),now.getMonth(),now.getDate())
if (lastRunDate) {
	lastRunDate = new Date(lastRunDate.getFullYear(),lastRunDate.getMonth(),lastRunDate.getDate())
	if (Stringify(lastRunDate) === Stringify(today)) {
	    var i = Platform.Function.Lookup(dePage,'Page',['AutoType'],['ContactDE']);
	} else {
	    var i = 1;
	}
} else {
	var i = 1;
}
do { //This is the count Do/While
    do { //This is the time based Do/While
	var url = externalDomain + '/coolData/?$Page=' + i + '&$PageSize=2000';
	var contentType = 'application/json'
	var payload = ''
	var res = HTTP.Post(url, 'application/json',payload,["Authorization"],[authToken]);
	var response = Platform.Function.ParseJSON(res.Response[0]);
        //Gathers the count from the response for verification
	var count = response.count;
	var items = response.items;
	i++;
    } while((new Date().getTime() - start) < timeOut)
} while(count == 2000) 
//Takes the count set in the loop and compares it. If it does not match the page limit (2000) then it breaks the loop
if (count < 2000 && count > 0) {
	i = 1
} 
//Resets i to 1 if loop is broken due to count
var rows = Platform.Function.UpsertData(dePage, ['AutoType'],['ContactDE'],['Page','Date'],[i,new Date()]);