Download Paging

Overview

The following psuedocode describes a typical approach for implementing paging with our download API. At a high level, this builds a list of downloads that are monitored until they complete. As each download completes, you will get the maximum Id from the result set and then request the download again starting at that RowId + 1. Once you get 0 lines in your result set, you will have all the rows for that type.

Data Structures

pending_queue => list of all in-process downloads along with status

Methods

run =>
    start_download "branding-appointments"
    start_download "branding-contacts"
    ... start any other downloads to want to page through
    
    while pending_queue is not empty
        poll_download with status-token
        if complete
            process_download
        end
    end
end

start_download( kind ) =>
    send startdownload request with RowId = 0 and kind
    add kind, status-token, RowId to pending_queue
end

poll_download =>
    request download status
    if complete
        download file
        return complete
    end
    return not complete
end

process_download =>
    open-downloaded-file
    skip-headers
    line_count = 0;
    foreach line in file
        read_column "Id"
        increment line_count
    end
    if line_count > 0
        continue_download with Id
    else
        finish_download
    end
end

continue_download =>
    send startdownload request with RowId = Id + 1 and kind
    update queue entry with RowId and status-token
end

finish_download =>
    remove from pending_queue
    delete contents from table and load all downloaded files for the kind
end