How to use the Zapier Platform and Zapier Transfer to migrate data

Like many growing businesses, we use a combination of homespun systems and commercial cloud services at Zapier, and are constantly looking for ways to be more efficient. This often means a lot of data migrations and a need to make them run as streamlined as possible.

If you’re like us, you’re always looking for examples of how other teams get the job done. We’ll walk you through a recent data migration project where we moved a subset of issues from GitHub and our own bug tracker software to Jira Software Cloud. 

We’ll share some background on our data migration decision-making process. Then we’ll go deep into how we used the Zapier Platform development environment coupled with Transfer by Zapier to make it happen. Experience with Zapier, JavaScript development, and the pain and suffering of doing data migrations is a plus if you want to follow along.

Buy or build?

The scope of each data migration helps us make the decision on whether we should look for help or make our own go of it. Migrating our support email and documentation to Zendesk? The scope involved millions of transactions and affected almost everyone at Zapier (and our customers). We went directly to buy, using Zendesk Professional Services to help.

Moving 600+ issues from GitHub and our own bug tracker software to Jira? The scope of that project was much smaller, so we decided to go the build route and do that one ourselves.

Does build just mean CSV export/import?

Today, most software enables some sort of export/import of CSV data. For many small migrations, this might be enough. As we experimented with CSV export/import though, we ran into some issues:

1. Non-text-based information, like images and attachments, weren’t included in our exports.

2. Fields with multiple values were problematic. For example, how could we build an export where we could have an indeterminate number of columns? How could we import this same CSV where we needed to match those same columns?

3. Many field formats were different between export and import—I guess that’s why there’s a T in ETL. For simple field transforms, this meant a run through Google Sheets and a formula. But for others, we couldn’t see an easy way to transform the data.

4. The CSV import was brittle. Test imports would work, then we’d add another field to the CSV and would have to reconstruct the whole import.

5. Our internally-developed bug tracking software doesn’t support CSV export.

Going with build +

Based on this, we decided we needed to extend what “build” meant for our data migration project. We’d also just released the beta version of Transfer by Zapier. 

This looked really promising, but the integrations we needed for our project (GitHub, Jira, and our own bug tracker software) weren’t supported yet. But we did have some documentation on how Zapier integration triggers could be extended to support Transfer and realized we could also use it for private integrations. 

A private integration runs through Zapier but is only available to the developer and those they invite to use it. We didn’t need a Zapier team or a Zapier partner to add a Transfer-enabled trigger to their integrations—we could roll our own! 

These realizations helped solidify the scope of our project.

Now we ended up with two distinct data migrations: 

Migrating GitHub issues to Jira Software Cloud bugs

Here are the steps we needed to take for the first migration:

1. Do what we can with a CSV export of GitHub issues.

2. Enhance CSV data with some Google Sheets formulas.

3. Develop our own Transfer-enabled GitHub trigger that processes GitHub issues and their linked comments.

4. Create a Transfer that:

   1. uses a new trigger to import GitHub issues,

   2. writes out Issue Comments using Google Sheets’ Create Row action.

5. Add this new Comments column to our CSV.

6. Import the CSV into Jira.

Migrating Zapier bug tracker software bugs to Jira Software Cloud bugs

Here are the steps we needed to take for the second migration:

1. Update our own internal integration to enhance a trigger to support Transfer.

2. Create a Zap that:

   1. Uses that new trigger to import bugs from Zapier’s bug tracker application,

   2. transforms fields using Zapier’s built-in Formatter tool,

   3. creates a new issue in Jira, and

   4. closes bugs in our bug tracker application.

3. Use Transfer by Zapier’s Transfer Existing Data option to run this more complex Zap through Transfer.

We’ll focus first on the development side and how we added Transfer support to our private integrations.

Building our own GitHub integration

Zapier does have an existing GitHub integration, but extending it to support Transfer for our data migration project looked pretty daunting. We decided it was best to create our own, simpler private integration that just had one trigger—New Issues—then use that to migrate our issue comments. We also just happened to use GitHub for our example in building integrations.

That way, we knew if we got stuck, we’d have some backup.

As the Zapier example was done using the Zapier Platform CLI, we also decided to use that environment for our development. Looking at the example code, there were six changes we needed to make:

1. Change the authentication to OAUTH.

2. Update the code to use async/await instead of promise/then (just because).

3. Update the input to include a labels field to help filter requests.

4. Update the noun for the trigger.

5. Add pagination to the trigger definition and trigger code.

6. Add code to get all the comments associated with each issue.

We downloaded the example code, registered it with zapier register and started the work. 

You can follow along with this documentation for how we accomplished step 1.

Steps 2 and 3 resulted in an updated issue.js source file with async/await, and the new input field “labels”. By adding this field we could label all of the issues we wanted to import into GitHub with “import” and minimize the number of records we’d need to process in our Transfer.

Expand to see the code we used

For Step 4, we needed to update the noun associated with the trigger so that it’s easily recognized in the Transfer UI. The noun is also used in the Zapier editor when a user gets a sample record, so we wanted it to work in both places. 

We added Transfer enabled to make it easy to spot:

module.exports = {

  key: "issue",

  noun: "Issue (Transfer enabled)",

 ….

}

For Step 5, we first added the canPaginate: true to our trigger definition:  

….

 ],

    perform: triggerIssue,

    canPaginate: true,

    sample: sample

  }

};

Our next change was to delineate that Transfer is calling our code and set a pagination variable (pageSize) determining how many records we want per request. 

Since we were updating the original code—and we might want to use this integration in the future—we decided to set this variable for each state our trigger code is called from. These calling states are the Zap Editor, a running Zap, and via Transfer. 

These calling states can be discovered using bundle.meta object properties. isBulkRead is the one we specifically need, as it’s set to true for Transfer. Setting the exact pagination page size per each request type required some testing. 

GitHub defaults to returning 100 records per request from their issue API endpoint. We started with that and found that when adding a comment endpoint request to each issue request, 100 records took too long to process. 

We have 30 seconds, but 30 seconds is a long time to look at a progress circle in a browser. Instead, we went with 50 records, which would usually return in about 20 seconds. We also set the “regular” polling size to 30, as that also has 30 seconds to work. It’s only looking for new items and Zapier polls every 5 to 15 minutes, so 30 records seemed like a good fit for that request cycle.  

Finally, we added a check for isLoadingSample, as we know the Zapier Editor only displays three new records per request—no reason for us to grab 30.

Expand to view the resulting code

The next change we made was updating the query parameters we send in the request to support this pagination. We sent our new pageSize in the per_page parameter and also a new page property, as defined by GitHub’s documentation.  

Zapier keeps track of this property, updating and incrementing it for each request and storing it in the bundle.meta.page object. GitHub starts their pagination at 1, so we needed to add 1 to this, as bundle.meta.page starts at 0.

…

 params: {

      filter: bundle.inputData.filter,

      state: bundle.inputData.state,

      sort: 'updated',

      direction: 'desc'

      per_page: pageSize,

      page: bundle.meta.page + 1,

    }

…

Now we needed to get the comments for each issue, so we created a function for that using a URL we get from the issue endpoint. GitHub returns an array of issue comment objects from this comments endpoint.

const getComments = async (z, bundle, url) => {

  const response = await z.request({

    url: url,

    method: "GET",

    headers: {

      Accept: "application/json",

      Authorization: `token ${bundle.authData.access_token}`,

    },

  });

  return response.data;

};

Finally, we needed to use that array and build a nicely formatted string that includes the comments URL, followed by each comment and some — for delineation.

const issuesArray = response.data;

for (const issue of issuesArray) {

    commentsArray = await getComments(z, bundle, issue.comments_url);

    issue.comment_body = issue.comments_url;

    commentsArray.map((comment) => {

      issue.comment_body = issue.comment_body + "n-----n" + comment.body;

    });

  }

  return issuesArray;

Expand to see what the new issues.js looked like

Or you can download it here.

Once our code passed zapier validate, we used zapier push and uploaded our new Transfer-enabled code to Zapier.

Zapier’s bug tracker application

Zapier developed its own bug tracking application about five years ago to leverage some of our other internal services. We made sure it had an API, and with that created our own integration so we could automate tasks associated with bug reporting. 

As part of our data migration project, we needed to move a set of issues out of our bug tracker and into Jira issues. Given the lack of a CSV export, we focused on the integration and Transfer. 

Unfortunately, we didn’t have a lot of documentation to go on, so we had to do our own digging in the integration. We noticed it supported a New Issue trigger, which looked like a good candidate to update for Transfer. 

“New” triggers tend to be the best triggers for Transfer use, as they usually request a set of records, sorted by date. To support Transfer, we just needed to paginate that request. When we examined it, though, we realized it was a webhook trigger.

This didn’t mean we couldn’t use it, but we would have to rely on the performList, and verify if we could extend that beyond just retrieving samples.

Fortunately, the performList was already in code mode, and the code looked promising, as it was using a page size.

Expand to see the code

After this review and some positive thinking, we landed on these tasks to update this trigger for Transfer:

1. Update the noun for the trigger. 

2. Add pagination to the trigger definition and trigger code.

For Step 1, the Zapier Platform UI provides a noun field, so we used the same language as our GitHub trigger:

For Step 2, much like our earlier work with our GitHub application, we needed to add pagination to the trigger definition and trigger code. In the Zapier Platform UI, you usually add pagination support in the UI for the trigger. But since this is a webhook trigger, this was on by default.

That meant we just needed to update our request code. Much like our decisions in GitHub, we needed a new variable for pageSize. In this case, however, we knew this trigger would never make a request from a running Zap, as it’s a REST Hook. So we ended up with 3 records for our sample, and, based on testing, 100 records for each Transfer request.

We passed this new value as our page_size parameter, and again used the bundle.meta.page to choose which page to request (again starting at 1). We also added some new inputData fields to help us filter the available records via type and tags in our request. 

Expand to see the resulting code

How we used Transfer

GitHub Issues migration

Now that we had our new Transfer ready triggers, it was time to use them. Remember for our GitHub issues migration we needed to create a transfer that:

• used a new trigger to import GitHub issues, and

• wrote out Issue Comments using a Google Sheets Create Row action.

Here are a couple of specifics to note for our data migration. 

Our source app dropdown is larger than the default, as it includes our new private GitHub Issues integration, other dev apps that we’ve worked on, and finally the Zapier certified Transfer apps. 

Our new trigger’s noun shows when we choose that as the source data. Once we selected that, we then chose Google Sheets as our destination to create new rows for our issue comments.

Once we authenticated into GitHub, we could then configure it for the transfer. We selected our repo and then allowed the defaults to be used for the other input fields.

As noted in the Transfer guide, you can preview your data before transferring. Our main concern was that the Comments Body field for each issue would be formatted correctly, with the Comment URL available if someone needed to look at the original thread. Fortunately, it looked just right.

The Transfer was successful, and our result was a two-column Google Sheet that we could merge in with our other already-constructed CSV to create a meta CSV for our import to Jira. Data Migration item number one done!

Zapier bug tracker application issue migration

Just a reminder, here was our workflow for this part of the project:

1. Create a Zap that:

   1. uses our new trigger to import bugs from Zapier’s bug tracker application,

   2. transforms fields using Formatter by Zapier,

   3. creates a new issue in Jira, and

   4. closes bugs in our bug tracker application.

2. Use Transfer by Zapier’s Transfer Existing Data option to run this more complex Zap through Transfer.

Fortunately, we were able to leverage our skills at building Zap workflows, then allow Transfer to use our Zap for our legacy data.

Here’s the outline for our Zap:

Step 1 is the trigger we modified for Transfer support. Notice is that it doesn’t look any different from a normal Zap trigger, but we know that when invoked from Transfer it will paginate and request all records, not just new records.

Step 2 uses an action from our bug tracker integration to look up all the users associated with the record.

Steps 3 and 4 transform the data from our bug tracker app into usable fields for Jira using formatter steps. This is a really powerful feature of using a Zap for a Transfer, where we can leverage built-in Zapier tools like Formatter to modify our data for the import process. 

There’s no limit to the number of steps we could add here to make sure the data is in the correct format for what we want to see in Jira.

Step 5 creates a Jira issue with our newly transformed record data.

Step 6 also cleans up the original data source. In this case, we added a comment to the issue so other users can know the issue has been migrated.

Using a Zap for Transfer gives you one other advantage—you can test it! Running a sample through all of your Zap steps or actually running the Zap and creating a new issue (in our case) lets us validate the end-to-end Transfer process before we do it with our real data. This is very important when we’re not just writing to a CSV for later import but the actual destination application.

Once we validated our Zap would work as advertised, we just needed to invoke it with Transfer. This is done on the Zaps page using the Transfer Existing Data menu available for our Zap.

As we’d already configured our Zap steps, this took us directly into Transfer, and showed us the records from the trigger that will be transferred. 

In this case, however, when we selected all, we knew it would also run through our other steps to transform the record before it’s created in Jira. 

With our records chosen, we pressed Next to get to our final confirmation screen, where we confirmed the count of the records we wanted to transfer. We noted our Source and Destination were the same, as our trigger read the records, and our final action marked them as closed.

Send Data started our transfer and executed our Zap for each found record. We confirmed the results in Zap History, as well as Jira and our bug tracker and then did a little jig – Data Migration item number two was finished! 

Tell us about your data migration projects

Every data migration project is a journey. Hopefully our steps using the Zapier Platform and Zapier Transfer inspire your next project. Let us know in the Zapier Community.