UPDATE: The imapsync project accepted a pull request I made to support Google’s .json files. This will make things a little bit simpler. The next release isn’t in the public git repo yet, but you can test it via the repo I used to make the pull request from. I will update this tutorial using the .json files once it’s in the public repo.
Back to your regularly scheduled programming…
To use imapsync to migrate a Google Apps domain (without requiring passwords for every account), you’ll need to setup OAUTH2 access.
OAUTH2 access to Gmail requires two main steps:
1.) Creating a service account (with a service account key) for the Google API in Google’s Cloud Console
2.) Giving that service account access to your Google Apps domain
And once those steps are completed, you just need the service account name and key to access any account on your domain via imapsync.
Creating a Service Account and Key
Go to https://console.cloud.google.com and login.
The site will look something like this:
Go to the “My Project” drop down and create a new project:
You can call it what you like. I’m calling it “Gmail Migration”, though Google will make a second cutesy id name for you no matter what you do (“sinuous-bedrock-132419”):
It will say “Creating Project” briefly in the lower left, and within a minute the new project should show up automatically:
On the drop down (which will probably be your project name), choose “Manage All Projects:
On the subsequent screen, select service accounts:
It may ask you to select a project. Choose the one we’ve created for the migration:
Finally, we’re on the screen where we can create the service account:
Create the service account. Make sure “furnish a private key” is checked, and select P12 for the key, NOT JSON. Additionally, enable “Domain-Wide Delegation” and hit “Create”:
It will automatically start the download for the .p12 file with an automatically generated name, along with a screen confirming it:
We’ve created our service account, “email@example.com”, as well as created our service account key. We need one last piece of information, and that is the oauth client id for the service account.
On the service accounts screen, select “view client id” on your new service account:
That will take us to the following screen with our client id:
Service Account Summary
We’ve created our service account! Here are the details for mine:
- Service Account name: firstname.lastname@example.org
- Service Account OAUTH ID: 116135892746269078114
- Service Account Keyfile: Gmail Migration-c4218ef51eab.p12
Authorizing Your Service Account for Gmail
Now, we need to give this service account access to your Google Apps Gmail domain.
Login to the Google Apps Admin Console, and choose security:
On the security screen click show more:
Open up the “Advanced” section, and choose “Manage API Client Access”.
Special note: If you have a “Manage OAuth domain key” as in my screenshot, ignore it. It’s not relevant to this tutorial. It’s an old OAuth method that is no longer supported by Google.
Finally, we’re where the magic happens. This is where we can give our service account access to our gmail domain.
Enter the Service Account Client ID in the “Client Name” field (“116135892746269078114” in this example).
Enter “https://mail.google.com/” in the “One or More API Scopes” field.
You should now see the client id in the list with Read/Write/Send privileges to Email:
We’re now ready to use our XOAUTH2 service account with imapsync!
Here is how you use your XOAUTH2 service account with imapsync. I’m operating under the assumption that you already know or can figure out how to use imapsync on your own. This is not a tutorial for imapsync.
- Put the .p12 file into the directory you’re going to run imapsync from
- Rename the file so there are no spaces. Google has using spaces by default in the .p12 files, so you almost definitely will need to do this, as imapsync’s XOAUTH2 support doesn’t parse filenames with spaces correctly. I’m the jerk that wrote the XOAUTH2 code, so I apologize profusely.
- Format your –passwordX argument as follows: –passwordX “
Here is an example imapsync session, using the –justlogin option to demonstrate the XOAUTH2 functionality:
[dev : 2:12PM] ls -alh total 16K drwxr-xr-x 3 mrgus mrgus 4.0K May 27 14:03 . drwxrwxrwt 16 root root 4.0K May 27 14:13 .. -rw-r--r-- 1 mrgus mrgus 2.5K May 27 14:01 gmailmigration.p12 [dev : 2:12PM] gmaildemotest>imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 email@example.com --password1 '116135892746269078114;gmailmigration.p12' --host2 imap.gmail.com --ssl2 -authmech2 xoauth2 --user2 firstname.lastname@example.org --password2 '116135892746269078114;gmailmigration.p12' Transfer started at Fri May 27 14:13:10 2016 PID is 21874 Log file is LOG_imapsyncemail@example.com ( to change it, use --logfile path ; or use --nolog to turn off logging ) $RCSfile: imapsync,v $ $Revision: 1.678 $ $Date: 2016/01/21 19:47:02 $ Here is a [linux] system (Linux shpadoinkle 3.2.0-91-generic #129-Ubuntu SMP Wed Sep 9 10:56:06 UTC 2015 x86_64) With perl 5.18.2 Mail::IMAPClient 3.35 Command line used: /home/testaccount/bin/imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 firstname.lastname@example.org --password1 MASKED --host2 imap.gmail.com --ssl2 -authmech2 xoauth2 --user2 email@example.com --password2 MASKED Temp directory is /tmp ( to change it use --tmpdir dirpath ) PID file is /tmp/imapsync.pid ( to change it use --pidfile filepath ; to avoid it use --pidfile "" ) Modules version list: Mail::IMAPClient 3.35 IO::Socket 1.36 IO::Socket::IP 0.25 IO::Socket::INET 1.33 IO::Socket::SSL 1.965 Net::SSLeay 1.58 Compress::Zlib 2.06 Digest::MD5 2.52 Digest::HMAC_MD5 1.01 Digest::HMAC_SHA1 1.03 Term::ReadKey 2.31 File::Spec 3.40 Time::HiRes 1.9725 Unicode::String 2.09 IO::Tee 0.64 File::Copy::Recursive 0.38 Authen::NTLM ? URI::Escape 3.31 Data::Uniqid 0.12 JSON 2.61 JSON::WebToken 0.07 Crypt::OpenSSL::RSA 0.28 LWP 6.05 HTML::Entities 3.69 Filesys::DfPortable ? ( use --no-modules_version to turn off printing this Perl modules list ) Use --sslargs1 SSL_verify_mode=1 for SSL_VERIFY_PEER on host1 Use --sslargs2 SSL_verify_mode=1 for SSL_VERIFY_PEER on host2 ssl debug mode level is --debugssl 1 (can be set from 0 meaning no debug to 4 meaning max debug) Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1. Host1: will try to use XOAUTH2 authentication on host1 Host2: will try to use XOAUTH2 authentication on host2 Host1: imap connexion timeout is 120 seconds Host2: imap connexion timeout is 120 seconds Host1: IMAP server [imap.gmail.com] port  user [firstname.lastname@example.org] Host2: IMAP server [imap.gmail.com] port  user [email@example.com] Host1: connecting and login on host1 [imap.gmail.com] port  with user [firstname.lastname@example.org] Use of uninitialized value $sockargs in join or string at /usr/share/perl5/Mail/IMAPClient.pm line 360. Host1 banner: * OK Gimap ready for requests from 2607:fa18:0:2::118 ad6mb151641860pac Host1: imap.gmail.com says it has CAPABILITY for AUTHENTICATE XOAUTH2 Host1: success login on [imap.gmail.com] with user [email@example.com] auth [XOAUTH2] Host2: connecting and login on host2 [imap.gmail.com] port  with user [firstname.lastname@example.org] Use of uninitialized value $sockargs in join or string at /usr/share/perl5/Mail/IMAPClient.pm line 360. Host2 banner: * OK Gimap ready for requests from 2607:fa18:0:2::118 t7mb317346986pag Host2: imap.gmail.com says it has CAPABILITY for AUTHENTICATE XOAUTH2 Host2: success login on [imap.gmail.com] with user [email@example.com] auth [XOAUTH2] Host1: state Authenticated Host2: state Authenticated Host1 capability: IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT APPENDLIMIT=35882577 LIST-EXTENDED LIST-STATUS Host2 capability: IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT APPENDLIMIT=35882577 LIST-EXTENDED LIST-STATUS Host2: found quota, presented in raw IMAP Sending: 4 GETQUOTAROOT INBOX Sent 22 bytes Read: * QUOTAROOT "INBOX" "" * QUOTA "" (STORAGE 1000731 15728640) 4 OK Success Host2: Quota current storage is 1024748544 bytes. Limit is 16106127360 bytes. So 6.36 % full