Creating Creating OAUTH2 Credentials With Google Apps

UPDATE: The imapsync project accepted a pull request I made to support Google’s .json files. This will make things a little bit simpler. The next release isn’t in the public git repo yet, but you can test it via the repo I used to make the pull request from. I will update this tutorial using the .json files once it’s in the public repo.

Back to your regularly scheduled programming…

To use imapsync to migrate a Google Apps domain (without requiring passwords for every account), you’ll need to setup OAUTH2 access.

OAUTH2 access to Gmail requires two main steps:

1.) Creating a service account (with a service account key) for the Google API in Google’s Cloud Console
2.) Giving that service account access to your Google Apps domain

And once those steps are completed, you just need the service account name and key to access any account on your domain via imapsync.

Creating a Service Account and Key

Go to https://console.cloud.google.com and login.

The site will look something like this:

Screen Shot 2016-05-27 at 12.07.03 PM

Go to the “My Project” drop down and create a new project:

Screen Shot 2016-05-27 at 12.08.39 PM

You can call it what you like. I’m calling it “Gmail Migration”, though Google will make a second cutesy id name for you no matter what you do (“sinuous-bedrock-132419”):

Screen Shot 2016-05-27 at 12.09.43 PM

It will say “Creating Project” briefly in the lower left, and within a minute the new project should show up automatically:

On the drop down (which will probably be your project name), choose “Manage All Projects:

Screen Shot 2016-05-27 at 12.12.43 PM

On the subsequent screen, select service accounts:

Screen Shot 2016-05-27 at 12.14.05 PM

It may ask you to select a project. Choose the one we’ve created for the migration:

Screen Shot 2016-05-27 at 12.14.46 PM

Finally, we’re on the screen where we can create the service account:

Screen Shot 2016-05-27 at 12.15.58 PM

Create the service account. Make sure “furnish a private key” is checked, and select P12 for the key, NOT JSON. Additionally, enable “Domain-Wide Delegation” and hit “Create”:

Screen Shot 2016-05-27 at 12.36.41 PM

It will automatically start the download for the .p12 file with an automatically generated name, along with a screen confirming it:

Screen Shot 2016-05-27 at 12.39.18 PM

We’ve created our service account, “gmailmigration@sinuous-bedrock-132419.iam.gserviceaccount.com”, as well as created our service account key. We need one last piece of information, and that is the oauth client id for the service account.

On the service accounts screen, select “view client id” on your new service account:

Screen Shot 2016-05-27 at 12.43.10 PM

That will take us to the following screen with our client id:

Screen Shot 2016-05-27 at 12.44.09 PM

Service Account Summary

We’ve created our service account! Here are the details for mine:

  • Service Account name: gmailmigration@sinuous-bedrock-132419.iam.gserviceaccount.com
  • Service Account OAUTH ID: 116135892746269078114
  • Service Account Keyfile: Gmail Migration-c4218ef51eab.p12

Authorizing Your Service Account for Gmail

Now, we need to give this service account access to your Google Apps Gmail domain.

Login to the Google Apps Admin Console, and choose security:

Screen Shot 2016-05-27 at 12.50.01 PM

On the security screen click show more:

Screen Shot 2016-05-27 at 12.50.52 PM

Open up the “Advanced” section, and choose “Manage API Client Access”.

Special note: If you have a “Manage OAuth domain key” as in my screenshot, ignore it. It’s not relevant to this tutorial. It’s an old OAuth method that is no longer supported by Google.

Screen Shot 2016-05-27 at 12.52.01 PM

Finally, we’re where the magic happens. This is where we can give our service account access to our gmail domain.

Enter the Service Account Client ID in the “Client Name” field (“116135892746269078114” in this example).

Enter “https://mail.google.com/” in the “One or More API Scopes” field.

Hit Authorize

Screen Shot 2016-05-27 at 12.56.47 PM

You should now see the client id in the list with Read/Write/Send privileges to Email:

Screen Shot 2016-05-27 at 12.57.17 PM

We’re now ready to use our XOAUTH2 service account with imapsync!

imapsync

Here is how you use your XOAUTH2 service account with imapsync. I’m operating under the assumption that you already know or can figure out how to use imapsync on your own. This is not a tutorial for imapsync.

  • Put the .p12 file into the directory you’re going to run imapsync from
  • Rename the file so there are no spaces. Google has using spaces by default in the .p12 files, so you almost definitely will need to do this, as imapsync’s XOAUTH2 support doesn’t parse filenames with spaces correctly. I’m the jerk that wrote the XOAUTH2 code, so I apologize profusely.
  • Format your –passwordX argument as follows: –passwordX “;“

Here is an example imapsync session, using the –justlogin option to demonstrate the XOAUTH2 functionality:

[dev : 2:12PM] ls -alh
total 16K
drwxr-xr-x  3 mrgus mrgus 4.0K May 27 14:03 .
drwxrwxrwt 16 root  root  4.0K May 27 14:13 ..
-rw-r--r--  1 mrgus mrgus 2.5K May 27 14:01 gmailmigration.p12
[dev : 2:12PM] gmaildemotest>imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 testaccount@notearthday.org --password1 '116135892746269078114;gmailmigration.p12' --host2 imap.gmail.com --ssl2 -authmech2 xoauth2 --user2 testaccount@notearthday.org --password2 '116135892746269078114;gmailmigration.p12'
Transfer started at Fri May 27 14:13:10 2016
PID is 21874
Log file is LOG_imapsync/2016_05_27_14_13_10_testaccount@notearthday.org.txt ( to change it, use --logfile path ; or use --nolog to turn off logging )
$RCSfile: imapsync,v $ $Revision: 1.678 $ $Date: 2016/01/21 19:47:02 $
Here is a [linux] system (Linux shpadoinkle 3.2.0-91-generic #129-Ubuntu SMP Wed Sep 9 10:56:06 UTC 2015 x86_64)
With perl 5.18.2 Mail::IMAPClient  3.35
Command line used:
/home/testaccount/bin/imapsync --justlogin --host1 imap.gmail.com --ssl1 --authmech1 xoauth2 --user1 testaccount@notearthday.org --password1 MASKED --host2 imap.gmail.com --ssl2 -authmech2 xoauth2 --user2 testaccount@notearthday.org --password2 MASKED
Temp directory is /tmp  ( to change it use --tmpdir dirpath )
PID file is /tmp/imapsync.pid ( to change it use --pidfile filepath ; to avoid it use --pidfile "" )
Modules version list:
Mail::IMAPClient     3.35
IO::Socket           1.36
IO::Socket::IP       0.25
IO::Socket::INET     1.33
IO::Socket::SSL      1.965
Net::SSLeay          1.58
Compress::Zlib       2.06
Digest::MD5          2.52
Digest::HMAC_MD5     1.01
Digest::HMAC_SHA1    1.03
Term::ReadKey        2.31
File::Spec           3.40
Time::HiRes          1.9725
Unicode::String      2.09
IO::Tee              0.64
File::Copy::Recursive 0.38
Authen::NTLM         ?
URI::Escape          3.31
Data::Uniqid         0.12
JSON                 2.61
JSON::WebToken       0.07
Crypt::OpenSSL::RSA  0.28
LWP                  6.05
HTML::Entities       3.69
Filesys::DfPortable  ?
( use --no-modules_version to turn off printing this Perl modules list )
Use --sslargs1 SSL_verify_mode=1 for SSL_VERIFY_PEER on host1
Use --sslargs2 SSL_verify_mode=1 for SSL_VERIFY_PEER on host2
ssl debug mode level is --debugssl 1 (can be set from 0 meaning no debug to 4 meaning max debug)
Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1.
Host1: will try to use XOAUTH2 authentication on host1
Host2: will try to use XOAUTH2 authentication on host2
Host1: imap connexion timeout is 120 seconds
Host2: imap connexion timeout is 120 seconds
Host1: IMAP server [imap.gmail.com] port [993] user [testaccount@notearthday.org]
Host2: IMAP server [imap.gmail.com] port [993] user [testaccount@notearthday.org]
Host1: connecting and login on host1 [imap.gmail.com] port [993] with user [testaccount@notearthday.org]
Use of uninitialized value $sockargs[13] in join or string at /usr/share/perl5/Mail/IMAPClient.pm line 360.
Host1 banner: * OK Gimap ready for requests from 2607:fa18:0:2::118 ad6mb151641860pac
Host1: imap.gmail.com says it has CAPABILITY for AUTHENTICATE XOAUTH2
Host1: success login on [imap.gmail.com] with user [testaccount@notearthday.org] auth [XOAUTH2]
Host2: connecting and login on host2 [imap.gmail.com] port [993] with user [testaccount@notearthday.org]
Use of uninitialized value $sockargs[9] in join or string at /usr/share/perl5/Mail/IMAPClient.pm line 360.
Host2 banner: * OK Gimap ready for requests from 2607:fa18:0:2::118 t7mb317346986pag
Host2: imap.gmail.com says it has CAPABILITY for AUTHENTICATE XOAUTH2
Host2: success login on [imap.gmail.com] with user [testaccount@notearthday.org] auth [XOAUTH2]
Host1: state Authenticated
Host2: state Authenticated
Host1 capability: IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT APPENDLIMIT=35882577 LIST-EXTENDED LIST-STATUS
Host2 capability: IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT APPENDLIMIT=35882577 LIST-EXTENDED LIST-STATUS

Host2: found quota, presented in raw IMAP
Sending: 4 GETQUOTAROOT INBOX
Sent 22 bytes
Read: 	* QUOTAROOT "INBOX" ""
  	* QUOTA "" (STORAGE 1000731 15728640)
  	4 OK Success

Host2: Quota current storage is 1024748544 bytes. Limit is 16106127360 bytes. So 6.36 % full

  • Posted on 27. May 2016
  • Written by mrgus
  • Categories: Email, Migration
Leave a comment

There are no comments yet.

Leave a Reply

Your email address will not be published. Required fields are marked *