Definitions: Due to overlap in OAuth 2 terminology and variations in our internal lexicon, the following definitions should be used to interpret this specific document. Reseller - A party which resells AML services to merchants. Also referred to as a Subscriber. Client - An application which performs actions through the AML External API. External API - A collection of web services that resellers can use to manage their AML accounts. OAuth Provider - An OAuth 2 provider service and associated user interfaces. OAuth 2 Specification - https://tools.ietf.org/html/rfc6749 Client ID - An OAuth Client ID, per the OAuth 2 specification. Client Secret - An OAuth client secret, per the OAuth 2 specification. User Management UI - A user interface for managing users of the OAuth Provider. Used by AML. Client ID Management UI - A user interface for managing Client IDs and tokens. Used by resellers. Realm - A reseller's SAPI realm, also the subdomain of said SAPI realm. External API Overall Security Architecture: The External API enables trusted third parties and resellers to perform actions on a reseller's SAPI stack programmatically. Requests to the External API are authenticated using a Spring Security based OAuth 2 implementation referred to in this document as the OAuth Provider. OAuth specific requests to the OAuth Provider are authenticated according to the RFC for OAuth 2. All other requests to the OAuth Provider are authenticated using HTTP Basic Authentication. Since neither HTTP Basic Authentication nor OAuth 2 are cryptography based protocols, the traffic to and from these services must be encrypted at all times using TLS. The OAuth provider is Spring Security based and provides a really basic UI to manage API users and their Client IDs. When a new API user is created, it also creates a SAPI user for each API user behind the scenes. This SAPI authentication information is stored in a database that is used by the External API to talk to the appropriate SAPI instance on behalf of the API user. This allows the SAPI UI to show changes made by the External API users as being performed by " API User" instead of the overall stack level admin user. The External API authenticates requests using OAuth 2 and checks tokens against the OAuth Provider to ensure that the granted realm for the token matches the requested subdomain. After successful authentication and realm authorization, during processing of the requests, the External API uses the stored SAPI credentials for the realm to perform requests against the targeted SAPI realm. Deployment Overview: OAuth Provider and External API are each Tomcat war deployments. They have fairly low resource requirements as they do not do much work and can probably be hosted on the same server, but it is advised that the production instance to have at least 4 cores(or 2 cores that support HT) for stability as serving one External API request involves concurrently serving a check_token request. -Databases There are two databases in use by the OAuth Provider and External API. oauth : accessible only to the OAuth Provider war. sapi_auth : accessible only to the OAuth Provider and External API wars. -Firewalls Both of these services should be firewalled off from the world and access should be opened up for our offices, our servers and whitelisted resellers' offices and servers only. It is anticipated that only larger clients would have the necessary development resources to make use of the External API, but our tech-ops and client services might make use of the External API for monitoring and reporting in the future. Additionally, the OAuth Provider and External API will need access to services on all SAPI stacks that they are intended to serve. This should occur through port 8443 as port 443 does not forward all the services(I assume) and acquiring an access token from SAPI involves a GET request where the username and password are query params. This also allows us to firewall the services separately from the reseller and other UIs. -Proxy The OAuth Provider UI and External API are both hosted on port 8080 internally, but it is intended to be used by our tech-ops and resellers' developer or tech-ops personnel only. It is desirable to proxy it to 8443 externally just for the purposes of encryption. The External API services are hosted on separate functionality specific context roots(merchants, taxonomies, deployments, etc.) so should be proxied in bulk from (.api.admaxlocal.com:8443/ to :8080/ ) to avoid reconfiguration upon release of new external apis. -DNS The OAuth Provider and External API both resolve the SAPI instance to target by pulling the lowest level subdomain from the request and prepending .admaxlocal.com. To avoid confusion with the SAPI instance that is .admaxlocal.com, all External API requests should be served by .api.admaxlocal.com rather than complex proxy configurations. Additionally, it is desirable for the OAuth Provider to respond on a specific subdomain (oauth.admaxlocal.com) as resellers will use this address to manage their Client Ids and tokens. Deployment Instructions: -Database Setup Use the AML Data Tool to create and seed the sapi_auth and oauth schemas. This will seed the 'admin' user with a password reset token of 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' but no password set. This means someone will have to go to http://oauth.admaxlocal.com/aml-oauth-provider/password.jsp?userId=1 and set a valid password for the new environment once the system is operational. Create or configure a file at: /usr/local/tsa/oauth-provider/conf/aml-oauth-provider-ds.properties Start by copying the file src/main/resources/aml-oauth-provider-ds.properties from the aml-oauth-provider repository. Change the host and port of the clientDetailsDs.url and the sapiAuthDs.url to match the same values used with the AML Data Tool. Additionally, set the username and password for clientDetailsDs and sapiAuthDs to match what was used for the AML Data Tool. -Mail Setup (Optional - default uses the same sendgrid account as SAPI) Create or configure a file at: /usr/local/tsa/oauth-provider/conf/aml-oauth-provider-mail.properties Start by copying the file src/main/resources/aml-oauth-provider-mail.properties from the aml-oauth-provider repository. Change the host, port, username and password to match the appropriate mail account for the environment(may be the same). -Tomcat Configuration The OAuth Provider and External API would best be run on a separate port from other services. This allows us to lock down the OAuth provider firewall more strictly than other services as only our offices, our servers, and specific reseller IP addresses need have access to the OAuth Provider or the External API. To configure a second set of ports for these services, please add this additional entry to the ${catalina_base}/conf/server.xml on the host machine right before the tag at the end of the file: The war files for both the OAuth Provider and External API should then be deployed to: ${catalina_base}/webapps_8082 -SAPI Configuration SAPI does not need any specific configuration itself, but the stacks which should be reachable through the External API need to have their ports open to the External API host as well as the OAuth Provider host. See the Proxy Configuration section for more details. -HTTPD Proxy Configuration (SAPI) All traffic between the External API and SAPI and between the resellers and the External API as well as the OAuthProvider must be encrypted at all times. This means all traffic should go through the HTTPD proxy layer or load balancer that terminates SSL. SAPI stacks should have port 8443 proxied through to 8080 to enable secure access to the SAPI services that are usually inaccessible external to our intranet. Port 8443 should then be firewalled off allowing only the External API and OAuth Provider host(s) access to this port. The OAuth Provider and External API host(s) should have only port 8443 open and only to specific IP addresses which should include our intranet external IP addresses as well as resellers' IPs who have been granted access to the External API. Update the /etc/httpd/conf/httpd.conf file on the SAPI host adding the following for production hosts(this step is not necessary for non SSL enabled environments like QA and dev): Listen 8443 ServerAdmin webmaster@dummy-host.example.com DocumentRoot /www/docs/dummy-host.example.com ServerName dummy-host.example.com ErrorLog logs/dummy-host.example.com-error_log CustomLog logs/dummy-host.example.com-access_log common ProxyPass / http://localhost:8080 ProxyPassReverse / http://localhost:8080 For dev and QA (or any non ssl enabled environment) this step is unnecessary. -HTTPD Proxy Configuration (OAuth Provider and AML External API Host) This configuration should be identical to the normal SAPI proxy configuration. ProxyPass / http://localhost:8080 ProxyPassReverse / http://localhost:8080 QA Instructions: In the QA environment we are using the same username and password as the rest of the system for the databases, this is to avoid confusion and reduce the time to deliver the feature. When you run AML Data Tool and configure the datasources, just use the normal spike:tar63t username and password and aml-db-primary port 3306. You will also have to create the two databases listed in the Deployment Overview:-Databases section of this document. You can create these using the schema.sql files in the following folders in bitbucket: aml-data-tool:src/main/resources/db/oauth aml-data-tool:src/main/resources/db/sapi_auth If you are still running standalone for sapi, each time you restart Tomcat you will have to erase any entries except the one for admin+tsa@thesearchagency.com in the table sapi_users in the schema sapi_auth as well as delete all users except admin in the OAuth User Management UI, then recreate the users for testing. This is because the SAPI auth information is created by the OAuth Provider in SAPI for each new account. This will not be a problem in production or sandbox because they both have a MySQL backed SAPI database. The OAuth User Management UI does not delete entries in sapi_auth in production because it reuses them. Suggested test cases: A user with ROLE_ADMIN can: Create, disable, and delete any user. A user must have at least one role and must have a realm assigned that actually is valid. Edit the roles of a user, a user must have at least one role When an Admin user disables a user, the user can no longer generate tokens. Access tokens that have already been generated can still be used until they expire. When an Admin user deletes a user, the user can no longer do anything and all of their access tokens can no longer be used. When an Admin user creates a user, the user's email receives an activation email with a link to a page which allows them to set their password. (This link is hard coded to be https as the user should never be able to set their password without encryption. If the QA environment does not have SSL enabled, you will need to remove the https and replace it will http) A user with ROLE_USER(but not ROLE_ADMIN) can: Create or delete Client IDs. When a Client ID is deleted, all of the access tokens for that Client ID can no longer be used. See their active Client IDs, refresh tokens, and access tokens in the UI. Generate a refresh token using a password_grant and their Client ID and client secret. Generate an access token using a refresh token and their Client ID and client secret. See the document I sent out earlier in the thread, or the OAuth2 specification for instructions to generate these tokens. A user with realm FOO can only access merchants or other objects that are created in the realm FOO. Likewise a user with realm BAR can only access merchants or other objects that are created in realm BAR. - Merchant API Anyone with a valid(non-expired) access token can make requests on behalf of the realm's reseller against the merchant API. If the scope of the particular access token includes read (scope is inherited from the Client ID by default, but may be narrowed per the OAuth 2 spec when generating tokens) then the user can list merchants, lookup merchants by id or merchant id. If the scope of the particular access token includes write then the user can create or update merchants. Only the merchants within the realm of the External API user should be listed or updated, and any created merchants should be created in the realm for the user. All successful requests to the Merchant API should be logged to the merchant-api-audit log file including the Client ID that was used for the request. Tech-Ops Instructions: In order for the External API to work for a stack, some firewall configurations are required for the target SAPI stack. Please ensure that any target stacks are reachable on port 8443 from the host(s) for the OAuth Provider and the External API. -Creating External API OAuth Users Access to the External API is controlled using the OAuth Provider's User Management UI. This UI can create and manage OAuth users. A user has access to a specific SAPI realm. This is governed by the Realm property of the user. This Realm property must match the subdomain for the SAPI realm that the user should have access to. For instance, if a user should be able to use the External API to manage foo.admaxlocal.com, the Realm property should be "foo". The Realm property is case insensitive, so "FOO", "Foo", "foo", and "foO" are all acceptable and equivalent. There are currently only two roles in the system which are ROLE_ADMIN and ROLE_USER. ROLE_ADMIN users have access to the AML User Management UI, this means they can create, delete, disable other users. Only AML internal Tech-Ops and technical leadership should be granted this role. By logging in as a user with ROLE_ADMIN granted, you will be shown a listing of users and their realm, roles, and status. The Roles, and enabled status are mutable for users, the username(email) and realm are not. The ROLE_ADMIN should only be granted after the user has been trained fully on how the system is to be managed. Impropper management may result in loss of access by resellers which may affect SLA compliance and could cause a financial loss to AML. Please use caution. It is advisable to have a new admin user created for each member of the AML team who needs access to the system. This allows us to know who has performed what operations in the system and when. This also allows us to discriminately disable access to the system for specific users if they leave the company or have their credentials compromised. ROLE_USER users have access only to the OAuth Client ID Management UI. This is the only role that External API users should ever be granted. -Managing Existing External API OAuth Users The following operations may be performed on OAuth users: Editing the Roles property - Multiple roles may be granted to the user. The current roles are only ROLE_USER and ROLE_ADMIN, but more roles may be created in the future to manage what scopes a user can create Client IDs for. For instance, it might be decided that we want to only grant Taxonomy API access to a given reseller. This could be implemented by creating ROLE_TAX_ONLY which would have a custom OAuth scope granted to prevent access to the Merchant API. ROLE_USER has access to two scopes when creating Client IDs which are 'read' and 'write'. These roles control the operations the tokens generated for a Client ID can perform against the External API. Enabling or Disabling a User - An enabled user may log into the OAuth Client ID Management UI. This means they can create and delete Client IDs and generate tokens for existing Client IDs. A disabled user may do none of these things, but access tokens they have already generated will continue to grant access until the tokens expire. Refresh tokens will not be disabled, but the user will not be able to use them to generate new access tokens. Deleting a user- If a user is deleted all of their tokens will cease to function immediately. This action cannot be undone. The associated reseller will lose all of their tokens permanently. The user can be recreated, but the External API user will have their password reset and will have to log back into the management interface and recreate all of their Client IDs and generate all new tokens for all of their apps in addition to distributing tokens on their systems. It is advisable to instead disable the user unless a stack is being decommissioned or a reseller had its login credentials to the AML OAuth Provider compromised and requested this change. Admin users cannot manage tokens for users. It is up to the External API user(reseller) to manage their own tokens. -Obtaining a log of all transactions for an API user A separate log file called merchant-api-audit contains all of the inbound requests to the Merchant API. Each API service has such a log file. To filter this log file for a specific reseller, run the following command after substituting the realm alias(subdomain) for the SAPI realm. awk '/{realm alias}/,/--------------------------------------/' merchant-api-audit To filter to just a single Client ID for a particular realm use the following command syntax. awk '/{realm alias}-{Client ID}/,/--------------------------------------/' merchant-api-audit This output will contain timestamps, full Client ID including realm alias, the path, HTTP method and the description string for the API method as well as specific information about the request itself. Reseller Instructions: -Account Activation Once access has been granted to the AML External API, a reseller is sent an email with a password activation link and a password activation key. The reseller user follows this link and enters the password activation key and creates a password. The password must have all of lowercase, uppercase, numbers and symbols and must be at least 12 characters long or must be at least 16 characters long have at least 2 of lowercase, uppercase, numbers and symbols. -Password Reset An AML Tech Ops person can generate an activation link to reset a user's password. The workflow for the reseller user is identical to the Account Activation flow. The old password for an account will cease to function when the activation link is generated. -Client ID Management An AML External API user can generate as many Client IDs as they wish, each being granted at least one scope of access but as many scopes of access as the user's role permits. At this time there are two scopes of access (read, write) and only one type of AML External API user. Upon activation of their account and successful setting of their password, a user can log into the Client ID Management UI. Once in the UI they can generate as many Client IDs as they require. Client IDs allow resellers to separately manage their credentials by application or however they see fit. This allows the user to control how much of their system might need to be reconfigured in the event a Client ID's refresh token and Client Secret were compromised. A Client ID can be deleted to invalidate all of its access and refresh tokens at any time. By deleting a Client ID all access granted by that Client IDs tokens is immediately revoked. You cannot directly manage access tokens, once they are generated, they are valid for one hour as long as the Client ID is still valid. -OAuth2 There are many resources on the internet about the OAuth2 protocol. There are also several different flows and implementations. The OAuth2 specification offers significant flexibility to implementers, which can cause some confusion. The AML External API implements access control using the following flow based on the password and refresh_token grant types defined in the specification. Step 1 - Obtain a Client ID and Client Secret This is performed by logging into the AML External API and using the form to generate a new Client ID. Once a Client ID is created, the user is presented with a Client Secret which is a long string of random alpha-numeric characters. The Client Secret is only presented one time for a Client ID and is not recoverable once the modal dialog presenting it is dismissed. The user must save this Client Secret to be used in generating refresh tokens and access tokens. Step 2 - Generate a Refresh Token A Refresh Token can be used along with the Client Secret to obtain and refresh access tokens. This is an alternative to distributing the username and password for the OAuth2 server on all systems which need access to the AML External API. Each Client ID is allowed up to 5 Refresh Tokens. If an additional refresh token is generated past the fifth, the oldest refresh token becomes invalid and all of its access tokens will expire immediately. This is performed by issueing a POST request to the following URL template replacing the {username} and {password} with the credentials used to access the OAuth2 Client ID Management UI. This request must also present a standard HTTP Basic Authorization header using the Client ID as the username and Client Secret as the password. https://oauth.admaxlocal.com/aml-oauth-provider/oauth/tokens?grant_type=password&username={username}&password={password} The response to a successful password grant request is an access token and refresh token, for example : { "access_token": "2da0240f-17df-4b66-8cb7-bd4a952ec843", "token_type": "bearer", "refresh_token": "ca7ab70e-1706-4ba9-b978-79bdad34f160", "expires_in": 3599, "scope": "read write" } Step 3 - Refresh your Access Tokens (repeat as necessary, but at least once per hour) Every Access Token expires one hour after it is generated. Refresh Tokens do not expire and may be used along with the Client ID and Client Secret to obtain new Access Tokens. This is performed by issueing a POST request to the following URL template replacing the {refresh_token} with the Refresh Token value generated in Step 2. This request must also present a standard HTTP Basic Authorization header using the Client ID as the username and Client Secret as the password. https://oauth.admaxlocal.com/aml-oauth-provider/oauth/tokens?grant_type=refresh&refresh_token={refresh_token} The response to a successful refresh_token grant request is an access token and the associated refresh token, for example : { "access_token": "74619ea6-7f10-11e5-8bcf-feff819cdc9f", "token_type": "bearer", "refresh_token": "ca7ab70e-1706-4ba9-b978-79bdad34f160", "expires_in": 3599, "scope": "read write" } For best performance, access tokens should be reused until they expire since each request for an access token takes additional processing time. Step 4 - Use your Access Token to interact with the AML External API Any request to which you are granted permission can be authenticated using any Access Token which has the necessary scope. Authentication is performed by adding the following header and value to your request replacing {access_token} with your access token. key value Authentication Bearer {access_token} Step 5 - Handle Access Token Expiry When the Access Token expires, it will have to be refreshed by repeating Step 3. This can be done preemptively if the client is keeping track of the expiration time. This can also be done by handling the timeout response by refreshing the token and retrying requests using the new access token. An External API request using an expired token will respond with the following if a token is no longer valid. { "error": "invalid_token", "error_description": "418ac767-5f75-4c99-818a-d98639271151" } If you want to check if a token is valid, you can make a POST request to the following URL. This request must also present a standard HTTP Basic Authorization header using the Client ID as the username and the Client Secret as the password for this request. https://oauth.admaxlocal.com/aml-oauth-provider/oauth/check_token?token={access_token} The response will look like the following if the token is valid. { "exp": {expiry time}, "user_name": "{username for oauth provider}", "scope": [ "read", "write" ], "authorities": [ "REALM_{realm_name}", "ROLE_USER" ], "aud": [ "aml-external-api" ], "client_id": "{clientId}" } The response will look like the following if the token is invalid. { "error": "invalid_token", "error_description": "Token was not recognised" } /* * This language is intended to start a conversation about how to communicate * responsibility boundaries to resellers. It should not be considered a finished * product. */ -Loss Mitigation The AML External API user holds full responsibility for managing their login credentials to the Client ID Management UI, and all Client IDs, Client Secrets, and Refresh and Acccess Tokens. AML Tech Ops is only responsible for granting or revoking access to the Client ID Management UI itself. In the event that a reseller believes their credentials may be compromised for the Client ID Management UI, they should submit an urgent support request to AML Tech Ops immediately. AML Tech Ops can disable the user and issue a password reset email for the reseller to reset their password. This will result in the access tokens which are currently active remaining active. Additionally any refresh tokens and client secrets will continue to function to refresh access tokens. Once the credentials to the Client ID Management UI have been secured again, the reseller can remove any Client IDs which were generated by third parties. In the event that a reseller believes an unauthorized party ever may have had access to the Client Secret for a Client ID, the Client ID should be deleted. It may be recreated with the same name, but by deleting the Client ID, all Refresh and Access Tokens will be revoked immediately. A full audit of External API actions should also be performed for any such Client IDs. AML Tech Ops can retrieve a log of all actions performed by a Client ID at the request of the reseller. The reseller is responsible for assessing the damage caused by any such breach and determining if action is required to repair such damage. The reseller is also responsible for executing any such repairs.