Colm O hEigeartaigh

Securing an Apache Kafka broker - part IV

Colm O hEigeartaigh - Wed, 09/28/2016 - 13:24
This is the fourth in a series of articles on securing an Apache Kafka broker. The first post looked at how to secure messages and authenticate clients using SSL. The second post built on the first post by showing how to perform authorization using some custom logic. The third post showed how Apache Ranger could be used instead to create and enforce authorization policies for Apache Kafka. In this post we will look at an alternative authorization solution called Apache Sentry.

1) Build the Apache Sentry distribution

First we will build and install the Apache Sentry distribution. Download Apache Sentry (1.7.0 was used for the purposes of this tutorial). Verify that the signature is valid and that the message digests match. Now extract and build the source and copy the distribution to a location where you wish to install it:
  • tar zxvf apache-sentry-1.7.0-src.tar.gz
  • cd apache-sentry-1.7.0-src
  • mvn clean install -DskipTests
  • cp -r sentry-dist/target/apache-sentry-1.7.0-bin ${sentry.home}
Apache Sentry has an authorization plugin for Apache Kafka, amongst other big data projects. In addition it comes with an RPC service which stores authorization privileges in a database. For the purposes of this tutorial we will just configure the authorization privileges in a configuration file locally to the broker. Therefore we don't need to do any further configuration to the distribution at this point.

2) Configure authorization in the broker

Configure Apache Kafka as per the first tutorial. To enable authorization using Apache Sentry we also need to follow these steps. First edit 'config/server.properties' and add:
  • authorizer.class.name=org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer
  • sentry.kafka.site.url=file:./config/sentry-site.xml
Next copy the jars from the "lib" directory of the Sentry distribution to the Kafka "libs" directory. Then create a new file in the config directory called "sentry-site.xml" with the following content:

This is the configuration file for the Sentry plugin for Kafka. It essentially says that the authorization privileges are stored in a local file, and that the groups for authenticated users should be retrieved from this file. Finally, we need to specify the authorization privileges. Create a new file in the config directory called "sentry.ini" with the following content:

This configuration file contains three separate sections. The "[users]" section maps the authenticated principals to local groups. The "[groups]" section maps the groups to roles, and the "[roles]" section lists the actual privileges. Now we can start the broker as in the first tutorial:
  • bin/kafka-server-start.sh config/server.properties 
3) Test authorization

Now lets test the authorization logic. Start the producer:
  • bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties
Send a few messages to check that the producer is authorized correctly. Now start the consumer:
  • bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer
If everything is configured correctly then it should work as in the first tutorial. 
Categories: Colm O hEigeartaigh

Securing an Apache Kafka broker - part III

Colm O hEigeartaigh - Mon, 09/26/2016 - 18:13
This is the third in a series of blog posts about securing Apache Kafka. The first post looked at how to secure messages and authenticate clients using SSL. The second post built on the first post by showing how to perform authorization using some custom logic. However, this approach is not recommended for non-trivial deployments. In this post we will show at how we can create flexible authorization policies for Apache Kafka using the Apache Ranger admin UI. Then we will show how to enforce these policies at the broker.

1) Install the Apache Ranger Kafka plugin

The first step is to download Apache Ranger (0.6.1-incubating was used in this post). Verify that the signature is valid and that the message digests match. Now extract and build the source, and copy the resulting plugin to a location where you will configure and install it:
  • tar zxvf apache-ranger-incubating-0.6.1.tar.gz
  • cd apache-ranger-incubating-0.6.1
  • mvn clean package assembly:assembly -DskipTests
  • tar zxvf target/ranger-0.6.1-kafka-plugin.tar.gz
  • mv ranger-0.6.1-kafka-plugin ${ranger.kafka.home}
Now go to ${ranger.kafka.home} and edit "install.properties". You need to specify the following properties:
  • COMPONENT_INSTALL_DIR_NAME: The location of your Kafka installation
  • POLICY_MGR_URL: Set this to "http://localhost:6080"
  • REPOSITORY_NAME: Set this to "KafkaTest".
Save "install.properties" and install the plugin as root via "sudo ./enable-kafka-plugin.sh". The Apache Ranger Kafka plugin should now be successfully installed (although not yet configured properly) in the broker.

2) Configure authorization in the broker

Configure Apache Kafka as per the first tutorial. There are a number of steps we need to follow to configure the Ranger Kafka plugin before it is operational:
  • Edit 'config/server.properties' and add the following: authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer
  • Add the Kafka "config" directory to the classpath, so that we can pick up the Ranger configuration files: export CLASSPATH=$KAFKA_HOME/config
  • Copy the Apache Commons Logging jar into $KAFKA_HOME/libs. 
  • The ranger plugin will try to store policies by default in "/etc/ranger/KafkaTest/policycache". As we installed the plugin as "root" make sure that this directory is accessible to the user that is running the broker.
Now we can start the broker as in the first tutorial:
  • bin/kafka-server-start.sh config/server.properties
3) Configure authorization policies in the Apache Ranger Admin UI 

At this point we should have configured the broker so that the Apache Ranger plugin is used to communicate with the Apache Ranger admin service to download authorization policies. So we need to install and configure the Apache Ranger admin service. Please refer to this blog post for how to do this. Assuming the admin service is already installed, start it via "sudo ranger-admin start". Open a browser and log on to "localhost:6080" with the credentials "admin/admin".

First lets add some new users that match the SSL principals we have created in the first tutorial. Click on "Settings" and "Users/Groups". Add new users for the principals:
  • CN=Client,O=Apache,L=Dublin,ST=Leinster,C=IE
  • CN=Service,O=Apache,L=Dublin,ST=Leinster,C=IE
  • CN=Broker,O=Apache,L=Dublin,ST=Leinster,C=IE
Now go back to the Service Manager screen and click on the "+" button next to "KAFKA". Create a new service called "KafkaTest". Click "Test Connection" to make sure it can communicate with the Apache Kafka broker. Then click "add" to save the new service. Click on the new service. There should be an "admin" policy already created. Edit the policy and give the "broker" principal above the rights to perform any operation and save the policy. Now create a new policy called "TestPolicy" for the topic "test". Give the service principal the rights to "Consume, Describe and Publish". Give the client principal the rights to "Consum and Describe" only.


4) Test authorization

Now lets test the authorization logic. Bear in mind that by default the Kafka plugin reloads policies from the admin service every 30 seconds, so you may need to wait that long or to restart the broker to download the newly created policies. Start the producer:
  • bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties
Send a few messages to check that the producer is authorized correctly. Now start the consumer:
  • bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer
If everything is configured correctly then it should work as in the first tutorial.
Categories: Colm O hEigeartaigh

Integrating Apache Camel with Apache Syncope - part III

Colm O hEigeartaigh - Fri, 09/23/2016 - 18:00
This is the third in a series of blog posts about integrating Apache Camel with Apache Syncope. The first post introduced the new Apache Camel provisioning manager that is available in Apache Syncope 2.0.0, and gave an example of how we can modify the default behaviour to send an email to an administrator when a user was created. The second post showed how an administrator can keep track of user password changes for auditing purposes. In this post we will show how to integrate Syncope with Apache ActiveMQ using Camel.

1) The use-case

The use-case is that Apache Syncope is used for Identity Management in a large organisation. When users are created we would like to be able to gather certain information about the new users and process it dynamically in some way. In particular, we are interested in the age of the new users and the country in which they are based. Perhaps at the reception desk of the company HQ we display a map with the number of employees in each country highlighted. To decouple whatever applications are processing the data from Syncope itself, we will use a messaging solution, namely Apache ActiveMQ. When new users are created, we will modify the default Camel route to send a message to two topics corresponding to the age and location of the user.

2) Download and configure Apache ActiveMQ

The first step is to download Apache ActiveMQ (currently 5.14.0). Unzip it and start it via:
  • bin/activemq start 
Now go to the web interface of ActiveMQ - 'http://localhost:8161/admin/', logging in with credentials 'admin/admin'. Click on the "Queues" tab and create two new queues called 'age' and 'country'.

3) Download and configure Apache Syncope

Download and extract the standalone version of Apache Syncope 2.0.0. Before we start it we will copy the jars we need to get Camel working with ActiveMQ in Syncope. In the "webapps/syncope/WEB-INF/lib" directory of the Apache Tomcat instance bundled with Syncope, copy the following jars:
  • From $ACTIVEMQ_HOME/lib: activemq-client-5.14.0.jar + activemq-spring-5.14.0.jar + hawtbuf-1.11.jar + geronimo-j2ee-management_1.1_spec-1.0.1.jar
  • From $ACTIVEMQ_HOME/lib/camel: activemq-camel-5.14.0.jar + camel-jms-2.16.3.jar
  • From $ACTIVEMQ_HOME/lib/optional: activemq-pool-5.14.0.jar + activemq-jms-pool-5.14.0.jar + spring-jms-4.1.9.RELEASE.jar
Next we need to create a Camel spring configuration file containing a bean with the address of the broker. Add the following file to the Tomcat lib directory (called "camelRoutesContext.xml"):

Now we can start the embedded Apache Tomcat instance. Open a browser and navigate to 'http://localhost:9080/syncope-console' logging in with 'admin/password'. The first thing we need to do is to configure user attributes for "age" and "country". Go to "Configuration/Types" in the left-hand menu, and click on the "Schemas" tab. Create two plain (mandatory) schema types: "age" of type "String" and "country" of type "Long". Now click on the "AnyTypeClasses" tab and create a new AnyTypeClass selecting the two plain schema types we just created. Finally, click on the "AnyType" tab and edit the "USER". Add the new AnyTypeClass you created and hit "save".

Now we will modify the Camel route invoked when a user is created. Click on "Extensions/Camel Routes" in the left-hand configuration menu. Edit the "createUser" route and add the following above the "bean method" part:
  • <setBody><simple>${body.plainAttrMap[age].values[0]}</simple></setBody>
  • <to uri="activemq:age"/>
  • <setBody><simple>${exchangeProperty.actual.plainAttrMap[country].values[0]}</simple></setBody>
  • <to uri="activemq:country"/>
This should be fairly straightforward to follow. We are setting the message body to be the age of the newly created User, and dispatching that message to the "age" queue. We then follow the same process for the "country". We also need to change "body" in the "bean method" line to "exchangeProperty.actual", this is because we have redefined what the body is for each of the Camel routes above.


Now let's create some new users. Click on the "Realms" menu and select the "USER" tab. Create new users "alice" in country "usa" of age "25" and "bob" in country "canada" of age "27". Now let's look at the ActiveMQ console again. We should see two new messages in each of the queues as follows, ready to be consumed:



Categories: Colm O hEigeartaigh

Using SHA-512 with Apache CXF SOAP web services

Colm O hEigeartaigh - Thu, 09/22/2016 - 15:49
XML Signature is used extensively in SOAP web services to guarantee message integrity, non-repudiation, as well as client authentication via PKI. A digest algorithm crops up in XML Signature both as part of the Signature Method (rsa-sha1 for example), as well as in the digests of the data that are signed. As recent weaknesses have emerged with the use of SHA-1, it makes sense to use the SHA-2 digest algorithm instead. In this post we will look how to configure Apache CXF to use SHA-512 (i.e. SHA-2 with 512 bits) as the digest algorithm.

1) Configuring the STS to use SHA-512

Apache CXF ships with a SecurityTokenService (STS) that is widely deployed. The principal function of the STS is to issue signed SAML tokens, although it supports a wide range of other functionalities and token types. The STS (for more recent versions of CXF) uses RSA-SHA256 for the signature method when signing SAML tokens, and uses SHA-256 for the digest algorithm. In this section we'll look at how to configure the STS to use SHA-512 instead.

You can specify signature and digest algorithms via the SignatureProperties class in the STS. To specify SHA-512 for signature and digest algorithms for generated tokens in the STS add the following bean to your spring configuration:

Next you need to reference this bean in the StaticSTSProperties bean for your STS:
  • <property name="signatureProperties" ref="sigProps" />
2) Configuring WS-SecurityPolicy to use SHA-512

Service requests are typically secured at a message level using WS-SecurityPolicy. It is possibly to specify the algorithms used to secure the request, as well as the key sizes, by configuring an AlgorithmSuite policy. Unfortunately the last WS-SecurityPolicy spec is quite dated at this point, and lacks support for more modern algorithms as part of the default AlgorithmSuite policies that are defined in the spec. The spec only supports using RSA-SHA1 for signature, and only SHA-1 and SHA-256 for digest algorithms.

Luckily, Apache CXF users can avail of a few different ways to use stronger algorithms with web service requests. In CXF there is a JAX-WS property called 'ws-security.asymmetric.signature.algorithm' for AsymmetricBinding policies (similarly 'ws-security.symmetric.signature.algorithm' for SymmetricBinding policies). This overrides the default signature algorithm of the policy. So for example, to switch to use RSA-SHA512 instead of RSA-SHA1 simply set the following property on your client/endpoint:
  • <entry key="ws-security.asymmetric.signature.algorithm" value="http://www.w3.org/2001/04/xmldsig-more#rsa-sha512"/>
There is no corresponding property to explicitly configure the digest algorithm, as the default AlgorithmSuite policies already support SHA-256 (although one could be added if there was enough demand). If you really need to support SHA-512 here, an option is to use a custom AlgorithmSuite (which will obviously not be portable), or to override one of the existing ones.

It's pretty straightforward to do this. First you need to create an AlgorithmSuiteLoader implementation to handle the policy. Here is one used in the tests that creates a custom AlgorithmSuite policy called 'Basic128RsaSha512', which extends the 'Basic128' policy to use RSA-SHA512 for the signature method, and SHA-512 for the digest method. This AlgorithmSuiteLoader can be referenced in Spring via:


The policy in question looks like:
  • <cxf:Basic128RsaSha512 xmlns:cxf="http://cxf.apache.org/custom/security-policy"/>
Categories: Colm O hEigeartaigh

Invoking on the Talend ESB STS using SoapUI

Colm O hEigeartaigh - Wed, 09/21/2016 - 17:11
Talend ESB ships with a powerful SecurityTokenService (STS) based on the STS that ships with Apache CXF. The Talend Open Studio for ESB contains UI support for creating web service clients that use the STS to obtain SAML tokens for authentication (and also authorization via roles embedded in the tokens). However, it is sometimes useful to be able to obtain tokens with a third party client. In this post we will show how SoapUI can be used to obtain SAML Tokens from the Talend ESB STS.

1) Download and run Talend Open Studio for ESB

The first step is to download Talend Open Studio for ESB (the current version at the time of writing this post is 6.2.1). Unzip it and start the container via:
  • Runtime_ESBSE/container/bin/trun
The next step is to start the STS itself:
  • tesb:start-sts
2) Download and run SoapUI

Download SoapUI and run the installation script. Create a new SOAP Project called "STS" using the WSDL:
  • http://localhost:8040/services/SecurityTokenService/UT?wsdl
The WSDL of the STS defines a number of different services. The one we are interested in is the "UT_Binding", which requires a WS-Security UsernameToken to authenticate the client. Click on "UT_Binding/Issue/Request 1" in the left-hand menu to see a sample request for the service. Now we need to do some editing of the request. Remove the 'Context="?"' attribute from RequestSecurityToken. Then paste the following into the Body of the RequestSecurityToken:
  • <t:TokenType xmlns:t="http://docs.oasis-open.org/ws-sx/ws-trust/200512">http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV2.0</t:TokenType>
  • <t:KeyType xmlns:t="http://docs.oasis-open.org/ws-sx/ws-trust/200512">http://docs.oasis-open.org/ws-sx/ws-trust/200512/Bearer</t:KeyType>
  • <t:RequestType xmlns:t="http://docs.oasis-open.org/ws-sx/ws-trust/200512">http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</t:RequestType>
Now we need to configure a username and password to use when authenticating the client request. In the "Request Properties" box in the lower left corner, add "tesb" for the "username" and "password" properties. Now right click in the request pane, and select "Add WSS Username Token" (Password Text). Now send the request and you should receive a SAML Token in response.

Bear in mind that if you wish to re-use the SAML Token retrieved from the STS in a subsequent request, you must copy it from the "Raw" tab and not the "XML" tab of the response. The latter adds in whitespace that breaks the signature on the token. Another thing to watch out for is that the STS maintains a cache of the Username Token nonce values, so you will need to recreate the UsernameToken each time you want to get a new token.

3) Requesting a "PublicKey" KeyType

The example above uses a "Bearer" KeyType. Another common use-case, as is the case with the security-enabled services developed using the Talend Studio, is when the token must have the PublicKey/Certificate of the client embedded in it. To request such a token from the STS, change the "Bearer" KeyType as above to "PublicKey". However, we also need to present a certificate to the STS to include in the token.

As we are just using the test credentials used by the Talend STS, go to the Runtime_ESBSE/container/etc/keystores and extract the client key with:
  • keytool -exportcert -rfc -keystore clientstore.jks -alias myclientkey -file client.cer -storepass cspass
Edit client.cer + remove the first and end lines (that contain BEGIN/END CERTIFICATE). Now go back to SOAP-UI and add the following to the RequestSecurityToken Body:
  • <t:UseKey xmlns:t="http://docs.oasis-open.org/ws-sx/ws-trust/200512"><ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:X509Data><ds:X509Certificate>...</ds:X509Certificate></ds:X509Data></ds:KeyInfo></t:UseKey>
where the content of the X.509 Certificate is the content in client.cer. This time, the token issued by the STS will contain the public key of the client embedded in the SAML Subject.

Categories: Colm O hEigeartaigh

Securing an Apache Kafka broker - part II

Colm O hEigeartaigh - Mon, 09/19/2016 - 15:49
In the previous post, we looked at how to configure an Apache Kafka broker to require SSL client authentication. In this post we will add authorization to the example, making sure that only authorized producers can send messages to the broker. In addition, we will show how to enforce authorization rules per-topic for consumers.

1) Configure authorization in the broker

Configure Apache Kafka as per the previous tutorial. To enforce some custom authorization rules in Kafka, we will need to implement the Kafka Authorizer interface. This interface contains an "authorize" method, which supplies a Session Object, where you can obtain the current principal, as well as the Operation and Resource upon which to enforce an authorization decision.

In terms of the example detailed in the previous post, we created broker, service (producer) and client (consumer) principals. We want to enforce authorization decisions as follows:
  • Let the broker principal do anything
  • Let the producer principal read/write on all topics
  • Let the consumer principal read/describe only on topics starting with "test".
There is a sample Authorizer implementation available in some Kafka unit test I wrote in github that can be used in this example - CustomAuthorizer:

Next we need to package up the CustomAuthorizer in a jar so that it can be used in the broker. You can do this by checking out the testcases github repo, and invoking "mvn clean package jar:test-jar -DskipTests" in the "apache/bigdata/kafka" directory. Now copy the resulting test jar in "target" to the "libs" directory in your Kafka installation. Finally, edit the "config/server.properties" file and add the following configuration item:
  • authorizer.class.name=org.apache.coheigea.bigdata.kafka.CustomAuthorizer
2) Test authorization

Now lets test the authorization logic. Restart the broker and the producer:
  • bin/kafka-server-start.sh config/server.properties
  • bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties
Send a few messages to check that the producer is authorized correctly. Now start the consumer:
  • bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer
If everything is configured correctly then it should work as in the first tutorial. Now we will create a new topic called "messages":
  • bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic messages
Restart the producer to send messages to "messages" instead of "test". This should work correctly. Now try to consume from "messages" instead of "test". This should result in an authorization failure, as the "client" principal can only consume from the "test" topic according to the authorization rules.
Categories: Colm O hEigeartaigh

Securing an Apache Kafka broker - part I

Colm O hEigeartaigh - Fri, 09/16/2016 - 18:19
Apache Kafka is a messaging system for the age of big data, with a strong focus on reliability, scalability and message throughput. This is the first part of a short series of posts on how to secure an Apache Kafka broker. In this post, we will focus on authenticating message producers and consumers using SSL. Future posts will look at how to authorize message producers and consumers.

1) Create SSL keys

As we will be securing the broker using SSL client authentication, the first step is to create some keys for testing purposes. Download the OpenSSL ca.config file used by the WSS4J project. Change the "certificate" value to "ca.pem", and the "private_key" value to "cakey.pem". You will also need to create a directory called "ca.db.certs", and make an empty file called "ca.db.index". Now create a new CA key and cert via:
  • openssl req -x509 -newkey rsa:1024 -keyout cakey.pem -out ca.pem -config ca.config -days 3650
Just accept the default options. Now we need to convert the CA cert into jks format:
  • openssl x509 -outform DER -in ca.pem -out ca.crt
  • keytool -import -file ca.crt -alias ca -keystore truststore.jks -storepass security
Now we will create the client key, sign it with the CA key, and put the signed client cert and CA cert into a keystore:
  • keytool -genkey -validity 3650 -alias myclientkey -keyalg RSA -keystore clientstore.jks -dname "CN=Client,O=Apache,L=Dublin,ST=Leinster,C=IE" -storepass cspass -keypass ckpass
  • keytool -certreq -alias myclientkey -keystore clientstore.jks -file myclientkey.cer -storepass cspass -keypass ckpass
  • echo 20 > ca.db.serial
  • openssl ca -config ca.config -policy policy_anything -days 3650 -out myclientkey.pem -infiles myclientkey.cer
  • openssl x509 -outform DER -in myclientkey.pem -out myclientkey.crt
  • keytool -import -file ca.crt -alias ca -keystore clientstore.jks -storepass cspass
  • keytool -import -file myclientkey.crt -alias myclientkey -keystore clientstore.jks -storepass cspass -keypass ckpass
Now follow the same template to create a "service" key in servicestore.jks, with store password "sspass" and key password "skpass". In addition, we will create a "broker" key in brokerstore.jks, with storepass "bspass" and key password "bkpass".  

2) Configure the broker

Download Apache Kafka and extract it (0.10.0.1 was used for the purposes of this tutorial). Copy the keys created in section "1" into $KAFKA_HOME/config. Start Zookeeper with:
  • bin/zookeeper-server-start.sh config/zookeeper.properties
Now edit 'config/server.properties' and add the following:
  • ssl.keystore.location=./config/brokerstore.jks
  • ssl.keystore.password=bspass
  • ssl.key.password=bkpass
  • ssl.truststore.location=./config/truststore.jks
  • ssl.truststore.password=security
  • security.inter.broker.protocol=SSL
  • ssl.client.auth=required
  • listeners=SSL://localhost:9092
and start the broker and then create a "test" topic with:
  • bin/kafka-server-start.sh config/server.properties
  • bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
3) Configure the producer

Now we will configure the message producer. Edit 'config/producer.properties' and add the following:
  • security.protocol=SSL
  • ssl.keystore.location=./config/servicestore.jks
  • ssl.keystore.password=sspass
  • ssl.key.password=skpass
  • ssl.truststore.location=./config/truststore.jks
  • ssl.truststore.password=security
and start the producer with:
  • bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties
Send a few messages to the topic to make sure that everything is working ok.

4) Configure the consumer

Finally we will configure the message consumer. Edit 'config/consumer.properties' and add the following:
  • security.protocol=SSL
  • ssl.keystore.location=./config/clientstore.jks
  • ssl.keystore.password=cspass
  • ssl.key.password=ckpass
  • ssl.truststore.location=./config/truststore.jks
  • ssl.truststore.password=security
and start the consumer with:
  • bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer.properties --new-consumer
The messages sent by the producer should appear in the console window of the consumer. So there it is, we have configured the Kafka broker to require SSL client authentication. In the next post we will look at adding simple authorization support to this scenario.
Categories: Colm O hEigeartaigh

Integrating Apache Camel with Apache Syncope - part II

Colm O hEigeartaigh - Fri, 09/09/2016 - 16:58
A recent blog post introduced the new Apache Camel provisioning manager that is available in Apache Syncope 2.0.0. It also covered a simple use-case for the new functionality, where the "createUser" Camel route is modified to send an email to an administrator when a User is created, with some details about the created User in the email. In this post, we will look at a different use-case, where the Camel provisioning manager is used to extend the functionality offered by Syncope.

1) The use-case

Apache Syncope stores users in internal storage in a table called "SyncopeUser". This table contains information such as the User Id, name, password, creation date, last password change date, etc. In addition, if there is an applicable password policy associated with the User realm, a list of the previous passwords associated with the User is stored in a table called "SyncopeUser_passwordHistory":

As can be seen from this screenshot, the table stores a list of Syncope User Ids with a corresponding password value. The main function of this table is to enforce the password policy. So for example, if the password policy states that a user can't change back to a password for at least 10 subsequent password changes, this table provides the raw information that is needed to enforce the policy.

Now, what if the administrator wants a stronger audit trail for User password changes other than what is provided by default in the Syncope internal storage? In particular, the administrator would like a record of when the User changes a password. The "SyncopeUser" table only stores the last password change date. There is no way of seeing when each password stored in the "SyncopeUser_passwordHistory" table was changed. Enter the Camel provisioning manager...

2) Configure Apache Syncope

Download and install  Apache Syncope (I used the "standalone" download for the purposes of this demo). Start Apache Syncope and log on to the admin console. First we will create a password policy. Click on "Configuration" and then "Policies". Click on the "Password" tab and then select the "+" button to create a new policy. Give a description for the policy and then select a history length of "10".

Now let's create a new user in the default realm. Click on "realms" and hit the "edit" button. Select the password policy you have just created and click "Finish". This means that all users created in this realm will have the applicable policy applied to them. Now click on the "User" tab and the "+" button to add a new user called "alice". Click on "Password Management" and enter a password for "alice". Now we have a new user created, we want to be able to see when she updates her password from now on.

Click on "Extensions" and "Camel Routes". Find the "updateUser" route (might not be on the first page) and edit it. Create a new Camel "filter" (as per the screenshot below) just above the "bean method=" line with the following content:
  • <simple>${body.password} != null</simple>
  • <setHeader headerName="CamelFileName"><simple>${body.key}.passwords</simple></setHeader>
  • <setBody><simple>New password '${body.password.value}' changed at time '${date:now:yyyy-MM-dd'T'HH:mm:ss.SSSZ}'\n</simple></setBody>
  • <to uri="file:./?fileExist=Append"/>
So what we are doing here is to audit the password changes for a particular user, by writing out the password + Timestamp to a file associated with that user. Let's examine what each of these statements do in turn. The first statement is the filter condition. It states that we should execute the following statements if the password is not null. The password will only be non null if it is being changed. So for example, if the user just changes a given attribute and not the password, the filter will not be invoked.

The second statement sets the Camel Header "CamelFileName" to the format of "<user.id>.passwords". This header is used by the Camel File component as the file name to write out to. The third statement sets the exchange Body (the file content) to show the password value along with a Timestamp. Finally, the fourth statement is an invocation of the Camel File component, which appends the exchange Body to the given file name. As we have overridden the message Body in the second statement above, we need to change the ${body} in the create call to ${exchangeProperty.actual}, which is the saved Body. Click on "save" to save the modified route.


Now let's update the user "alice" and set a new password a couple of times. There should be a new file in the directory where you launched Tomcat containing the audit log for password changes for that particular user. With the power of Apache Camel, we could audit to a database, to Apache Kafka, etc etc.


Categories: Colm O hEigeartaigh

Apache CXF Fediz 1.2.3 and 1.3.1 released

Colm O hEigeartaigh - Thu, 09/08/2016 - 19:45
Apache CXF Fediz 1.2.3 and 1.3.1 have been released. The 1.3.1 release contains the following significant features/fixes:
  • An update to use Apache CXF 3.1.7 
  • Support for Facebook Login as a Trusted IdP.
  • A fix for SAML SSO redirection on ForceAuthn or token expiry.
  • A bug fix to support multiple realms in the IdP.
  • A fix to enforce that mandatory claims are present in the received token.
In addition, both 1.2.3 and 1.3.1 contain a fix for a new security advisory - CVE-2016-4464:
Apache CXF Fediz is a subproject of Apache CXF which implements the WS-Federation Passive Requestor Profile for SSO specification. It provides a number of container based plugins to enable SSO for Relying Party applications. It is possible to configure a list of audience URIs for the plugins, against which the AudienceRestriction values of the received SAML tokens are supposed to be matched. However, this matching does not actually take place.

This means that a token could be accepted by the application plugin (assuming that the signature is trusted) that is targeted for another service, something that could potentially be exploited by an attacker.
    Categories: Colm O hEigeartaigh

    Integrating Apache Camel with Apache Syncope - part I

    Colm O hEigeartaigh - Wed, 08/31/2016 - 12:29
    Apache Syncope is an open-source Identity Management solution. A key feature of Apache Syncope is the ability to pull Users, Groups and Any Objects from multiple backend resources (such as LDAP, RDMBS, etc.) into Syncope's internal storage, where they can then be assigned roles, pushed to other backend resources, exposed via Syncope's REST API, etc.

    However, what if you wanted to easily perform some custom task as part of the Identity Management process? Wouldn't it be cool to be able to plug in a powerful integration framework such as Apache Camel, so that you could exploit Camel's huge list of messaging components and routing + mediation rules? Well with Syncope 2.0.0 you can do just this with the new Apache Camel provisioning manager. This is a unique and very powerful selling point of Apache Syncope in my opinion. In this article, we will introduce the new Camel provisioning manager, and show a simple example of how to use it.

    1) The new Apache Camel provisioning manager

    As stated above, a new provisioning manager is available in Apache Syncope 2.0.0 based on Apache Camel. A set of Camel routes are available by default which are invoked when the User, Groups and Any Objects in question are changed in some way. So for example, if a new User is created, then the corresponding Camel route is invoked at the same time. This allows the administrator to plug in custom logic on any of these state changes. The routes can be viewed and edited in the Admin Console by clicking on "Extensions" and then "Camel Routes".

    Each of the Camel routes uses a new "propagate" Camel component available in Syncope 2.0.0. This component encapsulates some common logic involved in using the Syncope PropagationManager to create some tasks, and to execute them via the PropagationTaskExecutor. All of the routes invoke this propagate component via something like:
    • <to uri="propagate:<propagateType>?anyTypeKind=<anyTypeKind>&options"/>
    Where propagateType is one of:
    • create
    • update
    • delete
    • provision
    • deprovision
    • status
    • suspend
    • confirmPasswordReset
    and anyTypeKind is one of:
    • USER
    • GROUP
    • ANY
    2) The use-case

    In this post, we will look at a simple use-case of sending an email to an administrator when a User is created, with some details about the created User in the email. Of course, this could be handled by a Notification Task, but we'll discuss some more advanced scenarios in future blog posts. Also note that a video available on the Tirasa blog details more or less the same use-case. For the purposes of the demo, we will set up a mailtrap account where we will receive the emails sent by Camel.

    3) Configure Apache Syncope

    Download and install  Apache Syncope (I used the "standalone" download for the purposes of this demo). Before starting Apache Syncope, we need to copy a few jars that are required by Apache Camel to actually send emails. Copy the following jars to $SYNCOPE/webapps/syncope/WEB-INF/lib:
    • http://repo1.maven.org/maven2/org/apache/camel/camel-mail/2.17.3/camel-mail-2.17.3.jar
    • http://repo1.maven.org/maven2/com/sun/mail/javax.mail/1.5.5/javax.mail-1.5.5.jar
    Now start Apache Syncope and log on to the admin console. Click on "Extensions" and then "Camel Routes". As we want to change the default route when users are created, click on the "edit" image for the "createUser" route. Add the following information just above the "bean method=" line:
    • <setHeader headerName="subject"><simple>New user ${body.username} created in realm ${body.realm}</simple></setHeader> 
    • <setBody><simple>User full name: ${body.plainAttrMap[fullname].values[0]}</simple></setBody>
    • <to uri="smtp://mailtrap.io?username=<username>&amp;password=<password>&amp;contentType=text/html&amp;to=dummy@apache-recipient.org"/>
    Let's examine what each of these statements do. The first statement is setting the Camel header "Subject" which corresponds to the Subject of the Email. It simply states that a new user with a given name is created in a given realm. The second statement sets the message Body, which is used as the content of the message by Camel. It just shows the User's full name, extracted from the "fullname" attribute, as an example of how to access User attributes in the route.

    The third statement invokes on the Camel smtp component. You'll need to substitute in the username + password you configured when setting up the mailtrap account. The recipient is configured using the "to" part of the URI. One more change is required to the existing route. As we have overridden the message Body in the second statement above, we need to change the ${body} in the create call to ${exchangeProperty.actual}, which is the saved Body. Click on "save" to save the modified route.


    Before creating a User, we need to add a "fullname" User attribute as the route expects. Go to "Configuration" and "Types", and click on the "Schemas" tab. Click on the "+" button under "PLAIN" and add a new attribute called "fullname". Then click on "AnyTypeClasses", and add the "fullname" attribute to the BaseUser AnyTypeClass.

    Finally, go to the "/" realm and create a new user, specifying a fullname attribute. A new email should be available in the mailtrap account as follows:

    Categories: Colm O hEigeartaigh

    Pulling users and groups from LDAP into Apache Syncope 2.0.0

    Colm O hEigeartaigh - Fri, 08/26/2016 - 17:54
    A previous tutorial showed how to synchronize (pull) users and roles into Apache Syncope 1.2.x from an LDAP backend (Apache Directory). Interacting with an LDAP backend appears to be a common use-case for Apache Syncope users. For this reason, in this tutorial we will cover how to pull users and groups (previously roles) into Apache Syncope 2.0.0 from an LDAP backend via the Admin Console, as it is a little different from the previous 1.2.x releases.

    1) Apache DS

    The basic scenario is that we have a directory that stores user and group information that we would like to import into Apache Syncope 2.0.0. For the purposes of this tutorial, we will work with Apache DS. The first step is to download and launch Apache DS. I recommend installing Apache Directory Studio for an easy way to create and view the data stored in your directory.

    Create two new groups (groupOfNames) in the default domain ("dc=example,dc=com") called "cn=employee,ou=groups,ou=system" and "cn=boss,ou=groups,ou=system". Create two new users (inetOrgPerson) "cn=alice,ou=users,ou=system" and "cn=bob,ou=users,ou=system". Now edit the groups you created such that both alice and bob are employees, but only alice is a boss. Specify "sn" (surname) and "userPassword" attributes for both users.

    2) Pull data into Apache Syncope

    The next task is to import (pull) the user data from Apache DS into Apache Syncope. Download and launch an Apache Syncope 2.0.x instance. Make sure that an LDAP Connector bundle is available (see here).

    a) Define a 'surname' User attribute

    The inetOrgPerson instances we created in Apache DS have a "sn" (surname) attribute. We will map this into an internal User attribute in Apache Syncope. The Schema configuration is quite different in the Admin Console compared to Syncope 1.2.x. Select "Configuration" and then "Types" in the left hand menu. Click on the "Schemas" tab and then the "+" button associated with "PLAIN". Add "surname" for the Key and click "save". Now go into the "AnyTypeClasses" tab and edit the "BaseUser" item. Select "surname" from the list of available plain Schema attributes. Now the users we create in Syncope can have a "surname" attribute.




    b) Define a Connector

    The next thing to do is to define a Connector to enable Syncope to talk to the Apache DS backend. Click on "Topology" in the left-hand menu, and on the ConnId instance on the map. Click "Add new connector" and create a new Connector of type "net.tirasa.connid.bundles.ldap". On the next tab select:
    • Host: localhost
    • TCP Port: 10389
    • Principal: uid=admin,ou=system
    • Password: <password>
    • Base Contexts: ou=users,ou=system and ou=groups,ou=system
    • LDAP Filter for retrieving accounts: cn=*
    • Group Object Classes: groupOfNames
    • Group member attribute: member
    • Click on "Maintain LDAP Group Membership".
    • Uid attribute: cn
    • Base Context to Synchronize: ou=users,ou=system and ou=groups,ou=system
    • Object Classes to Synchronize: inetOrgPerson and groupOfNames
    • Status Management Class: net.tirasa.connid.bundles.ldap.commons.AttributeStatusManagement
    • Tick "Retrieve passwords with search".
    Click on the "heart" icon at the top of tab to check to see whether Syncope is able to connect to the backend resource. If you don't see a green "Successful Connection" message, then consult the logs. On the next tab select all of the available capabilities and click on "Finish".

    c) Define a Resource

    Next we need to define a Resource that uses the LDAP Connector.  The Resource essentially defines how we use the Connector to map information from the backend into Syncope Users and Groups. Click on the Connector that was created in the Topology map and select "Add new resource". Just select the defaults and finish creating the new resource. When the new resource is created, click on it and add some provisioning rules via "Edit provision rules".

    Click the "+" button and select the "USER" type to create the mapping rules for users. Click "next" until you come to the mapping tab and create the following mappings:


    Click "next" and enable "Use Object Link" and enter "'cn=' + username + ',ou=users,ou=system'". Click "Finish" and "save". Repeat the process above for the "GROUP" type to create a mapping rule for groups as follows:
    Similar to creating the user mappings, we also need to enable "Use Object Link" and enter "'cn=' + name + ',ou=groups,ou=system'". Click "Finish" and "save".

    d) Create a pull task

    Having defined a Connector and a Resource to use that Connector, with mappings to map User/Group information to and from the backend, it's time to import the backend information into Syncope.  Click on the resource and select "Pull Tasks". Create a new Pull Task via the "+" button. Select "/" as the destination realm to create the users and groups in. Choose "FULL_RECONCILIATION" as the pull mode. Select "LDAPMembershipPullActions"  (this will preserve the fact that users are members of a group in Syncope) and "LDAPPasswordPullActions" from the list of available actions. Select "Allow create/update/delete". When the task is created,  click on the "execute" button (it looks like a cogged wheel). Now switch to the "Realms" tab in the left-hand menu and look at the users and groups that have been imported in the "/" realm from Apache DS.


    Categories: Colm O hEigeartaigh

    OpenId Connect in Apache CXF Fediz 1.3.0

    Colm O hEigeartaigh - Fri, 08/12/2016 - 18:02
    Previous blog posts have described support for OpenId Connect protocol bridging in the Apache CXF Fediz IdP. What this means is that the Apache CXF Fediz IdP can bridge between the WS-Federation protocol and OpenId Connect third party IdPs, when the user must be authenticated in a different security domain. However, the 1.3.0 release of Apache CXF Fediz also sees the introduction of a new OpenId Connect Idp which is independent of the existing (WS-Federation and SAML-SSO based) IdP, and based on Apache CXF. This post will introduce the new IdP via an example.

    The example code is available on github:
    • cxf-fediz-oidc: This project shows how to use interceptors of Apache CXF to authenticate and authorize clients of a JAX-RS service using OpenId Connect.
    1) The secured service

    The first module available in the example contains a trivial JAX-RS Service based on Apache CXF which "doubles" a number that is passed as a path parameter via HTTP GET. The service defines via a @RolesAllowed annotation that only users allowed in roles "User", "Admin" or "Manager" can access the service.

    The service is configured via spring. The endpoint configuration references the service bean above, as well as the CXF SecureAnnotationsInterceptor which enforces the @RolesAllowed annotation on the service bean. In addition, the service is configured with the CXF OidcRpAuthenticationFilter, which ensures that only users authenticated via OpenId Connect can access the service. The filter is configured with a URL to redirect the user to. It also explicitly requires a role claim to enforce authorization.

    The OidcRpAuthenticationFilter redirects the browser to a separate authentication endpoint, defined in the same spring file for convenience. This endpoint has a filter called OidcClientCodeRequestFilter, which initiates the OpenId Connect authorization code flow to a remote OpenId Connect IdP (in this case, the new Fediz IdP). It is also responsible for getting an IdToken after successfully getting an authorization code from the IdP.

    2) The Fediz OpenId Connect IdP

    The second module contains an integration test which deploys a number of wars into an Apache Tomcat container:
    • The "double-it" service as described above
    • The Apache CXF Fediz IdP which authenticates users via WS-Federation
    • The Apache CXF Fediz STS which performs the underlying authentication of users
    • The Apache CXF Fediz OpenId Connect IdP
    The way the Apache CXF Fediz OpenId Connect IdP works (at least for 1.3.x) is that user authentication is actually delegated to the WS-Federation based IdP via a Fediz plugin. So when the user is redirected to the Fediz IdP, (s)he gets redirected to the WS-Federation based IdP for authentication, and then gets redirected back to the OpenId Connect IdP with a WS-Federation Response. The OpenId Connect IdP parses this (SAML) Response and converts it into a JWT IdToken. Future releases will enable authentication directly at the OpenId Connect service.

    After deploying all of the services, the test code makes a series of REST calls to create a client in the OpenId Connect IdP so that we can run the test without having to manually enter information in the client UI of the Fediz IdP. To run the test, simply remove the @org.junit.Ignore assertion on the "testInBrowser" method. The test code will create the clients in Fediz and then print out a URL in the console before sleeping. Copy the URL and paste it into a browser. Authenticate using the credentials "alice/ecila".
    Categories: Colm O hEigeartaigh

    Introducing Apache Syncope 2.0.0

    Colm O hEigeartaigh - Thu, 08/11/2016 - 17:17
    Apache Syncope is a powerful and flexible open-source Identity Management system that has been developed at the Apache Software Foundation for several years now. The Apache Syncope team has been busy developing a ton of new features for the forthcoming new major release (2.0.0), which will really help to cement Apache Syncope's position as a first class Identity Management solution. If you wish to experiment with these new features, a 2.0.0-M4 release is available. In this post we will briefly cover some of the new features and changes. For a more comprehensive overview please refer to the reference guide.

    1) Domains

    Perhaps the first new concept you will be introduced to in Syncope 2.0.0 after starting the (Admin) console is that of a domain. When logging in, as well as specifying a username, password, and language, you can also specify a configured domain. Domains are a new concept in Syncope 2.0.0 that facilitate multi-tenancy. Domains allow the physical separation of all data stored in Syncope (by storing the data in different database instances). Therefore, Syncope can facilitate users, groups etc. that are in different domains in a single Syncope instance.

    2) New Console layout

    After logging in, it becomes quickly apparent that the Syncope Console is quite different compared to the 1.2.x console. It has been completely rewritten and looks great. Connectors and Resources are now managed under "Topology" in the menu on the left-hand side. Users and Groups (formerly Roles) are managed under "Realms" in the menu. The Schema types are configured under "Configuration". A video overview of the new Console can be seen here.


    3) AnyType Objects

    With Syncope 1.2.x, it was possible to define plain/derived/virtual Schema Types for users, roles and memberships, but no other entities. In Syncope 2.0.0, the Schema Types are decoupled from the entity that uses them. Instead, a new concept called an AnyType class is available which is a collection of schema types. In turn, an AnyType object can be created which consists of any number of AnyType classes. AnyType objects represent the type of things that Apache Syncope can model. Besides the predefined Users and Groups, it can also represent physical things such as printers, workstations, etc. With this new concept, Apache Syncope 2.0.0 can model many different types of identities.

    4) Realms

    Another new concept in Apache Syncope 2.0.0 is that of a realm. A realm encapsulates a number of Users, Groups and Any Objects. It is possible to specify account and password policies per-realm (see here for a blog entry on custom policies in Syncope 2.0.0). Each realm has a parent realm (apart from the pre-defined root realm identified as "/"). The realm tree is hierarchical, meaning that Users, Groups etc. defined in a sub-realm, are also defined on a parent realm. Combined with Roles (see below), realms facilitate some powerful access management scenarios.

    5) Groups/Roles

    In Syncope 2.0.0, what were referred to as "roles" in Syncope 1.2.x are now called "groups". In addition, "roles" in Syncope 2.0.0 are a new concept which associate a number of entitlements with a number of realms. Users assigned to a role can exercise the defined entitlements on any of the objects in the given realms (any any sub-realms).

    Syncope 2.0.0 also has the powerful concept of dynamic membership, which means that users can be assigned to groups or roles via a conditional expression (e.g. if an attribute matches a given value).

    6) Apache Camel Provisioning

    An exciting new feature of Apache Syncope 2.0.0 is the new Apache Camel provisioning engine, which is available under "Extensions/Camel Routes" in the Console. Apache Syncope comes pre-loaded with some Camel routes that are executed as part of the provisioning implementation for Users, Groups and Any Objects. The real power of this new engine lies is the ability to modify the routes to perform some custom provisioning rules. For example, on creating a new user, you may wish to send an email to an administrator. Or if a user is reactivated, you may wish to reactivate the user's home page on a web server. All these things and more are possible using the myriad of components that are available to be used in Apache Camel routes. I'll explore this feature some more in future blog posts.

    7) End-User UI

    As well as the Admin console (available via /syncope-console), Apache Syncope 2.0.0 also ships with an Enduser console (available via /syncope-enduser). This allows a user to edit only details pertaining to his/her-self, such as editing the user attributes, changing the password, etc. See the following blog entry for more information on the new End-User UI.


    8) Command Line Interface (CLI) client

    Another new feature of Apache Syncope 2.0.0 is that of the CLI client. It is available as a separate download. Once downloaded, extract it and run (on linux): ./syncopeadm.sh install --setup. Answer the questions about where Syncope is deployed and the credentials required to access it. After installation, you can run queries such as: ./syncopeadm.sh user --list.

    9) Apache CXF-based testcases

    I updated the testcases that I wrote before to use Apache Syncope 2.0.0 to authenticate and authorize web services calls using Apache CXF. The new test-cases are available here
    Categories: Colm O hEigeartaigh

    Installing the Apache Ranger Key Management Server (KMS)

    Colm O hEigeartaigh - Mon, 08/08/2016 - 13:40
    The previous couple of blog entries have looked at how to install the Apache Ranger Admin Service as well as the Usersync Service. In this post we will look at how to install the Apache Ranger Key Management Server (KMS). KMS is a component of Apache Hadoop to manage cryptographic keys. Apache Ranger ships with its own KMS implementation, which allows you to store the (encrypted) keys in a database. The Apache Ranger KMS is also secured via policies defined in the Apache Ranger Admin Service.

    1) Build the source code

    The first step is to download the source code, as well as the signature file and associated message digests (all available on the download page). Verify that the signature is valid and that the message digests match. Now extract and build the source, and copy the resulting KMS archive to a location where you wish to install it:
    • tar zxvf apache-ranger-incubating-0.6.0.tar.gz
    • cd apache-ranger-incubating-0.6.0
    • mvn clean package assembly:assembly 
    • tar zxvf target/ranger-0.6.0-kms.tar.gz
    • mv ranger-0.6.0-kms ${rangerkms.home}
    2) Install the Apache Ranger KMS Service

    As the Apache Ranger KMS Service stores the cryptographic keys in a database, we will need to setup and configure a database. We will also configure the KMS Service to store audit logs in the database. Follow the steps given in section 2 of the tutorial on the Apache Ranger Admin Service to set up MySQL. We will also need to create a new user 'rangerkms':
    • CREATE USER 'rangerkms'@'localhost' IDENTIFIED BY 'password';
    • FLUSH PRIVILEGES; 
    You will need to install the Apache Ranger KMS Service using "sudo". If the root user does not have a JAVA_HOME property defined, then edit ${rangerkms.home}/setup.sh + add in, e.g.:
    • export JAVA_HOME=/opt/jdk1.8.0_91
    Next edit ${rangerkms.home}/install.properties and make the following changes:
    • Change SQL_CONNECTOR_JAR to point to the MySQL JDBC driver jar (see previous tutorial).
    • Set (db_root_user/db_root_password) to (admin/password)
    • Set (db_user/db_password) to (rangerkms/password)
    • Change KMS_MASTER_KEY_PASSWD to a secure password value.
    • Set POLICY_MGR_URL=http://localhost:6080
    • Set XAAUDIT.DB.IS_ENABLED=true
    • Set XAAUDIT.DB.FLAVOUR=MYSQL 
    • Set XAAUDIT.DB.HOSTNAME=localhost 
    • Set XAAUDIT.DB.DATABASE_NAME=ranger_audit 
    • Set XAAUDIT.DB.USER_NAME=rangerlogger
    • Set XAAUDIT.DB.PASSWORD=password
    Now you can run the setup script via "sudo ./setup.sh".

    3) Starting the Apache Ranger KMS service

    After a successful installation, first start the Apache Ranger admin service with "sudo ranger-admin start". Then start the Apache Ranger KMS Service via "sudo ranger-kms start". Now open a browser and go to "http://localhost:6080/". Log on with "keyadmin/keyadmin". Note that these are different credentials to those used to log onto the Apache Ranger Admin UI in the previous tutorial. Click on the "+" button on the "KMS" tab to create a new KMS Service. Specify the following values:
    • Service Name: kmsdev
    • KMS URL: kms://http@localhost:9292/kms
    • Username: keyadmin
    • Password: keyadmin
    Click on "Test Connection" to make sure that the KMS Service is up and running. If it is showing a connection failure, log out and log into the Admin UI using credentials "admin/admin". Go to the "Audit" section and click on "Plugins". You should see a successful message indicating that the KMS plugin can successfully download policies from the Admin Service:


    After logging back in to the UI as "keyadmin" you can start to create keys. Click on the "Encryption/Key Manager" tab. Select the "kmsdev" service in the dropdown list and click on "Add New Key". You can create, delete and rollover keys in the UI:



    Categories: Colm O hEigeartaigh

    Syncing users and groups from LDAP into Apache Ranger

    Colm O hEigeartaigh - Fri, 07/22/2016 - 16:37
    The previous post covered how to install the Apache Ranger Admin service. The Apache Ranger Admin UI supports creating authorization policies for various Big Data components, by giving users and/or groups permissions on resources. This means that we need to import users/groups into the Apache Ranger Admin service from some backend service in order to create meaningful authorization policies. Apache Ranger supports syncing users into the Admin service from both unix and ldap. In this post, we'll look at syncing in users and groups from an OpenDS LDAP backend.

    1) The OpenDS backend

    For the purposes of this tutorial, we will use OpenDS as the LDAP server. It contains a domain called "dc=example,dc=com", and 5 users (alice/bob/dave/oscar/victor) and 2 groups (employee/manager). Victor, Oscar and Bob are employees, Alice and Dave are managers. Here is a screenshot using Apache Directory Studio:



    2) Build the Apache Ranger usersync module

    Follow the steps in the previous tutorial to build Apache Ranger and to setup and start the Apache Ranger Admin service. Once this is done, go back to the Apache Ranger distribution that you have built and copy the usersync module:
    • tar zxvf target/ranger-0.6.0-usersync.tar.gz
    • mv ranger-0.6.0-usersync ${usersync.home}
    3) Configure and build the Apache Ranger usersync service 

    You will need to install the Apache Ranger Usersync service using "sudo". If the root user does not have a JAVA_HOME property defined, then edit ${usersync.home}/setup.sh + add in, e.g.:
    • export JAVA_HOME=/opt/jdk1.8.0_91
    Next edit ${usersync.home}/install.properties and make the following changes:
    • POLICY_MGR_URL = http://localhost:6080
    • SYNC_SOURCE = ldap
    • SYNC_INTERVAL = 1 (just for testing purposes....)
    • SYNC_LDAP_URL = ldap://localhost:2389
    • SYNC_LDAP_BIND_DN = cn=Directory Manager,dc=example,dc=com
    • SYNC_LDAP_BIND_PASSWORD = test
    • SYNC_LDAP_SEARCH_BASE = dc=example,dc=com
    • SYNC_LDAP_USER_SEARCH_BASE = ou=users,dc=example,dc=com
    • SYNC_GROUP_SEARCH_BASE=ou=groups,dc=example,dc=com
    Now you can run the setup script via "sudo ./setup.sh". 

    4) Start the Usersync service

    The Apache Ranger Usersync service can be started via "sudo ./ranger-usersync-services.sh start". After 1 minute (see SYNC_INTERVAL above), it should successfully copy the users/groups from the OpenDS backend into the Apache Ranger Admin. Open a browser and go to "http://localhost:6080", and click on "Settings" and then "Users/Groups". You should see the users and groups synced successfully from OpenDS.

    Categories: Colm O hEigeartaigh

    Installing the Apache Ranger Admin UI

    Colm O hEigeartaigh - Tue, 07/19/2016 - 14:22
    Apache Ranger 0.6 has been released, featuring new support for securing Apache Atlas and Nifi, as well as a huge amount of bug fixes. It's easiest to get started with Apache Ranger by downloading a big data sandbox with Ranger pre-installed. However, the most flexible way is to grab the Apache Ranger source and to build and deploy the artifacts yourself. In this tutorial, we will look into building Apache Ranger from source, setting up a database to store policies/users/groups/etc. as well as Ranger audit information, and deploying the Apache Ranger Admin UI.

    1) Build the source code

    The first step is to download the source code, as well as the signature file and associated message digests (all available on the download page). Verify that the signature is valid and that the message digests match. Now extract and build the source, and copy the resulting admin archive to a location where you wish to install the UI:
    • tar zxvf apache-ranger-incubating-0.6.0.tar.gz
    • cd apache-ranger-incubating-0.6.0
    • mvn clean package assembly:assembly 
    • tar zxvf target/ranger-0.6.0-admin.tar.gz
    • mv ranger-0.6.0-admin ${rangerhome}
    2) Install MySQL

    The Apache Ranger Admin UI requires a database to keep track of users/groups as well as policies for various big data projects that you are securing via Ranger. In addition, we will use the database for auditing as well. For the purposes of this tutorial, we will use MySQL. Install MySQL in $SQL_HOME and start MySQL via:
    • sudo $SQL_HOME/bin/mysqld_safe --user=mysql
    Now you need to log on as the root user and create three users for Ranger. We need a root user with admin privileges (let's call this user "admin"), a user for the Ranger Schema (we'll call this user "ranger"), and finally a user to store the Ranger audit logs in the DB as well ("rangerlogger"):
    • CREATE USER 'admin'@'localhost' IDENTIFIED BY 'password';
    • GRANT ALL PRIVILEGES ON * . * TO 'admin'@'localhost' WITH GRANT OPTION;
    • CREATE USER 'ranger'@'localhost' IDENTIFIED BY 'password';
    • CREATE USER 'rangerlogger'@'localhost' IDENTIFIED BY 'password'; 
    • FLUSH PRIVILEGES;
    Finally,  download the JDBC driver jar for MySQL and put it in ${rangerhome}.

    3) Install the Apache Ranger Admin UI

    You will need to install the Apache Ranger Admin UI using "sudo". If the root user does not have a JAVA_HOME property defined, then edit ${rangerhome}/setup.sh + add in, e.g.:
    • export JAVA_HOME=/opt/jdk1.8.0_91
    Next edit ${rangerhome}/install.properties and make the following changes:
    • Change SQL_CONNECTOR_JAR to point to the MySQL JDBC driver jar that you downloaded above.
    • Set (db_root_user/db_root_password) to (admin/password)
    • Set (db_user/db_password) to (ranger/password)
    • Change "audit_store" from "solr" to "db"
    • Set "audit_db_name" to "ranger_audit"
    • Set (audit_db_user/audit_db_password) to (rangerlogger/password).
    Now you can run the setup script via "sudo ./setup.sh".

    4) Starting the Apache Ranger admin service

    After a successful installation, we can start the Apache Ranger admin service with "sudo ${rangerhome}./ews/start-ranger-admin.sh". Now open a browser and go to "http://localhost:6080/". Log on with "admin/admin" and you should be able to create authorization policies for a desired big data component.


    Categories: Colm O hEigeartaigh

    A new REST interface for the Apache CXF Security Token Service - part II

    Colm O hEigeartaigh - Fri, 06/17/2016 - 12:38
    The previous blog entry introduced the new REST interface of the Apache CXF Security Token Service. It covered issuing, renewing and validating tokens via HTTP GET and POST with a focus on SAML tokens. In this post, we'll cover the new support in the STS to issue and validate JWT tokens via the REST interface, as well as how we can transform SAML tokens into JWT tokens, and vice versa. For information on how to configure the STS to support JWT tokens (via WS-Trust), please look at a previous blog entry.

    1) Issuing JWT Tokens

    We can retrieve a  JWT Token from the REST interface of the STS simply by making a HTTP GET request to "/token/jwt". Specify the "appliesTo" query parameter to insert the service address as the audience of the generated token. You can also include various claims in the token via the "claim" query parameter.

    If an "application/xml" accept type is specified, or if multiple accept types are specified (as in a browser), the JWT token is wrapped in a "TokenWrapper" XML Element by the STS:

    If "application/json" is specified, the JWT token is wrapped in a simple JSON Object as follows:

    If "text/plain" is specified, the raw JWT token is returned. We can also get a JWT token using HTTP POST by constructing a WS-Trust RequestSecurityToken XML fragment, specifying "urn:ietf:params:oauth:token-type:jwt" as the WS-Trust TokenType.

    2) Validating JWT Tokens and token transformation

    We can also validate JWT Tokens by POSTing a WS-Trust RequestSecurityToken to the STS. The raw (String) JWT Token must be wrapped in a "TokenWrapper" XML fragment, which in turn is specified as the value of "ValidateTarget". Also, an "action" query parameter must be added with value "validate".

    A powerful feature of the STS is the ability to transform tokens from one type to another. This is done by making a validate request for a given token, and specifying a value for "TokenType" that corresponds to a token type that is desired.  In this way we can validate a SAML token and issue a JWT Token, and vice versa.

    To see some examples of how to do token validation as well as token transformation, please take a look at the following tests, which use the CXF WebClient to invoke on the REST interface of the STS.
    Categories: Colm O hEigeartaigh

    A new REST interface for the Apache CXF Security Token Service - part I

    Colm O hEigeartaigh - Thu, 06/16/2016 - 17:46
    Apache CXF ships a Security Token Service (STS) that can issue/validate/renew/cancel tokens via the (SOAP based) WS-Trust interface. The principal focus of the STS is to deal with SAML tokens, although other token types are supported as well. JAX-RS clients can easily obtain tokens using helper classes in CXF (see Andrei Shakirin's blog entry for more information on this).

    However, wouldn't it be cool if a JAX-RS client could dispense with SOAP altogether and obtain tokens via a REST interface? Starting from the 3.1.6 release, the Apache CXF STS now has a powerful and flexible REST API which I'll describe in this post. The next post will cover how the STS can now issue and validate JWT tokens, and how it can transform JWT tokens into SAML tokens and vice versa.

    One caveat - this REST interface is obviously not a standard, as compared to the WS-Trust interface, and so is specific to CXF.

    1) Configuring the JAX-RS endpoint of the CXF STS

    Firstly, let's look at how to set up the JAX-RS endpoint needed to support the REST interface of the STS. The STS is configured in exactly the same way as for the standard WS-Trust based interface, the only difference being that we are setting up a JAX-RS endpoint instead of a JAX-WS endpoint. Example configuration can be seen here in a system test.


    2) Issuing Tokens

    The new JAX-RS interface supports a number of different methods to obtain tokens. In this post, we will just focus on the methods used to obtain XML based tokens (such as SAML).

    2.1) Issue tokens via HTTP GET

    The easiest way to get a token is by doing a HTTP GET on "/token/{tokenType}". Here is a simple example using a web browser:

     The supported tokenType values are:
    • saml
    • saml2.0
    • saml1.1
    • jwt
    • sct
    A number of optional query parameters are supported:
    • keyType - The standard WS-Trust based URIs to indicate whether a Bearer, HolderOfKey, etc. token is required. The default is to issue a Bearer token.
    • claim - A list of requested claims to include in the token. See below for the list of acceptable values.
    • appliesTo - The AppliesTo value to use
    • wstrustResponse - A boolean parameter to indicate whether to return a WS-Trust Response or just the desired token. The default is false. This parameter only applies to the XML case.
    Note that for the "Holder of Key" keytype, the STS will try to get the client key via the TLS client certificate, if it is available. The (default) supported claim values are:
    • emailaddress
    • role
    • surname
    • givenname
    • name
    • upn
    • nameidentifier
    The format of the returned token depends on the HTTP application type:
    • application/xml - Returns a token in XML format
    • application/json - JSON format for JWT tokens only.
    • text/plain - The "plain" token is returned. For JWT tokens it is just the raw token. For XML-based tokens, the token is BASE-64 encoded and returned.
    The default is to return XML if multiple types are specified (e.g. in a browser).

    2.2) Issue tokens via HTTP POST

    While the GET method above is very convenient to use, it may be that you want to pass other parameters to the STS in order to issue a token. In this case, you can construct a WS-Trust RequestSecurityToken XML fragment and POST it to the STS. The response will be the standard WS-Trust Response, and not the raw token as you have the option of receiving via the GET method.

    3) Validating/Renewing/Cancelling tokens

    It is only possible to renew or validate or cancel a token using the POST method detailed in section 2.2 above. You construct the RequestSecurityToken XML fragment in the same way as for Issue, except including the token in question under "ValidateTarget", "RenewTarget", etc. For the non-issue use case, you must specify a "action" query parameter, which can be "issue" (default), "validate", "renew", "cancel".
    Categories: Colm O hEigeartaigh

    SAML SSO support in the Fediz 1.3.0 IdP

    Colm O hEigeartaigh - Fri, 05/27/2016 - 17:12
    The Apache CXF Fediz Identity Provider (IdP) has had the ability to talk to third party IdPs using SAML SSO since the 1.2.0 release. However, one of the new features of the 1.3.0 release is the ability to configure the Fediz IdP to use the SAML SSO protocol directly, instead of WS-Federation. This means that Fediz can be used as a fully functioning SAML SSO IdP.

    I added a new test-case to github to show how this works:
    • cxf-fediz-saml-sso: This project shows how to use the SAML SSO interceptors of Apache CXF to authenticate and authorize clients of a JAX-RS service. 
    The test-case consists of two modules. The first is a web application which contains a simple JAX-RS service, which has a single GET method to return a doubled number. The method is secured with a @RolesAllowed annotation, meaning that only a user in roles "User", "Admin", or "Manager" can access the service. The service is configured with the SamlRedirectBindingFilter, which redirects unauthenticated users to a SAML SSO IdP for authentication (in this case Fediz). The service configuration also defines an AssertionConsumerService which validates the response from the IdP, and sets up the session for the user + populates the CXF security context with the roles from the SAML Assertion.

    The second module deploys the Fediz IdP and STS in Apache Tomcat, as well as the "double-it" war above. It uses Htmlunit to make an invocation on the service and check that access is granted to the service. Alternatively, you can comment the @Ignore annotation of the "testInBrowser" method, and copy the printed out URL into a browser to test the service directly (user credentials: "alice/ecila").

    The IdP configuration is defined in entities-realma.xml. Note that under "supportedProtocols" for the "idp-realmA" configuration is the value "urn:oasis:names:tc:SAML:2.0:profiles:SSO:browser". In addition, the default authentication URI is "saml/up". These are the only changes that are required to switch the IdP for "realm A" from WS-Federation to SAML SSO.
    Categories: Colm O hEigeartaigh

    An interop demo between Apache CXF Fediz and Facebook

    Colm O hEigeartaigh - Fri, 05/06/2016 - 17:49
    The previous post showed how to configure the Fediz IdP to interoperate with the Google OpenID Connect provider. In addition to supporting WS-Federation, SAML SSO and OpenID Connect, from the forthcoming 1.3.1 release the Fediz IdP will also support authenticating users via Facebook Connect. Facebook Connect is based on OAuth 2.0 and hence the implementation in Fediz shares quite a lot of common code with the OpenID Connect implementation. In this post, we will show how to configure the Fediz IdP to interoperate with Facebook.

    1) Create a new app in the Facebook developer console

    The first step is to create a new application in the Facebook developer console. Create a Facebook developer account. Login and register as a Facebook developer. Click on "My Apps" and "Add a New App". Select the "website" platform and create a new app with the name "Fediz IdP". Enter your email, select a Category and click on "Create App ID". Enter "https://localhost:8443/fediz-idp/federation" for the Site URL. Now go to the dashboard for the application and copy the App ID + App Secret and save for later.

    2) Testing the newly created Client

    To test that everything is working correctly, open a web browser and navigate to, substituting the client id saved in step "1" above:
    • https://www.facebook.com/dialog/oauth?response_type=code&client_id=<app-id>&redirect_uri=https://localhost:8443/fediz-idp/federation&scope=email
    Login using your Facebook credentials and grant permission on the OAuth consent screen. The browser will then attempt a redirect to the given redirect_uri. Copy the URL + extract the "code" query String. Open a terminal and invoke the following command, substituting in the secret and code extracted above:
    • curl --data "client_id=<app-id>&client_secret=<app-secret>&grant_type=authorization_code&code=<code>&redirect_uri=https://localhost:8443/fediz-idp/federation" https://graph.facebook.com/v2.6/oauth/access_token
    You should see a successful response containing the OAuth 2.0 Access Token.

    3) Install and configure the Apache CXF Fediz IdP and sample Webapp

    Follow a previous tutorial to deploy the latest Fediz IdP + STS to Apache Tomcat, as well as the "simpleWebapp". Note that you will need to use at least Fediz 1.3.1-SNAPSHOT here for Facebook support. Test that the "simpleWebapp" is working correctly by navigating to the following URL (selecting "realm A" at the IdP, and authenticating as "alice/ecila"):
    • https://localhost:8443/fedizhelloworld/secure/fedservlet 
    3.1) Configure the Fediz IdP to communicate with the Facebook IdP

    Now we will configure the Fediz IdP to authenticate the user in "realm B" by using the Facebook Connect protocol. Edit 'webapps/fediz-idp/WEB-INF/classes/entities-realma.xml'. In the 'idp-realmA' bean:
    • Change the port in "idpUrl" to "8443". 
    In the 'trusted-idp-realmB' bean:
    • Change the "url" value to "https://www.facebook.com/dialog/oauth".
    • Change the "protocol" value to "facebook-connect".
    • Delete the "certificate" and "trustType" properties.
    • Add the following parameters Map, filling in a value for the client secret extracted above: <property name="parameters">
                  <util:map>
                      <entry key="client.id" value="<app-id>" />
                      <entry key="client.secret" value="<app-secret>" />
                  </util:map>
              </property>
    It is possible to specify custom scopes via a "scope" parameter. The default value for this parameter is "email". The token endpoint can be configured via a "token.endpoint" parameter (defaults to "https://graph.facebook.com/v2.6/oauth/access_token"). Similarly, the API endpoint used to retrieve the user's email address can be configured via "api.endpoint", defaulting to "https://graph.facebook.com/v2.6". The Subject claim to be used to insert into a SAML Assertion is defined via "subject.claim", which defaults to "email".

    3.2) Update the TLS configuration

    By default, the Fediz IdP is configured with a truststore required to access the Fediz STS. However, this would mean that the requests to the Facebook IdP over TLS will not be trusted. to change this, edit 'webapps/fediz-idp/WEB-INF/classes/cxf-tls.xml', and change the HTTP conduit name from "*.http-conduit" to "https://localhost.*". This means that this configuration will only get picked up when communicating with the STS (deployed on "localhost"), and the default JDK truststore will get used when communicating with the Facebook IdP.

    3.3) Update Fediz STS claim mapping

    The STS will receive a SAML Token created by the IdP representing the authenticated user. The Subject name will be the email address of the user as configured above. Therefore, we need to add a claims mapping in the STS to map the principal received to some claims. Edit 'webapps/fediz-idp-sts/WEB-INF/userClaims.xml' and just copy the entry for "alice" in "userClaimsREALMA", changing "alice" to your email address used to log into Facebook.

    Finally, restart Fediz to pick up the changes (you may need to remove the persistent storage first).

    4) Testing the service

    To test the service navigate to:
    • https://localhost:8443/fedizhelloworld/secure/fedservlet
    Select "realm B". You should be redirected to the Facebook authentication page. Enter the user credentials you have created. You will be redirected to Fediz, where it creates a SAML token representing the act of authentication via the trusted third party IdP,  and redirects to the web application.
    Categories: Colm O hEigeartaigh

    Pages

    Subscribe to Talend Community Coders aggregator - Colm O hEigeartaigh