Versions Compared

Old Version 2

changes.mady.by.user Marko Ivancic

Saved on

compared with

New Version Current

changes.mady.by.user Marko Ivancic

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Conformance IdP

The Conformance IdP is a

Test environment is set up with several docker containers with reverse proxy in front of them, all on a single virtual machine (VM).

Git repo of the whole setup is internally available here: https://gitlab.software.geant.org/TI_Incubator/saml-signature-validation-test-env

Test IdP

Test IdP is SimpleSAMLphp v2.1 instance with configured IdP and installed 'conformance' module (authentication processing filter) which can modify SAML Responses sent to the trusted SPs, with per-configured trust for several test SPs.SAML2 IdP instance with:

  • default authentication source which automatically authenticates a sample (dummy) user
  • Conformance module installed which provides
    • authentication processing filter that can modify SAML Responses, that is, create invalid ones in order to test SP behavior
    • ability to run Nuclei tests from the module UI
    • ability to ask SP contacts for consent for running tests on SP
    • API which enables programmatic control and execution of tests

IdP metadata: https://conformance-idp.maiv1.incubator.geant.org/module.php/saml/idp/metadata

Admin dashboard: https://conformance-idp.maiv1.incubator.geant.org/module.php/admin/

Conformance module UI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/setup

Conformance module repo: https://github.com/cicnavi/simplesamlphp-module-conformance

Test modification endpoints

...

Running tests

Prior to running tests, of course, the SAML trust between the Conformance IdP and particular SP has to be established
(SAML metadata exchanged). Check the "SP metadata handling" on how to add the SP metadata you want to test.

Also, this instance of Conformance IdP is configured to require SP consent to run tests on it. When the test for particular SP is initiated,
Conformance IdP will check if it has consent for it. If not, it will send an invitation to accept the test to the SP contacts emails. Just to showcase
this scenario, a catch-all SMTP service is brought up at https://mailbox.maiv1.incubator.geant.org/, where all the invitations can be
inspected / accepted (emails won't actually be sent to SP contacts).

Manual testing

For manual testing and validation, SP can simply initiate the authentication using the Conformance IdP
(SP initiated flow), in which case the Conformance IdP will present the screen with available test options prior to 
returning the SAML response:

Image Added

After the test selection, Conformance IdP will return the (modified) SAML response so one can observe if the SP is
behaving appropriately.

Running single tests from Conformance module UI

On the other hand, administrator can use the Conformance IdP UI to initiate all tests using the Nuclei tool.
To do that, navigate to SimpleSAMLphp administration area > Configuration > Conformance (Details area) > Run Nuclei Test.

Choose the desired SP and click Run. This will invoke the Nuclei tool in the backend and start testing using the
predefined testing templates, and then stream all of its output to the screen.

After the testing is done, results can be seen on the Nuclei Results page.

Bulk testing

Bulk testing can be executed using [SimpleSAMLphp Cron module](https://github.com/simplesamlphp/simplesamlphp/blob/master/modules/cron/docs/cron.md).
As you can see in Cron documentation, a cron tag can be invoked using HTTP or CLI. Of course, with Conformance testing
using CLI is the preferred way, since testing can take a (relatively) long time depending on the number of SPs. 
However, you are free to test execution using the HTTP version, in which case the maximum execution time
will correspond to the 'max_execution_time' INI setting.

Only one test runner instance can run at given point in time. By maintaining internal state, test runner can first check
if there is another runner active. If yes, the latter will simply exit and let the active test runner do its work.
This way one is free to invoke the cron tag at any time, since only one test runner will ever be active.

For this test environment, feel free to initiate bult testing from the Cron module UI here:

https://conformance-idp.maiv1.incubator.geant.org/module.php/cron/info

Bulk testing uses the same Nuclei logic, and all test results will be available as described with single tests in UI.

SP metadata handling

Conformance IdP is configured with PDO metadata storage handler (it can use database to store SP metadata) in addition to plain PHP metadata files.

Conformance module exposes an HTML form which can be used to manually add additional SP metadata, either by pasting the SP metadata XML or by uploading the metadata XML file. 

The UI form is available here: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/metadata/add

Also, there is an API endpoint which can be used to provision SP metadata dynamically (described below).

API endpoints

API endpoints are protected with Authorization Bearer token. In order to access the API, you must provide the token in the HTTP request as the Authorization header, with Bearer scheme. For example:

GET /resource HTTP/1.1
Host: server.example.com
Authorization: Bearer sometoken


Test setup for next authentication event

Endpoint to define the next test for particular SP. This will determine the shape of the SAML response in the next
authentication event for the given service provider.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/test/setup

HTTP method: GET

Parameters:

  • testId
    • valid values: standardResponse | noSignature | invalidSignature
    • example: noSignature
  • spEntityId
    • valid values: any trusted SP Entity ID
    • example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp


For example, to define specify that the next test for the SP 'urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp' should be the one which that doesn't sign the SAML Response:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/test/setup?testId=noSignature&spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

Running Nuclei tests using predefined templates

Endpoint which can be used to run all Nuclei tests on a particular service provider using the predefined templates.
This is the same as running Nuclei tests from the Conformance module UI. 
The HTTP response is a streamed output from the Nuclei tool.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/run

HTTP method: GET

Parameters:

  • spEntityId
    • valid values: any trusted SP Entity ID
    • example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
  • acsUrl (optional) - if not provided, a default SP ACS URL will be used
    • valid values: ACS URL belonging to the SP defined with spEntityId
    • example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
  • enableDebug (optional) - parameter forwarded to Nuclei which determines if HTTP requests and responses should be shown
    • valid values: 1 (true), 0 (false, default)
    • example: 1
  • enableVerbose (optional) - parameter forwarded to Nuclei which determines if verbose output should be shown
    • valid values: 1 (true), 0 (false, default)
    • example: 1

For example, to run tests for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`, make an
HTTP GET request to:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/run?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

Test results list

Endpoint to fetch test results in JSON format. By default, all results from all SPs will be returned. This can be 
filter out using parameters below.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get

HTTP method: GET

Parameters:

  • spEntityId (optional): can be used to fetch results for particular SP only
    • valid values: any trusted SP Entity ID
    • example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
  • latestOnly (optional): can be used to only fetch latest result per SP
    • valid values: `1` (true), `0` (false, default)
    • example: `1`

For example, to fetch all available results, make an HTTP GET request to:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get

To fetch only latest results for all SPs:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?latestOnly=1

To fetch all results for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

To fetch only the latest result for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp&latestOnly=1

Test result details

Endpoint to fetch particular test result details. This will also return data like full Nuclei findings and Nuclei JSON
report.

URI: https://conformance-idp.example.com/module.php/conformance/nuclei/results/get/{testResultId}

HTTP method: GET

Parameters:

  •  testResultId (route parameter): ID of the specific test result. It can be found in the test results list.
    • valid values: any test result ID from the test results list
    • example: `74`

For example, to fetch the details of test result with an ID of 74, make an HTTP GET request to:

https://conformance-idp.example.com/module.php/conformance/nuclei/results/get/74

Test result images list

Endpoint to list particular test result images. This will return a list of images with their id, name and URI where
they can be fetched from.

URI: https://conformance-idp.example.com/module.php/conformance/nuclei/results/get/{testResultId}/images

HTTP method: GET

Parameters:

  •  testResultId (route parameter): ID of the specific test result. It can be found in the test results list.
    • valid values: any test result ID from the test results list
    • example: `74`

For example, to fetch images list of test result with an ID of 74, make an HTTP GET request to:

https://conformance-idp.example.com/module.php/conformance/nuclei/results/get/74/images

SP metadata provisioning

Endpoint to provision SP metadata which will be trusted by the Conformance IdP.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/metadata/persist

HTTP method: POST

Parameters:

IdP Initiated Login

IdP initiated login can be performed as per SimpleSAMLphp documentation: https://simplesamlphp.org/docs/2.1/simplesamlphp-idp-more.html

Sample URI to initiate login to SP 'urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp':

https://conformance-idp.maiv1.incubator.geant.org/saml2/idp/SSOService.php?spentityid=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

SP Consents

Conformance module has ability to ask SP contacts for consent before running tests for SP.

For testing / demo purposes, in this environment a dummy smtp server was provisioned which is used to catch all outgoing email invitations for consents (emails won't really be sent to  SP contacts).

All emails will be available for inspection at: https://mailbox.maiv1.incubator.geant.org/

Sample SPs and Related Apps

...

SimpleSAMLphp v2.1 instance with configured SPs listed below . It has is featuring a code modification to skip signature checks for 'bad' SP, SPs for simulation purposes.

Admin dashboard: https://simplesamlphp-sp.maiv1.incubator.geant.org/simplesaml/module.php/admin/

...

App: https://simplesamlphp-sp.maiv1.incubator.geant.org/php-app-bad-ssp-sp/

KeyCloak

Instance with 'conformance' realm with two SPs in with different configurations regarding signature checking, available here: https://keycloak.maiv1.incubator.geant.org/

Authentication on either SP can be initiated by going to https://keycloak.maiv1.incubator.geant.org/realms/conformance/account > Personal info > Click on the appropriate IdP / SP  on the "Or sign in with" section (choose good or bad SP). 

Settings

Good SP: https://keycloak.maiv1.incubator.geant.org/admin/master/console/#/conformance/identity-providers/saml/geant-incubator-conformance-idp-with-good-sp/settings

Bad SP: https://keycloak.maiv1.incubator.geant.org/admin/master/console/#/conformance/identity-providers/saml/geant-incubator-conformance-idp-with-bad-sp/settings

Shibboleth

Two different Shibboleth v3 SPs as Docker container instances, one acting as a good SP (checks signature), and one as a bad SP which has signature check disabled using NullSecurity Rule - Service Provider 3 - Confluence (atlassian.net) and XMLSigning Rule - Service Provider 3 - Confluence (atlassian.net) set to false. 

Good SP / App

Metadata: https://shibb-good-sp.maiv1.incubator.geant.org/Shibboleth.sso/Metadata

App: https://shibb-good-sp.maiv1.incubator.geant.org/

TODO configuration

Shibboleth

Bad SP / App

Metadata: https://shibb-bad-sp.maiv1.incubator.geant.org/Shibboleth.sso/Metadata

App: https://shibb-bad-sp.maiv1.incubator.geant.org/TODO