...
- default authentication source which automatically authenticates a sample (dummy) user
- Conformance module installed which provides
- authentication processing filter that can modify SAML Responses, that is, create invalid ones in order to test SP behavior
- ability to run Nuclei tests from the module UI
- ability to ask SP contacts for consent for running tests on SP
- API which enables programmatic control and execution of tests
...
Conformance module repo: https://github.com/cicnavi/simplesamlphp-module-conformance
Running tests
Prior to running tests, of course, the SAML trust between the Conformance IdP and particular SP has to be established
(SAML metadata exchanged). Check the "SP metadata handling" on how to add the SP metadata you want to test.
Also, this instance of Conformance IdP is configured to require SP consent to run tests on it. When the test for particular SP is initiated,
Conformance IdP will check if it has consent for it. If not, it will send an invitation to accept the test to the SP contacts emails. Just to showcase
this scenario, a catch-all SMTP service is brought up at https://mailbox.maiv1.incubator.geant.org/, where all the invitations can be
inspected / accepted (emails won't actually be sent to SP contacts).
Manual testing
For manual testing and validation, SP can simply initiate the authentication using the Conformance IdP
(SP initiated flow), in which case the Conformance IdP will present the screen with available test options prior to
returning the SAML response:
After the test selection, Conformance IdP will return the (modified) SAML response so one can observe if the SP is
behaving appropriately.
Running single tests from Conformance module UI
On the other hand, administrator can use the Conformance IdP UI to initiate all tests using the Nuclei tool.
To do that, navigate to SimpleSAMLphp administration area > Configuration > Conformance (Details area) > Run Nuclei Test.
Choose the desired SP and click Run. This will invoke the Nuclei tool in the backend and start testing using the
predefined testing templates, and then stream all of its output to the screen.
After the testing is done, results can be seen on the Nuclei Results page.
Bulk testing
Bulk testing can be executed using [SimpleSAMLphp Cron module](https://github.com/simplesamlphp/simplesamlphp/blob/master/modules/cron/docs/cron.md).
As you can see in Cron documentation, a cron tag can be invoked using HTTP or CLI. Of course, with Conformance testing
using CLI is the preferred way, since testing can take a (relatively) long time depending on the number of SPs.
However, you are free to test execution using the HTTP version, in which case the maximum execution time
will correspond to the 'max_execution_time' INI setting.
Only one test runner instance can run at given point in time. By maintaining internal state, test runner can first check
if there is another runner active. If yes, the latter will simply exit and let the active test runner do its work.
This way one is free to invoke the cron tag at any time, since only one test runner will ever be active.
For this test environment, feel free to initiate bult testing from the Cron module UI here:
https://conformance-idp.maiv1.incubator.geant.org/module.php/cron/info
Bulk testing uses the same Nuclei logic, and all test results will be available as described with single tests in UI.
SP metadata handling
Conformance IdP is configured with PDO metadata storage handler (it can use database to store SP metadata) in addition to plain PHP metadata files.
...
GET /resource HTTP/1.1
Host: server.example.com
Authorization: Bearer sometoken
Test
...
setup for next authentication event
Endpoint to define the next test for particular SP. This will determine the shape of the SAML response in the next
authentication event for the given service provider.
URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/test/setup
HTTP method: GET
Parameters:
...
For example, to specify that the next test for the SP 'urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp' should be the one that doesn't sign the SAML Response:
https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/test/setup?testId=noSignature&spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
Running Nuclei tests using predefined templates
Endpoint which can be used to run all Nuclei tests on a particular service provider using the predefined templates.
This is the same as running Nuclei tests from the Conformance module UI.
The HTTP response is a streamed output from the Nuclei tool.
URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/run
HTTP method: GET
Parameters:
- spEntityId
- valid values: any trusted SP Entity ID
- example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
- acsUrl (optional) - if not provided, a default SP ACS URL will be used
- valid values: ACS URL belonging to the SP defined with spEntityId
- example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
- enableDebug (optional) - parameter forwarded to Nuclei which determines if HTTP requests and responses should be shown
- valid values: 1 (true), 0 (false, default)
- example: 1
- enableVerbose (optional) - parameter forwarded to Nuclei which determines if verbose output should be shown
- valid values: 1 (true), 0 (false, default)
- example: 1
For example, to run tests for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`, make an
HTTP GET request to:
https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/run?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
Test results list
Endpoint to fetch test results in JSON format. By default, all results from all SPs will be returned. This can be
filter out using parameters below.
URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get
HTTP method: GET
Parameters:
- spEntityId (optional): can be used to fetch results for particular SP only
- valid values: any trusted SP Entity ID
- example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
- latestOnly (optional): can be used to only fetch latest result per SP
- valid values: `1` (true), `0` (false, default)
- example: `1`
For example, to fetch all available results, make an HTTP GET request to:
https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get
To fetch only latest results for all SPs:
https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?latestOnly=1
To fetch all results for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`:
https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
To fetch only the latest result for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`:
https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp&latestOnly=1
Test result details
Endpoint to fetch particular test result details. This will also return data like full Nuclei findings and Nuclei JSON
report.
URI: https://conformance-idp.example.com/module.php/conformance/nuclei/results/get/{testResultId}
HTTP method: GET
Parameters:
- testResultId (route parameter): ID of the specific test result. It can be found in the test results list.
- valid values: any test result ID from the test results list
- example: `74`
For example, to fetch the details of test result with an ID of 74, make an HTTP GET request to:
https://conformance-idp.example.com/module.php/conformance/nuclei/results/get/74
Test result images list
Endpoint to list particular test result images. This will return a list of images with their id, name and URI where
they can be fetched from.
URI: https://conformance-idp.example.com/module.php/conformance/nuclei/results/get/{testResultId}/images
HTTP method: GET
Parameters:
- testResultId (route parameter): ID of the specific test result. It can be found in the test results list.
- valid values: any test result ID from the test results list
- example: `74`
For example, to fetch images list of test result with an ID of 74, make an HTTP GET request to:
https://conformance-idp.example.com/module.php/conformance/nuclei/results/get/74/images
SP metadata provisioning
Endpoint to provision SP metadata which will be trusted by the Conformance IdP.
...
SP Consents
Conformance module has ability to ask SP contacts for consent before running tests for SP.
For testing / demo purposes, in this environment a dummy smtp server was provisioned which is used to catch all outgoing email invitations for consents (emails won't really be sent to SP contacts).
All emails will be available for inspection at: https://mailbox.maiv1.incubator.geant.org/
Sample SPs and Related Apps
...