This document mindmaps pro's and con's for having a test IdP in eduGAIN.
(the sequence does not in any way suggest weight or order)

Test IdP IN eduGAIN mindmap

Pro		Con
If the SP is not in eduGAIN, then support for the SP is unlikely to be provided through the eduGAIN support team or national federations. It must be provided by the test IdP service, and a central service like that is not guaranteed to know the details about any future federation that the SP will join.		Support framework for a testIdP service has not yet been established
		It is unclear which federation would be willing an able to support a test IdP for eduGAIN
		Should an eduGAIN test IdP be concerned with national level requirements?
		Do federations currently understand (or care) about retirements from other federations?
		How likely is it requirements from new federations will (be able to) deviate very much from what is best current practice in eduGAIN? (as that would adversely influence inter-op with eduGAIN)
Allows for testing with currently registered SPs		Many SPs simply bulk load available IdPs from eduGAIN. A test IdP will in that case put a risk to these services
		As a consequence of the above, a test idP on eduGAIN MUST have technical measures to prevent unintentional usage to login to such SPs, but we cannot take responsibility of it goes wrong. Such measure may however create barriers for SPs for testing
		Any SP already in eduGAIN will likely be able to import metadata from a separate test IdP anyway
		Testing new attribute requirements on a production SP is probably not a good idea, best practice is too have dev/qa platforms for that, who may or may not be in eduGAIN
An SP registered in eduGAIN has gone through metadata checks (well-formed, validation, sense-checking). The test IdP would have to duplicate many processes that the federations and eduGAIN already perform.		If we mandate the test IdP should also work without being in eduGAIN, the testIdP must already support metadata checks (well-formed, validation, sense-checking) anyway, though perhaps not to the extend as is done by a national federation.
		We should test on eduGAIN metadata requirements, which should be acceptable for any national federation (as that is just what they already get from eduGAIN itself ). Additional national metadata requirements are considered out of scope.
And note that the UK federation has occasionally had SPs join which can't generate metadata and we have helped them construct it.		We propose to leverage SAML metadata as the bootstrapping of the test Idp relation with the SP.
		We are aware of this issue, and considering assisting the SP with metadata creation in support of SP products that do not or not full support eduGAIN meatdata requirements. Ultimately however, metadata generation is the responsibility of the SP.
If the SP is only integrated through eduGAIN, then it can use a single, well-defined metadata ingest process. Otherwise, you require the test IdP service team to make the metadata integrations; and the SP may end up with two distinct metadata ingest mechanisms (bilateral with the test IdP, multilateral with local federation or eduGAIN)		If we mandate the test IdP should also work without being in eduGAIN, the testIdP must already support metadata checks (well-formed, validation, sense-checking) anyway, though perhaps not to the extend as is done by a national federation. Also unclear how to proceed if national metadata requirement deviate form eduGAIN metadata requirements
Registering with a federation provides good support for entity categories - can we expect SPs to annotate their own metadata appropriately?		This is only only relevant for R&S (I think). In a test environment we might accept R&S as issued by the SP as we are not really interested in the trust aspect of R&S, but just in the ability to exchange attributes in a technically correct way

Open Questions

Should the test IdP feature national federation attributes and entity categories as part of the testing, given that one cannot join eduGAIN without joining a national federation first? Would that also require 'federation policy' to be implemented on a per federation basis?
→ this would be very difficult to maintain centrally, in addition such attributes and entity categories would not be available via other national federations and hence of no value to the SP (in the context of eduGAIN)
What if a national fed metadata requirements exceed eduGAIN metadata requirements?
→ Given that a national fed should already be able to consume eduGAIN metadata (or it would not be able to be make use of eduGAIN at all), may we assume the eduGAIN metadata requirements are the lowest common denominator we will support for SP metadata
What if a national fed does not support preferred eduGAIN metadata elements like e.g. SIRTFI or R&S?
→ If this national fed is the only option for the SP to get into eduGAIN, the test IdP cannot resolve this anyway, so we declare this problem out of scope.
If the test IdP is part of eduGAIN, to what level should it provide protection against unintended use?
→ The current proposal is to only engage with SPs that have explicitly registered their SP with the test IdP and to retire that connection after a given time ( e.g. 1 month)

Conclusion

From feedback received at the public Sprint demo the consensus opinion was the Test IdP should not be a part of eduGAIN, particularly due to the possible security risks. So at this point we have decided not to consider it within eduGAIN and leave any decision about whether to include it and the necessary requirements to do this to the eduGAIN steering committee.