The agreegated picture is here Seamless Access Deployment Architecture
We have four sites in production currently.
We have one Load balaner, two thiss-js servers, two thiss-mdq servers and one Medata Agreegator & Publisher server per site. They are all virtual machines managed by SUNET but have both geographic and network redundancy.
The relationship between these servers in combination with the services in our CDN provider Fastly are described in below diagrams. More details follow for each of these components further in this documentation.
The number of servers and sites in Beta enviornment are limited but has the same relationship between them as Production.
Knowledge of SUNET's puppet & cosmos structure https://wiki.sunet.se/display/sunetops/Cosmos
Fastly is CDN (content delivery network) provider. We use CDN to provide greater rechability by take advantage of their content caching nodes spread all over the world.
We have added our HAproxy load balancers as backends for the SeamlessAccess services in Fastly. The cache nodes of Fastly load balances between our HAproxy servers by forwarding requests from users to them. Over time, the speed and availability of the replies do not depend on our HAproxy load balancers any more as they are saved in nearest Fastly cache servers where the users are located and served to the users with minimum time. In addition to this, Fastly user interface offers statistical data with graphical presentation that shows the hits, misses, errors of the HTTPS requests sent to our HAproxy servers, both in real time and historically. It is good to have a look at this data to see a picture of how much traffic and requests are handled by Fastly and our servers.
Services that are hosted in Fastly are
The configuration of these services reside here https://manage.fastly.com/services/all
For md.seamlessaccess.org, Fastly monitors the status of our are load balancer for thiss-js servers by sending GET /manifest.json requests to them.
For service.seamlessaccess.org, Fastly monitors the status of our load balancer for thiss-mdq servers by sending HEAD /status requests to them.
If a load balancer is marked unhealthy due to health checks, Fastly will stop attempting to send requests to it.
https://docs.fastly.com/en/guides/checking-cache, curl -I -H"Fastly-debug:1" https://service.seamlessaccess.org |
curl -s "https://api.fastly.com/content/edge_check?url=https://service.seamlessaccess.org/990.js" -H 'Fastly-Key:xxx' |
https://wiki.sunet.se/pages/viewpage.action?pageId=83493119
| Name | Location | Env |
|---|---|---|
| meta.aws1.geant.eu.seamlessaccess.org | Frankfurt, AWS | Production |
| meta.aws2.geant.eu.seamlessaccess.org | N. California, AWS | |
| meta.ntx.sunet.eu.seamlessaccess.org | Nutanix, SUNET | |
| meta.se-east.sunet.eu.seamlessaccess.org | STO1v2, Safespring | |
| a-1.thiss.io | STO1v2, Safespring | Beta |
| a-staging-2.thiss.io | STO1v2, Safespring | Staging |
The servers with the name meta.*seamlessaccess.org run PyFF (https://pyff.io) in production environment. In Beta & Staging they are named a-*.thiss.io.
PyFF is short for python Federation Feeder - is a simple SAML metadata aggregator
In this environment, PyFF aggregates metadata from 3 federations - SWAMID, EduGAIN, InCommon & OpenAthens and publish them under /var/www/html/ using the script /usr/local/sbin/run-pyff running as a cronjob.
# Puppet Name: publish */30 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name publish -- /usr/local/sbin/run-pyff /opt/pyff/mdx.fd /var/www/html/metadata.json /var/www/html/metadata_sp.json |
They aggreagate 'general' metadata in /var/www/html/metadata.json and SP trust metadata in /var/www/html/metadata_sp.json. They are created every 30 minutes by running PyFF in a docker container momentarily.
The script also checks manually the fingerprint on the metadata and PyFF does the same thing again.
Read details about the sources and certificates of federation metadata in SeamlessAccess Metadata Feeds.
The servers also runs Apache in a docker container service called sunet-md_publisher to expose and publish the metadata JSON files on port 443 which are accisible only by the servers running thiss-mdq (md-*.seamlessaccess.org) belonging to the same site.
We monitor ages of all the metadata files in https://monitor.seamlessaccess.org/nagios4/. They are
/opt/pyff/metadata//var/www/html/metadata.json/var/www/html/metadata_sp.json.Take help of the 'Description & Troubleshooting' section above to troubleshoot the alarms. Se also GeneralTroubleshooting.
sunet-md_publisher are upgraded by chaging the versions in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml. The puppet manifests for production, beta and staging are separate. thiss::pyff_prod:
pyff_version: 2.1.3
output: /var/www/html/metadata.json
output_trust: /var/www/html/metadata_sp.json
thiss::md_publisher_prod:
watch: /var/www/html/metadata.json
watch_sp: /var/www/html/m |
sunet-md_publisher if you have upgraded the metdata publishing service. See GeneralTroubleshootingthiss-mdq servers with the name md-*.seamlessaccess.org should be able to fetch the metadata from the Aggregator & Publisher servers. Make sure it is all 'green' for those servers too.thiss-mdq servers and run /usr/local/bin/get_metadata.sh and see that they are able to fetch metadata files without any issues.| Name | Location | Env |
|---|---|---|
| md[1-2].aws1.geant.eu.seamlessaccess.org | Frankfurt, AWS | Production |
| md[1-2].aws2.geant.eu.seamlessaccess.org | N. California, AWS | |
| md[1-2].ntx.sunet.eu.seamlessaccess.org | Nutanix, SUNET | |
| md[1-2].se-east.sunet.eu.seamlessaccess.org | STO1v2, Safespring | |
| md[1-2].thiss.io | STO1v2, Safespring | Beta |
md-staging-2.thiss.io | STO1v2, Safespring | Staging |
The servers with the name md-.*seamlessaccess.org run thiss-mdq in production environment. In Beta & Staging they are named md-*.thiss.io.
The code for thiss-mdq lives here https://github.com/TheIdentitySelector/thiss-mdq.
The thiss-mdq is a REST-like API for requesting and receiving arbitrary metadata. It exposes the metadata in the URL https://md.seamlessaccess.org/entities. The URL for beta is https://md.thiss.io/entities.
The aggregator servers running PyFF expose their port 443 to thiss-mdq servers. The thiss-mdq servers run a cronjob that fetch metadata JSON files from Aggregator servers. Next the script checks to see if there are changes in the metadata JSON files by comparing to the local copies under /etc/thiss , if there is, it updates the files in the docker container running thiss-mdq and restart it. The script also does a pre-check to see if the fetched metadata files are empty or not, if empty, it will exit.
# Puppet Name: thiss__mdq_prod_fetch_metadata */5 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name thiss__mdq_prod_fetch_metadata -- /usr/local/bin/get_metadata.sh |
The servers run the thiss-mdq service on port 80 which is only open to the HAproxy load balancers with names md.*.seamlessaccess.org belonging to the same site.
We monitor the date when the metadata JSON files are last modified in both monitor.seamlessaccess.org and nagiosxi.nordu.net. The check warns if the metadata is 2 days old, it becomes critical if it is 5 days old.
We have this check on thiss-mdq servers, on HAproxy Load balancer servers connected to Fastly and on the top Fastly level which is https://md.seamlessaccess.org.
A simple check on the URLs on each level will show information about the metadata. The one on thiss-mdq server level can only be run locally from that server or from the HAproxy server belonging to the same site.
➜ ~ curl https://md.seamlessaccess.org
{"version":"1.5.8","start_time":"2025-11-22T00:35:20.300Z","metadata":{"last_modified":"2025-11-22T14:15:02.808Z","last_created":"2025-11-22T14:15:02.808Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:02.952Z","last_created":"2025-11-22T14:15:02.952Z","size":156}}% |
➜ ~ curl -k https://md.ntx.sunet.eu.seamlessaccess.org
{"version":"1.5.8","start_time":"2025-11-22T00:35:03.460Z","metadata":{"last_modified":"2025-11-22T14:15:02.125Z","last_created":"2025-11-22T14:15:02.125Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:02.212Z","last_created":"2025-11-22T14:15:02.212Z","size":156}}% |
root@md-1: ~ # curl -k http://localhost
{"version":"1.5.8","start_time":"2025-11-22T00:35:05.392Z","metadata":{"last_modified":"2025-11-22T14:15:03.281Z","last_created":"2025-11-22T14:15:03.281Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:03.409Z","last_created":"2025-11-22T14:15:03.409Z","size":156}} |
We also have nagios checks on the accisibility of these web links on each level. Chek also GeneralTroubleshooting.
The process is described in below link along with verification for both production and beta environments.
Seamless Access Software Deployment Guide#Backend(md.seamlessaccess.org)
Seamless Access Software Deployment Guide#Backend(md.thiss.io)
| Name | Location | Env |
|---|---|---|
| static[1-2].aws1.geant.eu.seamlessaccess.org | Frankfurt, AWS | Production |
| static[1-2].aws2.geant.eu.seamlessaccess.org | N. California, AWS | |
| static[1-2].ntx.sunet.eu.seamlessaccess.org | Nutanix, SUNET | |
| md[1-2].se-east.sunet.eu.seamlessaccess.org | STO1v2, Safespring | |
| static[1-2].thiss.io | STO1v2, Safespring | Beta |
static[1-2].aws2.thiss.io | N. California, AWS | Beta |
The servers with the name static-.*seamlessaccess.org run thiss-js in production environment. In Beta, they are named static-*.thiss.io.
The code for thiss-js lives here https://github.com/TheIdentitySelector/thiss-js. The is the code behind the discovery service exposed in the URL https://service.seamlessaccess.org. User search for their login organization here and the search quesries are sent to md.seamlessaccess.org.
The servers run the code in Docker containers. They run the thiss-js service on port 80 which is only open to the HAproxy load balancers with names static.*.seamlessaccess.org belonging to the same site.
We monitor both the version of the code and the accsibility of the service in in both monitor.seamlessaccess.org and nagiosxi.nordu.net.
We have this check on servers running thiss-js, on HAproxy Load balancer servers serving the code to Fastly and on Fastly level which is https://service.seamlessaccess.org.
A simple check on the URLs on each level will show information about the software. The one onthiss-js server level can only be run locally from that server or from the HAproxy server belonging to the same site.
➜ ~ curl https://service.seamlessaccess.org/manifest.json
{
"short_name": "Seamless Access",
"name": "Seamless Access Identity Selector",
"description": "See https://seamlessaccess.org",
"version": "2.1.98"
} |
➜ ~ curl -k https://static.se-east.sunet.eu.seamlessaccess.org/manifest.json
{
"short_name": "Seamless Access",
"name": "Seamless Access Identity Selector",
"description": "See https://seamlessaccess.org",
"version": "2.1.98"
} |
root@static-1: ~ # curl -k -4 http://localhost/manifest.json
{
"short_name": "Seamless Access",
"name": "Seamless Access Identity Selector",
"description": "See https://seamlessaccess.org",
"version": "2.1.160"
} |
We also have nagios checks on the accisibility of these web links on each level. Chekc also GeneralTroubleshooting.
The process is described in below link along with verification for both production and beta environments.
Seamless Access Software Deployment Guide#Frontend(service.seamlessaccess.org)
Seamless Access Software Deployment Guide#Frontend(use.thiss.io)
| Name | Location | Env |
|---|---|---|
| md.aws1.geant.eu.seamlessaccess.org | Frankfurt, AWS | Production |
| md.aws2.geant.eu.seamlessaccess.org | N. California, AWS | |
| md.ntx.sunet.eu.seamlessaccess.org | Nutanix, SUNET | |
| md.se-east.sunet.eu.seamlessaccess.org | STO1v2, Safespring | |
| md-lb.thiss.io | STO1v2, Safespring | Beta |
There is one load balancer server running HAproxy which is placed in front of the two thiss-mdq servers per site. These server have the names md.*.seamlessaccess.org. These HAproxy servers are added in Fastly for the service md.seamlessaccess.org. Fastly forwards the non-cached HTTPS GET requests invoked by the users to one of these HAproxy servers which in turn forwards them to one of the thiss-mdq servers using round robin algorithm. These HTTPS requests handle metadata queires.
The HAproxy service runs in a docker container and the configuration of it is supplied by puppet manifests.
We have three specific checks for these load balancers for each site in https://monitor.seamlessaccess.org/nagios4/
https://<site link>/manifest.jsonhttps://<site link>/statushttp://<site link>:8404/stats. This link is accesible only by SUNET VPN for SUNET NOC members and the monitor server.The site links are
https://md.ntx.sunet.eu.seamlessaccess.org
https://md.se-east.sunet.eu.seamlessaccess.org
https://md.aws2.geant.eu.seamlessaccess.org
https://md.aws1.geant.eu.seamlessaccess.org
Take help of GeneralTroubleshooting for fixing alarms. It may happen that thiss-mdq servers are unavailable which will cause alarm in HAproxy servers, then check the section for thiss-mdq servers to troubleshoot them.
SeamlessAccess HAproxy Upgrade
| Name | Location | Env |
|---|---|---|
| static.aws1.geant.eu.seamlessaccess.org | Frankfurt, AWS | Production |
| static.aws2.geant.eu.seamlessaccess.org | N. California, AWS | |
| static.ntx.sunet.eu.seamlessaccess.org | Nutanix, SUNET | |
| static.se-east.sunet.eu.seamlessaccess.org | STO1v2, Safespring | |
| static.thiss.io | STO1v2, Safespring | Beta |
There is one load balancer server running HAproxy which is placed in front of the two thiss-js servers per site. These server have the names static.*.seamlessaccess.org. These HAproxy servers are added in Fastly for the service service.seamlessaccess.org. Fastly forwards the non-cached HTTPS GET requests invoked by the users from https://service.seamlessaccess.org to one of these HAproxy servers which in turn forwards them to one of the servers running thiss-js code using round robin algorithm.
The HAproxy service runs in a docker container and the configuration of it is supplied by puppet manifests.
We have below checks for these load balancers for each site in https://monitor.seamlessaccess.org/
http://<site link>:8404/stats. This link is accesible only by SUNET VPN for SUNET NOC members and the monitor server.The site links are
https://static.ntx.sunet.eu.seamlessaccess.org
https://static.se-east.sunet.eu.seamlessaccess.org
https://static.aws1.geant.eu.seamlessaccess.org
https://static.aws2.geant.eu.seamlessaccess.org
Take help of GeneralTroubleshooting for fixing alarms. It may happen that thiss-js servers are unavailable which will cause alarm in HAproxy servers, then check the section for thiss-js servers to troubleshoot them.
SeamlessAccess HAproxy Upgrade
| Name | Location | Env |
|---|---|---|
| monitor.ntx.sunet.eu.seamlessaccess.org | STO1v2, Safespring | Production |
This is a monitor server which runs Nagios4 to monitor the health and operations of the virtual servers in Production, Beta and Staging. The GUI is here https://monitor.seamlessaccess.org/.
Infromation regarding access is given here https://wiki.sunet.se/display/sunetops/Monitoring.
service nagios4 status to check the statusservice nagios4 restart to restart the application./var/log/nagios4/nagios.log/etc/nagios4/ are controlled by puppet manifests.No proper guide is available. It is usually upgrade when there's a newer version of Nagios available when we upgrade the OS of the server.
| Name | Location | Env |
|---|---|---|
| sp-test.seamlessaccess.org | STO1v2, Safespring | Mixed |
This server runs Demo SP (service provider) applications for both Production and Beta. They are exposed in respectively https://demo.seamlessaccess.org/ and https://demo.beta.seamlessaccess.org.
The application runs in a docker container.
No specific monitoring is done for this service.
GeneralTroubleshooting can be used for troubleshooting.
By setting the version parameter in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml or in the thiss-ops/global/overlay/etc/puppet/modules/thiss/manifests/demo_sp.pp.
Almost all services run in docker containers. They are addes as systemd units. The names start with sunet-*.
journalctl -fu <service name of the system unit> /var/log/syslog for older logsdocker logs -f <docker container name>service <service name of the system unit> restartFor deeper troubleshooting knowledge of SUNET's puppet & cosmos structure is needed as mentioned in the Prerequisites section above.
The puppet manifests that deploy and manage the internal components are found here https://github.com/TheIdentitySelector/thiss-ops/tree/master/global. Those who have write acces to it are mentioned here https://wiki.sunet.se/pages/viewpage.action?pageId=83493119
SUNET has its own CA, http://ca.sunet.se/infra/, for internal use.
We use them in below servers.
| Servers | Purpose |
|---|---|
| HAproxy Load balancers | For authentication with Fastly |
| Aggregators & Publishers | For authentication with thiss-mdq servers |
We monitor the expiry of these certificates in https://monitor.seamlessaccess.org/
Follow SeamlessAccess SUNET INFRA cert update to update them.
We use SUNET Fleetlock service in our virtual machines to coordinate upgrades/reboots which are controlled by running of cosmos so only a given number of machines do it at the same time.
We have configured so that cosmos can run one at a time in Aggregataors, one thiss-js and one thiss-mdq server per site and two HAproxy servers at a time.
After a successful cosmos run in a server, this application runs specific health checks to see that the service running in that server is healthy and let other servers in the same group run cosmos and perform health checks.
The relevant configurations can be found in https://github.com/TheIdentitySelector/thiss-ops/tree/master
Read about Fleetlock, https://wiki.sunet.se/pages/viewpage.action?pageId=147522142.
The rules are implemented in the servers using nftables. The same rules are mirrored in security groups of Safepsring's openstack platform and in AWS.
Some of the nftable rules are implemented through this puppet manifest https://github.com/TheIdentitySelector/thiss-ops/blob/master/global/overlay/etc/puppet/modules/thiss/manifests/firewall_rules.pp
| Server type | Rules |
|---|---|
| All | SSH via SUNET's designated jump hosts |
| All | NRPE to monitor.seamlessaccess.org & nagiosxi.nordu.net |
| All | Egress/ougoing packets from all ports |
| HAproxy Load Balancer for thiss-js | HTTPS to internet TCP 8404 (HAproxy stats port) to vpn1.sunet.se & monitor.seamlessaccess.org |
| HAproxy Load Balancer for thiss-mdq | HTTPS to internet TCP 8404 (HAproxy stats port) to vpn1.sunet.se & monitor.seamlessaccess.org |
| thiss-js | HTTP to HAproxy Load Balancer for |
| thiss-mdq for Production & Beta | HTTP to HAproxy Load Balancer for |
| thiss-mdq for staging | HTTPS and HTTP to SUNET Load Balancers |
| Aggregator & publishers for Production & Staging | HTTPS to |
| Aggregator & publishers for Beta | HTTPS to |
| Monitor | HTTPS to vpn1.sunet.se HTTP to internet (for ACME challenges to renew Let's Encrypt certificate) |
| Demo Application | HTTPS to internet |
SUNET only hosts the thiss-mdq service for staging which is https://md-staging.thiss.io. It is served by SUNET load balancers.
The Discovery service for Staging which is https://staging.build.thiss.io is not managed by SUNET.
https://wiki.sunet.se/pages/viewpage.action?pageId=83493119
https://wiki.sunet.se/display/sunetops/Monitoring