...

Internal Components

Aggregator & Publisher

Descripton & Troubleshooting

The servers with the name meta.*seamlessaccess.org run PyFF (https://pyff.io) in production environment. In Beta & Staging they are named a-*.thiss.io.

PyFF is short for python Federation Feeder - is a simple SAML metadata aggregator

In this environment, PyFF aggregates metadata from 3 federations - SWAMID, EduGAIN, InCommon & OpenAthens and publish them under /var/www/html/ using the script /usr/local/sbin/run-pyff running as a cronjob. 

Servers

Descripton & Troubleshooting

The servers with the name meta.*seamlessaccess.org run PyFF (https://pyff.io) in production environment. In Beta & Staging they are named a-*.thiss.io.

PyFF is short for python Federation Feeder - is a simple SAML metadata aggregator

In this environment, PyFF aggregates metadata from 3 federations - SWAMID, EduGAIN, InCommon & OpenAthens and publish them under /var/www/html/ using the script /usr/local/sbin/run-pyff running as a cronjob. 

# Puppet Name: publish
*/30 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name publish -- /usr/local/sbin/run-pyff /opt/pyff/mdx.fd /var/www/html/metadata.json /var/www/html/metadata_

# Puppet Name: publish
*/30 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name publish -- /usr/local/sbin/run-pyff /opt/pyff/mdx.fd /var/www/html/metadata.json /var/www/html/metadata_sp.json

They aggreagate 'general' metadata in /var/www/html/metadata.json and SP trust metadata in /var/www/html/metadata_sp.json. They are created every 30 minutes by running PyFF in a docker container momentarily. 

...

• After commiting and bump-taging the changes, run cosmos in the concerned servers, better to do it one at a time & check that the service is working.
• If PyFF is upgraded, run the aforementioned cronjob for PyFF to see that it doesn't show any error.
• You have to restart sunet-md_publisher if you have upgraded the metdata publishing service.
• Check https://monitor.seamlessaccess.org/nagios3/ for any alarms.
• The MDQ servers with the name md-*.seamlessaccess.org should be able to fetch the metadata from the Aggregator & Publisher servers. Make sure it is all 'green' for those servers too.
• You can log in to the MDQ servers and run /usr/local/bin/get_metadata.sh and see that they are able to fetch metadata files without any issues.
• As a last & final check, visit any SP for example wiki.sunet.se and see that it is possible to login using SA discovery service or check the login using https://demo.seamlessaccess.org/ for production upgrades and https://demo.beta.seamlessaccess.org for Beta upgrades.

MDQ

Descripton & Troubleshooting

The servers with the name md-.*seamlessaccess.org run thiss-mdq in production environment. In Beta & Staging they are named md-*.thiss.io.

The code for thiss-mdq lives here https://github.com/TheIdentitySelector/thiss-mdq.

The MDQ is a REST-like API for requesting and receiving arbitrary metadata. It exposes the metadata in the URL https://md.seamlessaccess.org/entities. The URL for beta is https://md.thiss.io/entities.

The aggregator servers running PyFF expose their port 443 to MDQ servers. The MDQ servers run a cronjob that fetch metadata JSON files from Aggregator servers. Next the script checks to see if there are changes in the metadata JSON files by comparing to the local copies under /etc/thiss , if there is, it updates the files in the docker container running thiss-mdq and restart it. The script also does a pre-check to see if the fetched metadata files are empty or not, if empty, it will exit. 

# Puppet Name: thiss__mdq_prod_fetch_metadata
*/5 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name thiss__mdq_prod_fetch_metadata -- /usr/local/bin/get_metadata.sh

The servers run the MDQ service on port 80 which is only open to the HAproxy load balancers with names md.*.seamlessaccess.org belonging to the same site.

Mointoring

We monitor the date when the metadata JSON files are last modified in both monitor.seamlessaccess.org and nagiosxi.nordu.net. The check warns if the metadata is 2 days old, it becomes critical if it is 5 days old.

We have this check on MDQ servers, on HAproxy Load balancer servers connected to Fastly and on the top Fastly level which is https://md.seamlessaccess.org.

A simple check on the URLs on each level will show information about the metadata. The one on MDQ server level can only be run locally from that server or from the HAproxy server belonging to the same site.

...

Servers

Descripton & Troubleshooting

The servers with the name md-.*seamlessaccess.org run thiss-mdq in production environment. In Beta & Staging they are named md-*.thiss.io.

The code for thiss-mdq lives here https://github.com/TheIdentitySelector/thiss-mdq.

The MDQ is a REST-like API for requesting and receiving arbitrary metadata. It exposes the metadata in the URL https://md.seamlessaccess.org/entities. The URL for beta is https://md.thiss.io/entities.

The aggregator servers running PyFF expose their port 443 to MDQ servers. The MDQ servers run a cronjob that fetch metadata JSON files from Aggregator servers. Next the script checks to see if there are changes in the metadata JSON files by comparing to the local copies under /etc/thiss , if there is, it updates the files in the docker container running thiss-mdq and restart it. The script also does a pre-check to see if the fetched metadata files are empty or not, if empty, it will exit. 

# Puppet Name: thiss__mdq_prod_fetch_metadata
*/5 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name thiss__mdq_prod_fetch_metadata -- /usr/local/bin/get_metadata.sh

The servers run the MDQ service on port 80 which is only open to the HAproxy load balancers with names md.*.seamlessaccess.org belonging to the same site.

Mointoring

We monitor the date when the metadata JSON files are last modified in both monitor.seamlessaccess.org and nagiosxi.nordu.net. The check warns if the metadata is 2 days old, it becomes critical if it is 5 days old.

We have this check on MDQ servers, on HAproxy Load balancer servers connected to Fastly and on the top Fastly level which is https://md.seamlessaccess.org.

A simple check on the URLs on each level will show information about the metadata. The one on MDQ server level can only be run locally from that server or from the HAproxy server belonging to the same site.

➜  ~ curl -k https://md.ntx.sunet.eu.seamlessaccess.org
{"version":"1.5.8","start_time":"2025-11-22T00:35:0320.460Z300Z","metadata":{"last_modified":"2025-11-22T14:15:02.125Z808Z","last_created":"2025-11-22T14:15:02.125Z808Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:02.212Z952Z","last_created":"2025-11-22T14:15:02.212Z952Z","size":156}}% 

root@md-1:➜  ~ # curl -k httphttps://localhostmd.ntx.sunet.eu.seamlessaccess.org
{"version":"1.5.8","start_time":"2025-11-22T00:35:05.392Z","metadata":{"last_modified":"2025-11-22T14:15:03.281Z","last_created":"2025-11-22T14:15:03.281Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:03.409Z","last_created":"2025-11-22T14:15:03.409Z","size":156}}

We also have nagios checks on the accisibility of these web links on each level.

Upgrade

The process is described in below link along with verification for both production and beta environments.

Seamless Access Software Deployment Guide#Backend(md.seamlessaccess.org) 

Seamless Access Software Deployment Guide#Backend(md.thiss.io)

...

03.460Z","metadata":{"last_modified":"2025-11-22T14:15:02.125Z","last_created":"2025-11-22T14:15:02.125Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:02.212Z","last_created":"2025-11-22T14:15:02.212Z","size":156}}%

root@md-1: ~ # curl -k http://localhost
{"version":"1.5.8","start_time":"2025-11-22T00:35:05.392Z","metadata":{"last_modified":"2025-11-22T14:15:03.281Z","last_created":"2025-11-22T14:15:03.281Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:03.409Z","last_created":"2025-11-22T14:15:03.409Z","size":156}}

We also have nagios checks on the accisibility of these web links on each level.

Upgrade

The process is described in below link along with verification for both production and beta environments.

Seamless Access Software Deployment Guide#Backend(md.seamlessaccess.org) 

Seamless Access Software Deployment Guide#Backend(md.thiss.io)

Thiss-js

Servers

Descripton & Troubleshooting

...

Seamless Access Software Deployment Guide#Frontend(use.thiss.io)

HAproxy Load Balancer

Servers

Descripton & Troubleshooting

It forwards the HTTP GET requests invoked by the users from frontend to one of the MDQ servers using round robin algorithm

Mointoring

Upgrade

SeamlessAccess HAproxy Upgrade

Monitor

Server

Descripton & Troubleshooting

Mointoring

Upgrade

Demo Application

Server

Descripton & Troubleshooting

...