<Incomplete, but the existing information is correct>

Environments

• Production - domain is seamlessaccess.org
• Beta & staging - domain is thiss.io
• Staging - domain is thiss.io

SeamlessAccess Services

• The Discovery Service – service.seamlessaccess.org
• Metadata Query Service– md.seamlessaccess.org
• Thiss Beta Discovery Service – use.thiss.io
• Thiss Beta Metadata Query Service – md.thiss.io
• Staging Metadata Query Service - md-staging.thiss.io
• Demo of the service in production - https://demo.seamlessaccess.org/
• Demo of the service in Beta - https://demo.beta.seamlessaccess.org

...

Take help of the 'Description & Troubleshooting' section above to troubleshoot the alarms. Se also GeneralTroubleshooting.

Upgrade & Verification

• Both PyFF and sunet-md_publisher are upgraded by chaging the versions in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml. The puppet manifests for production, beta and staging are separate.

   thiss::pyff_prod:
      pyff_version: 2.1.3
      output: /var/www/html/metadata.json
      output_trust: /var/www/html/metadata_sp.json
   thiss::md_publisher_prod:
      watch: /var/www/html/metadata.json
      watch_sp: /var/www/html/m

• After commiting and bump-taging the changes, run cosmos in the concerned servers, better to do it one at a time & check that the service is working.
• If PyFF is upgraded, run the aforementioned cronjob for PyFF to see that it doesn't show any error.
• You have to restart sunet-md_publisher if you have upgraded the metdata publishing service. See GeneralTroubleshooting
• Check https://monitor.seamlessaccess.org/nagios3/ for any alarms.
• The thiss-mdq servers with the name md-*.seamlessaccess.org should be able to fetch the metadata from the Aggregator & Publisher servers. Make sure it is all 'green' for those servers too.
• You can log in to the thiss-mdq servers and run /usr/local/bin/get_metadata.sh and see that they are able to fetch metadata files without any issues.
• As a last & final check, visit any SP for example wiki.sunet.se and see that it is possible to login using SA discovery service or check the login using https://demo.seamlessaccess.org/ for production upgrades and https://demo.beta.seamlessaccess.org for Beta upgrades.

thiss-mdq

Servers

...

md-staging-2.thiss.io

...

Descripton & Troubleshooting

The servers with the name md-.*seamlessaccess.org run thiss-mdq in production environment. In Beta & Staging they are named md-*.thiss.io.

The code for thiss-mdq lives here https://github.com/TheIdentitySelector/thiss-mdq.

The thiss-mdq is a REST-like API for requesting and receiving arbitrary metadata. It exposes the metadata in the URL https://md.seamlessaccess.org/entities. The URL for beta is https://md.thiss.io/entities.

The aggregator servers running PyFF expose their port 443 to thiss-mdq servers. The thiss-mdq servers run a cronjob that fetch metadata JSON files from Aggregator servers. Next the script checks to see if there are changes in the metadata JSON files by comparing to the local copies under /etc/thiss , if there is, it updates the files in the docker container running thiss-mdq and restart it. The script also does a pre-check to see if the fetched metadata files are empty or not, if empty, it will exit. 

# Puppet Name: thiss__mdq_prod_fetch_metadata
*/5 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name thiss__mdq_prod_fetch_metadata -- /usr/local/bin/get_metadata.sh

The servers run the thiss-mdq service on port 80 which is only open to the HAproxy load balancers with names md.*.seamlessaccess.org belonging to the same site.

Mointoring & Troubleshooting

We monitor the date when the metadata JSON files are last modified in both monitor.seamlessaccess.org and nagiosxi.nordu.net. The check warns if the metadata is 2 days old, it becomes critical if it is 5 days old.

We have this check on thiss-mdq servers, on HAproxy Load balancer servers connected to Fastly and on the top Fastly level which is https://md.seamlessaccess.org.

A simple check on the URLs on each level will show information about the metadata. The one on thiss-mdq server level can only be run locally from that server or from the HAproxy server belonging to the same site.

...

We run docker.sunet.se/pyff as docker image. ci.sune.se (Jenkins)  builds that image every day with newer versions of dependencies and push it to docker.sunet.se.

We should make sure to run an image that is manually tagged specially in production and has been tested beforehand either in SeamlessAcess beta environment or in other test environments in SUNET. We should not run an image with a tag that is automatically added by ci.sunet.se. By running a manual tag that has been tested, we do not risk running an image with dependencies that may not be compabile with underlying code for pyFF. Currently we are running a 'stable' tag that has been tested in EIDAS project.

• Both PyFF and sunet-md_publisher are upgraded by chaging the versions in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml. The puppet manifests for production, beta and staging are separate.

   thiss::pyff_prod:
      pyff_version: eidas-2.1.3-stable
      output: /var/www/html/metadata.json
      output_trust: /var/www/html/metadata_sp.json
   thiss::md_publisher_prod:
      watch: /var/www/html/metadata.json
      watch_sp: /var/www/html/m

• After commiting and bump-taging the changes, run cosmos in the concerned servers, better to do it one at a time & check that the service is working.
• If PyFF is upgraded, run the aforementioned cronjob for PyFF to see that it doesn't show any error.
• You have to restart sunet-md_publisher if you have upgraded the metdata publishing service. See GeneralTroubleshooting
• Check https://monitor.seamlessaccess.org/nagios3/ for any alarms.
• The thiss-mdq servers with the name md-*.seamlessaccess.org should be able to fetch the metadata from the Aggregator & Publisher servers. Make sure it is all 'green' for those servers too.
• You can log in to the thiss-mdq servers and run /usr/local/bin/get_metadata.sh and see that they are able to fetch metadata files without any issues.
• As a last & final check, visit any SP for example wiki.sunet.se and see that it is possible to login using SA discovery service or check the login using https://demo.seamlessaccess.org/ for production upgrades and https://demo.beta.seamlessaccess.org for Beta upgrades.

thiss-mdq

Servers

Descripton & Troubleshooting

The servers with the name md-.*seamlessaccess.org run thiss-mdq in production environment. In Beta & Staging they are named md-*.thiss.io.

The code for thiss-mdq lives here https://github.com/TheIdentitySelector/thiss-mdq.

The thiss-mdq is a REST-like API for requesting and receiving arbitrary metadata. It exposes the metadata in the URL https://md.seamlessaccess.org/entities. The URL for beta is https://md.thiss.io/entities.

The aggregator servers running PyFF expose their port 443 to thiss-mdq servers. The thiss-mdq servers run a cronjob that fetch metadata JSON files from Aggregator servers. Next the script checks to see if there are changes in the metadata JSON files by comparing to the local copies under /etc/thiss , if there is, it updates the files in the docker container running thiss-mdq and restart it. The script also does a pre-check to see if the fetched metadata files are empty or not, if empty, it will exit. 

# Puppet Name: thiss__mdq_prod_fetch_metadata
*/5 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name thiss__mdq_prod_fetch_metadata -- /usr/local/bin/get_metadata.sh

The servers run the thiss-mdq service on port 80 which is only open to the HAproxy load balancers with names md.*.seamlessaccess.org belonging to the same site.

Mointoring & Troubleshooting

We monitor the date when the metadata JSON files are last modified in both monitor.seamlessaccess.org and nagiosxi.nordu.net. The check warns if the metadata is 2 days old, it becomes critical if it is 5 days old.

We have this check on thiss-mdq servers, on HAproxy Load balancer servers connected to Fastly and on the top Fastly level which is https://md.seamlessaccess.org.

A simple check on the URLs on each level will show information about the metadata. The one on thiss-mdq server level can only be run locally from that server or from the HAproxy server belonging to the same site.

➜  ~ curl https://md.seamlessaccess.org
{"version":"1.5.8","start_time":"2025-11-22T00:35:20.300Z","metadata":{"last_modified":"2025-11-22T14:15:02.808Z","last_created":"2025-11-22T14:15:02.808Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:02.952Z","last_created":"2025-11-22T14:15:02.952Z","size":156}}% 

➜  ~ curl -k https://md.ntx.sunet.eu.seamlessaccess.org
{"version":"1.5.8","start_time":"2025-11-22T00:35:03.460Z","metadata":{"last_modified":"2025-11-22T14:15:02.125Z","last_created":"2025-11-22T14:15:02.125Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:02.212Z","last_created":"2025-11-22T14:15:02.212Z","size":156}}%

root@md-1: ~ # curl -k http://localhost
{"version":"1.5.8","start_time":"2025-11-22T00:35:05.392Z","metadata":{"last_modified":"2025-11-22T14:15:03.281Z","last_created":"2025-11-22T14:15:03.281Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:03.409Z","last_created":"2025-11-22T14:15:03.409Z","size":156}}

We also have nagios checks on the accisibility of these web links on each level. Chek also GeneralTroubleshooting.

Upgrade & verification

The process is described in below link along with verification for both production and beta environments.

Seamless Access Software Deployment Guide#Backend(md.seamlessaccess.org) 

Seamless Access Software Deployment Guide#Backend(md.thiss.io)

Thiss-js

Servers

Descripton 

The servers with the name static-.*seamlessaccess.org run thiss-js in production environment. In Beta, they are named static-*.thiss.io.

The code for thiss-js lives here https://github.com/TheIdentitySelector/thiss-js. The is the code behind the discovery service exposed in the URL https://service.seamlessaccess.org. User search for their login organization here and the search quesries are sent to md.seamlessaccess.org.

The servers run the code in Docker containers. They run the thiss-js service on port 80 which is only open to the HAproxy load balancers with names static.*.seamlessaccess.org belonging to the same site.

Mointoring & Troubleshooting

We monitor both the version of the code and the accsibility of the service in in both monitor.seamlessaccess.org and nagiosxi.nordu.net.

We have this check on servers running thiss-js, on HAproxy Load balancer servers serving the code to Fastly and on Fastly level which is https://service.seamlessaccess.org.

A simple check on the URLs on each level will show information about the software. The one onthiss-js server level can only be run locally from that server or from the HAproxy server belonging to the same site.

...

➜  ~ curl -k https://md.ntx.sunet.eu.seamlessaccess.org
{"version":"1.5.8","start_time":"2025-11-22T00:35:03.460Z","metadata":{"last_modified":"2025-11-22T14:15:02.125Z","last_created":"2025-11-22T14:15:02.125Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:02.212Z","last_created":"2025-11-22T14:15:02.212Z","size":156}}%

service.seamlessaccess.org/manifest.json
{
  "short_name": "Seamless Access",
  "name": "Seamless Access Identity Selector",
  "description": "See https://seamlessaccess.org",
  "version": "2.1.98"
} 

➜  ~root@md-1: ~ # curl -k httphttps://static.se-east.sunet.eu.seamlessaccess.org/manifest.json
{
  "short_name": "Seamless Access",
  "name": "Seamless Access Identity Selector",
  "description": "See https://seamlessaccess.org",
  "version": "2.1.98"
}

root@static-1: ~ # curl -k -4 http://localhost/manifest.json
{
  "short_name": "Seamless Access",
  "name": "Seamless Access Identity Selector",
  "description": "See https://seamlessaccess.org",
  "version": "2.1.160"
localhost
{"version":"1.5.8","start_time":"2025-11-22T00:35:05.392Z","metadata":{"last_modified":"2025-11-22T14:15:03.281Z","last_created":"2025-11-22T14:15:03.281Z","size":17502},"trust_metadata":{"last_modified":"2025-11-22T14:15:03.409Z","last_created":"2025-11-22T14:15:03.409Z","size":156}}

We also have nagios checks on the accisibility of these web links on each level. Chek also Chekc also GeneralTroubleshooting.

Upgrade &

...

verification 

The process is described in below link along with verification for both production and beta environments.

Seamless Access Software Deployment Guide#BackendGuide#Frontend(mdservice.seamlessaccess.org) 

Seamless Access Software Deployment Guide#Backend(md.thiss.io)

Thiss-js

Servers

...

static[1-2].aws2.thiss.io

...

Descripton 

The servers with the name static-.*seamlessaccess.org run thiss-js in production environment. In Beta, they are named static-*.thiss.io.

The code for thiss-js lives here https://github.com/TheIdentitySelector/thiss-js. The is the code behind the discovery service exposed in the URL https://service.seamlessaccess.org. User search for their login organization here and the search quesries are sent to md.seamlessaccess.org.

The servers run the code in Docker containers. They run the thiss-js service on port 80 which is only open to the HAproxy load balancers with names static.*.seamlessaccess.org belonging to the same site.

Mointoring & Troubleshooting

We monitor both the version of the code and the accsibility of the service in in both monitor.seamlessaccess.org and nagiosxi.nordu.net.

We have this check on servers running thiss-js, on HAproxy Load balancer servers serving the code to Fastly and on Fastly level which is https://service.seamlessaccess.org.

A simple check on the URLs on each level will show information about the software. The one onthiss-js server level can only be run locally from that server or from the HAproxy server belonging to the same site.

➜  ~ curl https://service.seamlessaccess.org/manifest.json
{
  "short_name": "Seamless Access",
  "name": "Seamless Access Identity Selector",
  "description": "See https://seamlessaccess.org",
  "version": "2.1.98"
} 

...

Guide#Frontend(use.thiss.io)

HAproxy Load Balancer for thiss-mdq

Servers

Descripton 

There is one load balancer server running HAproxy which is placed in front of the two thiss-mdq servers per site. These server have the names md.*.seamlessaccess.org. These HAproxy servers are added in Fastly for the service md.seamlessaccess.org. Fastly forwards the non-cached HTTPS GET requests invoked by the users to one of these HAproxy servers which in turn forwards them to one of the thiss-mdq servers using round robin algorithm. These HTTPS requests handle metadata queires.

The HAproxy service runs in a docker container and the configuration of it is supplied by puppet manifests.

Mointoring & Troubleshooting

We have three specific checks for these load balancers for each site in https://monitor.seamlessaccess.org/nagios4/

1. Monitor the date when the metadata JSON files are last modified from the https://<site link>/manifest.json
2. SSL check and availability of the site links
3. The string 'OK' is found in https://<site link>/status
4. Monitor that both backends are up by checking HAproxy stats from http://<site link>:8404/stats. This link is accesible only by SUNET VPN for SUNET NOC members and the monitor server.

The site links are

https://md.ntx.sunet.eu.seamlessaccess.org

https://md.se-east.sunet.eu.seamlessaccess.org

https://md.aws2.geant.eu.seamlessaccess.org

https://md.aws1.geant.eu.seamlessaccess.org

Take help of GeneralTroubleshooting for fixing alarms. It may happen that thiss-mdq servers are unavailable which will cause alarm in HAproxy servers, then check the section for thiss-mdq servers to troubleshoot them.

Upgrade

SeamlessAccess HAproxy Upgrade

HAproxy Load Balancer for thiss-js

Servers

...

...

Descripton & Troubleshooting

There is one load balancer server running HAproxy which is placed in front of the two thiss-js servers per site. These server have the names static.*.seamlessaccess.org. These HAproxy servers are added in Fastly for the service service.seamlessaccess.org. Fastly forwards the non-cached HTTPS GET requests invoked by the users from https://service.seamlessaccess.org to one of these HAproxy servers which in turn forwards them to one of the servers running  thiss-js code using round robin algorithm.

The HAproxy service runs in a docker container and the configuration of it is supplied by puppet manifests.

Mointoring & Troubleshooting

We have below checks for these load balancers for each site in https://monitor.seamlessaccess.org/

1. SSL check and availability of the site links
2. Monitor that both backends are up by checking HAproxy stats from http://<site link>:8404/stats. This link is accesible only by SUNET VPN for SUNET NOC members and the monitor server.

The site links are

https://static

root@static-1: ~ # curl -k -4 http://localhost/manifest.json
{
  "short_name": "Seamless Access",
  "name": "Seamless Access Identity Selector",
  "description": "See https://seamlessaccess.org",
  "version": "2.1.160"
}

We also have nagios checks on the accisibility of these web links on each level. Chekc also GeneralTroubleshooting.

Upgrade & verification 

The process is described in below link along with verification for both production and beta environments.

Seamless Access Software Deployment Guide#Frontend(service.seamlessaccess.org)

Seamless Access Software Deployment Guide#Frontend(use.thiss.io)

HAproxy Load Balancer for thiss-mdq

Servers

...

.ntx.sunet.eu.seamlessaccess.org

...

https://static.se-east.sunet.eu.seamlessaccess.org

...

https://static.aws1.geant.eu.seamlessaccess.org

https://static.aws2.geant.eu.seamlessaccess.org

Take help of GeneralTroubleshooting for fixing alarms. It may happen that thiss-js servers are unavailable which will cause alarm in HAproxy servers, then check the section for thiss-js servers to troubleshoot them.

Upgrade

SeamlessAccess HAproxy Upgrade

Monitor

Server

Descripton

This is a monitor server which runs Nagios4 to monitor the health and operations of the virtual servers in Production, Beta and Staging. The GUI is here

Descripton 

There is one load balancer server running HAproxy which is placed in front of the two thiss-mdq servers per site. These server have the names md.*.seamlessaccess.org. These HAproxy servers are added in Fastly for the service md.seamlessaccess.org. Fastly forwards the non-cached HTTPS GET requests invoked by the users to one of these HAproxy servers which in turn forwards them to one of the thiss-mdq servers using round robin algorithm. These HTTPS requests handle metadata queires.

The HAproxy service runs in a docker container and the configuration of it is supplied by puppet manifests.

Mointoring & Troubleshooting

We have three specific checks for these load balancers for each site in https://monitor.seamlessaccess.org/nagios4/

1. Monitor the date when the metadata JSON files are last modified from the https://<site link>/manifest.json
2. SSL check and availability of the site links
3. The string 'OK' is found in https://<site link>/status
4. Monitor that both backends are up by checking HAproxy stats from http://<site link>:8404/stats. This link is accesible only by SUNET VPN for SUNET NOC members and the monitor server.

The site links are

https://md.ntx.sunet.eu.seamlessaccess.org

https://md.se-east.sunet.eu.seamlessaccess.org

https://md.aws2.geant.eu.seamlessaccess.org

https://md.aws1.geant.eu.seamlessaccess.org

Take help of GeneralTroubleshooting for fixing alarms. It may happen that thiss-mdq servers are unavailable which will cause alarm in HAproxy servers, then check the section for thiss-mdq servers to troubleshoot them.

Upgrade

SeamlessAccess HAproxy Upgrade

HAproxy Load Balancer for thiss-js

Servers

...

.

Infromation regarding access is given here https://wiki.sunet.se/display/sunetops/Monitoring.

Mointoring & Troubleshooting

• Run service nagios4 status to check the status
• Run service nagios4 restart to restart the application.
• Check logs in /var/log/nagios4/nagios.log
• Many of the configurations from /etc/nagios4/ are controlled by puppet manifests.

Upgrade

No proper guide is available. It is usually upgrade when there's a newer version of Nagios available when we upgrade the OS of the server.

Demo Application

Server

Descripton

This server runs Demo SP (service provider) applications for both Production and Beta. They are exposed in respectively https://demo.seamlessaccess.org/ and https://demo.beta.seamlessaccess.org.

The application runs in a docker container.

Mointoring & Troubleshooting

No specific monitoring is done for this service.

GeneralTroubleshooting can be used for troubleshooting.

Upgrade

By setting the version parameter in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml or in the thiss-ops/global/overlay/etc/puppet/modules/thiss/manifests/demo_sp.pp.

Log

Server

Descripton

The servers runs a syslog application to collect logs from service.seamlessaccess.org. The server is specifically allowed in Fastly configuration, you can check that under Logging for the current version of service.seamlessacces.org configuration running in Fastly.

We have added Enrique Perez's SSH key and IP address so he can fetch the logs from under /var/log with the names sa.log. 

This is how it looks in /root/.ssh/authorized_keys of the server.

command="/usr/bin/rrsync -ro /var/log/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-ed25519 AAAAB3NzaC1yc2EAAAADAQABAAABAQDWOTGSoPh/+uNglvrLifb4jVhDLzGnAQlH3jagVnWFQKVieUNB2vlhrTtW/89+9uRUtjICa1gevGxICkavgaP8MIvOrgksgR+j+CakbwKe1gGmC5AqFb1kmbUOpeUrGDHYbWp46fOc0zTBxTqT1u93LAw/ZUHUMB3ETnmScrbvxC3JwA0qsU7bw73QCLM24epy8dvstFTLcNPcPC2TOCh86IkZpvJj38Hy5uqanWN6KceOtQBtOORJE6rAsBTpmhiVtE/AsvkEWKNk1g5uArULK/Dd6K7fMxkr0rv+YT9qot/z0xUqHe5RDn3E5w3ojV8x47/0V9l3eh9jrEf3l6u9 -var-log--command_key

There is a configuration in logrotate so the sa.log(s) are rotated for 30 days and will be removed afterwards.

Mointoring & Troubleshooting

• Check /var/log/syslog if there's any issue with access for Enrique or any issue with rsyslog functionality.
• Take help of applicable puppet manifests to understand the configuration and troubleshoot further.
• Check in Fastly if there's any warning message in the service configuration for Logging for service.seamlessaccess.org.

General Troubleshooting

Almost all services run in docker containers. They are addes as systemd units. The names start with sunet-*.

• journalctl -fu <service name of the system unit>
• Check /var/log/syslog for older logs
• docker logs -f <docker container name>
• service <service name of the system unit> restart

For deeper troubleshooting knowledge of SUNET's puppet & cosmos structure is needed as mentioned in the Prerequisites section above.

The puppet manifests that deploy and manage the internal components are found here https://github.com/TheIdentitySelector/thiss-ops/tree/master/global. Those who have write acces to it are mentioned here https://wiki.sunet.se/pages/viewpage.action?pageId=83493119

Use of SUNET INFRA cert

SUNET has its own CA, http://ca.sunet.se/infra/, for internal use.

We use them in below servers.

We monitor the expiry of these certificates in https://monitor.seamlessaccess.org/

Follow SeamlessAccess SUNET INFRA cert update to update them.

Use of Fleetlock

We use SUNET Fleetlock service in our virtual machines to coordinate upgrades/reboots which are controlled by running of cosmos so only a given number of machines do it at the same time.

We have configured so that cosmos can run one at a time in Aggregataors, one thiss-js and one thiss-mdq server per site and two HAproxy servers at a time.

After a successful cosmos run in a server, this application runs specific health checks to see that the service running in that server is healthy and let other servers in the same group run cosmos and perform health checks.

The relevant configurations can be found in https://github.com/TheIdentitySelector/thiss-ops/tree/master

Read about Fleetlock, https://wiki.sunet.se/pages/viewpage.action?pageId=147522142.

Firewall Restrictions

The rules are implemented in the servers using nftables. The same rules are mirrored in security groups of Safepsring's openstack platform and in AWS.

Some of the nftable rules are implemented through this puppet manifest

Descripton & Troubleshooting

There is one load balancer server running HAproxy which is placed in front of the two thiss-js servers per site. These server have the names static.*.seamlessaccess.org. These HAproxy servers are added in Fastly for the service service.seamlessaccess.org. Fastly forwards the non-cached HTTPS GET requests invoked by the users from https://service.seamlessaccess.org to one of these HAproxy servers which in turn forwards them to one of the servers running  thiss-js code using round robin algorithm.

The HAproxy service runs in a docker container and the configuration of it is supplied by puppet manifests.

Mointoring & Troubleshooting

We have below checks for these load balancers for each site in https://monitor.seamlessaccess.org/

1. SSL check and availability of the site links
2. Monitor that both backends are up by checking HAproxy stats from http://<site link>:8404/stats. This link is accesible only by SUNET VPN for SUNET NOC members and the monitor server.

The site links are

https://static.ntx.sunet.eu.seamlessaccess.org

https://static.se-east.sunet.eu.seamlessaccess.org

https://static.aws1.geant.eu.seamlessaccess.org

https://static.aws2.geant.eu.seamlessaccess.org

Take help of GeneralTroubleshooting for fixing alarms. It may happen that thiss-js servers are unavailable which will cause alarm in HAproxy servers, then check the section for thiss-js servers to troubleshoot them.

Upgrade

SeamlessAccess HAproxy Upgrade

Monitor

Server

Descripton

This is a monitor server which runs Nagios4 to monitor the health and operations of the virtual servers in Production, Beta and Staging. The GUI is here https://monitor.seamlessaccess.org/.

Infromation regarding access is given here https://wiki.sunet.se/display/sunetops/Monitoring.

Mointoring & Troubleshooting

• Run service nagios4 status to check the status
• Run service nagios4 restart to restart the application.
• Check logs in /var/log/nagios4/nagios.log
• Many of the configurations from /etc/nagios4/ are controlled by puppet manifests.

Upgrade

No proper guide is available. It is usually upgrade when there's a newer version of Nagios available when we upgrade the OS of the server.

Demo Application

Server

Descripton

This server runs Demo SP (service provider) applications for both Production and Beta. They are exposed in respectively https://demo.seamlessaccess.org/ and https://demo.beta.seamlessaccess.org.

The application runs in a docker container.

Mointoring & Troubleshooting

No specific monitoring is done for this service.

GeneralTroubleshooting can be used for troubleshooting.

Upgrade

By setting the version parameter in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml or in the thiss-ops/global/overlay/etc/puppet/modules/thiss/manifests/demo_sp.pp.

General Troubleshooting

Almost all services run in docker containers. They are addes as systemd units. The names start with sunet-*.

• journalctl -fu <service name of the system unit>
• Check /var/log/syslog for older logs
• docker logs -f <docker container name>
• service <service name of the system unit> restart

For deeper troubleshooting knowledge of SUNET's puppet & cosmos structure is needed as mentioned in the Prerequisites section above.

The puppet manifests that deploy and manage the internal components are found here https://github.com/TheIdentitySelector/thiss-ops/treeblob/master/global. Those who have write acces to it are mentioned here https://wiki.sunet.se/pages/viewpage.action?pageId=83493119

Use of SUNET INFRA cert

SUNET has its own CA, http://ca.sunet.se/infra/, for internal use.

We use them in below servers.

...

/global/overlay/etc/puppet/modules/thiss/manifests/firewall_rules.pp

We monitor the expiry of these certificates in https://monitor.seamlessaccess.org/

Follow SeamlessAccess SUNET INFRA cert update to update them.

Use of Fleetlock

We use SUNET Fleetlock service in our virtual machines to coordinate upgrades/reboots which are controlled by running of cosmos so only a given number of machines do it at the same time.

We have configured so that cosmos can run one at a time in Aggregataors, one thiss-js and one thiss-mdq server per site and two HAproxy servers at a time.

After a successful cosmos run in a server, this application runs specific health checks to see that the service running in that server is healthy and let other servers in the same group run cosmos and perform health checks.

The relevant configurations can be found in https://github.com/TheIdentitySelector/thiss-ops/tree/master

Read about Fleetlock, https://wiki.sunet.se/pages/viewpage.action?pageId=147522142.

...

Staging Metadata Service

SUNET only hosts the MDQ thiss-mdq service for staging which is https://md-staging.thiss.io. It is served by SUNET load balancers.

...