...

• The Discovery Service – service.seamlessaccess.org
• Metadata Query Service– md.seamlessaccess.org
• Thiss Beta Discovery Service – use.thiss.io
• Thiss Beta Metadata Query Service – md.thiss.io
• Staging Metadata Query Service - md-staging.thiss.io
• Demo of the service in production - https://demo.seamlessaccess.org/
• Demo of the service in Beta - https://demo.beta.seamlessaccess.org

...

The configuration of these services reside here https://manage.fastly.com/services/all

Fastly monitors the status of our load balancer of thiss-js servers by sending GET /manifest.json requests to them.

Fastly monitors the status of our load balancer of thiss-js servers by sending GET /manifest.json requests to them.

Troubleshooting

• How to check cache for an URL- 

...

https://wiki.sunet.se/pages/viewpage.action?pageId=83493119

Internal Components

General Troubleshooting

Almost all services from internal components run in docker containers. They are addes as systemd units. The names start with sunet-*.

• journalctl -fu <service name of the system unit>
• Check /var/log/syslog for older logs
• docker logs -f <docker container name>
• service <service name of the system unit> restart

For deeper troubleshooting knowledge of SUNET's puppet & cosmos structure is needed as mentioned in the Prerequisites section above.

The puppet manifests that deploy and manage the internal components are found here https://github.com/TheIdentitySelector/thiss-ops/tree/master/global. Those who have write acces to it are mentioned here https://wiki.sunet.se/pages/viewpage.action?pageId=83493119

Aggregator & Publisher

Servers

...

Aggregator & Publisher

Servers

Descripton 

The servers with the name meta.*seamlessaccess.org run PyFF (https://pyff.io) in production environment. In Beta & Staging they are named a-*.thiss.io.

PyFF is short for python Federation Feeder - is a simple SAML metadata aggregator

In this environment, PyFF aggregates metadata from 3 federations - SWAMID, EduGAIN, InCommon & OpenAthens and publish them under /var/www/html/ using the script /usr/local/sbin/run-pyff running as a cronjob. 

# Puppet Name: publish
*/

Descripton & Troubleshooting

The servers with the name meta.*seamlessaccess.org run PyFF (https://pyff.io) in production environment. In Beta & Staging they are named a-*.thiss.io.

PyFF is short for python Federation Feeder - is a simple SAML metadata aggregator

In this environment, PyFF aggregates metadata from 3 federations - SWAMID, EduGAIN, InCommon & OpenAthens and publish them under /var/www/html/ using the script /usr/local/sbin/run-pyff running as a cronjob. 

# Puppet Name: publish
*/30 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name publish -- /usr/local/sbin/run-pyff /opt/pyff/mdx.fd /var/www/html/metadata.json /var/www/html/metadata_sp.json

...

The servers also runs Apache in a docker container service called sunet-md_publisher to expose and publish the metadata JSON files on port 443 which are accisible only by the servers running MDQ (md-*.seamlessaccess.org) belonging to the same site.

Mointoring & Troubleshooting

We monitor ages of all the metadata files in https://monitor.seamlessaccess.org/nagios3nagios4/. They  They are

• Metadta XML files for each federation  in /opt/pyff/metadata/
•  /var/www/html/metadata.json
• /var/www/html/metadata_sp.json.

Take help of the 'Description & Troubleshooting' section above to troubleshoot the alarms. Se also GeneralTroubleshooting.

Upgrade & Verification

• Both PyFF and sunet-md_publisher are upgraded by chaging the versions in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml. The puppet manifests for production, beta and staging are separate.

...

• After commiting and bump-taging the changes, run cosmos in the concerned servers, better to do it one at a time & check that the service is working.
• If PyFF is upgraded, run the aforementioned cronjob for PyFF to see that it doesn't show any error.
• You have to restart sunet-md_publisher if you have upgraded the metdata publishing service. See GeneralTroubleshooting
• Check https://monitor.seamlessaccess.org/nagios3/ for any alarms.
• The MDQ servers with the name md-*.seamlessaccess.org should be able to fetch the metadata from the Aggregator & Publisher servers. Make sure it is all 'green' for those servers too.
• You can log in to the MDQ servers and run /usr/local/bin/get_metadata.sh and see that they are able to fetch metadata files without any issues.
• As a last & final check, visit any SP for example wiki.sunet.se and see that it is possible to login using SA discovery service or check the login using https://demo.seamlessaccess.org/ for production upgrades and https://demo.beta.seamlessaccess.org for Beta upgrades.

...

thiss-mdq

Servers

Descripton & Troubleshooting

The servers with the name md-.*seamlessaccess.org run thiss-mdq in production environment. In Beta & Staging they are named md-*.thiss.io.

...

The servers run the MDQ service on port 80 which is only open to the HAproxy load balancers with names md.*.seamlessaccess.org belonging to the same site.

Mointoring & Troubleshooting

We monitor the date when the metadata JSON files are last modified in both monitor.seamlessaccess.org and nagiosxi.nordu.net. The check warns if the metadata is 2 days old, it becomes critical if it is 5 days old.

...

We also have nagios checks on the accisibility of these web links on each level. Chek also GeneralTroubleshooting.

Upgrade & verification

The process is described in below link along with verification for both production and beta environments.

...

...

Descripton 

The servers with the name static-.*seamlessaccess.org run thiss-js in production environment. In Beta, they are named static-*.thiss.io.

...

The servers run the code in Docker containers. They run the thiss-js service on port 80 which is only open to the HAproxy load balancers with names static.*.seamlessaccess.org belonging to the same site.

Mointoring & Troubleshooting

We monitor both the version of the code and the accsibility of the service in in both monitor.seamlessaccess.org and nagiosxi.nordu.net.

We have this check on servers running thiss-js, on HAproxy Load balancer servers serving the code to Fastly and on Fastly level which is https://service.seamlessaccess.org.

...

root@static-1: ~ # curl -k -4 http://localhost/manifest.json
{
  "short_name": "Seamless Access",
  "name": "Seamless Access Identity Selector",
  "description": "See https://seamlessaccess.org",
  "version": "2.1.160"
}

We also have nagios checks on the accisibility of these web links on each level. Chekc also GeneralTroubleshooting.

Upgrade

The process is described in below link along with verification for both production and beta environments.

Seamless Access Software Deployment Guide#Frontend(service.seamlessaccess.org)

Seamless Access Software Deployment Guide#Frontend(use.thiss.io)

HAproxy Load Balancer

Servers

Descripton & Troubleshooting

It forwards the HTTP GET requests invoked by the users from frontend to one of the MDQ servers using round robin algorithm

Mointoring

Upgrade

  "description": "See https://seamlessaccess.org",
  "version": "2.1.160"
}

We also have nagios checks on the accisibility of these web links on each level. Chekc also GeneralTroubleshooting.

Upgrade & verification 

The process is described in below link along with verification for both production and beta environments.

Seamless Access Software Deployment Guide#Frontend(service.seamlessaccess.org)

Seamless Access Software Deployment Guide#Frontend(use.thiss.io)

HAproxy Load Balancer for MDQ

Servers

Descripton 

There is one load balancer server running HAproxy which is placed in front of the two MDQ servers per site. These server have the names md.*.seamlessaccess.org. These HAproxy servers are added in Fastly for the service md.seamlessaccess.org. Fastly forwards the non-cached HTTPS GET requests invoked by the users to one of these HAproxy servers which in turn forwards them to one of the MDQ servers using round robin algorithm. These HTTPS requests handle metadata queires.

The HAproxy service runs in a docker container and the configuration of it is supplied by puppet manifests.

Mointoring & Troubleshooting

We have three specific checks for these load balancers for each site in https://monitor.seamlessaccess.org/nagios4/

1. Monitor the date when the metadata JSON files are last modified from the https://<site link>/manifest.json
2. SSL check and availability of the site links
3. The string 'OK' is found in https://<site link>/status
4. Monitor that both backends are up by checking HAproxy stats from http://<site link>:8404/stats. This link is accesible only by SUNET VPN for SUNET NOC members and the monitor server.

The site links are

 https://md.ntx.sunet.eu.seamlessaccess.org/

 https://md.se-east.sunet.eu.seamlessaccess.org

https://md.aws1.geant.eu.seamlessaccess.org

https://md.aws2.geant.eu.seamlessaccess.org

Take help of GeneralTroubleshooting for fixing alarms. It may happen that MDQ servers are unavailable which will cause alarm in HAproxy servers, then check the section for MDQ servers to troubleshoot them.

Upgrade

SeamlessAccess HAproxy Upgrade

HAproxy Load Balancer for thiss-js

Servers

Descripton & Troubleshooting

There is one load balancer server running HAproxy which is placed in front of the two thiss-js servers per site. These server have the names static.*.seamlessaccess.org. These HAproxy servers are added in Fastly for the service service.seamlessaccess.org. Fastly forwards the non-cached HTTPS GET requests invoked by the users from https://service.seamlessaccess.org to one of these HAproxy servers which in turn forwards them to one of the servers running  thiss-js code using round robin algorithm.

The HAproxy service runs in a docker container and the configuration of it is supplied by puppet manifests.

Mointoring

...

Upgrade

Monitor

Server

Descripton & Troubleshooting

Mointoring

Upgrade

Demo Application

Server

Descripton & Troubleshooting

Mointoring

Upgrade

Use of SUNET INFRA cert

add details

SeamlessAccess SUNET INFRA cert update

Use of Fleetlock

General Troubleshooting

Almost all services run in docker containers. They are addes as systemd units. The names start with sunet-*.

...

The puppet manifests that deploy and manage the internal components are found here https://github.com/TheIdentitySelector/thiss-ops/tree/master/global. Those who have write acces to it are mentioned here https://wiki.sunet.se/pages/viewpage.action?pageId=83493119

Firewall Restrictions

Staging Metadata Service

Access to Internal Components

...