You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next » 

<Incomplete, but the existing information is correct>

Environments

  • Production - domain is seamlessaccess.org
  • Beta & staging - domain is thiss.io

SeamlessAccess Services


Overview of the Structure 

The agreegated picture is here Seamless Access Deployment Architecture

Production

service.seamlessaccess.org

SA service structure


md.seamlessaccess.org

SA MD structure

Beta

use.thiss.io

SA beta service structure

md.thiss.io

SA beta MD structure

Prerequisites for management & troubleshooting 

External Component

Fastly 

Fastly is CDN (content delivery network) provider. We use CDN to provide greater rechability accross the world, take advantage of their cache nodes

Services that are hosted in Fastly are

The configuration of these services reside here https://manage.fastly.com/services/all

Troubleshooting

  • How to check cache for an URL- 
https://docs.fastly.com/en/guides/checking-cache, curl -I -H"Fastly-debug:1" https://service.seamlessaccess.org
  • How to check cache in an Edge node of Fastly, you need to create an API key for this
curl -s "https://api.fastly.com/content/edge_check?url=https://service.seamlessaccess.org/990.js" -H 'Fastly-Key:xxx' 


Access

https://wiki.sunet.se/pages/viewpage.action?pageId=83493119

Internal Components

Aggregator & Publisher

Descripton & Troubleshooting

The servers with the name meta.*seamlessaccess.org run PyFF (https://pyff.io) in production environment. In Beta & Staging they are named a-*.thiss.io.

PyFF aggregates metadata from 3 federations - SWAMID, EduGAIN, InCommon & OpenAthens and publish them under /var/www/html/ using the script /usr/local/sbin/run-pyff running as a cronjob. 

# Puppet Name: publish
*/30 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name publish -- /usr/local/sbin/run-pyff /opt/pyff/mdx.fd /var/www/html/metadata.json /var/www/html/metadata_sp.json

They aggreagate 'general' metadata in /var/www/html/metadata.json and SP trust metadata in /var/www/html/metadata_sp.json. They are created every 30 minutes by running PyFF in a docker container momentarily. 

The script also checks manually the fingerprint on the metadata and PyFF does the same thing again.

Read details about the sources and certificates of federation metadata in SeamlessAccess Metadata Feeds.

The servers also runs Apache in a docker container service called sunet-md_publisher to expose and publish the metadata JSON files on port 443 which are accisible only by the servers running MDQ (md-*.seamlessaccess.org).

Mointoring

We monitor ages of all the metadata files in https://monitor.seamlessaccess.org/nagios3/. They are

  • Metadta XML files for each federation  in /opt/pyff/metadata/
  •  /var/www/html/metadata.json
  • /var/www/html/metadata_sp.json.

Take help of the 'Description & Troubleshooting' section to troubleshoot the alarms. 

Upgrade

  • Both PyFF and sunet-md_publisher are upgraded by chaging the versions in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml. The puppet manifests for production, beta and staging are separate.
   thiss::pyff_prod:
      pyff_version: 2.1.3
      output: /var/www/html/metadata.json
      output_trust: /var/www/html/metadata_sp.json
   thiss::md_publisher_prod:
      watch: /var/www/html/metadata.json
      watch_sp: /var/www/html/m


  • After commiting and bump-taging the changes, run cosmos in the concerned servers, better to do it one at a time & check that the service is working.
  • If PyFF is upgraded, run the aforementioned cronjob for PyFF to see that it doesn't show any error.
  • You have to restart sunet-md_publisher if you have upgraded the metdata publishing service.
  • Check https://monitor.seamlessaccess.org/nagios3/ for any alarms.
  • The MDQ servers (md-*.seamless.org) should be able to fetch the metadata from the Aggregator & Publisher servers. Make sure it is all 'green' for those servers too.
  • You can log in to the MDQ servers and run /usr/local/bin/get_metadata.sh and see that they are able to fetch metadata files without any issues.
  • As a last & final check, visit any SP for example wiki.sunet.se and see that it is possible to login using SA discovery service or check the login using https://demo.seamlessaccess.org/ for production upgrades and https://demo.beta.seamlessaccess.org for Beta upgrades.

MDQ

Descripton & Troubleshooting

Mointoring

Upgrade

Thiss-js

Descripton & Troubleshooting

Mointoring

Upgrade

Load Balancer HAproxy

Descripton & Troubleshooting

Mointoring

Upgrade

SeamlessAccess HAproxy Upgrade

Monitor

Descripton & Troubleshooting

Mointoring

Upgrade

Demo Application

Descripton & Troubleshooting

Mointoring

Upgrade

General Troubleshooting

Almost all services run in docker containers. They are addes as systemd units. The names start with sunet-*.

  • journalctl -fu <service name of the system unit>
  • Check /var/log/syslog for older logs
  • docker logs -f <docker container name>
  • service <service name of the system unit> restart

For deeper troubleshooting knowledge of SUNET's puppet & cosmos structure is needed as mentioned in the Prerequisites section above.

The puppet manifests that deploy and manage the internal components are found here https://github.com/TheIdentitySelector/thiss-ops/tree/master/global. Those who have write acces to it are mentioned here https://wiki.sunet.se/pages/viewpage.action?pageId=83493119

Use of SUNET INFRA cert

add details

SeamlessAccess SUNET INFRA cert update


Use of Fleetlock


Firewall Restriction

Access to Internal Components

https://wiki.sunet.se/pages/viewpage.action?pageId=83493119

Overall Monitoring

https://wiki.sunet.se/display/sunetops/Monitoring


  • No labels