<Incomplete, but the existing information is correct>
prod, staging, beta, from cdn to bottom
The agreegated picture is here Seamless Access Deployment Architecture
https://wiki.geant.org/spaces/gn43wp5/pages/1228374096/Seamless+Access+Component+Description+TroubleshootingKnowledge of SUNET's puppet & cosmos structure https://wiki.sunet.se/display/sunetops/Cosmos
Fastly is CDN (content delivery network) provider. We use CDN to provide greater rechability accross the world, take advantage of their cache nodes
Services that are hosted in Fastly are
The configuration of these services reside here https://manage.fastly.com/services/all
https://docs.fastly.com/en/guides/checking-cache, curl -I -H"Fastly-debug:1" https://service.seamlessaccess.org |
curl -s "https://api.fastly.com/content/edge_check?url=https://service.seamlessaccess.org/990.js" -H 'Fastly-Key:xxx' |
https://wiki.sunet.se/pages/viewpage.action?pageId=83493119
The servers with the name meta.*seamlessaccess.org run PyFF (https://pyff.io) in production environment. In Beta & Staging they are named a-*.thiss.io.
PyFF aggregates metadata from 3 federations - SWAMID, EduGAIN, InCommon & OpenAthens and publish them under /var/www/html/ using the script /usr/local/sbin/run-pyff running as a cronjob.
# Puppet Name: publish */30 * * * * /usr/local/bin/scriptherder --mode wrap --syslog --name publish -- /usr/local/sbin/run-pyff /opt/pyff/mdx.fd /var/www/html/metadata.json /var/www/html/metadata_sp.json |
They aggreagate 'general' metadata in /var/www/html/metadata.json and SP trust metadata in /var/www/html/metadata_sp.json. They are created every 30 minutes by running PyFF in a docker container momentarily.
The script also checks manually the fingerprint on the metadata and PyFF does the same thing again.
Read details about the sources and certificates of federation metadata in SeamlessAccess Metadata Feeds.
The servers also runs Apache in a docker container service called sunet-md_publisher to expose and publish the metadata JSON files on port 443 which are accisible only by the servers running MDQ (md-*.seamlessaccess.org).
We monitor ages of all the metadata files in https://monitor.seamlessaccess.org/nagios3/. They are
/opt/pyff/metadata//var/www/html/metadata.json/var/www/html/metadata_sp.json.Take help of the 'Description & Troubleshooting' section to troubleshoot the alarms.
sunet-md_publisher are upgraded by chaging the versions in thiss-ops/global/overlay/etc/puppet/cosmos-rules.yaml. The puppet manifests for production, beta and staging are separate. thiss::pyff_prod:
pyff_version: 2.1.3
output: /var/www/html/metadata.json
output_trust: /var/www/html/metadata_sp.json
thiss::md_publisher_prod:
watch: /var/www/html/metadata.json
watch_sp: /var/www/html/m |
sunet-md_publisher if you have upgraded the metdata publishing service.md-*.seamless.org) should be able to fetch the metadata from the Aggregator & Publisher servers. Make sure it is all 'green' for those servers too./usr/local/bin/get_metadata.sh and see that they are able to fetch metadata files without any issues.SeamlessAccess HAproxy Upgrade
Almost all services run in docker containers. They are addes as systemd units. The names start with sunet-*.
journalctl -fu <service name of the system unit> /var/log/syslog for older logsdocker logs -f <docker container name>service <service name of the system unit> restartFor deeper troubleshooting knowledge of SUNET's puppet & cosmos structure is needed as mentioned in the Prerequisites section above.
The puppet manifests that deploy and manage the internal components are found here https://github.com/TheIdentitySelector/thiss-ops/tree/master/global. Those who have write acces to it are mentioned here https://wiki.sunet.se/pages/viewpage.action?pageId=83493119
add details
SeamlessAccess SUNET INFRA cert update
https://wiki.sunet.se/pages/viewpage.action?pageId=83493119
https://wiki.sunet.se/display/sunetops/Monitoring