IBM Spectrum Scale Strategy Days
2019-03-20 - 2019-03-21 | IBM Client Center Ehningen, IBM-Allee 1, Ehningen, 71139, Germany
This event is run by IBM and is mostly spoken in German.
Software-Defined Storage ist gegenwertig eines der heißesten Themen in Unternehmen. Bei dieser kostenfreien Veranstaltung können Sie sich auf den aktuellen Stand rund um das Thema Spectrum Scale bringen. Der Einsteigertag am 19. März vermittelt Ihnen die Grundlagen und gibt einen Rundumblick zum Einordnen, Ausloten und Auffrischen der Kenntnisse. An den Expertentagen am 20. und 21. März vertiefen Sie Ihr Wissen. Melden Sie sich jetzt an, wahlweise für die gesamte Veranstaltung, nur für den Einsteigertag oder die Expertentage. Die Teilnahme ist kostenlos und die Veranstaltungssprache überwiegend Deutsch.
The role that IT infrastructures plays is especially visible in IBM Systems Storage solutions by increasing the effectiveness and reducing the costs of the storage and compute resources that support enterprise applications and drive sustainable and immediate enterprise value generation.

US Spectrum Scale user group meeting at NCAR
2019-04-16 - 2019-04-17 | NCAR Green Center, 3080 Center Green Drive, Boulder, CO, 80301, United States
This Spectrum Scale user group event has a fantastic agenda of Spectrum Scale experts and will take place over 2-days. The event will provide you with the opportunity to hear the latest Spectrum Scale updates from filesystem experts as well as hear from other users on their experiences.
Whilst the user group is open to all Spectrum Scale users, please note that the user group is a forum for technical discussion and is not a sales opportunity.
Due to limited space at our venue, if you want to send more than 3 people from your organization, please contact us via email prior to registering (usa-principal@gpfsug.org or usa-co-principal@gpfsug.org)
Agenda:

Duration
Start
End
Title
Speaker

Tuesday, April 16th

30
8:30
9:00
Registration and Networking
n/a

20
9:00
9:20
Welcome
Kristy Kallback-Rose / Bob Oesterlin (Chair) / Ted Hoover (IBM)

25
9:20
9:45
Spectrum Scale: The past, the present, the future
Wayne Sawdon (IBM)

25
9:45
10:10
Accelerating AI workloads with IBM Spectrum Scale
Ted Hoover (IBM)

20
10:10
10:30
Nvidia: Doing large scale AI faster – scale innovation with multi-node-multi-GPU computing and a scaling data pipeline
Jacci Cenci (Nvidia)

30
10:30
11:00
Coffee and Networking
n/a

25
11:00
11:25
AI ecosystem and solutions with IBM Spectrum Scale
Piyush Chaudhary (IBM)

30
11:25
11:55
Spectrum Scale Update
Chris Maestas (IBM)

5
11:55
12:00
Meet the devs
Ulf Troppens (IBM)

60
12:00
13:00
Lunch and Networking
n/a

20
13:00
13:20
NCAR site update
Pamela Hill (NCAR)

15
13:20
13:35
ESS Update
Rezaul Islam (IBM)

15
13:35
13:50
Support Update
Bob Simon (IBM)

40
13:50
14:30
Memory Consumption in Spectrum Scale
Tomer Perry (IBM)

30
14:30
15:00
Coffee and Networking
n/a

40
15:00
15:40
Panel: Data Migration
TBD

20
15:40
16:00
Container & Cloud Update
Ted Hoover (IBM)

20
16:00
16:20
Towards Proactive Service with Call Home
Ulf Troppens (IBM)

10
16:20
16:30
Break
n/a

30
16:30
17:00
Advanced metadata management with Spectrum Discover
Deepavali Bhagwat (IBM)

20
17:00
17:20
High Performance Tier
Tomer Perry (IBM)

40
17:20
18:00
Meet the Devs – Ask us Anything
All

120
18:00
20:00
Get Together
n/a

13:00 – 17:15 Breakout Session: Getting Started with Spectrum Scale
Chris Maestas (IBM)

Wednesday, April 17th

30
8:30
9:00
Coffee und Networking
n/a

30
8:30
9:00
Spectrum Scale Licensing
Carl Zetie (IBM)

60
9:00
10:00
Spectrum Scale Use Cases (Beginner)
Spectrum Scale Protocols (Overview) (Beginner)
Ulf Troppens (IBM
Tomer Perry (IBM) / Ulf Troppens (IBM)

Spectrum Scale backup and SOBAR
Chris Maestas (IBM)

Getting started with AFM (Advanced)
Venkat Puvvada (IBM)

60
10:00
11:00
How to design a Spectrum Scale environment? (Beginner)
Tomer Perry (IBM)

Spectrum Scale on Google Cloud
Jeff Ceason (IBM)

Spectrum Scale Trial VM
Spectrum Scale Vagrant
Chris Maestas (IBM
Ulf Troppens (IBM)

60
11:00
12:00
Spectrum Scale GUI (Beginner)
Spectrum Scale REST API (Beginner)
Chris Maestas (IBM)
Chris Maestas (IBM

Spectrum Scale Network flow
Tomer Perry (IBM)

Spectrum Scale Watch Folder (Advanced)
Spectrum Scale File System Audit Logging
Deepavali Bhagwat (IBM)
Deepavali Bhagwat (IBM)

60
12:00
13:00
Lunch and Networking
n/a

20
13:00
13:20
Excelero NVMesh: Supercharging IBM SpectrumScale with Software-defined NVMe Storage (Sponsor Talk)
Joe Harlan (Excelero) / Sam Eckhouse (Excelero)

20
13:20
13:40
AWE site update
Paul Tomlinson (AWE)

20
13:40
14:00
Lenovo storage update (Sponsor Talk)
Ray Padden (Lenovo)

30
14:00
14:30
Coffee and Networking
n/a

30
14:30
15:00
New Storage Options for Spectrum Scale
Carl Zetie (IBM)

30
15:00
15:30
AFM Update
Venkat Puvvada (IBM)

20
15:30
15:50
TCT Update
Rob Basham

10
15:50
16:00
Wrap-up
Kristy Kallback-Rose / Bob Oesterlin

Food and Socialzing
April 16th:
Breakfast, Lunch, and afternoon snack provided on-site at NCAR (no charge for attendees)
Evening Reception at NCAR (beer, wine, and hors d’oeuvres )
April 17th:
Breakfast, Lunch, and afternoon snack provided on-site at NCAR (no charge for attendees)
Directions from Denver International Airport
Fastest Route (w/ Tolls)
Take Peña Blvd exiting the Airport
Take exit 6B toward Boulder/Ft. Collins
Merge onto E-470 N
Keep to the left and continue on Northwest Pkwy
Slight right to merge onto US-36 W to Boulder
Take the CO-157 N / Foothills Pkwy exit
Use the 2nd from the left lane to turn left onto Valmont Rd
Turn right onto Center Green Dr
Destination is at the end of the cul-de-sac
Accommodation
There are a number of hotels nearby to suit a range of budgets. We do not provide a booking service for accommodation and you will need to make your own arrangements. Book your hotel early to get the best rate.
Residence Inn by Marriott Boulder
3030 Center Green Dr, Boulder, CO 80301
(303) 449-5545. 800-331-3131
marriott.com
Hilton Garden Inn Boulder
2701 Canyon Blvd, Boulder, CO 80302
(303) 443-2200
hiltongardeninn3.hilton.com
Embassy Suites by Hilton Boulder
2601 Canyon Blvd, Boulder, CO 80302
(303) 443-2600
embassysuites3.hilton.com
Hyatt Place Boulder/Pearl Street
2280 Junction Pl, Boulder, CO 80301
(303) 442-0160
hyatt.com
Courtyard by Marriott Boulder
4710 Pearl E. Circle, Boulder, CO 80301
303-440-4700
marriott.com
The Millennium Harvest House
1345 28th Street, Boulder, CO 80302
(303) 998-3822
millenniumhotels.com
Any questions?
Contact us by email: usa-principal@gpfsug.org or usa-co-principal@gpfsug.org
FAQs
Where can I contact the organizer with any questions?
Contact us by email: usa-principal@gpfsug.org or usa-co-principal@gpfsug.org
Is this an IBM run event?
No. The Spectrum Scale user group is not part of IBM, we are a community of users, ISVs, VARs and have a number of IBM staff as members of the group. To enable us to bring developers from IBM to speak, we work closely with IBM to run user group events and are very grateful for their support.
I have an idea for a talk, how can I get a slot?
Contact Kristy Kallback-Rose or Bob Oesterlin, co-principal by email: usa-principal@gpfsug.org or usa-co-principal@gpfsug.org
How do I join the group?
The group is free to join, please visit www.spectrumscale.org/join for details of the group and how to join the mailing list.
There are no tickets left, can I still attend?
We are limited to 125 people by the space available. We will be running a wait-list, however if you do not have a ticket by event day, we are unlikely to be able to accommodate you.
I can no longer attend, what should I do?
Contact Kristy Kallback-Rose or Bob Oesterlin, co-principal by email: usa-principal@gpfsug.org or usa-co-principal@gpfsug.org
We will then be able to offer your ticket to another member.

Register here: https://www.eventbrite.com/e/spectrum-scale-gpfs-user-group-us-spring-2019-meeting-tickets-57035376346

UK and World Wide Spectrum Scale User Group Meeting
2019-05-08 - 2019-05-09 | IBM South Bank, 76/78 Upper Ground, London, SE1 9PZ, United Kingdom
The UK/World Wide 2019 Spectrum Scale user group is planned to take place on 8th and 9th May, once again overlooking the River Thames at IBM’s South Bank Client Centre.
The draft agenda for the event is posted below and has a mix of IBM, customer and sponsor talks. This year we have a particular focus on AI with Spectrum Scale and it promises to be another great two days.
Thanks to our sponsors, we’re able to run an evening dinner/networking event. We’ll provide more details on this on the day, but make sure you plan to attend our evening event!
Agenda
Day 1: 8th May:

StartEndTitleSpeaker

6009:0010:00ARRIVALS & REGISTRATION

1510:0010:15Opening & welcomeSimon Thompson (Chair) / Ted Hoover (IBM)

2510:1510:40Spectrum Scale: The past, the present, the futureWayne Sawdon (IBM)

2510:4011:05Accelerating AI workloads with IBM Spectrum ScaleTed Hoover (IBM)

2011:0511:25Sponsor Talk [NVIDIA]: Doing large scale AI faster – scale innovation with multi-node-multi-GPU computing and a scaling data pipelineMiguel Martínez (NVIDIA)

2511:2511:50AI ecosystem and solutions with IBM Spectrum ScalePiyush Chaudhary (IBM)

1011:5012:00Meet the devsUlf Troppens (IBM)

6012:0013:00LUNCH

3013:0013:30Spectrum Scale UpdatePuneet Chaudhary

1513:3013:45ESS UpdatePuneet Chaudhary

2013:4514:05Customer Talk - Why Spectrum Scale, a Customer ViewMark Vilensky (Weizmann Institute of Science)

2014:0514:25Sponsor Talk [DDN]: "Optimizing Storage Stacks for AI"Sven Oehme

2014:2514:45Sponsor Talk [Lenovo] - "Storage, Fast and Slow - Lenovo updates from both ends of the storage hierarchy"Michael Hennecke

3014:4515:15BREAK

1515:1515:30Support UpdateKai Korkow (IBM)

2015:3015:50New Storage Options for Spectrum ScaleLuis Bolinches (IBM)

2015:5016:10Customer Talk - Why Spectrum Scale, a Customer ViewPatrick Dekkers (Amsterdam UMC Locactie VUmc)

2016:1016:30Sponsor Talk [e8 Storage]: E8 Shared NVMe for High Performance Spectrum Scale ClustersStuart Campbell (e8 Storage) / Peter Childs (QMUL)

2016:3016:50Sponsor Talk [Mellanox] : Low Latency Ethernet Storage FabricsGadi Godanyan (Mellanox Technologies)

4016:5017:30Meet the Devs - Ask us AnythingAll

1017:3017:40WRAP UPSimon Thompson (Chair)

13518:3022:45User Group Networking / DinnerRegistration and Badge/Lanyard required

During the afternoon, we will also be running a breakout session “Getting Started with Spectrum Scale”. This is intended for newer users of Spectrum Scale and will be run subject to demand.
Day 2: 9th May:

StartEndTopicSpeaker

3008:3009:00ARRIVALS

3008:3009:00Spectrum Scale LicensingCarl Zetie (IBM)

6009:0010:00AI at Scale at the example of Autonomous Driving (Beginner)
IBM Aspera Sync for Spectrum ScaleNils Haustein (IBM)

Cloud Update
Spectrum Scale Support for ContainerHarald Seipp (IBM)

Towards Proactive Service with Call Home
Spectrum Scale Reliability, Availability und ServiceabilityMathias Dietz (IBM)

6010:0011:00Spectrum Scale installation and upgrade with Install ToolkitKedar Karmarkar (IBM)

Tiering to tape with Spectrum ArchiveKhanh Ngo (IBM)

Spectrum Scale Watch Folder
Lessons from the fieldOlaf Weiser (IBM)
Kai Korkow (IBM) / Luis Bolinches (IBM)

6011:0012:00Overview: Spectrum Scale support for NFS and SMB
Spectrum Scale Best Practices for SnapshotsUlf Troppens (IBM)
Kedar Karmarkar (IBM)

Spectrum Scale Policy Engine: Introduction and Use CasesNils Haustein (IBM)

Spectrum Scale Network FlowJohn Lewars (IBM)

6012:0013:00LUNCH

3013:0013:30Lightning Talks - My favourite/worst Spectrum Scale featureAll - including attendees

3013:3014:00Memory Consumption in Spectrum ScaleJohn Lewars (IBM)

3014:0014:30BREAK

2014:3014:50Customer Talk - Migrating Spectrum Scale Filesystems with minimal downtimeLuke Sudbery (University of Birmingham)

2014:5015:10Sponsor Talk [OCF]: Atempo - Large Filesystem Migration & BackupThomas Ahrens (Atempo)

2015:1015:30Customer Talk - Experience with Spectrum Scale AFMAlvise Dorigo (PSI)

2015:3015:50High Performance TierWayne Sawdon (IBM)

1015:5016:00CLOSESimon Thompson (Chair)

This event is sponsored by IBM, OCF, e8 Storage, Lenovo, DDN Storage, Mellanox Technologies and NVIDIA.

Spectrum Scale user group meeting @ISC19
2019-06-17 | ISC Conference Venue, Messe Frankfurt Ludwig-Erhard-Anlage 1, Frankfurt am Main, 60327, Germany
IBM will host a joint “IBM Spectrum Scale and IBM Spectrum LSF User Meeting” at ISC. As with other user group meetings, the agenda will include user stories, updates on IBM Spectrum Scale & IBM Spectrum LSF, and access to IBM experts and your peers.
The meeting is planned for:
Monday June 17th, 2019 – 1pm-6pm
ISC Frankfurt, Germany
Venue:
Room Applaus – Forum
Tor Ost (East Gate) Hall 3
Brüsseler Strasse 60327
Frankfurt am Main
Please register here to properly plan for refreshments:
https://www-01.ibm.com/events/wwe/grp/grp308.nsf/Agenda.xsp?openform&seminar=6DQG9UES
Messe Frankfurt

13:0013:15WelcomeGabor Samu (IBM) / Ulf Troppens (IBM)

13:1513:35What is new in Spectrum Scale and ESS?Christopher Maestas (IBM)

13:3513:55Sharing files in Spectrum Scale with Nextcloud Sync&ShareFrank Karlitschek (Nextcloud)

13:5514:15Whats new in IBM Spectrum LSF SuiteBill McMillan (IBM)

14:1514:40Caches all the way down: Building a wide area data fabric with Spectrum ScaleDavid Abramson (University of Queensland)

14:4015:00Spectrum Scale Reliability, Availability und ServiceabilityMathias Dietz

15:0015:30Break (w/refreshments)n/a

15:3015:50Spectrum Scale Erasure Code Edition (ECE)Christopher Maestas (IBM)

15:5016:10I/O Profiling with Ellexus at Diamond Light Source on an IBM Spectrum Scale clusterRosemary Francis (Ellexus)

16:1016:35AI at scale with Spectrum LSFFlorin Manailla (IBM)

16:3516:55Best Practices for Spectrum Scale with Lenovo DE4000/DE6000 StorageOliver Kill (pro-com)

16:5517:15High Performance TierSimon Lorenz (IBM) / Tomer Perry (IBM)

17:1517:30Wrap-upGabor Samu (IBM) / Ulf Troppens (IBM)

17:3018:00Get Together (w/refreshments)n/a

Spectrum Scale Meet-up in Buenos Aires
2019-07-16 17:00 - 19:00 (America/Argentina/Buenos_Aires)
IBM will host a “IBM Spectrum Scale Meet-up” along IBM Technical University Buenos Aires. This is the first user meeting in South America. All sessions will be in Spanish.
Date:
July 16, 2019 – 5:00 to 7:00 pm
Location:
Intercontinental Buenos Aires
809 Moreno St, C1091 AAQ,
Buenos Aires,Argentina
Room : Dali
Registration:
To attend please send a mail to Eliane Vido (enadur@br.ibm.com).
Agenda:
5:00 – Welcome
5:15 – Spectrum Scale Overview – Christopher Maestas
5:40 – Replication and High availability – Julian Cachua
6:05 – Spectrum Scale a BigData and Analytics solution – Ismael Solis
6:30 – Networking & Refreshment

Spectrum Scale/GPFS Days (part of HPCXXL)
2019-09-23 - 2019-09-24 | NERSC/LBNL Berkeley, CA, 1 Cyclotron Rd, Berkeley, CA, 97420, United States
GPFS Days are free of charge, but you still need to register. See the registration form for details, link is below under “Website:”. Click on the “Register” button and choose to register for the Free GPFS Days.
We are still working on the agenda. Here is the latest draft which is subject to change.
Monday, Sep 23th 2019

1513:0013:15WelcomeTina Declerck (NERSC) / Angela Encizo Pholphiboun (IBM)

2513:1513:40Spectrum Scale: Past, Present and FutureWayne Sawdon (IBM)

2513:4014:05What is new in Spectrum Scale and ESS?Chris Maestas (IBM)

2514:0514:30AI ecosystem and solutions with IBM Spectrum ScaleAngela Encizo Pholphiboun (IBM)

2514:3014:55Nvidia: The Storage Layer in the NVIDIA DGX SuperPODJacci Cenci (NVIDIA)

2514:5515:20Spectrum Scale Solution for Nvidia DGXConstantine Arnold (IBM)

3015:2015:50Breakn/a

2515:5016:15Unified Metadata Management with IBM Spectrum DiscoverStephen Edel (IBM)

2516:1516:40AWE: Site UpdateMark Roberts (AWE)

2516:4017:05High Performance Tier (HPT)Wayne Sawdon (IBM)

4517:0517:50Spectrum Scale Memory UsageTomer Perry (IBM)

1017:5018:00Wrap-upKristy Kallback-Rose (User Group)

Tuesday, Sep 24th 2019

6009:0010:00Spectrum Scale Erasure Code Edition (ECE)Nikhil Khandelwal

3010:0010:30ContainerisationVasily Tarasov (IBM)

3010:3011:00Breakn/a

2511:0011:25NERSC - Site updateKristy Kallback-Rose (NERSC), Ravi Cheema (NERSC)

2511:2511:50FZJ - User migrationMartin Lischweski (FZJ)

4011:5012:30Discussion: Data Migration ChallengesKristy Kallback-Rose (User Group)

6012:3013:30Lunchn/a

1513:3013:45License UpdateCarl Zetie (IBM)

3013:4514:15Spectrum Scale Watch FolderLukas Rupprecht (IBM)

2514:1514:40Sponsor Talk: VAST DataVAST Data

2514:4015:05Sponsor Talk: DDNAndreas Krause (DDN)

2515:0515:30Sponsor Talk: Flexible Configurations in the Data CenterRaymond Paden (Lenovo)

3015:3016:00Breakn/a

4516:0016:45Give or take a blockTomer Perry (IBM)

6016:4517:45Ask us anythingAll

1517:4518:00Wrap-upKristy Kallback-Rose (User Group)

18:30?Dinner outRichter Room, Triple Rock Brewing, 1920 Shattuck Ave, Berkeley, CA 94704

Spectrum Scale NYC User Meeting 2019
2019-10-10 09:00 - 16:30 (America/New_York) | NYU Langone Health, 435 East 30th Street, New York, NY, 10016, United States
NYU Langone and IBM will host a Spectrum Scale User Meeting on October 10.
Please register here to attend:
https://www.eventbrite.com/e/2019-spectrum-scale-user-group-nyc-tickets-73471036849
Draft agenda

3008:3009:00Registration and Morning Coffeen/a

1509:0009:15WelcomeAmy Hirst (IBM) / John Speakman (NYU Langone)

2509:1509:40What is new in Spectrum Scale?Felipe Knop (IBM)

2509:4010:05What is new in ESS?Steve Duersch (IBM)

2510:0510:30Spectrum Scale Erasure Code Edition (ECE)Puneet Chaudhary (IBM)

2010:3010:50Coffee and Networkingn/a

2510:5011:15Customer TalkMichael Costantino (NYU Langone)

1511:1511:30What is new in licensing?Nariman Nasef (IBM)

2011:3011:50Service UpdateJames Livulpi (IBM)

2511:5012:15Scaling data transferChris Black (The New York Genome Center)

4512:1513:00Lunch and Networkingn/a

2513:0013:25AI ecosystem and solutions with IBM Spectrum ScaleTed Hoover (IBM)

2513:2513:50Nvidia: Doing large scale AI faster – scale innovation with multi-node-multi-GPU computing and a scaling data pipelineTBD (Nvidia)

2513:5014:15Spectrum Scale Solutions for AI workloadsPiyush Chaudhary (IBM)

2014:1514:35Coffee and Networkingn/a

4014:3515:15Technical Deep Dive - Topic NetworkJohn Lewars (IBM)

3015:1515:45Containerisation DiscussionTed Hoover (IBM)

6015:4516:45Open DiscussionAll

1516:4517:00Wrap-upRichard Rupp (IBM)

12017:0019:00Social EventAll

Australian Spectrum Scale User Group (after IBM Systems Technical University)
2019-10-18 | IBM St Leonards Sydney, 601 Pacific Highway St Leonards, Sydney, NSW, Australia
18th October | Sydney, Australia
IBM St Leonards
Level 13,
601 Pacific Highway
St Leonards, NSW 2065

8am – 5pm (Registration from 8 until 9am, prompt kickoff at 9am).
This will include refreshments and Lunch.

A social event is being worked on, perhaps either an event Thursday night or something afterwards on Friday. Looking for sponsors!

Agenda to come…
Please reach out to Chris Schlipalius if you would like to present your user case study at the meeting or you would like to sponsor a networking event.

This day may also include afternoon sessions for Spectrum Protect, Spectrum Archive and LSF as time and content planning allows.

Event details: http://bit.ly/2YOFQ3u

Spectrum Scale China User Meeting 2019
2019-11-05 09:00 - 16:00 (Europe/London) | IBM Shanghai Client Center, 55 Chuanhe Road, Zhangjiang, Pudong, Shanghai, China
IBM will host a Spectrum Scale User Meeting in Shanghai on November 5.
Please register via mail if you plan to attend:
yingjunsha@jianuodigital.com
Draft agenda

09:0009:30Registration and Morning Coffeen/a

09:3009:40WelcomeMatt Geiser (IBM)

09:4010:10What is new in Spectrum Scale/ESS?Matt Geiser (IBM) / TBD

10:1010:40Spectrum Scale Erasure Code Edition (ECE)Shen Lin Feng (IBM)

10:4011:10Spectrum DiscoverFrank Lee (IBM)

11:1011:30Customer TalkTBD

11:3012:00Spectrum Scale for containerGong Wei (IBM)

12:0013:00Lunch

13:0013:30Spectrum Scale for CloudMeng He (AWS China)

13:3014:00Spectrum Scale for AIMeng Wei (IBM)

14:0014:30Spectrum Scale Advanced Data Management with Spectrum Archive EE and IBM Cloud Object StorageXu Tai Ming (IBM)

14:3015:00Spectrum Scale Customer Service UpdateLi Guang Lei (IBM)

15:0015:20IBM Spectrum Storage Online LabLiu Xin (IBM)

15:2016:00Roundtable and Q&AAll

Spectrum Scale user group meeting @SC19
2019-11-17 09:00 - 17:30 (America/Denver) | Hyatt Regency Denver, 650 15th Street, Denver, CO, 80202, United States
Start your SC19 experience off with the Spectrum Scale User Group! We’ll be hosting our yearly gathering on Sunday November 17, from 9:30-6:00 PM. This global user-led meeting provides an opportunity for peer-to-peer learning and interaction with IBM’s technical leadership team on the latest IBM Spectrum Scale roadmaps, latest features, ecosystem, licensing, and applications for AI.
This is a free event, but registration in advance is required: www.ibm.com/events/2019/SC19_BC
IBM Spectrum Scale User Group Meeting
Sunday, November 17, 2019
9:30 am – 12:00
The morning session is dedicated to new and potential users of IBM Spectrum Scale. Experienced users and technical leaders will host an introductory session featuring key use cases, basic concepts and an interactive Q&A.
12:30 pm – 6:00 pm
The afternoon session is the main user group meeting. Topics include recent enhancements, product strategy and AI ecosystems. Client and partner presenters will present details of their deployment or their solutions. This year’s agenda will also feature lightning talks on user-generated topics. A cocktail reception will follow the main agenda at 6 pm to continue the discussion.
See you in Denver!
Draft agenda

09:3009:45New users: Welcome

09:4510:15New users: Spectrum Scale use casesUlf Troppens (IBM)

10:1511:15New users: Spectrum Scale concepts and features Tomer Perry (IBM)

11:1511:45New users: Q&AAll

11:4512:30Lunch breakn/a

12:3012:45WelcomeKristy Kallback-Rose & Bob Oesterlin (User Group) / Amy Hirst (IBM)

12:4513:05Spectrum Scale & ESS updateChris Maestas (IBM)

13:0513:25Evolving a campus wide research storage capture, analysis and management strategyJake Carrol (University of Queensland)

13:2513:40Starfish Storage Zones: Enabling Users to Manage their Own Storage (Partner Talk)Jacob Famer (Starfish)

13:4014:00AI ecosystems and solutions with Spectrum Scale (update)Piyush Chaudhary (IBM)

14:0014:15It’s the network again (even on Windows): A turorial on how to use nsdperf to pinpoint complex networking issues (Sponsor Talk)Michael Hennecke (Lenovo)

14:1514:45Spectrum Scale Erasure Code Edition (ECE)Nikhil Khandelwal (IBM) / Olaf Weiser (IBM)

14:4515:15Coffee breakn/a

15:1515:45Lightning talksSimon Thompson (Moderator)

15:4516:00Tip the Scale in your favor: How an IBM “Full Stack” Partner can help (Sponsor Talk)Chris Bogan (Mark III)

16:0016:15Fastest File Servers in Public CloudPinkesh Valdria (Oracle Cloud)

16:1516:35Spectrum Scale strategy updateTed Hoover (IBM) / Wayne Sawdon (IBM)

16:3516:55A Review of Spectrum Scale Migrations and Challenges at St. JudeWill Schmied (St. Jude Children's Research Hospital)

16:5517:15Spectrum Scale Data Migration With AFM - A Cautionary TaleBob Oesterlin (Nuance)

17:1517:45Deep Dive - Block and inode allocation in Spectrum ScaleTomer Perry (IBM)

17:4518:00Ask us anythingAll

18:0020:00Get togethern/a

Spectrum Scale user group @Computing Insights
2019-12-05 11:30 - 13:00 (Europe/London) | Manchester Central, Windmill St, Manchester, M2 3GX, United Kingdom
This event is part of Computing Insights UK and attendees must be registered for CIUK to attend the meeting.

StartEndTitleSpeaker

1511:3011:45Big Data and AI at ScaleTomer Perry (IBM)

2011:4512:05Experience of CES as an NFS ServerFrederik Ferner (Diamond Light Source)

2512:0512:30TBCSven Oehme (DDN)

3012:3013:00Deep Dive – Block and inode allocation in Spectrum ScaleTomer Perry (IBM)

CANCELLED: Spectrum Scale User Group @SC Asia 2020
2020-02-25 12:30 - 19:30 (Australia/Perth)
The SC Asia event unfortunately is cancelled today as of the 13th Feb 2020 due to travel restrictions that the respective organisations of the sponsor and organiser, have now chosen to implement.

German Spectrum Scale User Meeting 2020 (New User Day)
2020-03-03 | IBM Client Center Ehningen, IBM-Allee 1, Ehningen, 71139, Germany
The Spectrum Scale Strategy days in Germany are organised by IBM. Full details are available on the IBM Event site.

German Spectrum Scale User Meeting 2020
2020-03-04 - 2020-03-05 | IBM Client Center Ehningen, IBM-Allee 1, Ehningen, 71139, Germany

CANCELLED: Worldwide/UK Spectrum Scale 2020 new user day
2020-05-12 | IBM South Bank, 76/78 Upper Ground, London, SE1 9PZ, United Kingdom
Due to the ongoing COVID-19 outbreak, this in-person event is cancelled. We hope to bring a digital version!

The new user day is something we’ve tried to squeeze around the main agenda in the past, this year is running as a completely separate day prior to the main event.
Please note that registration for the new user day is separate to the main 2 day event.
Agenda
The agenda for the new user day is currently being developed but the outline is provided here to help you plan travel.

StartEndTitleSpeaker

3009:3010:00ARRIVALS & REGISTRATION

1510:0010:15Opening, welcome

Spectrum Scale Use Cases

Spectrum Scale Features & Functions

6012:0013:00LUNCH

Live Demo

Orientation for New Users

3015:0015:30BREAK

How to design a Spectrum Scale environment

3017:3018:00Q&A and WRAP UP

CANCELLED: Worldwide/UK 2020 Spectrum Scale User Group
2020-05-13 - 2020-05-14 | IBM South Bank, 76/78 Upper Ground, London, SE1 9PZ, United Kingdom
Due to the ongoing COVID-19 outbreak, this in-person event is cancelled. We hope to bring a digital version!

This year’s Spectrum Scale user group event has a fantastic agenda of Spectrum Scale experts and will take place over 2-days, again at IBM’s client centre overlooking the River Thames in London. 2020 sees 10 years of the user group in the UK! The event will provide you with the opportunity to hear the latest Spectrum Scale updates from filesystem experts as well as hear from other users on their experiences. Similar to previous years, we’re also holding smaller interactive workshops to allow for more detailed discussion.
In addition to the main event, we’re also running a new user day on 12th May. Please note that registration for the new user day is separate.
Sponsors
This event wouldn’t be possible without the support of our sponsors and we thank them for their support!

Agenda
The agenda is currently being developed, the outline agenda to give an indication of timings is given below but is subject to change.
Day 1 – May 13th 2020

StartEndTitleSpeaker

6009:0010:00ARRIVALS & REGISTRATION

3010:0010:30Opening, welcome & pop quizSimon Thompson (Chair)

6012:0013:00LUNCH

3014:4515:15BREAK

4016:5017:30Meet the Devs - Ask us AnythingAll

1017:3017:40WRAP UPSimon Thompson (Chair)

13518:3022:45User Group Networking / DinnerRegistration and Badge/Lanyard required

Day 2 – May 14th 2020

StartEndTopicSpeaker

3008:3009:00ARRIVALS

3008:3009:00Spectrum Scale Licensing

6009:0010:00

6010:0011:00

6011:0012:00

6012:0013:00LUNCH

3013:0013:30

3013:3014:00

3014:0014:30BREAK

2014:3014:50

2014:5015:10

2015:1015:30

2015:3015:50

1015:5016:00CLOSESimon Thompson (Chair)

IBM Virtual User Groups Day
2020-05-19 - 2020-05-20
IBM are hosting an Virtual User groups day which includes a focus on Spectrum Scale

SSUG::Digital: 001 - What is new in Spectrum Scale 5.0.5?
2020-06-18 16:00 - 17:30 (Europe/London) | Digital Event
At each of our user group events, we pretty much always start off with “What’s new in XXX release?” and with Spectrum Scale 5.0.5 having just been release, we’re doing the same with the new series of SSUG::Digital events.

Download slides here

Blog Post: What is new in Spectrum Scale 5.0.5?
Q&A
Q: How would one go about obtaining an IBM Contact?
A: Do you have an IBM Sales rep already? That would be the first person to contact. If you purchased through a business partner, that is your first point of contact.
Q: Will EUS releases going to be consumed preferentially by the ESS code distributions? Maybe would make it easier to coordinate Spectrum Scale and ESS code levels when we need to update Spectrum Scale.
A: That’s the intention. Of course, based on exact timing of releases and ESS needs for new functions, it might not work out in all cases.
Q: Regarding Thin Provisioning support: Are there any test cycle for other vendors like Hitachi already happening?
A: Contact IBM to discuss the specific vendor requirements. If you have a specific piece of hardware you want to see supported, file an RFE.
Q: Does the support of compression mean that you will also support the FCM Modules in an ESS 3000 or in storages like the IBM Flash Systems 9000/7000/5000?
A: It is under evaluation to support FCM Modules of IBM Flash Systems and future ESS models with future Spectrum Scale releases.
Q: Any estimate on performance differences between Spectrum Scale 4.2.3 and Spectrum Scale 5.0.5?
A: There are incremental performance improvements in every Spectrum Scale releases. There is a significant performance jump from Spectrum Scale 4.2.3 to Spectrum Scale 5.0.0 to meet the performance commitments for CORAL. Some performance improvements have been covered at previous User Group Meetings and are available on (https://www.spectrumscaleug.org/presentations). It is also planned to provide a performance update in a future Expert Talk.
Q: Are there plans to set the all-to-all daemon connections to defaults?
A: No as default at the moment. See Expert Talk 004 “Performance Update” for more details:
https://www.spectrumscaleug.org/event/ssugdigital-spectrum-scale-expert-talks-update-on-performance-enhancements-in-spectrum-scale
Q: Is all-to-all connection establishment limited to the nodes inside a cluster or does it include all nodes from remote clusters that are already connected to a FS?
A: Local and remote.
Q: ”cp –preserver=xattr” feature is it something that 5.0.5 will enable for copies to “5.0.5”? Aka migrating data from 4.X to 5.X, or only from 5.0.5 to future versions?
A: You can copy files from Spectrum Scale 5.0.5 to previous and further version preserving extended attributes. You cannot copy the extended attributes from previous versions of Spectrum Scale to Spectrum Scale 5.0.5.
Q: Is the “cp –preserve…” a function of the RHEL release, or is there a version of cp included with Spectrum Scale?
A: The system calls listxattr, getxattr, and setxattr were extended to retrieve ACLs as extended attributes.
Q: So, then the version of the Spectrum Scale filesystem doesn’t matter, just the version of RHEL? (for the cp question).
A: Those system calls are extended by Spectrum Scale Scale at the VFS layer, so it would depend on the version of Spectrum Scale (and kernel extensions).
Q: Are there any plans for Scale Protocols to support SMB transparent failover?
A: As functionality is introduced into Samba code base, IBM look into how they can pick support up for that. It’s a topic being discussed in the Samba community for this type of failover. However, it’s known to be a hard problem.
Q: Any news on NFS4.1 support?
A: It is in plan to support it later this year (subject to IBM plan commitment disclaimer).
Q: On Spectrum Scale 4.2.3 and RHEL 7, we’ve had problems with the Ganesha daemon using steadily more memory over time, requiring us to failover / stop / start / failback periodically. Has Ganesha’s memory requirements been reduced in Spectrum Scale 5.0.5, or is there better visibility into what is driving memory usage?
A: This issue was traced back to a C library memory allocation fragmentation issue. There was a fix put into the Ganesha code to force the release of this unused fragmented memory. This fix was made available last year in the 5.0.x release stream.
Q: Hallo All, what is now the strategy to support object protocols like S3. It is missing the currency.
A: We have renewed focus on Object protocol. We plan to support the Train release in the fall release. Going forward we will try to update the Swift/S3 version once a year to make sure it stays current (subject to IBM plan commitment disclaimer). If you have specific interest in S3 applications, please contact us as we would like to hear about your requirements and use cases.
Q: We are working on a deployment of CSI 1.1.0. When is snapshot support happening.?
A: Snapshot support is planned to be available in a CSI driver update coming in late 3Q early 4Q 2020 (subject to IBM plan commitment disclaimer).
Q: Are there any news about the restriction of GUI HA with CSI?
A: This is a high priority requirement for the fall release. It’s not officially committed yet, but we are definitely trying to squeeze it in (subject to IBM plan commitment disclaimer).
User group host: Simon Thompson
Speakers:

Speaker NamePhotoBioSocial connections

Chris MaestasChris is an Executive Architect for IBM File and Object Storage Solutions with over 20 years of experience deploying and designing IT systems for clients in various spaces. He has experience scaling performance and availability with a variety of file systems technologies. He has developed benchmark frameworks to test out systems for reliability and validate research performance data. He also has led global enablement sessions online and face to face where discussing how best to position mature technologies like Spectrum Scale with emerging technologies in Cloud, Object, Container or AI spaces.Twitter: @cdmaestas
LinkedIn: https://www.linkedin.com/in/cdmaestas

Mathias DietzMathias works in the Spectrum Scale development team in Kelsterbach (Germany) as a software architect responsible for Reliability, Availability and Serviceability (RAS). Part of his role is to drive reliability improvement into Spectrum Scale and improve Health & Performance monitoring, Proactive Services and CES failover.

Ismael Solis MorenoIsmael works in the Spectrum Scale development team in Guadalajara Mexico as a data scientist and performance analyst. He is responsible for evaluating Spectrum Scale new features and releases performance. Part of his role is to analyze datasets to identify points of performance improvement providing insights to the development teams.LinkedIn: https://www.linkedin.com/in/ismaelsm

SSUG::Digital: 002 - Best Practices for building a stretched cluster
2020-07-13 16:00 - 17:30 (Europe/London) | Digital Event
Talk 2 in the SSUG::Digital series looks at how to build a stretched cluster. What are the best practices? What pitfalls are there? Why would you consider a stretched cluster built with Spectrum Scale, as opposed to one of the alternative approaches to high availability? How do stretched clusters work, and what considerations go into planning a successful stretched cluster? We will examine the theory behind Spectrum Scale stretched clusters, review some best practices for designing stretched clusters, and talk about a few cases where stretched clusters have been successfully deployed.

Download slides here

Q&A
Q: For DR use case where ClusterA and ClusterB are the 2 separate data centres (DC A and DCB), do I need my Tiebreaker Quorum node installed in Data Centre C?
A: (This is covered in the presentation). It is recommended to have the tiebreaker quorum node at a third site but it could be in one of the sites with the caveat that if that site goes down the second site will not be able to stay up.
Q: The documentation shows a high speed shared storage is needed…does it mean that san fabric should be merged over ISL for volume allocation across site?
A: When using Spectrum Scale replication for stretch clusters there is no need to for the SAN to be extended across the sites. The stretched cluster architecture described in the presentation works even when underlying storage does not replicate the data across sites.
Q: Will there be any performance difference between extended SAN and accessing NSD over network using their owner?
A: Well aside from the protocol difference block vs file, it depends on the type of connectivity you have to SAN vs network. Spectrum Scale has been placing more resiliency in recent releases for what to do for network behaviour (eg recently proactiveReconnet feature was added to Spectrum Scale).
Q: Does 10ms latency required between SiteA, SiteB and also Tiebreaker quorum node? Can my tiebreaker quorum node have higher latency?
A: Yes, the third site can have a higher latency but it should still be “within reason”. So maybe double that number ie 20ms. It is recommended to keep it under a second.
Q: Is tiebreaker node hosted on AWS or any other Cloud Providers a supported configuration?
A: Yes, we have customers who using a public cloud for their third site.
Q: What is the RPO and RTO?
A: Remember that this is synchronous replication. So as long as you don’t run out of space on your storage there is 0 RPO. RTO answer depends on your workload and infrastructure. It depends on the rate of change of data change, your storage and the WAN.
Q: How to check/measure the rate of data change?
A: This really depends on the application and the rate of data change by the application. If you already have implemented Spectrum Scale, you can use the historical data from performance monitoring within Spectrum Scale to estimate the rate of data change.
Q: Do you have any general tips/recommendations regarding CES in a stretched cluster?
A: The CES nodes in your cluster need to be split between the two sites as they are still part of single cluster. SMB performs its own locking with ctdb component. Thus the latency between the CES nodes needs to be fairly small value. Also be aware if you have different address spaces on two sites, there may not be an automated failover of services and you may need to manually perform the same.

User group host: Bill Anderson
Speakers:

Speaker NamePhotoBioSocial connections

Lindsay ToddAs part of the Software Defined Infrastructure team at the IBM Washington Systems Center (WSC), Lindsay provides deep technical expertise with Spectrum Scale (GPFS), leveraging prior experience he gained while using it as a customer himself at a university supercomputing center. Lindsay continues exploring and using Spectrum Scale for WSC infrastructure needs, as well as helping many clients use it to build innovative solutions to their business problems.Blog: https://parallelstorage.com
LinkedIn: https://linkedin.com/in/rltodd

SSUG::Digital: 003 - Strategy Update
2020-07-27 16:00 - 17:30 (Europe/London)
Spectrum Scale Strategy UpdateToday is the AI era and we are going through huge explosion of data. Besides the AI revolution, we have clouds, hybrid clouds and data is moving from “on-prem” to various clouds, multi-clouds and back. Coupled with this data growth, Hardware is evolving with an increasing factor of 10. The IBM Spectrum Scale team continues to Invest heavily in adding exciting new features and technology to maintain its leadership as a premier file system. In this session, Wayne Sawdon (CTO) and Ted Hoover (Program Director) of the Spectrum Scale development team will give an overview of recent, upcoming features and strategy for Spectrum Scale.

Download slides here
Q&A
None
User group host: Bob Oesterlin
Speakers:

Speaker NamePhotoBio

Wayne SawdonWayne joined IBM in 1982 and worked on a variety of research projects including the QuickSilver Transactional Operating System. He spend most of the 90's on educational leave at Carnegie Mellon University working on Distributed Shared Memory and Software Defined Computer Architecture. Upon returning he joined the TigerShark research project which became IBM's General Parallel File System. Although Wayne has worked on most of the file system, he only admits to working on its data management. These days, Wayne serves as the CTO for Spectrum Scale and ESS.

Ted HooverTed Hoover is a Program Director within IBM’s Spectrum Scale product development organisation. Ted is responsible for the worldwide development of Spectrum Scale cloud, container, and performance engineering teams.

Spectrum Scale Usergroup Australia
2020-08-27 10:00 - 16:00 (Australia/Sydney) | Digital Event
Hello so the next Australian Usergroup will be online.
All details including the draft program can be found at: https://bit.ly/2P1CXF5
Please pick up a ticket if you are interested in attending as only those on the attendees list will be sent the event link in coming weeks.
The event will be recorded for sharing later.
See you there!

SSUG::Digital: 004 - Update on Performance Enhancements in Spectrum Scale
2020-09-10 16:00 - 17:30 (Europe/London) | Digital Event
Update on File Create and MMAP performance, optimised code for small DIO.
Spectrum Scale is a highly scalable, high-performance storage solution for file and object storage. IBM continues to enhance Spectrum Scale performance, in response to recent hardware advancements and evolving workloads.
This presentation will discuss performance related improvements in Spectrum V5, focusing on enhancements made in support of AI and HPC use cases, including improvements to MMAP reads, file create performance, and small direct IO. In addition we will review some performance numbers measured on the IBM ESS 5000.

Download slides here
Q&A
Q: I assume copy of these charts will be posted to Spectrum Scale User Group “Presentations” web page?
A: Yes, for all episodes the slides and video should be posted afterwards.
Q: Please expand on other areas of performance improvements within GPFS that IBM is working on now?
A: Which areas would you like to see improved?
Q: Will prefetch still happen after the slow second IO?
A: Regarding ‘will prefetch still happen after the slow second IO’ – I know that Ulf said we should handle any more prefetch questions in another talk, but let me just comment on one case: we make decisions to prefetch after the associated I/Os are complete, so prior to prefetch kicking in, a slow I/O might delay the decision to start prefetching.
User group host: Simon Thompson
Speakers:

Speaker NamePhotoBio

John Lewars (IBM)John Lewars is a Senior Technical Staff Member leading performance engineering work in the IBM Spectrum Scale development team. He has been with IBM for over 20 years, working first on several aspects of IBM's largest high performance computing systems, and later on the IBM Spectrum Scale (formerly GPFS) development team. John's work on the Spectrum Scale team includes working with large customer deployments and improving network resiliency, along with co-leading development of the team's first public cloud and container support deliverables.

Jürgen Hannappel (DESY)Jürgen Hannappel works in the scientific computing group of the DESY IT department on data management for EuXFEL and Petra III. With a background in particle physics his interests shifted towards computing over time as his place of work moved from CERN and Bonn University to DESY

Olaf Weiser (IBM)Olaf works with GPFS for over 15 years now. He started his GPFS career in one of the worlds biggest telecommunication companies as a technical administrator. Since more than 10 years, Olaf is with IBM as storage consultant and performance specialist. Recently, he joined IBM Research and Development and works on enhancements in Spectrum Scale to adopt client and customer's needs in the product.

Support Webinar: AFM – Deep Dive and Debugging (NA/EU Session)
2020-09-16 11:00 - 12:00 (America/New_York)
IBM Spectrum Scale Webinars are hosted by IBM Spectrum Scale Support to share expertise and knowledge of the Spectrum Scale product, as well as product updates and best practices. This webinar will focus on AFM functionality in IBM Spectrum Scale.
>> Register here <<
Agenda:

Introduction to AFM
AFM basics and architecture
AFM commands for monitoring and debugging
AFM internal directories
AFM debugging workflow
Common AFM failures
AFM configuration parameters

Support Webinar: AFM – Deep Dive and Debugging (AP/JP Session)
2020-09-17 13:30 - 14:30 (Asia/Singapore)

SSUG::Digital: 005 - Update on functional enhancements in Spectrum Scale (inode management, vCPU scaling, considerations for NUMA)
2020-09-22 16:00 - 17:30 (Europe/London) | Digital Event
Spectrum Scale is a highly scalable, high-performance storage solution for file and object storage. It started more than 20 years ago as research project and is now used by thousands of customers. IBM continues to enhance Spectrum Scale, in response to recent hardware advancements and evolving workloads.
This presentation will discuss selected improvements in Spectrum V5, focusing on improvements for inode management, vCPU scaling and considerations for NUMA.
Part 1 (inode management):

Part 2 (vCPU scaling and NUMA considerations):

Download slides here
Q&A – inode management
Q: If I make the block size 8k, can an inode stuff a file of that size?
A: No , the max inode size is 4K. As discussed on the call, changing the blocksize (including the metadata blocksize) doesn’t impact the size of an inode, which is currently limited to 4K.
Q: When files are created an inode number is assigned from 1-Max. 32bit applications can only address inodes up to 4b. With millions of temp files created during application runs, inodes get used up very quickly. But once the job is finished the files are deleted, but the inodes are not recycled. This results in filling the inodes while the file system isn’t full. Can inodes/files that have been deleted, have their inode recycled for future use?
A: Inodes do get reused after deletion. How many independent filesets do you have? If you only had the root fileset and you set the maxInodes to less than 4b, then you can never have an inode number greater than 4b.
Q: Do we have any idea how long it takes to re-layout the inode allocation map? And can that be done while the FS is mounted?
A: The run-time for this operation will depend on the size of the existing inode allocation map file since we will be migrating data from the exiting map to the new map. In one customer engagement, the migration completed in an hour and in another case it took 18 hours.
While this operation can theoretically be done while fs is mounted, we have currently restricted it to be done with file system offline for safety reasons. We are evaluating to make this operation online in a future release. The re-layout parameters can however be tested with file system mounted.
Q: Are there counters that report the lock collisions/waiters for lock contention that would indicate whether a re-layout is desirable?
A: ‘mmfsadm dump ialloc’ provides counters on segment search. Grep for ‘inodes allocated from’. Ideally, we expect allocations to happen from ‘inodes allocated from list of prefetched/deleted inodes’ or ‘inodes allocated from current ialloc segment’. Also, long waiters during file creation is indication of inode space pressure.
Q: Why does a large number of NumNodes influence mmdf run time? (seen some minutes)
A: mmdf fetches cached data. This should not be impacted by cluster size.
Q: How does NumNodes relate to the number of segments?
A: The number of inode allocation map segments are chosen such that every node can find a segment with free inodes even if 75% of all segments are full. This has to do with the inode expansion getting triggered only when the inode space is 75% full. We want inode allocation to continue while the inode expansion is taking place. This means that the number of segments would be roughly 4 times NumNodes.
Q: Are there any general recommendations for initial inode allocation? I know this depends on the filesystem’s expected use. We typically just base it roughly off existing systems.
A: Use the default value of allocated inodes (by omitting NumInodesToPreallocate argument of –inode-limit option of mmcrfs/mmcrfileset) when creating file system/independent fileset and let the inodes expand on demand.
Q: How is the inode allocation map, and its segmentation, affected if metadata NSDs are added or deleted?
A: The inode allocation map is not affected by newly added NSDs as it only tracks inode state. The block allocation map is the one that tracks free/used disk blocks and will get updated on new disk add/delete.
Q: Can we shrink the inode space, if we by mistake allocate a large inode space using –inode-limit?
A: No.
Q: When files are deleted, does the recovery of free inode happen in lazy way? One customer has just reported that after deleting data from 5TB filesystem, free space is not reflecting on the filesystem.
A: Yes. The files are deleted asynchronously in the background. You can run ‘mmfsadm dump deferreddeletions’ to see the number of inodes that are queued for deletion in mounted file systems.
Q: At what version is automatic inode expansion available?
A: Since the earliest spectrum scale versions.
Q: How do you indentify the metanode?
A: Here is an example:
ls -i testfile
68608 testfile
Then find this inode number in ‘mmfsadm dump files’. (note that the mmfsadm dump command should be avoided in production)
===== dump files =====
[… search on inode]
inode 68608 snap 0 USERFILE nlink 1 genNum 0x49DE6F0F mode‑
The above is an example of how you might lookup the metanode for a file.
You can map the cluster name by looking at the ‘tscomm’ section of a dump, e.g.:
===== dump tscomm =====
[…]
Domain , myAddr <c1n2> (was: <c1p0>)[…]
UID domain 0x1800DF65038 (0xFFFFB6768DF65038) Name “c202f06fs04a-ib1.gpfs.net”
Q: Is metanode transient?
A: A metanode is a per file assignment. It lasts for as long as there are open instances of the file. The assignment is dynamic and the metanode role may automatically migrate to other nodes for better performance.
Q: If some of the node went down and if metanode unable to get update from those failed nodes: In such situation how updates are maintained by the metanode?
A: A non-metanode will sends its updates to the metanode before it writes any dependent blocks to disk. If the non-metanode went down before it could send its updates, then log recovery will ensure that there are no inconsistent modifications to disk data by the non-metanode. Spectrum Scale only guarantees persistence of data/metadata from the last sync window.
Q: Can we limit metanode to migrate to remote node? Also, will it help in improving performance if limit metanode in storage cluster?
A: Metanode performance depends on how many nodes are sending metanode updates and how expensive the network send is. The file system uses such heuristics to determine the optimal metanode placement. In most cases it is best to let the file system make this decision. The only known use case for preventing metanode migration to remote node is if the remote node is in a compute cluster which cannot afford the overhead of a metanode operation. For this rare case we have an undocumented configuration parameter to force the metanode to be in the storage cluster.
Q: Sometime when we delete large data, it takes significant time to show free space in df -h command output. Do we need to run mmrestripfs to reclaim the deleted space faster?
A: ‘df -h’ would return cached information on free space. It is likely that the large file that was deleted has not freed up its space as file deletes happen in the background. You can use ‘mmfsadm dump deferreddeletions’ to get a count of number of inodes that are queued for background deletion. If the node is not overloaded on I/O and you find that the number of to-be-deleted inodes are not reducing at a reasonable rate (depending on the file size and I/O througput of the node), then we would need to investigate further by collecting dumps and traces. Please open a ticket with IBM support in such a case. The mmrestripefs command is for restoring/rebalancing data and metadata replicas. It would not have any impact on speeding up background file deletion.
Q&A – vCPU scaling and NUMA considerations
Q: We see in the mmfs log now following messages, what does it mean? What is missing?
[W] NUMA BIOS/platform support for NUMA is disabled or not available. NUMA results are approximated and GPFS NUMA awareness may suffer.
A: That means libnuma was found but numa_available() returned false. This is a platform firmware functionality shortcoming. Spectrum Scale can still get a lot of information as some is derivable from /proc . File a ticket with your server vendor that libnuma :: numa_available() returns false .
Q: So, any recommendations on POWER9 for SMT settings? AIX versus Linux on Power? We used to suggest smaller SMT modes in the past.
A: We are running SMT-4 on some large POWER9 systems. Evaluate based on I/O vs workload needs as discussed verbally.
Q: Are there any special NUMA considerations for AMD systems which are different to NUMA considerations for Intel systems?
A: This is highly dependent upon the processor and chip set independent of brand and based on what that processor and chipset offer for tuning. We do not have any prescriptive guidance.
User group host: Simon Thompson
Speakers:

Speaker NamePhotoBio

Michael HarrisMike is a Senior Software Engineer on the Spectrum Scale Core Team. Mike has a deep background on OS kernel, device drivers, virtualization, and system software with focus on NUMA, atomics and concurrency, high cpu count concurrency. On GPFS focusing on NUMA and scaling as well as DMAPI and host file system integration and system calls.

Karthik IyerKarthik Iyer is a Senior Software Engineer in Spectrum Scale Core. Karthik has 18 years of design and development experience in distributed system software, specifically in the areas of file system core and database management. Karthik also specialises in trouble shooting Spectrum Scale corruption related issues.

SSUG::Digital: 006 - Persistent Storage for Kubernetes and OpenShift environments
2020-10-06 16:00 - 17:00 (Europe/London) | Digital Event
This episode will discuss the Spectrum Scale Container Storage Interface (CSI). CSI is a standard for exposing arbitrary block and file storage systems to containerized workloads on container orchestration systems like Kubernetes and OpenShift. Spectrum Scale CSI provides your containers fast access to files stored in Spectrum Scale with capabilities such as dynamic provisioning of volumes and read-write-many access.

Download slides here
Episode 2: Best Practices for building a stretched cluster
Q&A
Q: This slide (titled Spectrum Scale CSI Driver – Architecture) shows CPU architecture is x86
A: Yes, with Spectrum Scale CSI Driver 2.0.0 only x86 is supported. The support for other architectures (IBM Power and IBM Z) will be provided in upcoming releases (IBM usual roadmap disclaimers apply).
Q: Is the management of storage class available via Ansible?
A: Setting up a storage class is a one-time operation. While it might be done using Ansible (and Kubernetes integration modules), clients usually do the management using the Kubernetes or Openshift CLI or GUI.
Q: Will the slides be provided post this presentation?
A: Yes. You will find the chart decks, recordings, Q&A and related information for all past talks including this one at https://www.spectrumscaleug.org/experttalks/.
Q: Once you have CSI driver support for non x86_64 platforms, will the Spectrum Scale cluster be able to be heterogeneous (AIX, Linux, x86_64 and ppc64le)? Will this cluster support AIX NSD only nodes?
A: In the first release for non x86_64 platforms, all worker nodes that have Spectrum Scale client installed need to be of same CPU architecture and the same operating system. If there are AIX NSD nodes, those must be outside of Kubernetes cluster. AIX NSD only nodes might be integrated by remote mounting the storage cluster to a client Spectrum Scale cluster that runs the Kubernetes workload.
Q: Is Network Load Balancer a per-requisite must have for the CSI deployment?
A: No it isn’t.
Q: Is there a possibility to have Spectrum Scale clients installed within containers?
A: We are working on a capability called Container Native Spectrum Scale (CNSS) where Spectrum Scale will run inside a container. The initial release is planned for December 2020. (Disclaimer: All dates are subject to change; IBM usual roadmap disclaimers apply)
Q: Do we need to have an x86 “only” Spectrum Scale/OpenShift cluster and a ppc64le “only” Spectrum Scale/OpenShift cluster?
A: The requirement of same CPU architecture and same operating system is only for Spectrum Scale Client node which are part of Kubernetes/ Openshift cluster. NSD server can be of other platform (as per Spectrum Scale support matrix at https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html)
Q: Any plans for ability to self-provision Spectrum Scale clusters with containers?
A: Container Native Spectrum Scale (CNSS) will have an Operator that will deploy and configure a Spectrum Scale cluster automatically. It will also remote mount the filesystem from a Spectrum Scale Storage Cluster.
Q: One of the issues that we are trying to solve is to isolate Spectrum Scale IO with respect to each tenant/application/user on a single server just like how we can isolate cpu/network with cgroups. Would spectrum scale on containers help us in isolating or qos with storage IO?
A: Running Spectrum Scale in a container or CSI by itself will not address QoS. A new fileset based QoS capability, with CSI, will be able to handle this in a future release. (Disclaimer: All dates are subject to change; IBM usual roadmap disclaimers apply)
Q: Does OpenStack have to be managed via the web gui. Can it be controlled via a CLI?
A: You are free to use either CLI or GUI.
User group host: Bill Anderson
Speakers:

Speaker NamePhotoBio

Smita RautSmita Raut is a Senior Software Engineer with IBM Storage Labs in Pune, India. She works with the IBM Spectrum Scale development team as the architect for persistent storage for containers. In her nine years with IBM, she has lead the development on various projects, including Object protocol for IBM Spectrum Scale and enablement of IBM Spectrum Scale on public cloud. She is an active technical blogger and has published several blogs on object protocol and container storage interface driver.

Harald SeippHarald Seipp is a Senior Technical Staff Member with IBM Systems in Germany. He is the founder and Technical Leader of the Center of Excellence for Cloud Storage as part of the EMEA Storage Competence Center. He is providing guidance to worldwide IBM teams across organizations, and works with customers and IBM Business Partners across EMEA to create and implement complex storage cloud architectures. His more than 25 years of technology experience includes previous job roles as Software Developer, Software Development Leader, Lead Developer, and Architect for successful software products, and co-inventor of an IBM storage product. He holds various patents on storage and networking technology.

Renar GrunenbergRenar Grunenberg is since 27 years at HuK-Coburg. He leads the backup and storage team and is responsible for all the storage and backup stuff in his department and company. Renar has 15 years experience with Spectrum Scale including CES, CSI, ESS and normal core function. In this episode Renar will discuss a use case for Kafka self-service with K8s and Spectrum Scale CSI.

Simon ThompsonSimon Thompson is the Research Computing Infrastructure Architect within Advanced Research Computing at the University of Birmingham. He oversees the infrastructure and systems team, running the University's HPC and research data systems. This involves experimenting (and breaking) new technology. Simon is also chair of the Spectrum Scale user group in the UK.

SSUG::Digital: 007 - Manage the lifecycle of your files using the policy engine
2020-10-21 16:00 - 17:00 (Europe/London) | Digital Event
This episode will provide a comprehensive introduction to the IBM Spectrum Scale policy engine. It highlights the underlying architecture and how policies are executed in a IBM Spectrum Scale cluster. This episode also discusses example rules and policies facilitating Information Lifecycle Management accompanied with practical tips.

Download slides here
References

Whitepaper: IBM Spectrum Scale ILM and Archiving Policies – A practical Guide
Spectrum Scale ILM policy examples and scripts
Apache Tika

Q&A
Q: Which type of nodes participate in policy execution?
A: Depends on the nodes specified with the -N option of the mmapplypolicy command. If the -N option is not specified, then the command runs parallel instances of the policy code on the nodes that are specified by the defaultHelperNodes attribute of the mmchconfig command. If -N is specified then the command runs parallel instances on the nodes or node class specified with the -N option. For more information see the IBM Spectrum Scale knowledge center: https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1adm_mmapplypolicy.htm
Q: Can I somehow identify the type of a file via the policy engine, e.g. via the magicbyte? Or do I have to rely on the file extension?
A: The policy engine does not allow access to the data – only the file’s metadata including extended attributes can be evaluated by the policy engine. To identify the type of a file with the policy engine an EXTERNAL LIST rule can be used along with an external script that determines the type of files.
Q: Will the external tool process the filelist in parallel on all nodes, which are used to generate the filelist?
A: Yes, if an external tool or interface script is defined in an EXTERNAL POOL rule then this script is executed on all nodes that are specified with the -N option of the mmapplypolicy command. This assumes that all node specified with the -N option have access to the interface script. If this is not the case, then the policy run fails. You can control the number of instances of the external tool pool with the option -m and the number of files passed to one instance of the external pool with the option -B of the mmapplypolicy command.
Q: Are they any limitations or recommendations around length of rules in policy files? For example, we have ~750 filesets we want to place data on a specific pool. Should we just have one rule, or many rules for this?
A: Placement policies will be stored in a single file. The challenge is not so much the length of the file but the number of placement rules contained in the policy files. Whenever a file is created the policy engine must walk through all rules to find a match. If there are many rules, this will delay the file creation. Therefore I recommend to keep the number of placement rules low. For example, you could organize the placement policies by storage pools. There is a limit of eight storage pools, thus this would lead to maximal eight placement rules. In each rule you can use the FILESET statement to specify multiple filesets to be placed on a pool.
User group host: Bob Oesterlin
Speakers:

Speaker NamePhotoBio

Nils HausteinNils Haustein is Senior Technical Staff Member with IBM Systems. He is responsible for design and implementation of backup, archiving, file and object storage solutions. Nils provides guidance to IBM teams and consults with clients and business partners world wide. Nils has co-authored the book "Storage Networks explained". As leading IBM Master Inventor he has created more than 170 patents and is a respected mentor for the technical community world wide.

SSUG::Digital: 008 - Scalable multi-node training for AI workloads on NVIDIA DGX, Red Hat OpenShift and IBM Spectrum Scale
2020-11-04 16:15 - 17:45 (Europe/London) | Digital Event
Nvidia and IBM did a complex proof-of-concept to demonstrate the scaling of AI workload using Nvidia DGX, Red Hat OpenShift and IBM Spectrum Scale at the example of ResNet-50 and the segmentation of images using the Audi A2D2 dataset. The project team published an IBM Redpaper with all the technical details and will present the key learnings and results.

Download slides here
Related material

IBM Redpaper: Deployment and Usage Guide for Running AI Workloads on Red Hat OpenShift and NVIDIA DGX Systems with IBM Spectrum Scale
Episode 6: Persistent Storage for Kubernetes and OpenShift environments

User group host: Bob Oesterlin
Speakers:

Speaker NamePhotoBio

Thomas SchoenemeyerThomas Schoenemeyer is a Senior Solution Architect and works for the NVIDIA EMEA Automotive Enterprise Team in Munich since March 2019. His area of expertise covers defining the computing architecture and infrastructure for the development of autonomous vehicles. He is helping customers to solve their technical challenges by integrating NVIDIA datacenter products such as the DGX family, the DriveSim/Constellation and NVIDIA Mellanox Networking solutions.

Gero SchmidtGero Schmidt is a Software Engineer at IBM Germany Research and Development GmbH in the IBM Spectrum Scale development group.
He joined IBM in 2001 working at the European Storage Competence Center in Mainz, Germany, providing technical presales support for a broad range of IBM storage products with a primary focus on enterprise storage solutions, system performance, and IBM POWER systems. He participated in the product rollout of the IBM System Storage DS6000/DS8000 series, co-authored several IBM Redbooks and has been a frequent speaker at IBM international conferences. In 2015 he joined the storage research group at the IBM Almaden Research Center in California, USA, where he worked on IBM Spectrum Scale, compression of genomic data in next generation sequencing pipelines, and the development of a cloud-native backup solution for containerized applications in Kubernetes/Red Hat OpenShift. In 2018 Gero joined the Big Data & Analytics team in the IBM Spectrum Scale development group.

Simon LorenzSimon Lorenz is an IT Architect at IBM Research and Development in Frankfurt, Germany. He joined IBM Germany in 1993 and has held various positions within IBM and IBM Research and Development. During international assignments, he helped to improve fully automated chip factories in Asia and US. Simon joined the IBM Spectrum Scale development team in 2014 and since then worked on: Openstack Swift integration, designing and building System Health and Proactive Services solutions. Since March 2019 he is the world wide leader of the IBM Spectrum Scale Big Data and Analytics team. His role includes designing and building solutions in the area of AI such as the Data Accelerator for AI and Analytics.

SC20 Meeting (Session 1): Storage for AI
2020-11-16 11:00 - 12:30 (America/New_York)
The Spectrum Scale user group meets annually at the SC conference. For 2020 the user group will be split into two 90 minute digital sessions. The second session will be on Wednesday Nov 18th, 11 am EST
>> Session 2: What is new in Spectrum Scale 5.1? <<
This session will focus on Storage for AI including a customer case study and a deep dive in new Spectrum Scale features which optimize the analysis of datasets stored in a data lake. Live Q&A will be available.
The session will be hosted by IBM and moderated by the user group.
>> Please register here to join the session. <<
Continental: Deep Thought – An AI Project for Autonomous Driving Development
Continental, a tier-1 automotive supplier, runs a new high-performance cluster based on NVIDIA DGX, IBM Spectrum Scale and IBM ESS to boost autonomous driving development performance. Continental uses this system for deep learning, simulation, virtual data generation and related workloads. The new cluster reduces development time from weeks to hours. Speakers from Continental and the business partner SVA will present business requirements and derived workloads, the Spectrum and ESS implementation, and first operational experience.
https://www.continental.com/en-us/press-/press-releases/continental-supercomputer-nvidia-228624
Data Accelerator for AI and Analytics (DAAA)
This talk focuses on the need of data orchestration in enterprise data pipelines. It provides details about data orchestration and how to address typical challenges that customers face when dealing with large and ever-growing amounts of data for data analytics. While the amount of data increases steadily, AI workloads need to speed up in order to be able to deliver insights and business value in a timely manner. IBM addresses these requirements with the Data Accelerator for AI and Analytics (DAAA), an expertly engineered solution based on new features delivered with Spectrum Scale 5.1. The session will include a demo of a complex proof-of-concept and present the key learnings and results.

SSUG::Digital: 009 - Deep Thought: An AI Project for Autonomous Driving Development
2020-11-16 16:00 - 17:00 (Europe/London) | Digital Event
Continental, a tier-1 automotive supplier, runs a new high-performance cluster based on NVIDIA DGX, IBM Spectrum Scale and IBM ESS to boost autonomous driving development performance. Continental uses this system for deep learning, simulation, virtual data generation and related workloads. The new cluster reduces development time from weeks to hours. Speakers from Continental and the business partner SVA will present business requirements and derived workloads, the Spectrum and ESS implementation, and first operational experience.

Download slides here
Q&A
Did Continental ever look at Spectrum Discover for tagging? Any feedback on pros/cons for its usefulness at Continental?
We did not have a chance to look into Spectrum Discover.
Related material

Continental: Automotive supercomputer
Continental press release

User Group Host: Bill Anderson
Speaker

Speaker NamePhotoBio

David EnenkelDavid Enenkel is the Head of IT Operations at Continental - ADC GmbH. He has over 20 years of IT-Infrastructure experience and moved into the area of ADAS 2,5 years ago. He and his team are supporting the ADC engineers to find and establish feasible IT solutions/services for the specific ADAS demands as well as operating the platforms and data centers. He led the Supercomputer project as the IT project manager and is happy to give an insight into all ups and downs building up such an environment.

Viktor PálViktor Pál is working as a Deep Learning Infrastructure Architect at Continental and is currently also acting as Technical Team Lead. He has more than 14 years of IT Infrastructure experience and started working at Continental two years ago. Viktor and his team are supporting AI development by designing, implementing and operating the environment that enables Deep Learning at Continental. He contributed to the Supercomputer project as a Solution Architect and he will talk about the challenges of AI Infrastructure as Code.

Jochen ZellerJochen is IT Architect at www.sva.de. He has more than 20 years of IT experience, is part of the SVA IT Architects team and technical leader of the Spectrum Scale team. His Spectrum Scale and infrastructure projects span various industries and fields, such as HPC, health care, autonomous driving, banking, insurance, etc. When he is not working with his customers, he sits in his glider and flies away - at least in summer.
