Computing Resources#
Independent Data Access Centers (IDACs) and Scientific Processing Centers (SPCs) One of the goals of the In-Kind Program is to augment the available resources for data- and compute-intensive use cases for the Rubin community. IDACs and SPCs, following the guidelines laid out by Rubin Observatory, will provide significant computing, storage, data, and experience for such use cases.
What kinds of contributed computing resources will be available?#
IDACs and SPCs will collectively provide access to CPUs, data storage, databases, and GPUs.
Who has proposed computing resource contributions?#
The map, filters, search, and table below list the IDACs and SPCs expected to be active during Rubin Operations. Click a country in the table, or a marker on the map, to jump to its full profile.
| Country | Type | Host | Storage PB-yr |
CPU Mhrs |
Products |
|---|---|---|---|---|---|
| Lite IDAC | La Plata | TBD | TBD | 2 / 12 | |
| Lite IDAC | Melbourne | 20 | 65 | 2 / 12 | |
| Lite IDAC | Rio de Janeiro | 65 | 56 | 3 / 12 | |
| Lite IDAC | Victoria | 186 | 630 | 4 / 12 | |
| SPC | Rijeka | 9 | 30 | 1 / 12 | |
| Lite IDAC | Copenhagen | 65 | 280 | 4 / 12 | |
| Lite IDAC (x2) | Mitaka, Tokyo | 46 | 141 | 5 / 12 | |
| Lite IDAC | Mexico City | 20 | 31 | 3 / 12 | |
| Lite IDAC | Poznań | 65 | 56 | 2 / 12 | |
| Lite IDAC | Nova Gorica | 7 | 20 | 1 / 12 | |
| Lite IDAC | Daejeon | 7 | 20 | 3 / 12 | |
| Lite IDAC | Barcelona | 7 | 21 | 1 / 12 | |
| Data Facility (Full IDAC) | Edinburgh | 2005 | 61 | 11 / 12 |
Lite IDAC — La Plata, Facultad de Ciencias Astronómicas (UNLP)
Object Table (subset) Other Data Products
Capacity (13-yr): Storage TBD PB-yr · CPU TBD Mhrs
Software & services: hipsdb tables for fast search and cross matching. image stamps DIA analysis. Mask and global properties maps for sample selection
Complementary datasets: user generated dataproducts, selected transients and variable data from follow-ups datasets
Science use cases: Fast Correlation functions and cross-matching. Voids and Systems of Galaxies detection and classification. Streams and MW Satellite galaxies detection. DIA algorithms for heterogeneous transient follow-ups.
Lite IDAC — Melbourne, Swinburne University
Object Table (subset) Other Data Products GPUs
Capacity (13-yr): Storage 20 PB-yr · CPU 65 Mhrs · GPU 65 Mhrs · ~26 expected local users
Software & services: Multipurpose science platform, with Slurm for job scheduling, EasyBuild for software modules, Lustre for storage
Complementary datasets: Local copy of LIGO data, MeerKAT SKA pulsar timing data, Gaia, MWA, etc.
Science use cases: Cross-match LSST catalog objects with those from other optical (e.g. ESO, AAT) and radio survey (e.g. MWA) data held
Lite IDAC — Rio de Janeiro, LIneA
Object Table (subset) Co-added Images Other Data Products
Capacity (13-yr): Storage 65 PB-yr · CPU 56 Mhrs · ~26 expected local users
Software & services: LIneA Science Platform (LSP) with Jupyter notebooks, combining upgraded science server as deployed at NCSA with new user query (based on Daiquiri platform) and x-match service (based on LDSB); photo-z server, solar system portal, cluster analysis and visualization server (CANVAS); HPC cluster (Slurm, Open OnDemand); PostgreSQL.
Complementary datasets: Survey property maps (e.g. masks; depth); photo-z tables, spectroscopic redshift samples, training sets; cluster and stellar system candidate list updated with each DR; regularly updated list of occultation events; SDSS and DES object catalogs; all-sky hips image
Science use cases: Photo-z’s; galaxy clusters; resolved stars; visualization; crossmatches; occultation events
Contacts: Carlos Adean (carlosadean@linea.org.br), Luiz Da Costa (ldacosta@linea.org.br), Julia Gschwend (julia@linea.org.br)
Lite IDAC — Victoria, Canadian Astronomy Data Centre (NRC Herzberg)
Object Table (subset) Solar System Tables Co-added Images Other Data Products
Capacity (13-yr): Storage 186 PB-yr · CPU 630 Mhrs · ~150 expected local users
Software & services: CAOM interface, CANFAR science platform, Firefly/Rubin Portal, TAP, QServ/sharded DB, Multi-Messanger Alerts DB
Complementary datasets: SKA, Euclid, and CADC Archives
Contacts: Ryan Taylor (rptaylor@uvic.ca), Stephen Gwyn (Stephen.Gwyn@nrc-cnrc.gc.ca), Brian Major (Brian.Major@nrc-cnrc.gc.ca), John Ouellette (John.Ouellette@nrc-cnrc.gc.ca), JJ Kavelaars (JJ.Kavelaars@nrc-cnrc.gc.ca), Sharon Goliath (Sharon.Goliath@nrc-cnrc.gc.ca), Adrian Damian (Adrian.Damian@nrc-cnrc.gc.ca), Pat Dowler (patrick.dowler@nrc-cnrc.gc.ca), Isabella Ghiurea (Isabella.Ghiurea@nrc-cnrc.gc.ca)
Notes: Includes support for public archive
SPC — Rijeka, Bura supercomputer, University of Rijeka
Object Table (subset)
Capacity (13-yr): Storage 9 PB-yr · CPU 30 Mhrs · ~7 expected local users
Software & services: Access through secure VPN channel & IP filtering (need to be arranged in advance) OS Redhat, command prompt + Slurm workload manager, MPI for parallel computing, conda environment
Science use cases: Stellar astrophysics, including period finding, TRILEGAL Milky Way simulation, and microlensing light curves
Science Collaboration agreements: Solar System (May 2026) Transients and Variable Stars (pending) Stars Milky Way and Local Volume (pending)
Lite IDAC — Copenhagen, DARK / Niels Bohr Institute
Object Table (subset) DIASource Table Co-added Images Other Data Products GPUs
Capacity (13-yr): Storage 65 PB-yr · CPU 280 Mhrs · GPU 1 Mhrs · ~60 expected local users
Software & services: TBD
Complementary datasets: Community-derived data products; Auxiliary data products relevant for transient + galaxy science such as SDSS, PS, WISE, VEXAS, KIDS, GALEX, Euclid
Science use cases: Special emphasis on transients, cosmology, galaxies, dark matter (serving especially DESC, TVS, SLS); AI/ML applications Rubin/Euclid joint processing
Lite IDAC (x2) — Mitaka, Tokyo, National Astronomical Observatory of Japan (NAOJ) and Kavli IPMU
Object Table (subset) ForcedSource Table DIASource Table Co-added Images Other Data Products
Capacity (13-yr): Storage 46 PB-yr · CPU 141 Mhrs · ~35 expected local users
Software & services: Jupyter+Kubernetes platform built on HSC-SSP uses. Other interfaces such as Xpra desktop access and filesharing are under development. Database options include Postgresql, Citus, MapReduce distributed DB
Complementary datasets: HSC and PFS data How much and what types of Rubin data to be hosted: under discussion PFS spectra of transient, host galaxies and photo-z training sample taken by other in-kind contribution
Science use cases: Weak lensing cosmology, transients (incl. SNe), photo-z’s, solar system sciences
Lite IDAC — Mexico City, Universidad Nacional Autónoma de México (UNAM)
Object Table (subset) Object Table Co-added Images
Capacity (13-yr): Storage 20 PB-yr · CPU 31 Mhrs · ~10 expected local users
Software & services: Lustre file system, PostgreSQL, exploring RSP as interface
Science use cases: TBD
Lite IDAC — Poznań, Poznań Supercomputing and Networking Center
Object Table (subset) Other Data Products
Capacity (13-yr): Storage 65 PB-yr · CPU 56 Mhrs · ~26 expected local users
Software & services: Dedicated multipurpose science platform (Jupyter/SSH/UI on SLURM/Lustre using Docker Swarm/Singularity/CVMFS/Keycloak/OpenSearch/…)
Complementary datasets: Local copies of Gaia, OGLE, SDSS, Vipers, Akari, other catalogs
Science use cases: Rubin-Gaia cross-match, Rubin-OGLE-ZTF cross-match along the Milky Way disk, Rubin-OGLE-VMC-DECam cross-match for Magellanic System, Preparation of a catalog of known variable stars in the Rubin sky, Construction of deep color-color and color-magnitude diagrams for Magellanic Clouds, Data from alerts for microlensing, LSST large-scale structure (LSS) analysis, Photometric Redshift estimation of galaxies from the LSST data, Quasar redshift estimation and brightness curves, photometric redshift (PZ) estimation of galaxies
Lite IDAC — Nova Gorica, University of Nova Gorica
Object Table (subset) GPUs
Capacity (13-yr): Storage 7 PB-yr · CPU 20 Mhrs · GPU 3.25 Mhrs · ~5 expected local users
Software & services: TBD
Complementary datasets: TBD
Science use cases: TBD
Lite IDAC — Daejeon, Korea Astronomy and Space Science Institute (KASI)
Object Table (subset) Co-added Images Other Data Products
Capacity (13-yr): Storage 7 PB-yr · CPU 20 Mhrs · ~5 expected local users
Software & services: Rubin-dedicated hyperconverged infrastructure cluster to host Rubin Science Platform (Portal/Notebooks/APIs) and distributed PostgreSQL databases for object-lite catalog. Additional high-capacity object storage to serve coadded images.
Complementary datasets: user-generated data products
Contacts: Chang Hee Ree (chr@kasi.re.kr)
Lite IDAC — Barcelona, Port d’Informació Científica (PIC) / IFAE
Object Table (subset) GPUs
Capacity (13-yr): Storage 7 PB-yr · CPU 21 Mhrs · GPU 20 Mhrs · ~8 expected local users
Software & services: CosmoHub: Web portal for interactive exploration and distribution of massive cosmological datasets Full dataset plots (over all rows): May use sampling / Cone search tool / 1D histogram & 2D heatmap TAP + ADQL interfaces in development
Complementary datasets: ~15 different projects (Gaia, DES, PAU, Euclid, DESI, COSMOS, KiDS, GLADE, CFHTLenS, …) including DESC DC2 (Object and Truth-match tables)
Science use cases: Catalog validation / exploration / crossmatch Large Scale Structure algorithms (photo-z, 2-point correlation functions, …) Galaxy simulations
Contacts: Jordi Delgado (jordidem@pic.es), Francesc Torradeflot (torradeflot@pic.es), Jorge Carretero (carretero@pic.es), Nacho Sevilla (ignacio.sevilla@ciemat.es), Pau Tallada (tallada@pic.es)
Data Facility (Full IDAC) — Edinburgh, Royal Observatory / UK Data Facility
Object Table (subset) Object Table Source Table ForcedSource Table DIAObject Table DIASource Table Solar System Tables Co-added Images Visit Images Template Images Other Data Products
Capacity (13-yr): Storage 2005 PB-yr · CPU 61 Mhrs · ~437 expected local users
Software & services: Full RSP deployment Options to exploit technologies from ‘nearby’ programmes – such as SKA, Euclid, and Gaia Rucio for distributed data management (tbc); Ctrl-BPS/ Parsl/ Slurm – batch processing for HPC (tbc)
Complementary datasets: LSST-VISTA fused products; crossmatch catalogues for topical surveys, Edinburgh Wide-field Astronomy Unit survey catalogues (via TAP)
Science use cases: Full range of science cases Co-location with Lasair Community Broker for transient science
Contacts: George Beckett (george.beckett@ed.ac.uk), Mike Read (mar@roe.ac.uk)
Notes: Free CPU capacity does not include the UK Data Facility’s significant resources for supporting database queries of full copy of catalog data
What data and services will be available?#
The profiles above are built from the same underlying data as this spreadsheet, which reflects the current plans for the Rubin data, services, and potential use cases to be supported at individual IDACs.
What are some potential uses of contributed computing resources?#
The virtual workshop Supporting Computational Science with Rubin LSST, held in March 2023, featured discussion of a significant number of use cases submitted by members of the science community. Links to the use cases, presentations, recordings, notes, and background material are available on the workshop web page.
IDACs are considering a range of specific use cases, including time series analyses, solar system occultation predictions, and development of photometric redshift training sets, as well as general use. IDACs are also following the development of the use cases identified in the workshop “From Data to Software to Science with the Rubin Observatory LSST”, and may adopt some of these as a basis for specific datasets and services.
When will IDACs and SPCs be available to the community?#
As seen in this presentation, IDACs and SPCs are expected to start operations with the release of LSST DR1 sometime in 2026.
Want to know more?#
The IDACs Coordination Group maintains a space on Community for discussion and sharing knowledge amongst IDACs and SPCs and their user communities. Join the conversation!