The Gaia Archive. A. Mora, J. Gonzalez-Núñez, J. Salgado, R. Gutiérrez-Sánchez, J.C. Segovia, J. Duran ESA-ESAC Gaia SOC and ESDC

Size: px

Start display at page:

Download "The Gaia Archive. A. Mora, J. Gonzalez-Núñez, J. Salgado, R. Gutiérrez-Sánchez, J.C. Segovia, J. Duran ESA-ESAC Gaia SOC and ESDC"

Dylan Caldwell
5 years ago
Views:

1 The Gaia Archive A. Mora, J. Gonzalez-Núñez, J. Salgado, R. Gutiérrez-Sánchez, J.C. Segovia, J. Duran ESA-ESAC Gaia SOC and ESDC IAU Symposium 330. Nice, France ESA UNCLASSIFIED - For Official Use

2 Outline 1. Introduction 2. The Gaia Archive 3. Tables: catalogue, cross-match and light curves 4. Archive demo 5. How to select a stellar cluster (basic query) 6. How to create an HR diagram (2MASS cross-match) 7. How to reconstruct an RR Lyrae light curve 8. Additional resources 9. Conclusions ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 2

3 1. Introduction ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 3

4 ESA Gaia Archive: key points Code to the data (workflows) Gaia Archive contents. Provided by DPAC Catalogues (tables) Pre-computed cross-match (pivot tables) Light curves (one entry per point) Gaia Archive functionality ADQL: SQL dialect for queries reproducibility papers TAP+ VO standard: web data query, user tables, sharing, SAMP VO standard: interoperability 3 rd party software DR2+ evolution: ~PB size, parameters, spectra, epoch data, Learning curve: reproducing DR1 Brown plots ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 4

5 Code to the data (workflows) There is no correct workflow! if it works, it is good Traditional (bring the data to the code) Download tgas_source (~1GB), gaia_source (~1TB) and external catalogues Ingest in local supercomputer. Do cross-match Do science! Gaia archive (towards bringing the code to the data) Selection (ADQL query) at Gaia Archive Reproducibility! Refinement (in Archive): Xmatch, computations, external catalogues, share Download and do science! Data (size) reduction ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 5

6 2. The Gaia Archive ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 6

7 Gaia Archive contents: DR1 Where? Main Gaia archive: ESAC Partner data centres: AIP, ARI, ASI, BCN, CDS Affiliated and other data centres, enthusiasts: tens What? tgas_source (~2M 5 parameter astrometry): all gala_source (~1.1G positions): most Variability (599 cepheids, 2595 RR Lyrae), QSOs (2191): ESAC+ Xmatch (2MASS, PPMXL, SDSS9, UCAC4, URAT1, WISE): ESAC External catalogues (for Xmatch): ESAC+ What TAP+ (data base, user space, share), cross-match ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 7

8 Gaia Archive contents: DR1 Total DR1: 1.6 TB Gaia Gaia DR1 catalogue 1.1 billion rows TGAS billion rows External Catalogues Hipparcos & Hipparcos new red billion rows IGSL (Initial Gaia Source List) 1.2 billion rows 2MASS 0.47 billion rows Tycho billion rows UCAC billion rows Hubble Source Catalogue v billion rows Crossmatches Crossmatch tables between Hipparcos, 2MASS, Tycho2 and Gaia expressed as neighbourhood and best neighbour, e.g: AllWise-Gaia neighbourhood 0.31 billion rows ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide billion rows

9 The User Interface ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 9

validation or proprietary exploitation User Space User-uploaded data ESA

10 TAP+ Open APIs TAP+ I/F Command line tools Public area Publicly released catalogues External Apps Data Validation Restricted area Catalogues during validation or proprietary exploitation User Space User-uploaded data ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 10

11 Gaia Archive: TAP+ Schema Persistent Upload Server Crossmatch Table sharing ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 11

12 Visualization Visualization is a need Statistics provide holistic views. Big data techniques Validated static stats On-the-fly visualization (Lisbon University) ESASky integration Virtual Reality prototype ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 12

13 3. Tables: catalogue, cross-match and light curves ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 13

14 Gaia source: main table Columns Units Data: one entry (row) per object Data model ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 14

15 Variables: light curve One entry per object and epoch: 3194 stars, 233,181 rows ~Trillion size by DR5! Table: phot_variable_time_series_gfov ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 15

16 Pre-computed cross-matches DR1 one data set. Other catalogues: astrometry, velocities, photometry, parameters Ambiguity: resolution, wavelength, epoch, proper motion, Many to many relationship pivot tables needed traceability! Gaia Archive: catalogue copies and pre-computed cross-matches Speed: indices. Efficiency: avoids ~100 M rows traffic 2MASS PSC: Skrutskie et al ,992,970 entries AllWISE: Cutri et al. 2013, 747,634,026 entries GSC2.3: Lasker et al ,592,683 entries PPMXL: Roeser et al ,468,688 entries SDSS DR9: Ahn et al ,029,929 entries UCAC4: Zacharias et al ,728,883 entries URAT-1: Zacharias et al ,276,482 entries ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 16

17 Xmatch: neighbourhood ~Many to many relationship double entry one row per pair ω Cen core ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 17

18 Xmatch: neighbourhood ~Many to many relationship double entry one row per pair ω Cen core ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 18

19 Xmatch: best neighbour >0 other Gaia sources share best neighbour ω Cen core ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 19

20 VO standards 3 rd party Topcat (M. Taylor): Data query (TAP), reception (SAMP) & plotting ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 20

21 4. Archive demo ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 21

22 5. How to select a stellar cluster (basic query) ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 22

23 Tutorial: Pleiades gross selection SELECT * FROM gaiadr1.tgas_source WHERE CONTAINS( POINT('ICRS', ra, dec), CIRCLE('ICRS', 56.75, , 2) )=1 Cone search: 2º radius Apparent in proper motion ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 23

1167, 2) )=1 AND pmra IS NOT NULL AND pmra!= 0 AND pmdec IS NOT NULL AND pmdec!= 0 AND abs(pmra_error/pmra) < 0.

24 Tutorial: Pleiades gross selection SELECT * FROM gaiadr1.tgas_source WHERE CONTAINS( POINT('ICRS', ra, dec), CIRCLE('ICRS', 56.75, , 2) )=1 AND pmra IS NOT NULL AND pmra!= 0 AND pmdec IS NOT NULL AND pmdec!= 0 AND abs(pmra_error/pmra) < 0.10 AND abs(pmdec_error/pmdec) < 0.10 AND pmra BETWEEN 15 AND 25 AND pmdec BETWEEN -55 AND -40 PM filter ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 24

25 6. How to create an HR diagram (2MASS cross-match) ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 25

26 Brown+ 2016: HR diagram Plots are reproducible! ADQL queries in the paper Uses Gaia parallaxes and fluxes absolute magnitudes Uses pre-computed cross-match and external catalogue: 2MASS Histograms can be generated in the archive Code to the data DR2+ ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 26

27 Brown+ 2016: HR diagram select gaia.source_id, gaia.phot_g_mean_mag + 5 * log10(gaia.parallax)- 10 as g_mag_abs, gaia.phot_g_mean_mag-tmass.ks_m as g_min_ks from gaiadr1.tgas_source as gaia inner join gaiadr1.tmass_best_neighbour as xmatch on gaia.source_id = xmatch.source_id inner join gaiadr1.tmass_original_valid as tmass on tmass.tmass_oid = xmatch.tmass_oid where gaia.parallax/gaia.parallax_error >= 5 and ph_qual = 'AAA' and sqrt(power(2.5/log(10)*gaia.phot_g_mean_flux_error /gaia.phot_g_mean_flux,2) ) <= 0.05 and sqrt(power(2.5/log(10)*gaia.phot_g_mean_flux_error /gaia.phot_g_mean_flux,2) + power(tmass.ks_msigcom,2)) <= 0.05 ADQL: Learning curve ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 27

28 Brown+ 2016: HR diagram select gaia.source_id, gaia.phot_g_mean_mag + 5 * log10(gaia.parallax)- 10 as g_mag_abs, gaia.phot_g_mean_mag - tmass.ks_m as g_min_ks from gaiadr1.tgas_source as gaia inner join gaiadr1.tmass_best_neighbour as xmatch on gaia.source_id = xmatch.source_id inner join gaiadr1.tmass_original_valid as tmass on tmass.tmass_oid = xmatch.tmass_oid where gaia.parallax/gaia.parallax_error >= 5 and ph_qual = 'AAA' and sqrt(power(2.5/log(10)*gaia.phot_g_mean_flux_error /gaia.phot_g_mean_flux,2) ) <= 0.05 and sqrt(power(2.5/log(10)*gaia.phot_g_mean_flux_error /gaia.phot_g_mean_flux,2) + power(tmass.ks_msigcom,2)) <= 0.05 ADQL: on the fly computation Gaia + 2MASS: Pre-computed Xmatch ADQL: filters ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 28

29 Brown+ 2016: HR diagram select from ( g_min_ks_index / 10 as g_min_ks, g_mag_abs_index / 10 as g_mag_abs, count(*) as n select gaia.source_id, floor((gaia.phot_g_mean_mag+5*log10(gaia.parallax)-10) * 10) as g_mag_abs_index, floor((gaia.phot_g_mean_mag-tmass.ks_m) * 10) as g_min_ks_index from gaiadr1.tgas_source as gaia inner join gaiadr1.tmass_best_neighbour as xmatch on gaia.source_id = xmatch.source_id inner join gaiadr1.tmass_original_valid as tmass on tmass.tmass_oid = xmatch.tmass_oid where gaia.parallax/gaia.parallax_error >= 5 and )as subquery ph_qual = 'AAA' and group by g_min_ks_index, g_mag_abs_index 2D histogram in the archive: no billion object download DR2 sqrt(power(2.5/log(10)*gaia.phot_g_mean_flux_error/gaia.phot_g_mean_flux,2)) <= 0.05 and sqrt(power(2.5/log(10)*gaia.phot_g_mean_flux_error/gaia.phot_g_mean_flux,2) + power(tmass.ks_msigcom,2)) <= 0.05 ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 29

30 7. How to reconstruct an RR Lyrae light curve ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 30

31 Brown+ 2016: light curves Plots are reproducible! ADQL queries in the paper Uses Gaia epoch photometry and best fit period Light curves in a relational data base! Best strategy? DR2+ RR Lyrae: ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 31

32 Brown+ 2016: light curves select curves.observation_time, mod(curves.observation_time - rrlyrae.epoch_g, rrlyrae.p1) / rrlyrae.p1 as phase, curves.g_magnitude, 2.5/log(10)* curves.g_flux_error/ curves.g_flux as g_magnitude_error from gaiadr1.phot_variable_time_series_gfov as curves inner join gaiadr1.rrlyrae as rrlyrae on rrlyrae.source_id = curves.source_id where rrlyrae.source_id = ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 32

33 Brown+ 2016: light curves select curves.observation_time, mod(curves.observation_time - rrlyrae.epoch_g, rrlyrae.p1) / rrlyrae.p1 as phase, curves.g_magnitude, 2.5/log(10)* curves.g_flux_error/ curves.g_flux as g_magnitude_error from gaiadr1.phot_variable_time_series_gfov as curves inner join gaiadr1.rrlyrae as rrlyrae on rrlyrae.source_id = curves.source_id where rrlyrae.source_id = Light curve reconstruction Source selection ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 33

34 8. Additional resources ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 34

35 Partner data centres Alternative and complementary access at data release time Selected data. Custom interface. Additional functionality AIP (Leibniz-Institut für Astrophysik Potsdam, Germany) ARI (Astronomisches Rechen-Institut, Heidelberg, Germany) ASDC (ASI Science Data Center, Rome, Italy) CDS (Centre de Données astronomiques de Strasbourg, France) Affiliated Data Centres. Receive full release after a small delay efficient mirroring ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 35

36 Documentation and resources Gaia Archive. Includes help and tutorials Gaia DR1 papers Online documentation (361 pages) Data model documentation ADQL: GAVO short course, UK ROE cookbook Gaia Helpdesk. ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 36

37 9. Conclusions ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 37

38 Conclusions Archive: Helpdesk: Functionality: TAP+ (data base, user space, sharing), cross-match Data: main tables, TGAS, variables, QSO, Xmatch, external catalogues Bring code to the data: select, refine (ADQL, user tables) and download Prepare for DR2 (~10 9 sources). Archive might be the only way forward Add ADQL queries to your papers (Brown+ 2016, Clementini+ 2016) Reproducibility Use it. User demand is a key driver for future developments Ask us. Via Helpdesk for additional support. Please provide feedback Want to support Gaia? keep writing papers (and acknowledging) ESA UNCLASSIFIED - For Official Use A. Mora The Gaia Archive IAU Symposium Slide 38

Working with Gaia data

Working with Gaia data Alcione Mora ESA-ESAC Gaia SOC Gaia 2016 DR1 workshop ESA-ESAC 2016-11-02 Issue/Revision: 1.0 Reference: Presentation Reference Status: Issued Outline Ø Introduction Ø Gaia DR1 contents