2.1   Base DMCS and OCS Commandable Entities¶

The Alert Production hardware is divided into four commandable entities from the perspective of the OCS:

1. Archiver: responsible for archiving images in real time.
2. Catch-Up Archiver: responsible for archiving images that did not get captured in real time due to an outage of some part of the DM system.
3. EFD Replicator: responsible for replicating the EFD from the Summit to the Chilean DAC and the US DAC.
4. Alert Production Cluster: responsible for generating Level 1 data products.

Each commandable entity can be commanded by the OCS to configure, enable, or disable itself, along with obeying other generic OCS commands such as init, release, stop, and abort. Each commandable entity publishes events and telemetry to the OCS for use by the observatory operations staff. The command/action/response protocol used by the OCS is common to all subsystems and is a standard real-time system control mechanism used, for example, by the ATST.[5] The configure/enable/disable message pattern is also a common one; it is used, for example, in the LHCb control system.[6]

All these commandable entities are implemented in the Base DMCS. They all run on a single machine, which is the only one that communicates directly with the OCS. If it fails, as detected by heartbeat monitoring, it is powered down and a spare machine is enabled at the same IP address, possibly missing one or more visits.

The Base DMCS communicates with the OCS via the Data Distribution Service (DDS), through which it receives commands according to a well-defined asynchronous command protocol [7] and sends command result messages, status updates, events, and telemetry. It should be noted that the commandable entities do their processing while in the IDLE state from the perspective of the command protocol.

The Base DMCS will be booted before the start of each night’s observing to ensure that the system is in a clean configuration. When the Base DMCS cold boots, the Base DMCS performs a self test sequence to verify that it can communicate with the DM Event Services Broker (for DM-internal communications) and the OCS (via DDS). After the self test sequence, the commandable entities start up, in no particular defined configuration, and publish the OFFLINE state to the OCS.

The Base DMCS uses the Orchestration Manager (currently baselined to be implemented using HTCondor [8]) to start jobs on the replicators, distributors, and workers. The Orchestration Manager may run on the Base DMCS host or another machine.

The typical sequence of OCS commands after a cold boot will be init, configure, and enable for each commandable entity.

2.2   EFD replication¶

Not included in the Alert Production per se but closely tied to it is replication of the Engineering and Facility Database (EFD) from the Summit to the Chilean DAC and the Chilean DAC to the US DAC.

The replication is implemented by standard replication mechanisms for the selected database management system used to implement the EFD. The latency for the replication from the Summit to the Chilean DAC is anticipated to typically be in the milliseconds, although latencies of up to one visit time are acceptable. The latency for the replication from the Chilean DAC to the US DAC is to be as short as possible, constrained by the available bandwidth from Chile to the US, but no longer than 24 hours (except when a network outage occurs). The typical case for Chile-to-US replication is expected to be seconds or less.

The Alert Production computations will require telemetry stored in the EFD. The design does not rely on replication for this information, however. At the Base, the local Chilean DAC EFD replica is queried for some information, but the OCS telemetry stream is also monitored for more recent changes than are reflected in the results of the query. This essential data is then sent along with the image data to the Archive for processing. If the replication proves to have sufficiently low-latency and be sufficiently reliable, it will be easy to switch to an alternate mode where the US DAC EFD replica is queried for the information of interest.

Figure 2 Visit Sequence Diagram

2.3   Alert Production Hardware¶

We now describe the detailed operations performed by each Alert Production infrastructure component. The sequence of operations for a typical visit is shown in Figure 2.

All DM hardware is monitored by DM system administration tools, which publish results via the Archive DM Control System. Each machine verifies its software installation on boot (e.g. via hash or checksum).

2.4   Catch-Up Archiver¶

The Catch-Up Archiver transfers images from the camera that were not retrieved due to an error or outage. It also transfers images from the network outage buffer to the Archive Center.

The Catch-Up Archiver has its own replicators and distributors. These nodes communicate similarly to the replicators and distributors of the Archiver commandable entity.

The Base DMCS scans the Camera buffer for images that have not been archived to tape (or transmitted over the network). Each of those images triggers a replicator job. The oldest images will be submitted first. The Base DMCS also scans the network outage buffer for images that were not transmitted (as recorded in the Alert Production database). Those images also trigger a different replicator job that retrieves its data from the buffer instead of the camera.

Images handled by the Catch-Up Archiver are not processed by the normal Alert Production Cluster. The Base DMCS may be configured to submit worker jobs to a separate pool of workers for catch-up processing of these images.

2.5   Calibration image and engineering image modes¶

When the DM Archiver and Alert Production Cluster are configured in these modes, there are no visits. The startExposure event is used to trigger both replicator jobs and worker jobs (although another event could be used to trigger the workers). Worker processing only performs ISR (often just a subset), CCD assembly, PSF determination (if appropriate), and a subset of SDQA, as configured for the mode selected.

2.6   Daytime DM operations mode¶

In this mode, the Alert Production Cluster is used to perform SSObject detection and orbit fitting (DayMOPS) and other maintenance tasks, including updating DIAObject and DIASource caches and projecting SSObject orbits for the next night. The Archiver may be enabled while the Alert Production Cluster is in this mode, but no processing of any images will occur and the distributors will never receive requests from the workers. The Catch-Up Archiver may be enabled.

The Base DMCS will submit jobs to the Orchestration Manager as necessary to perform the daytime tasks.

2.7   Failure Modes¶

In the event of a failure of the Summit-to-Base network link or Base power and the consequent loss of DM functionality, the Summit has sufficient analysis capability to be able to proceed with observations independently, writing images to the CDS buffer. The Catch-Up Archiver will then be used to retrieve and archive these images when connectivity is restored. No alerts are produced, and no feedback telemetry from DM goes to the Camera or Telescope, of course.

In the event of a total failure of the Base-to-Archive network link, the replicators will detect loss of connection to the distributors, register themselves into the local-only pool, and write to the local tape system and the network outage buffer. The Network Operations team will be notified to investigate and resolve the issue. The Catch-Up Archiver is again used to retrieve and transmit these images to the Archive when connectivity is restored. Again, no alerts are produced, and no feedback telemetry goes from DM to the Camera or Telescope. (If desired, spare hardware at the Base such as the commissioning compute cluster could be assigned to a worker pool to do a limited amount of processing to provide feedback telemetry and even some alerts, but this is not part of the baseline.)

In both network failure cases, if the outage is a “black swan” that extends for longer than has been anticipated in the buffer sizes, media shipping will be used as a backup image transfer channel. Images and associated telemetry from an EFD replica will be copied onto a disk array (possibly solid state disk) at the Summit or Base, as appropriate. The array will then be shipped to the Base or Archive, respectively (and then shipped back once the data has been extracted). Multiple arrays will be required to handle expected shipping and data transfer times.

In the event of a partial failure (e.g. a slowdown) of the Base-to-Archive network link, the replicator jobs will detect that they are not completing in the expected amount of time. As they detect this, the replicator machines will re-register themselves in the local-only pool. If sufficient pairs do so, the Network Operations team will be notified to investigate and resolve the issue. After random time intervals, as long as heartbeat messages from their paired distributors continue to be received, the replicators will re-register themselves in the fully-operational pool so as to enable automatic recovery.

For a more comprehensive discussion of network failures and network operations, refer to the LSST Observatory Network Design (LSE-78) and the LSST Network Operations and Management Plan (Document-11918).

If a replicator, distributor, or worker dies, a spare will be used automatically by the Orchestration Manager. If the Base or Archive DMCS, the DM Event Services Broker, or the Orchestration Manager itself dies, a spare will be brought online. Since these machines maintain little state, a replacement should be available rapidly without missing many visits.

The network outage buffer is designed to be single-fault-tolerant. If the tape system or shared disk become unavailable due to faults, the Catch-Up Archiver can be used with the network outage buffer when they return.

If an EFD replica fails, queries can be directed to the next master up the chain (US DAC to Chilean DAC, Chilean DAC to Summit) until a new slave can be brought online and synchronized.

If the calibration, Level 1 catalog, or Alert Production control database fails, a hot spare replica will be reconfigured to be the master.

If the application software fails on a given sensor (or if the sensor itself does not produce data or produces invalid data), the Alert Production algorithms will be designed to continue processing in its absence. Job failures of this type will be communicated to the Orchestration Manager and will not be rescheduled.

If the Alert Production workers get behind, the Orchestration Manager will begin to schedule worker jobs on spare worker hardware. If so configured, it may also schedule jobs on the Data Release general compute pool. The worker jobs themselves are designed to process batches of DiaSources into Alerts so that at least some Alerts are issued for each visit before the latency deadline. Executing jobs that are beyond the deadline may be killed and scheduled for later reprocessing. If even that is not enough and unexecuted jobs pile up because processing is too slow, as determined by monitoring the Orchestration Manager’s queue length, the Base DMCS will kill the oldest unexecuted jobs to get below threshold. In addition, the Base DMCS configuration will allow sampling of visits for worst-case scenarios, in which only a fraction of visits actually spawn worker jobs.

2.8   Maintenance and Upgrades¶

New Alert Production software will be deployed during daytime maintenance periods. Full integration tests of the new configuration on both a dedicated integration cluster and the production hardware will be performed before the software is certified to go live for science observing. Each class of machine (e.g. replicator, distributor, worker, DMCS) will be uniform in terms of software, from the operating system through the application code. Cluster configuration management software like Chef or Puppet will be used to enable and ensure this.

The Alert Production compute load does not increase significantly with time. (Only moving object prediction and association get noticeably harder.) As a result, new hardware will be deployed primarily to replace failed components and at specified hardware refresh intervals to avoid obsolescence. Full integration tests of the production software on the new hardware will be performed before science observing, with fallback to the old hardware in case of difficulty. Since new hardware is expected to have at least the same performance as old hardware, heterogeneity of hardware within a machine class will be permitted. This simplifies the upgrade process and avoids the need to change out many machines at the same time.

4.1   Overall Sequence¶

Many of the Data Release Production algorithms are expected to involve computations across the full set of available images, at least in one region of the sky and possibly across the entire survey area. It is impractical to perform these computations in an incremental fashion. Therefore a “freeze date” must be chosen which delineates the latest image to be included in the DRP processing.

After the freeze date is selected, the Calibration Products Production is run in pre-DRP mode, which recalculates all of the master calibration images and the calibration database to be used for all the exposures up to that date.

Second, a region of the sky (about 5-10% of the total survey area) is processed through the entire DRP, treating it as if that were the entire survey. The results of this processing are analyzed and verified to ensure that the software is performing properly.

Finally, after any software fixes or configuration changes resulting from the single-region analysis, the entire sky is processed.

When the complete set of Level 2 data products has been generated, it is transferred to the Chilean DAC (and any other non-project stand-alone DACs that provide the necessary bandwidth resources). For the Chilean DAC, this transfer nominally occurs by writing the data products to disk and shipping the disk to Chile, although an alternative path via high-speed network is being considered.

4.2   Detailed Sequence¶

The DRP computation can be considered to have several major segments:

1. Single-frame processing
2. Global astrometric and photometric calibration
3. Coaddition, template generation, and difference imaging
4. MOPS
5. Object characterization and forced photometry

To initialize the DRP, Level 1 database visit metadata tables as of the “freeze date” are copied to the DRP temporary database along with the Calibration Database from the CPP and a verified “seed” Solar System Object catalog. A dataset repository is configured pointing to the raw images on tape and the master calibration images from the Calibration Products Production, along with the DRP temporary database. The visit metadata tables are scanned to extract the boresight RA/dec coordinates. These are used to determine which sky tiles are covered by each visit. The complete list of such sky tiles is gathered.

Then single-frame processing begins. The Archive DMCS steps through the list of sky tiles in sequence. Since the raw images on tape are organized spatially, this allows tapes to be read sequentially. For each sky tile in the list, the Archive DMCS submits a list of single-frame measurement tasks to the Orchestration Manager, one for each visit overlapping that sky tile. Each task reads the raw visit images from tape, performs instrument signature removal, characterizes the image by determining its WCS, PSF, and approximate photometric calibration, and detects and measures Sources on the image. SDQA metrics are derived from these results. The image characterization parameters and SDQA metrics are loaded into the DRP temporary database. The Source datasets are retained in DRP scratch space. The calibrated exposures are not retained, so all sky tiles can be processed as quickly as possible.

The Archive DMCS next submits tasks to the Orchestration Manager to perform global astrometric and photometric calibration using the Source datasets. The resulting astrometric and photometric models are loaded into the DRP temporary database.

Source ingestion into the nascent Level 2 database can occur at this point.

The Archive DMCS then steps through the list of sky tiles again, with the number of simultaneously processed tiles depending on available scratch disk space. For each tile, the Archive DMCS submits a task graph with dependencies. The task graph generally alternates between groups of tasks executed on a per-visit basis and groups of tasks executed on a per-coadd-patch basis, with each group of tasks depending (only) on the previous group. As each visit is deterministically matched with a list of coadd patches it covers, the entire task graph can be precomputed.

The first group of tasks regenerates calibrated exposures for each visit overlapping a sky tile and writes the exposure images to DRP scratch space. (This regeneration task is simplified to just instrument signature removal because it can use the image characterization results from the first pass.)

The second group of coadd tasks is issued for each coadd patch in the sky tile. (Tiles are defined so that they align with coadd patches.) Each such coadd task depends on the calibrated exposure regeneration tasks for the exposures that overlap the patch. These tasks warp the calibrated exposure to the patch frame using the astrometric models from global astrometric calibration and then generate all coadds (deep, short-period, best seeing, PSF-matched, and template) from the warped exposures. All coadd patches (except the templates) have detection and measurement performed. The CoaddSource datasets, including footprints, the deep coadd images, and the template images are retained in DRP scratch space.

Another group of tasks is issued for each sky tile, again one per visit, depending on the coadd tasks for the coadd patches covered by that visit. These difference imaging tasks use the calibrated exposure for the visit and the template images for the coadd patches to do image differencing and DiaSource detection and measurement. Each DiaSource is matched against known SSObjects. The DiaSource datasets and difference images are retained in DRP scratch space.

A group of tasks, one per coadd patch, is used to spatially associate DiaSources into DiaObjects.

Another group of tasks is executed to perform MOPS on all DiaObjects that have only one associated DiaSource. MOPS is expected to use a different data partitioning than the coadd patches. The result of MOPS is an updated list of SSObject orbits and a list of DiaObjects/DiaSources that were successfully linked.

The remaining unlinked DiaObjects (both single-DiaSource and multiple-DiaSource) are loaded into the DRP temporary database.

A group of tasks, one per visit, is used to do forced photometry on difference images at the positions of the DiaObjects in the database, resulting in DiaForcedSources. The difference images are then removed from the DRP scratch space.

DiaObjects, DiaSources, DiaForcedSources, and SSObjects can be ingested into the nascent Level 2 database.

Finally, a group of tasks is executed, one per coadd patch, to associate and deblend CoaddSources, Sources, and DiaObjects into Objects and then perform MultiFit object characterization and forced photometry using the calibrated exposures overlapping the patch, generating Object measurements and ForcedSources.

Objects and ForcedSources are ingested into the Level 2 database at this point.

The Level 2 catalogs are ingested into a temporary Level 2 database at the Archive Center as portions are generated as described above, and the metadata for the Level 2 image products is also ingested into the database.

SDQA is performed continuously at each step as the Level 2 data products are generated, with the resulting metrics ingested into the Level 2 database. Metrics from SDQA may be used in succeeding steps (e.g. to avoid low-quality images during coaddition). Additional SDQA (automated and manual) is performed after the data products are complete.

The Level 2 data products are sent to the Chilean DAC and installed there. They are copied from Data Release Production scratch space and the Data Release Production database to the US DAC.

The Level 2 database and images are then released simultaneously at the US and Chilean DACs.

4.3   Parallelization¶

In order to accomplish the heavy computational load required by the Data Release Production, parallelization across large numbers of cores is required. Most pipelines are parallelizable over obvious data units such as images, sky patches, or Objects. MOPS is parallelizable over lunation time periods. These data units will generate thousands to millions or even billions of independent tasks, which will be grouped into jobs of appropriate length, on the order of single-digit hours. In particular, the object characterization may be done on all Objects within a sky patch to minimize I/O of image pixels. These jobs will be submitted to the Orchestration Manager.

The global astrometric and photometric calibrations involve solving extremely large but sparse matrix algebra problems. Algorithms for doing these computations are parallelizable but require message passing as opposed to being independent tasks. These will be written using MPI as wrapped within the Inter-Process Messaging Services.

The start, status, and result information (including timing) for each job will be tracked by the Orchestration Manager and reported to the Archive DMCS for overall progress monitoring.

4.4   Input and Output¶

As jobs built using the Pipeline Construction Toolkit execute, they retrieve files from and persist files to tape, shared scratch disk, and the in-progress Level 2 data products storage using the Data Access Client Framework. They may also query the Data Release Production database using that framework. Jobs do not write directly to the database; instead, for efficiency and avoidance of transaction locking, they write files that are ingested into the database at a later time.

All data products are backed up to tape as they are written to the data products storage.

At the time of the data release, the in-progress data products are made available to the DAC while the oldest data release (except DR1, which is always kept) is removed.

4.5   Failure Modes¶

Since the Data Release Production is composed of many restartable jobs, hardware failures are typically handled by rescheduling on another node from the general compute pool. The shared disk systems are designed to be at least two-fault-tolerant (RAID 6).

If the application software fails on a given data element, which is expected to be a rare occurrence due to the planned implementation of automatic adaptive parameter configuration, the production can be instructed to continue without those results or a manual execution of the failed job with new configuration parameters can be performed. Such parameters are recorded for provenance purposes.

4.6   Maintenance and Upgrades¶

New Data Release Production software will be frozen at or before the data “freeze date”. While the software will have been tested on the dedicated integration test cluster, the initial 5-10% region processing also serves as a final verification pass on the production hardware. Each class of machine (e.g. replicator, distributor, worker, DMCS) will be uniform in terms of application software, although operating system and library versions may vary as long as results are reproducible to scientific equivalence. Cluster configuration management software like Chef or Puppet will be used to enable and ensure this.

New hardware will be deployed throughout the course of the survey to add capacity, to replace failed components, and at specified hardware refresh intervals to avoid obsolescence. New compute machines will be tested as part of the development and integration clusters before deployment in the operational compute cluster. Compute cluster machines targeted for removal will be drained of jobs, de-registered from the general compute pool, and then shut down. Disks will be handled similarly, with logical volumes expanded across new disks and contracted away from disks to be removed. In addition, there is downtime between the completion of one DR and the beginning of processing for the next in which major upgrades, especially to central services like the local area network, can be performed.

5.1   Databases¶

The Level 1 database will be updated in an append-only fashion in real time during observing as DIASources and DIAObjects are identified. It is intended to be queried for individual objects and small cone searches, not for large statistical queries.

The Engineering and Facilities Database replica will also be updated continuously in (near) real time.

At the beginning of the daytime DM operations mode processing, the Level 1 database will be snapshotted to a static replica. This replica may be used for statistical queries and analysis.

The Level 2 catalogs will be stored in a large-scale, parallel, distributed database to provide sufficient performance for not only full-table-scan queries but also near-neighbor queries.

The calibration database will be updated, in append-only fashion, at the conclusion of each execution of the Calibration Products Production.

All databases provide SQL interfaces (possibly with extensions or limitations) through standard (ODBC, mysqlapi, Python DB-API) APIs. In addition, an ADQL adapter layer may be provided, as well as adapters to produce FITS table or VOTable results. These adapters will be manifested as Web services.

Bulk downloads of database information, including databases from older Data Releases, will be provided in at least the same form as the tape backups of each database, either through separate Web services or through standard file transfer protocols (e.g. rsync, Globus Online).

5.2   Image Storage¶

Caches are maintained for raw, calibrated, and difference images, and the full set of raw images is accessible from the tape system at the Archive Center and the Base Center.

Recent coadd, template, and raw and master calibration images are stored on disk. Historical versions are accessible from tape.

An image regeneration Web service is provided to produce calibrated and difference images on demand. This service access the raw, master calibration, and template images from the DAC disk or from tape as required. Latency when tape is involved will obviously be greater.

The primary access to the image storage and regeneration service occurs through the image cutout Web service. This takes a request with appropriate parameters such as image type, date, spatial region on the sky, sensor identification, etc. and retrieves the appropriate image(s), trimming and mosaicking them as necessary. It can also be used to retrieve stacks of images (“postage stamps”) of a particular object each time it has been observed. The image cutout service has space dedicated to it in each of the raw, calibrated, and difference image caches so that multiple requests for the same area can be handled rapidly.

Additional bulk download interfaces, e.g. to Education and Public Outreach, will be provided through separate Web services and standard file transfer protocols.

5.3   Level 3 Storage and Compute¶

DAC users may be allocated database and file storage space for use by their own, possibly proprietary, computations. At least part of the database space will be distributed on the same nodes as the Level 2 query services database to facilitate joins with the large Object, Source, and ForcedSource catalogs. Transfer of data to and from Level 3 file storage will be via standard file transfer protocols.

A compute cluster is provided for Level 3 usage. An Orchestration Manager instance will control access to these nodes by pipelines written using the Pipeline Construction Toolkit that use the Data Access Client Framework for access to Level 1 and Level 2 data products. Level 3 pipelines may also use non-LSST-provided libraries, but only if that code can be installed with ordinary user privileges.

Allocations of space and compute resources to users will be performed according to project policy and enforced by resource management tools as part of the storage infrastructure and Orchestration Manager.

A significant use case for Level 3 compute capacity is expected to be processing of Data Release Production intermediate data products in a way different than the DRP itself. This could involve producing different coadds or different measurements, for example. This use case will be enabled by copying the DRP intermediates to Level 3 file storage as they are generated. Users may submit tasks and task graphs to the DAC Orchestration Manager that have data dependencies on these intermediates. The same facility that allows the Archive DMCS to sequence through the set of available visits or coadd patches can be used to generate these task submissions. Note that the one-way nature of the copy from DRP to Level 3 storage and the isolation of the Level 3 processes and compute hardware from the DRP hardware ensures that there is no possibility for problems with user code in Level 3 to impact the DRP. Also note that the Level 3 compute tasks are subject to the usual resource management, and, in addition, the compute time available to these tasks will be dependent on the amount of available Level 3 file storage, since new intermediate data products will overwrite old ones when the available space has filled up.

5.4   Failure Modes¶

The smaller databases will be replicated to slave backups that can be reconfigured as masters in the event of a failure.

The large Query Services database is designed to be at least single-fault-tolerant [10], as is the shared disk used by the image caches and storage.

The image regeneration and cutout services will be replicated across multiple nodes in a standard load-balanced Web service configuration.

5.5   Maintenance and Upgrades¶

When new data releases are to be published, data is copied from the Archive Center to the Data Access Center (either by local network, wide-area network, or disk shipping). The data is incorporated into the databases and image storage but not made accessible to external users. Testing of the new release to verify completeness, consistency, and accessibility is then performed. Finally, external user access is enabled. Access to the third-oldest release (except the first) is then disabled, and its space is reclaimed.

Level 1 database maintenance, including modifications to its schema, will occur on the Archive Center copy during the day with replication to the DAC copies during a daily maintenance period before the start of nighttime observing. Level 2 databases in new data releases need not have the same schema as in previous data releases. Further details on database schema evolution are in the LSST Database Design document (LDM-135).

When new versions of the DAC software services are to be deployed, they will be brought up on the same hosts as the operational services, but using different network ports. If the new versions require different internal data formats, additional reserved space on the storage media (tape, shared disk, local disk) will be used to hold the transformed data. After the new services are tested, the old and new versions will be swapped. Once the new version has been proved in operation, the old service will be disabled and any space used will be reclaimed.

As for the Data Release Production, new hardware will be deployed throughout the course of the survey to add capacity, to replace failed components, and at specified hardware refresh intervals to avoid obsolescence. Hardware for small databases will be brought up as slaves and then, during a brief shutdown, converted to masters. Hardware for the Query Services database will be brought up as additional replicas of current data and then added to the database cluster. Query Services database to be retired can just be shut down and removed. Web services hardware can be deployed and retired as needed. Image storage will be handled using logical volumes, expanding them across new disks and contracting them away from disks to be removed.