Wrong magnitudes with ELG_NGC tiling

So last week (Thursday), I woke up with a lot of mails from the observers complaining about issues with observing a plate. This was the first plate in the ELG_NGC chunk that we tiled last November. The issue were

1. SoS didn't realize that plate 9585 were and ELG plate and didn't trigger the special reductions.
2. The red camera SN2 were all coming out to be zero.

This was a critical issue as the observers had no ways of analysing the plate completeness while observing.
To correct the first issue, I first tried doing a re reduction to understand what is going wrong. Everything went well in the re reduction and the SoS was correctly recognizing the plate as an ELG plate. I do not know what went wrong during night observations. I just stopped the SoS-deamon and re-started it.

The second issue was more challenging. I first had to understand what is triggering the SoS SN2 to be 0. After much poking around, I found that there are no objects in the magnitude range [z-band 21-22] where the SN2 fitting is done. I checked this for all the plates in that tiling and none had any sources in this magnitude range. I compared it with several other ELG chunks [SGC, test plates] etc. Clearly, only this chunk had this issue. I plotted the magnitude distributions of ELG_NGC and ELG_SGC chunks for comparison. Clearly, there is a difference in the distributions. See the plot below.

If this magnitude difference is real, we had to devise a new strategy to estimate the plate SN2 and plate completeness. Else, we have to update the database with correct magnitudes. I contacted Johan and requested him to take a look at this. (He supplied the ELG targets).
The next morning, Johan verified that there was a bug in his targeting code which copied wrong magnitudes. He sent me a target file with updated magnitudes. Now, I basically had to modifiy all the relevant files in the database which have a dependence on magnitudes. I did a thorough scan of the platelist-dir and found out that the following files need to be modified,

1. '$PLATELIST_DIR/runs/2016.11.[a,b].eboss /plPlugMapP-????.par'
2. '$PLATELIST_DIR/plates/009?XX/00????/plateHoles-00????.par'
3. '$PLATELIST_DIR/plates/009?XX/00????/plateHolesSorted-00????.par'
4. '$PLATELIST_DIR/plates/009?XX/00????/plPlugMapP-????.par'

I did a sphere match of RA-DEC positions of sources in these files with that of Johan's file and updated the magnitudes. After that, I contacted the platedesing coordinator about updating the files. As usual, he never replied back. I was ready with the solution by Friday evening. But, because I didn't get a head nod, I refrained from committing the updated files to the svn. Kyle also asked me to wait to hear from him. The next day I just got a mail from him saying, 'Let me know if it is solved. Only then I will commit'. I replied to him that it is not. But, again silence from him. I waited for one more day to hear back from him. All these days, the night was super clear and observers was having trouble with these plates. I had requested them to burn 5 exposures on these plates if SoS SN2 were not sensible. This was costing us extra exposures because the sky was super clear and we were completing other plates in 3 exposures. On Sunday morning, I just decided to go ahead with the update as it was clearly affecting the eBOSS observations.

After my update, everything is working fine. What I did for in between days is that I updated the relevant files at the observatory and did a rerun early morning. After seeing the correct SoS output, I modified the plug request for that day. I had to wake up very early in the morning to do this re run so that I can send the modified plug request by 8 am in the morning.

Altogether, last week was too hectic and tiring. But, happy that I was able to resolve the issue all by myself.