Follow-up meeting on 04 August 2021

A meeting was held on 04/08/2021 between Emma Bland, Pasha Ponomarenko, Bill Bristow, Gareth Chisham and Kevin Sterne on the topic of accepting fitACF 3.0 as the default processing algorithm to generate fit-level data. The DAWG notified the PIs prior to the SuperDARN 2021 Workshop that a proposal for fitACF 3.0 to become the default would be made. After discussions amongst the PIs during the Workshop, Bill and Gareth provided a summary of the PI feedback to Emma, Pasha, and Kevin. During this meeting, the PI feedback was discussed and a plan was made to address it.

Meeting Summary

This summary reflects the consensus reached during the meeting and has been approved by all meeting attendees.

1. Scientific evaluation of the new fitting algorithm

What are the specific knowledge gaps that need to be filled regarding:

the fitted parameters in fitACF3.0
the use of the lag zero phase for calculating elevation angles
the despeckling routine

The PIs expressed some uncertainty about whether the lag zero phase is more strongly affected by ‘noise’ than the fitted phase.

No other specific knowledge gaps were identified, but there were some more general comments:

It takes considerable time and effort to read/understand/evaluate the work that has already been done on fitACF3.0. Most PIs have not been able to invest this time/effort.
PIs are not necessarily asking for data problems to be explained (e.g. interference). They are asking for data problems to not impact the fitted data
PIs want to be sure that the fitted data from their own radars is high quality. Therefore, the PIs believe that fitACF3.0 must be evaluated for all radars

What information is needed to fill these knowledge gaps?

Brief theoretical justification for using the lag zero phase rather than the fitted phase for calculating elevation angles
Quantitative evaluation of the despecking routine for all radars
Data plots comparing the fitted parameters from fitACF2.5 and fitACf3.0 from all radars
Data plots comparing the elevation angles determined from lag zero phase and fitted phase from all radars.

What is missing from previous analyses presented at the workshops in 2016, 2018 and 2021?

Consolidate previous work into a format that is easily analysed by the PIs
Sample code that can be adapted by each PI to evaluate the data from their own radars

What are the criteria for evaluating this information?

(Ref: earlier statement from the PIEC: ‘we will have objective criteria as a basis for a decision to adopt the new routine’)

No actual criteria for evaluating fitACF3.0 and the despeckling routine have been provided by the PIEC.
Individual PIs probably have their own minimum standards for fitted data quality. These standards may not be well-defined.
We are in a situation where each PI must be convinced that the scientific benefits of fitACF3.0 outweigh the costs based on their own individual assessments.

Conclusion: PIs currently have no common objective criteria for evaluating the fitting algorithm, and it is difficult to unanimously agree on any criteria, objective or otherwise.

Who is responsible for providing this information?

The perspective of the developer (Pasha) is that fitACF3.0 is already available to those who want to use it, and its benefits have already been demonstrated to a standard that would pass through peer review.
Large-scale testing for every radar in the network serves the purpose of satisfying each PI that hardware/operational problems with their own radars are successfully dealt with by the fitting and despeckling routines.
- PIs are in the best position to identify appropriate test datasets from their own radars (standard operating modes/frequencies, known technical problems). Choosing test datasets by the DAWG at random is not guaranteed to reveal a problem if it exists.
PIs need to take a keen interest in, and share responsibility for, the quality of the fitted data from their own radars. Therefore, we invite them to actively participate in the evaluation of fitACF3.0 for their own radars. DAWG will provide sample IDL code to reduce the workload.
Some PIs have already evaluated fitACF3.0 for their own radars using their own methods and criteria. PIs can decide for themselves whether any further evaluation is necessary for their radars.
Results should be freely shared between the DAWG and PIEC to support knowledge exchange and to provide some accountability. Bill shall be responsible for coordinating this effort among the PIs and for sharing results with the DAWG.

Conclusion: PIs and the DAWG share the responsibility to ensure that the outputs of fitACF are appropriate for scientific use

Actions:

Pasha will update the fitACF3.0 White Paper to include more information about why the lag zero phase should be used for calculating elevation
Pasha and Emma will amalgamate the previous workshop presentations into a single easy-to-follow document
Emma will write sample IDL code to perform the statistical analysis requested for evaluating the despeckling routine.
Pasha will write sample IDL code to produce 2D histogram plots comparing the fitted parameters from fitACF2.5 and fitACf3.0
Pasha will write sample IDL code to perform the statistical analysis requested for comparing elevation angles determined from lag zero phase and fitted phase.
PIs are strongly encouraged to use the sample code with their own datasets and share the results with all members of the PIEC and DAWG. A deadline will be set after the sample code is provided. The DAWG can help with interpreting the results on request.

2. Implementation of the despecking routine in RST

The PIEC requested that the despeckle routine be added to make_fit as a non-default command line option. Emma noted that this creates a dual purpose for make_fit (fitting and filtering). The existing standalone routine that performs the despecking (fit_speck_removal) will be retained.

Action: Emma will determine whether it is feasible to add a command line option to make_fit to perform the despecking

3. Exclusion of XCF fitted parameters in fitACF3.0

What is the scientific value of the XCF fitted parameters?

It is not known why these parameters were originally included in FITACF
To our knowledge, the XCF fitted parameters have never been used in a published paper
There are some anecdotal reports that some PI groups have used XCF data for monitoring interferometer data quality, but no PIs have declared that this data is regularly used, or that it has any scientific value

Conclusion: the XCF fitted parameters have no demonstrated scientific value

Actions:

Pasha will provide a brief explanation why the XCF fitted parameters were not included in fitACF3.0 (in progress)
Gareth has requested more information from PI groups that have used XCF fitted parameters

General notes:

RST development is driven by the people who are willing to spend their research time developing new data analysis routines and share them with the community. If a particular group/individual needs XCF power fitting to be included in fitACF3.0, they should liaise with the DAWG and consider investing their own time and resources to implement it. The DAWG should not be tasked with software development for an unused SuperDARN data product.
XCF fitted parameters continue to be available in fitACF2.5 in case anyone wants to use them.
We have agreed that the absence of XCF fitted parameters in fitACF3.0 should not prevent fitACF3.0 from being approved as the standard/default fitting algorithm for SuperDARN data.

4. Use of elv_low and elv_high fields for storing other information

This was going to be addressed by the Data Standards Working Group (DSWG). The PIEC has taken over responsibility for the DSWG tasks, and this issue has been identified as a top priority. As yet, no specific PI has been given responsibility for this action.

The DAWG has already proposed a solution to this issue to the DSWG in October 2020 (link)

The DAWG specifically requested the PIEC to discuss this issue and the proposed solution in February 2021 (link)

Action: Gareth will follow up with the PIEC to decide who shall take responsibility for solving this issue.

5. Christmas Valley data from November 2018

This interval has been analysed in detail by Emma and Pasha and the main findings were (link):

there was an internal interference/noise that was acknowledged by the PI on 14 January 2019 (link)
the number of averages is extremely low (<=5) so that the data do not meet the basic criteria for a statistically valid ACF.

The DAWG recommended blocklisting these data.

Action: No action required

6. Vertical stripes in Clyde River data for 29 May 2021

Note: This item was added after the meeting

This BOREALIS-specific issue has already been analysed and resolved (link), and the necessary corrections have been implemented in RST4.6 on 21 June 2021.

Plots for the 29 May 2021 example are available here
Kevin Sterne has confirmed that the data on davit.ece.vt.edu have been processed with the older software (http://davit.ece.vt.edu/images/tenplot/20210529.tenplot.pola.jpg). The same data processed with the newer version of the code show no such issue.