adding RESPONSEFORMAT, DALI error responses & OVERFLOW usage #64

molinaro-m · 2025-07-28T13:11:29Z

Covering issue #61, adding DALI RESPONSEFORMAT as optional to the ConeSearch query interface.
Consequently clarifying in the succesfull response behaviour that optional response formats are possible, while VOTable 1.x stays as the mandatory one.

Updating ivoatex to include also VOTable-1.5 reference.

mbtaylor

I don't think the text after this PR as it stands is very clear; the item 2.2 starts off by saying the service must respond with a VOTable, then says it can be some other format, then talks about the detailed format that the VOTable MUST have. The next item (2.3) then says that in case of error a service must respond with a VOTable error document, which is probably undesirable for a client requesting e.g. RESPONSEFORMAT=text/csv.

I think it would be clearer to say in 2.2:

a successful response must be a table containing ID, RA, Dec with certain conditions on the content
table format is VOTable by default, but if RESPONSEFORMAT is set it has to match that
if it's a VOTable, then it must obey particular rules (UCDs, types, RESOURCE structure etc).

and in 2.3:

by default, errors are reported using a VOTable stub with a PARAM/INFO having name="Error"
if a non-VOTable RESPONSEFORMAT is requested, the response SHOULD be a plain text error document instead

However, I'm not sure what the scope of this revision is supposed to be. If the idea is to make the minimum possible changes to the text while introducing more DALI-compatible options, maybe the PR as it stands would be OK.

The PR also includes hard-coded section/item numbers, though that also follows practice in the existing text (should we open a separate issue to address this?)

molinaro-m · 2025-07-28T14:35:57Z

Fair points.
I'll shuffle the content of 2.2 to make it clearer.
2.3 will be completely rewritten (doing it now)

As for the hardcoded sections, I'll remove them as soon as I realise how to have \ref+\label work in that case.

BTW: this won't end up as is in a proper WD before issues #62 and #18, at least, are covered, but possible also others. That is, there's time to fix text.

mbtaylor · 2025-07-28T14:41:32Z

Thanks Marco. I seem to recall that LaTeX enumeration items can get cross-referenced by using \refs in the right place, though I don't have my LaTeX manual by me right now. But maybe it would be better to replace that enumerate environment with \subsections anyway? Those parts are referred to as "subsections" in the text.

molinaro-m · 2025-07-28T14:49:08Z

I'm hesitating at moving those bullets into sections, that's what I tried for the "old" v.1.1 and it was deemed to be difficult when comparing 1.03 to 1.1.

I can retry, now that there won't be the deprecation sections (that was really confusing).

Maybe also limiting to subsections, rather than fully exploding into subsubsections for the parameters and fields. Up for discussion.

mbtaylor · 2025-07-28T14:53:31Z

I feel like just going to 2.* subsections for that one enumeration in Section 2 would help, since the text already refers to those parts by sec.subsec number, without need to go further. But if you've tried it and it doesn't work out, I'm happy to defer to your judgement.

molinaro-m · 2025-07-31T13:07:37Z

Following @mbtaylor comments and suggestions, I promoted numbered listings in Sec.2 to sub-sections and re-ordered the text there to make it more clear.

I should also have removed the hard-coded references.

Additionally, I ported here the re-writing of the error responses, following DALI rec.

This means that, when this is PR is clean and merged, it will close issues #61, #45, and #54

mbtaylor

@molinaro-m thanks for the changes. There are still a few items I think should be addressed as per my comments.

mbtaylor · 2025-08-01T09:29:50Z

ConeSearch.tex

@@ -167,11 +170,17 @@ \section{Service Interface Requirements}
 				\item[Example]
 					\url{http://mycone.org/cgi-bin/search?RA=180.567&DEC=-30.45&SR=0.0125}
 			\end{bigdescription}
-		\item As defined by DALI a service SHOULD also understand the following parameter:
+		\item As defined by DALI a service SHOULD also understand the following parameters:
 			\begin{description}
 				\item[\textbf{MAXREC}] to let the client limit the number of records returned
 					or require a service metadata response. Its usage is encouraged and preferred
 					to the SR=0 solution for metadata discovery.


This should probably say "its usage with a value of 0" rather than "its usage".
Note however it's a bit problematic preferring MAXREC=0 to SR=0 for metadata probing, since services that don't support the optional MAXREC parameter (including all pre-v1.1 services?) will ignore it and potentially give you a lot of data. Preferred usage for metadata probing would therefore be MAXREC=0&SR=0&. But maybe that's either obvious or too much detail to put in here.

I should have fixed this, adding a footnote for the additional consideration on MAXREC being optional

mbtaylor · 2025-08-01T09:31:44Z

ConeSearch.tex

+					formats other than the mandatory VOTable one. Indeed, a Simple
+					Cone Search service MUST provide responses in VOTable format \citep{2025ivoa.spec.0116O},
+					compliant with respect to what will
+					be detailed in the next subsection (Sec. 2.2), but should also


Hardcoded xref "Sec 2.2" should be "Sec \ref{subsec:response}"

thanks for spotting this! fixed.

mbtaylor · 2025-08-01T09:33:36Z

ConeSearch.tex

+	\item one numerical column representing the main value for the
+		right-ascension of the source in the ICRS coordinate system;
+	\item one numerical column representing the main value for the
+		declination of the source in the ICRS coordinate system.


Specify decimal degrees for RA & Dec output columns.

OK, I have added a sentence but this wasn't in the 1.03 version... I'll add it as a discussion point for the dedicated running meeting.

I see what you mean, this could in principle invalidate an existing CS 1.03-compliant service that returns RA and Dec in radians, so is maybe questionable. But I doubt that there are any such services, and being able to assume degrees here simplifies rigorous client processing a lot, so I'd say it's worth that small risk. As you say: let's discuss.

From discussion at DAL Running Meeting 21 the requirement to have RA and Dec in decimal degrees stays as is. No major issues foreseen from a quick investigation of the existing services behaviour.

mbtaylor · 2025-08-01T09:40:24Z

ConeSearch.tex

+VOTable\citep{2025ivoa.spec.0116O} format. However, if RESPONSEFORMAT is
+set (see Sec.~\ref{subsec:baseurl}) the response table \textbf{should}
+match the requested format, depending on the service capabilities.


According to DALI 1.1 sec 3.4.3 a service

"should fail (Section 4.2 ) where the RESPONSEFORMAT parameter specifies a format not supported by the service implementation."

So maybe reword

the response table \textbf{should} match the requested format; if this format is unsupported the request \textbf{should} fail.

check if it's fine now.

mbtaylor · 2025-08-01T09:46:44Z

ConeSearch.tex

+	\item There must be a single RESOURCE with \texttt{type=''results''} in the VOTable,
+		and that RESOURCE must contain a single TABLE. Additional RESOURCE(s) are
+		allowed to enable, e.g., DataLink service descriptors.


OVERFLOW signalling as described in DALI 1.1 section 4.4 should be mentioned here as well. Alternatively, just defer to DALI 1.1 sec 4.4 for the whole description of how to write DALI-compliant VOTable results.

Referenced DALI 1.1 Sec.4.4, even if I must say, these sort of references never look perfect to me.

Agreed referencing fixed sections of fixed versions in other standards is not great, though it does a usability service to readers for as long as the version in question is current. It may be that just referring to the DALI standard without quoting section or version is better, I don't have a strong opinion.

mbtaylor · 2025-08-01T09:48:23Z

ConeSearch.tex

+In the case the error is expressed in VOTable format, recommendation
+expressed in Section 4.4 of DALI \textbf{should} be followed,
+inluding the OVERFLOW behaviour in case the MAXREC parameter is set.


OVERFLOW is not an error condition, so it should not be mentioned here. OVERFLOW is marked as described in DALI 1.1 sec 4.4 as part of a successful response.

Removed, DALI is also already mentioned wit this respect.

molinaro-m · 2025-08-23T12:21:26Z

@mbtaylor , please review changes

gmantele

This is clearly an improvement compared to ConeSearch 1.03.
I just have a single comment in addition to what @mbtaylor has originally said about the metadata probing mechanism.

gmantele · 2025-08-28T08:16:04Z

ConeSearch.tex

+					or require a service metadata response. Its usage with a value of 0 is encouraged
+					and preferred to the SR=0 solution for metadata discovery
+					\footnote{However, keep in mind that, for back-compatibility reasons, MAXREC
+					is optional and thus SR=0 needs to be accepted for metadata probing.}.


So, I guess that MAXREC=0&SR=0 is equivalent SR=0 ; the user will get the metadata.

But what happens if we have just MAXREC=0? Should we get an error (because SR is not 0) or just the metadata?

I don't see a benefit in designating MAXREC=0 on its own an error. But clients doing this may find (if they're talking to a non-MAXREC-aware service) that they get more data than they want. The same applies to other values of MAXREC.

I would leave this question open.
I added a bullet to the running meeting discussion.

My view:

MAXREC=0 and missing SR is an error since SR is mandatory

MAXREC=0 and SR not 0 depends on the version the server complies to; so far v.103 only it should answer with records depending on SR value ignoring MAXREC (that cannot be understood), otherwise (let's say v1.1+) it's undefined as of now, it could be a flag for conflict or metadata tailored to the requested SR (if it makes sense)

@molinaro-m you are quite right; my comment missed the point that SR is mandatory.

commit 9366d93 should fix this with respect to DAL Running Meeting 21 discussiondiscussion

gmantele · 2025-08-28T08:26:32Z

ConeSearch.tex

+% remove the constraint of one RESOURCE only, clarifying it was there
+% only for the results
+%
+% added DALI MAXREC and RESPONSEFORMAT parameters, for response


These items of the Change log are commented. Should some of them be un-commented now?

Right, I'll update the changelog, maybe I'll wait the solutions to the above comments (i.e. the running meeting at least) before making this last update before merging.

commit c293414 should have fixed this

mbtaylor

Thanks @molinaro-m, that looks good to me now.

molinaro-m · 2025-09-05T12:03:08Z

OK, @mbtaylor and @gmantele please, review latest updates and requested changes replies

mbtaylor

Yes for me this is OK, except that one of the recent commits deleted the preceding backslash from a "textbf{should}", should be fixed.

The change lists in Appendix C are in need of some attention, since with the newly uncommented lines there are items duplicated between headings "Changes from WD-1.1-20200828" and "Changes from REC-1.03"; probably these headings should be consolidated but it doesn't need to be done in this PR.

molinaro-m · 2025-09-05T12:32:44Z

Should be fixed now.
I reworked the history, trying to convey that previous changes were incorporated in the new WD.

gmantele

It seems all good to me.

molinaro-m added 4 commits October 14, 2024 18:22

SR and MAXREC behaviour

4728fb7

Merge branch 'master' into responseformat

09a075f

syncing with latest ivoatex to have VOTable-1.5 to cite

4449e5c

adding RESPONSEFORMAT to cover issue ivoa-std#61

1f346c1

mbtaylor reviewed Jul 28, 2025

View reviewed changes

molinaro-m marked this pull request as draft July 28, 2025 14:52

molinaro-m added 3 commits July 29, 2025 14:50

moved numbered listing in Sec.2 to subsections

0eeff8b

reworked the flow of the successful response subsection

286f47c

replaced error response prescription, following DALI

08a8708

molinaro-m changed the title ~~adding RESPONSEFORMAT~~ adding RESPONSEFORMAT, DALI error responses & OVERFLOW usage Jul 31, 2025

molinaro-m marked this pull request as ready for review July 31, 2025 13:10

mbtaylor requested changes Aug 1, 2025

View reviewed changes

molinaro-m added 2 commits August 23, 2025 14:10

fixing following MBT feedback

d5eabd6

filled silly typo

fca7bea

gmantele requested changes Aug 28, 2025

View reviewed changes

mbtaylor previously approved these changes Sep 1, 2025

View reviewed changes

molinaro-m added 2 commits September 5, 2025 13:42

applied DAL RM#21 equivalence for MAXREC=0 and SR=0

9366d93

cleaned and updated the change history

c293414

molinaro-m dismissed mbtaylor’s stale review via c293414 September 5, 2025 11:51

mbtaylor requested changes Sep 5, 2025

View reviewed changes

typo fixing and history cleaned

0635b3d

mbtaylor approved these changes Sep 5, 2025

View reviewed changes

gmantele approved these changes Sep 9, 2025

View reviewed changes

molinaro-m merged commit ace4fe4 into ivoa-std:master Sep 9, 2025
1 check passed

This was referenced Sep 9, 2025

missing RESPONSEFORMAT #61

Closed

VOTable error response is DALI misaligned #54

Closed

recommending "OVERFLOW" response if applyable #45

Closed

molinaro-m deleted the responseformat branch September 9, 2025 13:30

adding RESPONSEFORMAT, DALI error responses & OVERFLOW usage #64

adding RESPONSEFORMAT, DALI error responses & OVERFLOW usage #64

Uh oh!

Conversation

molinaro-m commented Jul 28, 2025

Uh oh!

mbtaylor left a comment

Choose a reason for hiding this comment

Uh oh!

molinaro-m commented Jul 28, 2025

Uh oh!

mbtaylor commented Jul 28, 2025

Uh oh!

molinaro-m commented Jul 28, 2025

Uh oh!

mbtaylor commented Jul 28, 2025

Uh oh!

molinaro-m commented Jul 31, 2025

Uh oh!

mbtaylor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

molinaro-m Aug 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

molinaro-m commented Aug 23, 2025

Uh oh!

gmantele left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbtaylor left a comment

Choose a reason for hiding this comment

Uh oh!

molinaro-m commented Sep 5, 2025

Uh oh!

molinaro-m Aug 23, 2025 •

edited

Loading