%.IDENTIFICATION anafile.tex
%.PURPOSE        Man page of anafile, can be edited in LaTeX
%.AUTHOR         Francois Ochsenbein
%.VERSION   1.0  21-Jul-1992
%.VERSION   1.1  07-Oct-1992: Added Formats
%.VERSION   1.2  11-Dec-1992: 5 columns in format file
%.VERSION   1.3  05-Oct-1993: Ranges + required
%.VERSION   1.4  18-Dec-1993: New column statistics
%----------

\def\ifhtx{\iffalse}	% Ignored in HTX context
\ifhtx\else\def\A#1#2{#2}\fi
\def\pgm{{\bf anafile}\ }
\def\CDSdoc{{\em\A{http://vizier.u-strasbg.fr/doc/catstd.htx}
	{Standards for Astronomical Catalogues}}}
\begin{manpage} {1}{anafile}{analyze a file and print statistics.}
	{V7.2 (January 2007)}{file analysis}
\def\pfile{{\em file}}
\syntax{ \pgm [{\bf--v|--q}] [{\bf--l}] [{\bf--a}] [{\bf--u}] [{\bf--cc}[{\bf g}]]
	[{\bf --head}[{\bf=}$n$]|--hash]
	[{\bf --d}[{\em c}] [{\bf --f}[{\em x}] {\em format\_file}]
	[{\bf--w} {\em width}] 
	[[{\bf--t} {\em table\_structure}] {\em input\_file...}] }

\mansec{ Description }
The \pgm program is meant mainly to check the concordance of a {\em description
file} (generally named {\tt ReadMe}) with the actual data files, 
and to report the inconsistencies.

Without a description file, \pgm 
analyses the beginning or the complete \pfile(s),
assigns a {\em file class} to each file, and lists on request
the column (byte-per-byte) statistics.
The possible {\em file class} values are detailed in the section
{\em File Classes} below.

\mansec{Options}
Without any option, \pgm examines the first 2880 bytes, 
and assigns the {\em file class}. 

\option{--a}
	lists the position (line and column numbers) of each character
	which leads to a classification as {\em ascii binary} or {\em binary}
	file.
\option{--d}
	considers the input file as {\em tab-separated-values}
	({\bf--d} alone option), or
	{\em column-separated-values} where the character {\em c}
	represents the column separator. This option, combined with
	({\bf--cc}, generates statistics by columns (width of each column,
	and characters present there).
\option{--cc}
	generates a detailed list of the frequencies of every character in
	each column: the characters of each column are listed in order
	of their decreasing frequency.\\
	Associated to the {\bf--d} option, the contents of every field 
	(defined as the contiguous bytes between the column separator)
	are examined, and statistics on its contents are listed.
	With the addition of $g$ (i.e. with option {\bf--ccg}),
	the statistics include computations of min/max in each field,
	a proposal of a description to in include in a {\tt ReadMe} file
	and possible alignements (using the {\bf acut} program)
	are suggested.\\
	See also the {\bf--head} option if some top lines have to be skipped
	(or used as titles / units)
\option{--hash} is a shortcut for {\bf--headlines=\#}
\option{--headlines}
	specifies that the input file may contain heading line(s) -- i.e.
	lines that explain the contents. These heading lines (the {\em header})
	may be formed by a {\em number of lines} or a {\em leading character}.
	For instance, the existence of 2 heading lines is specified with
	{\fg{DarkRed}\tt-head=2}; the existence of heading lines
	starting by the {\em hash} sign is is specified with
	 {\fg{DarkRed}\tt-head='\#'}. The default is a singlle line
	 (i.e. equivalent to {\fg{DarkRed}\tt-head=1})
\option{--f{\em x} {\it format\_file}}
	uses the contents of {\it format\_file} to check the compliance
	of the file to the specified format. The {\em x} may be used for
	further options concerning this format file like computation
	of ranges, verifications against the {\em CDS Standards}, etc...
	(see the section {\em Format File} below).
\option{--l}
	asks to examine the complete files to assign the {\em file class},
	and lists the number of lines as well as the number of bytes
	of the longest line.
\option{--q}
	{\em suiet} mode: the messages are minimized.
\option{--t {\em table\_structure}}
	indicates that the next {\em file} argument designates a
	data file which contains data structured like
	{\em table\_structure} (for instance an excerpt of a table).
	{\em table\_structure} is therefore a name which must exist
	in one of the {\em Byte-by-byte Description ...} section in the
	{\tt ReadMe} file. \\
	A value of {\tt-} for {\em table\_structure} asks to stop this
	behaviour. \\
	Note that this option can only work when a {\bf--f} specification
	precedes {-bf--t}.
\option{--u}
	lists the columns which are constant 
	(i.e. have exactly the same contents)
	over all lines. This option may be used to check that e.g. that the
	decimal points are correctly aligned, or to find out the blank columns
	which could be removed with {\em trcol}(1).
\option{--v}
	is a verbose option.
\option{--w {\it width}}
	specifies the assumed column width for {\em ascii bulk} or {\em ebcdic}
	files; such files have no {\em linefeed} embedded, and the
	length of each line must be assumed.

\mansec{Format File}
	A typical format file (specified via the {\bf--f} option)
	contains the following:
\begin{verbatim}

Byte-per-byte Description of file: hbc
-------------------------------------------------------------------
   Bytes Format  Units   Label    Explanations
-------------------------------------------------------------------
   2-  4  I3     ---     HBC      [1,423]+ HBC number.
       5  A1     ---     NEBUL    [n] Nebulosity association flag.
       6  A1     ---     REMARK   [*] Remark flag.
   8- 18  A11    ---     NAME     [A-Z0-9@.+-]! Star name.
  20- 56  A37    ---     OTHER    Other designation.
  59- 60  I2     h       RAh      Hours of right ascension (1950.0).
  62- 63  I2     min     RAm      Minutes of right ascension.
  65- 69  F5.2   s       RAs      Seconds of right ascension.
      71  A1     -       DE-      Sign of declination (1950.0).
  72- 73  I2     deg     DEd      Degrees of declination.
  75- 76  I2     arcmin  DEm      Minutes of declination.
  78- 81  F4.1   arcsec  DEs      Seconds of declination.
  83- 90  A8     ---     REF      References to the position.
-------------------------------------------------------------------

\end{verbatim}
	
	The {\em format file} is made of five columns: the {\em byte position},
	the {\em format}, the {\em units}, the {\em label}
	and an {\em explanation text}. Such a file is
	interpreted by \pgm, is reedited in a standard form 
	(on the screen if the {\bf --v} option is present, or the
	{\em format file} is rewritten with the {\bf --fw} option),
	and is used for data check. Note also that special {\em label}s
	are understood by \pgm (see {\em Special Labels} below).

	The {\em explanation text} may contain further restrictions
	concerning the {\em range} for numeric fields or the 
	{\em character set} allowed for alphabetical field;
	refer to {\em Validity Checks} section below.

	Note that the {\em byte position} may be specified as 
	{\em relative} from the end of the previous field
	when followed by the {\tt X} letter, as it is in Fortran.
	The number preceding the {\tt X} therefore represents the 
	number of blanks between the two columns.
	
	With the {\bf --f.} (a dot following the f) option, the input
	format file does not include the {\em units} column; 
	the reedition fills this column with dashes {\bf---} 

	With the {\bf --f1} (a one following the f) option, the first column
	of the input format file contains only the {\em starting byte}
	of the column; the ending byte is derived from the {\em format} column.
	The reedition (with option {\bf--f1w})
	completes this column with the ending byte.

	With the {\bf --f1X} (a one and X following the f) option, 
	it is assumed that a blank always separates two adjacent
	columns; the contents of the {\em starting byte}, if existing, 
	is ignored.
	The reedition (with option {\bf--f1Xw})
	computes the {\em starting}--{\em ending} byte column.

	With the {\bf --fr}  option, the actual {\em ranges}
	(minimal / maximal values) of each column are computed.

	With the {\bf --fs}  option, the 
	{\em format file} is assumed to conform to the \CDSdoc;
	further 
	compliance checks (presence of titles, correctness of
	units, etc...) of this {\em format file} are then performed.

	With the {\bf --fw}  option, the 
	{\em format file} is rewritten according to standards.

	The {\em format options} may be combined, e.g. {\bf--f1.w}
	asks to rewrite {\em format file} where no units and no 
	ending columns were supplied.

\mansec{Special Labels}
Some labels are recognized and further checks are performed.
The complete list with implied defaults are listed in the
\CDSdoc. A few frequent such labels are:
\begin{itemize}
\item	Right ascension fields {\tt RAh}, {\tt RAm} and {\tt RAs} 
\item	Declination sign {\tt DE-} (only a sign or a blank is accepted)
\item 	Declination values {\tt DEd}, {\tt DEm} and {\tt DEs}
\item	Positions in decimal degrees {\tt RAdeg} (range $[0,360[$)
	 and {\tt DEdeg} (range $[-90,+90]$)
\item	Galactic positions {\tt GLON} (range $[0,360[$) 
	 and {\tt GLAT} (range $[-90,+90]$)
\item	Position angle {\tt PA}
\end{itemize}

\mansec{Validity Checks}
	The first word (i.e. set of characters followed by a blank)
	of the explanation of a column
	may specify {\em validity checks} to perform about:
	\begin{enumerate}
	\item the available {\em range} of the value in the column
	\item the possibility of {\em blank} or {\em NULL} values
	\item the {\em order} of the value within the table
		(increasing / decreasing value)
	\end{enumerate}

	For a {\em numerical} field, a {\em range} 
	can be specified in the
	format file if the explanation text starts with 
	a square bracket {\bf [ ]} as in the {\tt HBC}
	column in the above example. The opening
	bracket is {\bf[} if the lower value is included, and
	{\bf]} if the lower value is excluded --- {\em i.e.} 
	the standard mathematical conventions apply. Both
	lower and upper values are not required; for instance,
	the specification of any value lower than 100 (100 excluded)
	is specified by  {\bf [,100[}. Writing {\tt[]} is acceptable
	when no range checking applies --- {\em e.g.}
	to override the default range implied by the {\em label}
	(see {\em Special Labels} section above).
	
	For an {\em alphabetical} ({\em i.e.} A-format) field,
	the set of the allowed characters may be specified in the format
	file if the explanation text
	starts with a square bracket {\bf[}: permitted characters
	are surrounded by square brackets {\bf[\dots]}, and the
	dash indicates a range (in the ascii sequence). The closing
	bracket is accepted as a character permitted in the set if
	it is specified first (i.e. {\tt[]]} means that only the
	closing bracket is acceptable); the dash is accepted when it is first or
	last character. On the above example, only {\tt n} (or blank)
	is accepted in the {\tt NEBUL} column, and uppercase letters,
	digits, and the symbols {\tt @ . + -} in the {\tt NAME} column.

	An exclamation mark {\bf!} {\em following immediately} the
	range or character-set specification indicates that the field
	{\em cannot be blank}, i.e. cannot be filled with only blanks.
	In the above example, the Name field can  never be blank.

	A question mark {\bf?} {\em following immediately} the
	range or character-set specification indicates that the field
	{\em can be blank}, i.e. can be filled with only blanks.
	The default concerning blank fields is:
	\begin{itemize}
	\item {\em numeric} field: the column cannot be filled with
		blanks, unless the {\bf?} symbol exists;
	\item {\em alphabetic} (A-format) field: the column 
		can be filled with blanks; the {\bf!} means
		that a completely blank field can't exist.
	\end{itemize}
	The question mark may be followed immediated by a numeric
	value to designate non-blank {\em NULL} values, e.g.
	{\bf ?=99.99} telling that values $99.99$ indicate unknown
	values.
	
	The {\em order} within a column can be specified with the signs:
	\begin{itemize}
	\item[{\bf+}] indicates {\em strictly increasing} values in the table;
	\item[{\bf--}] indicates {\em strictly decreasing} values in the table;
	\item[{\bf=}], associated to {\bf+} or {\bf--},  indicates 
		that the variation is not strict, i.e.
		the row number $n+1$ may have a value larger than or
		equal to that of row $n$ if {\bf+=} is specified.
	\end{itemize}

\mansec{File Classes}
Outside the context of a description file, an input file is classified
into one of the following categories according to its contents:
\begin{itemize}
\item {{\em ascii file}}\\
	standard ascii file, where no other control characters than {\em tabs}, 
	{\em carriage-return} and {\em line-feed} can be found.
\item {{\em ascii mailable}}\\
	may include other control characters like
	{\em bell} or {\em escape} which can be sent over
	the network without problem.
\item {{\em ascii binary}}\\
	includes control characters which cannot be
	sent over the network, like {\em Control-Q}.
	(see also the {\bf --a} option)
\item {{\em ascii bulk}}\\
	does not include any {\em line-feed}, 
	which means that the file cannot be seen as a set of lines.
	(see {\bf--w} option)
\item {{\em ebcdic}}\\
	EBCDIC (IBM-specific) file
\item {{\em fits}}\\
	FITS file (bulk file with all headers)
\item {{\em hdf}}\\
	FITS header (starting with {\tt SIMPLE } or {\tt XTENSION})
	without the data part, and stored as a standard ascii file.
\item {{\em directory}}\\
	(not an actual file)
\item {{\em binary}}\\
	file which cannot be classified in one of the above categories.
\end{itemize}

\mansec{Returned Status}
The \pgm command returns 0 in case of success.
A non-zero status may indicate bad options or unreadable files;
it is also returned when the {\em format\_file} indicated by a 
{\bf--fs} option does not conform to the \CDSdoc.

\seealso{awk(1)    gawk(1)   trcol(1) ``\CDSdoc''}
\end{manpage}
