Table Filtering

Contents of This Section

• Table Filtering
• Examples
• Header Parameters
• Math Functions
• Region Filtering with Expanded Capabilities

Table Filtering

Table filtering allows a program to select rows from an table (e.g., X-ray event list) to process by checking each row against one or more expressions involving the elements in the table. When a table is filtered, only valid rows satisfying these expressions are passed through for processing. Typically, filter specifications are specified using bracket notation appended to the filename of the data being processed:

  foo.fits[pha==1&&pi==2]

  foo.fits[@my.reg]

When row filters are passed in bracket notation in this manner, the filtering is set up automatically when the file is opened and all processing occurs through the filter. Programs also can use the filter library API to open filters explicitly. Table filtering can be performed on columns of data in a FITS binary table or a raw event file. Table filtering is accomplished by means of table filter specifications. An table filter specification consists of one or more filter expressions Filter specifications also can contain comments and local/global processing directives.

Filter Expressions

  # comment until end of line
  # include the following file in the table descriptor
  @file
  # each row expression contains shapes separated by operators
  [filter_expression1], [filter_expression2], ...
  [filter_expression], [filter_expression], ...
  # the special row# keyword allows a range of rows to be processed
  row#=m:n

  pha>1
  energy == pha
  pha>1 && energy<=2
  max(pha,energy)>=2.5

  (status & 15) == 8           # decimal
  (status & 017) == 010        # octal
  (status & 0xf) == 0x8        # hex
  (status & 0b1111) == 0b1000  # binary

The special keyword row# allows you to process a range of rows. When row# is specified, the filter code skips to the designated row and only processes the specified number of rows. The "*" character can be utilized as the high limit value to denote processing of the remaining rows. Thus:

  row#=100:109

  row#=100:*

Table filtering is more easily described by means of examples. Consider data containing the following table structure:

• double TIME
• int X
• int Y
• short PI
• short PHA
• int DX
• int DY

Tables can be filtered on these columns using IRAF/QPOE range syntax or any valid C syntax. The following examples illustrate the possibilities:

pha=10

pha==10

select rows whose pha value is exactly 10

pha=10:50

select rows whose pha value is in the range of 10 to 50

pha=10:50,100

select rows whose pha value is in the range of 10 to 50 or is equal to 100

pha>=10 && pha<=50

select rows whose pha value is in the range of 10 to 50

pha < 10 || pha>50

select rows whose pha value is less than 10 or greater than 50

pi=1,2&&pha>3

select rows whose pha value is 1 or 2 and whose pi value is 3

pi=1,2 || pha>3

select rows whose pha value is 1 or 2 or whose pi value is 3

pha==pi+1

select rows whose pha value is 1 less than the pi value

(pha==pi+1) && (time>50000.0)

select rows whose pha value is 1 less than the pi value and whose time value is greater than 50000

(pi+pha)>20

select rows in which the sum of the pi and pha values is greater than 20

pi%2==1

select rows in which the pi value is odd

Currently, integer range list limits cannot be specified in binary notation (use decimal, hex, or octal instead). Please contact us if this is a problem.

Separators Also Are Operators

As mentioned previously, multiple filter expressions can be specified in a filter descriptor, separated by commas, new-lines, or semi-colons. When such a separator is used, the boolean AND operator is automatically generated in its place.

Note that the behavior of separators is different for filter expressions and region expressions. The former uses AND as the operator, while the latter user OR. See Combining Region and Table Filters for more information about these conventions and how they can be combined.

Header Parameters

The filter syntax supports comparison between a column value and a header parameter value of a FITS binary tables (raw event files have no such header). The header parameters can be taken from the binary table header or the primary header. For example, assuming there is a header value MEAN_PHA in one of these headers, you can select photons having exactly this value using:

• pha==MEAN_PHA

Math Functions

It also is possible to specify C math functions as part of the filter syntax. When the filter parser recognizes a function call, it automatically includes the math.h and links in the C math library. Thus, it is possible to filter rows by expressions such as these:

• (pi+pha)>(2+log(pi)-pha)
• min(pi,pha)*14>x
• max(pi,pha)==(pi+1)

Region Filtering with Expanded Capabilities

In addition to the region filtering described in the Spatial Region Filtering document, we have implemented special spatial region filtering for rows by allowing filtering to occur on any column. That is, each of the standard region shape specifications comes in two forms: the first is the same as the standard IRAF/PROS region specification:

• annulus(x,y,ri,ro)
• circle(x,y,r)
• ellipse(x,y,a,b[,angle])
• pie(x,y,r1,r2)
• point(x,y)
• polygon(x1,y1,x2,y2,...,xn,yn)
• box(x,y,w,h[,angle])

With the standard specification, filtering is applied to the columns that are being used to bin the data, i.e., to the columns pointed to by the bincols= switch or the FITS_BINCOLS or EVENTS_BINCOLS environment variables.

The second form has two additional argument that specify which columns to use as the "X" and "Y" columns:

• annulus(X,Y,x,y,ri,ro)
• circle(X,Y,x,y,r)
• ellipse(X,Y,x,y,a,b[,angle])
• pie(X,Y,x,y,r1,r2)
• point(X,Y,x,y)
• polygon(X,Y,x1,y1,x2,y2,...,xn,yn)
• box(X,Y,x,y,w,h[,angle])

Any table column can be specified as the X and Y arguments. For example, if the EVENT table above is binned on X and Y, you can specify a filter for all rows within a circular region of the detector coordinates DX=400, DY=350 using:

• circle(DX,DY,400,350,20)

Note that we also support the Chandra region syntax for specifying alternate columns for filtering:

• (X,Y)=annulus(x,y,ri,ro)
• (X,Y)=circle(x,y,r)
• (X,Y)=ellipse(x,y,a,b[,angle])
• (X,Y)=pie(x,y,r1,r2)
• (X,Y)=point(x,y)
• (X,Y)=polygon(x1,y1,x2,y2,...,xn,yn)
• (X,Y)=box(x,y,w,h[,angle])

Include Files

The special @filename directive specifies an include file containing filter expressions. This file is processed as part of the overall filter descriptor:

  foo.fits[pha==1,@foo]

Last updated: March 30, 2001