Differences

This shows you the differences between two versions of the page.

--- ch10_handbook:data_file_interpretation [2014/04/13 12:09] – bob
+++ ch10_handbook:data_file_interpretation [2014/07/16 15:18] (current) – Added links to TSPI and CAN Bus pages bob
@@ Line 1: / Line 1: @@
+~~ODT~~
 ===== DATA FILE INTERPRETATION =====
@@ Line 71: / Line 72: @@
 ==== Overall Data Packet Organization ====
- Overall data packet organization is shown in Figure 6-1. Data packets contain a standard
+Overall data packet organization is shown below. Data packets contain a standard
 header, a data payload containing one or multiple data messages, and a standard trailer. The
 standard header is composed of a required header, optionally followed by a secondary header.
@@ Line 100: / Line 101: @@
 | DATA CHECKSUM | **Packet Trailer** |
-**Data Packet Organization**
+<html><center><b>Data Packet Organization</b></center></html>
 Data packets must contain data. They are not allowed to only contain filler. Filler can be
@@ Line 136: / Line 137: @@
  The packet header contains information about the data payload such as time, packet
 length, data type, data version, and other information. The layout of a Chapter 10 packet header
-is shown in Figure 6-2. FIXME
+is shown below.
 <code>
 struct SuI106Ch10Header
     {
-    uint16_t uSync;         // Packet Sync Pattern
+    uint16_t    uSync;              // Packet Sync Pattern
-    uint16_t uChID;         // Channel ID
+    uint16_t    uChID;              // Channel ID
-    uint32_t ulPacketLen;   // Total packet length
+    uint32_t    ulPacketLen;        // Total packet length
-    uint32_t ulDataLen;     // Data length
+    uint32_t    ulDataLen;          // Data length
-    uint8_t ubyDataVer;     // Data Version
+    uint8_t     ubyDataVer;         // Data Version
-    uint8_t ubySeqNum;      // Sequence Number
+    uint8_t     ubySeqNum;          // Sequence Number
-    uint8_t ubyPacketFlags; // Packet Flags
+    uint8_t     ubyPacketFlags;     // Packet Flags
-    uint8_t ubyDataType;    // Data type
+    uint8_t     ubyDataType;        // Data type
-    uint8_t aubyRelTime[6]; // Reference time
+    uint8_t     aubyRelTime[6];     // Reference time
-    uint16_t uChecksum;     // Header Checksum
+    uint16_t    uChecksum;          // Header Checksum
     };
 </code>
-**Packet header structure.**
+<html><center><b>Packet header structure</b></center></html>
 The Channel ID field uniquely identifies the source of the data. The value of the
@@ Line 197: / Line 199: @@
  The optional secondary header is used to provide an absolute time (i.e. clock time) stamp
 for data packets. The secondary header time format can be interpreted several ways. The
-specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure in
+specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure below
-Figure 6-3 is used when secondary header time is to be interpreted as a Chapter 4 format value
+is used when secondary header time is to be interpreted as a Chapter 4 format value
-(Flag Bits 3-2 = 0). The structure in Figure 6-4 is used when secondary header time is to be
+(Flag Bits 3-2 = 0). The following structure is used when secondary header time is to be
 interpreted as an IEEE 1588 format value (Flag Bits 3-2 = 1).
@@ Line 213: / Line 215: @@
     };
 </code>
-**Optional secondary header structure with IRIG 106 Ch 4 time representation.**
+<html><center><b>Optional secondary header structure with IRIG 106 Ch 4 time representation</b></center></html>
- <code>
+<code>
 struct SuI106Ch10SecHeader_1588Time
     {
@@ Line 224: / Line 226: @@
     };
 </code>
-**Optional secondary header structure with IEEE 1588 time representation.**
+<html><center><b>Optional secondary header structure with IEEE 1588 time representation</b></center></html>
 ==== Data Payload ====
@@ Line 240: / Line 242: @@
 when there are no more data messages to read.
- Intra-packet headers, when they are present, typically contain one, or sometimes more
+Intra-packet headers, when they are present, typically contain one, or sometimes more
 than one, time stamp as well as other information about the data message that follows.
-Commonly used structures for intra-packet time data are shown in Figure 6-5, Figure 6-6, and
+Commonly used structures for intra-packet time data are shown in the three figures below.
-Figure 6-7. These three time structures will be referenced in most of the data format descriptions
+These three time structures will be referenced in most of the data format descriptions
 that follow.
@@ Line 253: / Line 255: @@
     };
 </code>
-**Intra-packet Time Stamp, 48-bit RTC.**
+<html><center><b>Intra-packet Time Stamp, 48-bit RTC</b></center></html>
 <code>
@@ Line 264: / Line 266: @@
     };
 </code>
-**Intra-packet Time Stamp, IRIG 106 Ch 4 binary**
+<html><center><b>Intra-packet Time Stamp, IRIG 106 Ch 4 binary</b></center></html>
 <code>
@@ Line 273: / Line 275: @@
     };
 </code>
-**Intra-packet Time Stamp, IEEE 1588**
+<html><center><b>Intra-packet Time Stamp, IEEE 1588</b></center></html>
-Computer Generated Data
+==== Data Formats ====
+[[Computer Generated Data]]
+[[PCM Data]]
+[[Time Data]]
+[[MIL-STD-1553 Data]]
+[[Analog Data]]
+[[Discrete Data]]
+[[Message Data]]
+[[ARINC 429 Data]]
+[[Video Data]]
+[[Image Data]]
+[[UART Data]]
+[[IEEE 1394 Data]]
+[[Parallel Data]]
+[[Ethernet Data]]
+[[TSPI Data]]
+[[CAN Bus Data]]
+==== Time Interpretation ====
+ Chapter 10 defines a 48-bit Relative Time Counter (RTC) as the basis for all packet and
+message time stamps. The RTC clock is 10 MHz, resulting in a clock resolution of 100
+nanoseconds. There is no constraint on the absolute value of the RTC; at recorder power on, it
+could be initialized to zero or some random number. Some recorder vendors will preset the RTC
+to a value based on absolute clock time, but for interoperability reasons it is unwise to assume
+the RTC value will be related to absolute clock time in any meaningful fashion.
+Absolute clock time comes into a recorder and is recorded much like any other data source. In
+fact, there may be multiple time sources recorded. Time Data, Format 1 data packets are used to
+record input time signals. Since Time Data, Format 1 packets contain both the absolute input
+time value and the RTC clock value at the instant the absolute time was valid, these packets can
+be used to relate RTC values to the input absolute time source. For example, if a time packet is
+recorded with a RTC value of 1,000,000 and an absolute time value of 100:12:30:25.000, then
+the clock time of a subsequent data packet with an RTC value of 1,150,000 could be deduced to
+be 100:12:30:25.015 (150,000 clock tics x 100 nsec per tic = 15 msec).
+ When multiple time channels are available, it is incumbent on the programmer or data
+analyst to determine and select the best source of time for a particular data set. For example,
+there may be separate time channels for time derived from IRIG B, Global Positioning System
+(GPS), and an internal battery backed up clock. In this scenario, all of these time sources are
+present in the data file as separate channels, each correlating the RTC to its own notion of clock
+time. The software application may allow the user to select which source of time to use for a
+given analysis. Alternatively, the software may decide the “best” source of time, depending on
+which time channels are providing valid time. In general, each time source will provide a
+slightly (or not so slightly) different clock time. It is usually most correct to select one time
+channel only and to use this channel exclusively to correlate RTC time to absolute clock time for
+all data packet types.
+ The stability of the RTC isn’t specified in Chapter 10 other than to require it to be at least
+as good as a common crystal oscillator. A good grade crystal oscillator can provide stability on
+the order of 10 ppm. Some vendors provide a RTC source considerably more stable than this. It
+is tempting for an application to find a single time packet early in a recording and to use those
+time values to subsequently derive clock time from relative time. It is better to use the clock and
+relative time values from a time packet that occurs near the current data packet as the data file is
+decoded since there is some drift in the RTC during a recording session. It also may be the case
+that there is a jump in input clock time during a recording, such as when GPS locks for the first
+time, or when an IRIG time source is reprogrammed.
+Index and Event Records
+ Often times it is useful to make an in-memory version of the data file index. This allows
+rapid access to recorded data packets based on time or the occurrence of events. A general
+algorithm for reading all root and node index packets is as follows:
+. If "R-x\IDX\E" is not equal to "T" then index does not exist.
+. Move read pointer to last packet of data file. Store file offset of this packet.
+. If last packet data type does not equal 0x03 (Computer Generated Data, Format 3)
+then index does not exist.
+. Get the index count from the CSDW.
+. For each root index contained in the packet,
+  * Read the Node Index offset value
+  * Move the read pointer to the Node Index offset value
+  * Read the Node Index packet
+  * Get the node index count from the CSDW
+  * For each node index contained in the packet read and store the time stamp, channel ID, data type, and data packet offset values.
+. Read last root node index. If offset value is equal to current root node packet offset
+(stored in Step 2) then done.
+. Else the move read pointer to the next Root Index packet offset value
+. Read the next Root Index packet.
+. Go to Step 4.
+==== Data Streaming ====
+ Chapter 10 recorders can stream their data over one of their download interface network
+ports using User Datagram Protocol (UDP)/IP and Chapter 10 UDP transfer headers. This is
+normally done over an Ethernet port, but any network connection that supports UDP/IP can use
+this method. The .PUBLISH command is used to control data streaming. Chapter 6 defines the
+use of .PUBLISH and has numerous examples of its use. Data can be streamed to one or more
+specific unicast and multicast IP addresses, or broadcast address. Different channels can be
+addressed to different addresses.
+ It is common to publish different groups of data to different multicast groups. According
+to RFC 3171, addresses 224.0.0.0 to 239.255.255.255 are designated as multicast addresses.
+Different multicast address regions are designated for different purposes. According to RFC
+, Chapter 10 data streaming should be directed to multicast addresses in the Local Scope
+address range 239.255.0.0 to 239.255.255.255.
+IP multicast packets are delivered by using the Ethernet MAC address range
+:00:5e:00:00:00 - 01:00:5e:7f:ff:ff. This is 23 bits of available address space. The lower
+bits of the 28-bit multicast IP address are mapped into the 23 bits of available Ethernet
+address space. This means that there is ambiguity in delivering packets. If two hosts on the
+same subnet each subscribe to a different multicast group whose address differs only in the first
+bits, Ethernet packets for both multicast groups will be delivered to both hosts, requiring the
+network software in the hosts to discard the packets which are not required. If multiple multicast
+addresses are used, be careful to choose multicast addresses that will result in different Ethernet
+multicast addresses.
+Multicast data is filtered by the Ethernet controller hardware, only passing subscribed
+packets to the software driver for decoding. This improves performance under high network
+traffic loads. Ethernet controllers only have a limited number of multicast addresses they can
+filter. 16 multicast addresses is a common hardware limit. If a workstation needs to subscribe to
+more multicast addresses than the Ethernet hardware provides for, then all multicast traffic is
+passed to the software driver for filtering, negating the benefit of multicast filtering in hardware.
+The size of a UDP packet is represented by a 16 bit value in the IPv4 IP and UDP headers, but
+some software implementation treat this as a signed value with a maximum value of 2^15 or
+. Because of this, the maximum size of a Chapter 10 streaming packet should be no more
+than 32724 bytes. Physical networks have a Maximum Transfer Unit (MTU), which is the
+largest data packet they can carry. If a UDP packet has a size larger than the network MTU, it
+will be fragmented into smaller packets by the IP software driver before sending them over the
+underlying physical network. The fragmented UDP packets are then reassembled into a larger
+packet by the IP software driver at the receiving end. There is a performance penalty for this
+fragmentation and reassembly. Better performance may be achieved by choosing a UDP packet
+small enough to avoid fragmentation and reassembly. Regular Ethernet supports a maximum
+size of 1500 bytes of data payload (IP header, UDP header, and UDP data) but some newer
+Ethernet technologies support larger jumbo frames.
+Chapter 10 data packets are sent in a UDP/IP packet by prepending a UDP transfer header to the UDP data payload. Chapter 10 data packet(s) smaller than the 32k maximum size will prepend the non-segmented UDP transfer header shown below.
+<code>
+struct SuUdpTransferHeaderNonseg
+    {
+    uint32_t uVersion    :  4;  // Version
+    uint32_t uType       :  4;  // Type of message
+    uint32_t uUdpSeqNum  : 24;  // UDP sequence number
+    };
+</code>
+<html><center><b>UDP Transfer Header, non-segmented data</b></center></html>
+A Chapter 10 data packet larger than the 32k maximum size will need to be segmented before transmission, and
+will prepend the segmented UDP transfer header shown below. IPv6 supports large data packets, negating the need for segmented data packets.
+<code>
+struct SuUdpTransferHeaderSeg
+    {
+    uint32_t uVersion    :  4;  // Version
+    uint32_t uType       :  4;  // Type of message
+    uint32_t uUdpSeqNum  : 24;  // UDP sequence number
+    uint32_t uChanID     : 16;  // Channel ID
+    uint32_t uChanSeqNum :  8;  // Channel sequence number
+    uint32_t uReserved   :  8;  //
+    uint32_t uSegOffset;        // Segment offset
+    };
+</code>
+<html><center><b>UDP Transfer Header, segmented data</b></center></html>
+Computer Generated Data, Format 3 (Recording Index) packets are meaningless in a
+network data stream. It is necessary that they be transmitted so that Channel ID 0 data packets
+will have contiguous sequence numbers for error detection. They should be ignored, though,
+when received.