soitanly

federal Small Business Innovation Research program for automated interpretation of EEG "brain wave" and other physiological data. During the last two years, Spectre has been adapted to analyze market price data. In addition to very pure spectral analysis of time series or price data to extract precise frequency domain measurements, Spectre uses a "family" of fast minus slow oscillators to extrapolate from past behavior. Ordinarily, only one such oscillator is employed using available technical analysis tools. But Spectre typically generates three to five significant (trending, long period) oscillators and a plethora of shorter cycles. In a family of such oscillators, the short periods are usually "shaping" factors we ordinarily visually identify as confined to a channel, a pennant formation, or other recognizable pattern. Spectre treats the oscillators as being coupled, damped and driven, just as in many physics models of collective behavior. Coupled means interacting: price data affects and is affected by not only adjacent prices, but more significantly by prices from distant “patterns” which recur during the price history. Damped means dissipative: eventually all patterns change, or even disappear. Driven means an external force is applied: in Spectre, the external force is derived from “sentiment” derivations, see below. Basically, the oscillators are semi-coherent collections of past prices. This means that price is not only dependent on past price, but also affects future prices, to the extent that coupling is identified. However, coupling can be disrupted by unforeseeable events in the external world. Such a disruption affects the shortest period oscillators the most. Their effect therefore weakens substantially the further an extrapolation proceeds. Far from a random walk, or even a log-normal distribution, an extrapolation using Spectre is not just a wavy line into the future, but uncannily resembles the past data in both the statistical and the visual sense. This is possible because the purity of retained "phase" information (a term used in sinusoidal wave math to indicate “timing” or position within the price history) from the analysis allows the family of coupled oscillators to operate into the future. The oscillators remember their entire interaction history on multiple time scales simultaneously, and merely continue to behave that way, obeying the math required for a valid analysis of statistical "non-stationary" data where the moments (mean, variance, skew etc.) can vary with time. Stationarity can be violated from tick to tick in financial time series due to the "human element" of market sentiment. In recent years sentiment has become numerically definable and usable in predictive systems. No other spectral analysis technique commercially available retains the essential phase data satisfactorily. Common Fourier Transform methods exhibit substantial corruption of identified phase as well as amplitude and period information (spectral “leakage”), so although they are used routinely their suitability for our purposes is destroyed. If a price chart exhibits more than one occurrence of a particular pattern, like a slow rise to a plateau followed by a steeper drop, the retained phase info enables Spectre to position the pattern in the extrapolation at a statistically meaningful place where the pattern’s spectral components re-align. Just as in Ichimoku (a Japanese currency trader's "One Glance Equilibrium Chart"), what is extrapolated into the future is not merely a price estimate, but a sophisticated estimate of "sentiment", which is really the prime market mover. Sentiment is gleaned from price extremes, volume, implied and historical volatility, and put/call ratio histories, considering both normal and stressed behaviors. These are derived similarly to the way Average True Range is defined, but again on multiple time scales simultaneously. In context, we use the term "price history" to refer to this multiple-source data set. The enhanced capability to do this in Spectre arises from the vastly increased level of complexity employed by the family of coupled fast minus slow oscillators and retained as phase information in the program for positioning. Long-period oscillators form the backbone of trending behavior, but require care because the extrapolation proceeds into a widening cone of uncertainty as required by formal prediction theory. Operationally this translates as meaning that volatility extremes can return seemingly out of the blue, but their return may be foreseeable from an analysis of price histories only if they have occurred before within that history. Similarly, calm periods called regions of “congestion” or “consolidation”, or in Ichimoku "confusion", can develop only if they have been seen before. A pattern such as a breakout developing at the "hard right edge" may simply fail to exhibit statistically rational behavior, and thus not extrapolate as such. In other words, patterns change all the time: trends end when they please, or break down soon after they begin due to the collective (but independent) reactions of seasoned traders to newly-evolving patterns.

operating entirely in the time domain, with a resolution and accuracy far superior to the Fourier Transform. The near-term extrapolation capability now in Spectre is based on an exploitation of non-stationary statistics. Normally, stationary data, where the first few moments (mean, variance, skew) are reasonably constant throughout the data set, allow limited predictions based on that stationarity. When stationarity is often violated, as is the case in almost all financial data, traditional methods for prediction must fail. But by partitioning the data into short segments that are stationary and collecting the changes and rates of change in the moments between segments, an interesting picture emerges. Somewhat like the famed VIX indicator, which is based on open interest in puts, the rise and fall of the moments can be used in a predictive tool for non-stationary data. It isn't elegant, just brute force data collection of the varying statistics, but on encountering the "hard right edge" of the data, these rates of change themselves form patterns. An extension to the original data is made, using the patterns emerging from the varying moments, and a typical FOS analysis is applied to the data plus its new extension. The extension is formed from random numbers, constrained within the observed ranges of the data moments. As an example, if VIX is rising rapidly, all traders know "fear" is becoming evident in the marketplace, and once the fear has peaked, it's a good time to buy back in. Crudely, the prediction models this process, building in a positive bias from the predicted peak in the VIX indicator. The process iterates several times to capture ranges of price predictions within established bands. But of course it isn't that simple. The skew, for instance, gives market data a characteristic tilt: prices tend to fall more rapidly than they rise, looking a bit like a shark's fin or the vertical stabilizer on World War II aircraft. Gradual rise, steep fall. Check out "skewness" at Wikipedia. The skew often disappears when a commodity is in a constricted trading range. When a new trend emerges, the details of the skewness distribution will have changed, and this must be inciorporated in any valid model. Spectre does all of this, and more, using the periodicities from FOS to form a graphical rendering of the moments. You can see "beneath" the data; the graphs are nicely illustrative, and the patterns are visually obvious. I did a lot of work to enable Spectre to pick patterns like these from the original data. Although not anywhere near stationary, those patterns show a remarkable persistence over reasonable lengths of time. It is the patterns I try to exploit, not some supposed stationarity that doesn't last for long in financial data. However, at this time, any attempt to project the data forward more than a few percent of the data set length encounters a breakdown in the patterns unless the patterns (the graphical data set composed of time-varying moments) looks either steady or periodic. In this regard, partitioning at a scale of about 400 points for a data set with 10000 points (on any time scale) works well as a default; experience enables the user to tune the partitioning to roughly match the sizes of characteristic patterns like breakouts from a congested base. The Feature DataBase, which is now loaded with physiology decision apparatus, now contains economic and financial patterns. Summarized averages from the partitions enables tracking volume and open interest as candidates for still more patterns. The goal is to achieve event coupling between enough phenomena to represent the contents of a good traders book or detailed experiences over several years in the markets. Now here is where it could get interesting: if macroeconomic data is examined synchronously with the price of a commodity, this multi-variate data can be used to extract coupling between, as a simple example, declining yields from oil exploration with increased demand from emerging economies. Easy to do yourself with only two or three channels to monitor, but the human ability to correlate cross-channel events for many channels decreases rapidly. Too much complexity. I know I can do the cross-channels event linkages automatically in Spectre for EEG, EKG, blood pressure, etc. from my NIH physiology SBIR grants. I now want to do this for financial data. Spectre is constantly adding new features, and CoDebris also does custom builds for a fee. CoDebris has had several nondisclosure agreements that were valid for one year. They have all expired, and the new features are now largely in the retail version. They largely involved methods for extrapolating the data forward. Over half our customers use this feature with varying reported success, but now Spectre uses greatly enhanced non-stationary data statistics and pattern-matching which is strikingly like that an observer would visually recognize as the data scrolls by. This requires training both the user and the program to become adept at visually recognizing data patterns, some of which are only eye-catching in the spectral domain. But the new Spectre version 7.11 also addresses the issues raised by several avid customers who want to extend their data forward in time... CoDebris 4491 Santa Cruz Avenue San Diego, CA 92107 (619) 211-0974 (858) 663-8817 codebris@yahoo.com ------------------- Foreword --------------------Spectre has been in use by data analysts for over 12 years. As users registered their copy, they were asked what features would be valuable to them in future upgrade, and so Spectre contains many of the features suggested to us. We encourage purchasers of this release to make us aware of their intended application, and how Spectre could be made better to accommodate the surprisingly diverse user community. It seems as though each new user applies Spectre to data analysis in a different manner, inspired by a new mind-set to solve problems they could formerly only wistfully dream of solving. We're amazed and pleased at how often users say they can now confidently tackle analysis projects which formerly daunted them. We feel that the original developer of the underlying mathematics, Dr. Michael J. Korenberg of Queens University in Kingston, Ontario, has created a very powerful and flexible System ID research tool. He has earned our collective gratitude and appreciation. -------------------- New algorithms -------------------- From the Spectre 7.11 documentation, "Blindly extrapolating Fast Orthogonal Search results forward generally produces poor results, as the identified periods are only valid inside the original data. The periods used are orthogonalized by FOS to optimally fit the data, and have no explicit validity beyond the endpoints. Often, wildly divergent behavior is the result as the periods are no longer orthogonalized and rapidly interact. The mathematics of least-squares spectral analysis precludes using all the identified periods to extrapolate the data. But some very reasonable-looking periodicities that make up a simple majority of the “power” in a data set can certainly be extended. The new algorithm synthetically extends the data to the right several points from the data end, adding about two sigma noise riding symmetric, varying amplitude ramps a settable amount above and below the extended EMA to present FOS with a substantial range of variation within the extended region, and bearing for a calculable time the statistical characteristics of the rightmost data segment, as embodied in the FOS features derived from the data." The operant phrase here is "bearing the statistical characteristics". Price data is notoriously very nonstationary (price distribution moments such as mean and variance vary with time), so Spectre includes a "partitioning" capability that saves the essence from each short piece of a long time series. The recurrences of certain statistical patterns, visually obvious on most charts, gives the method some credence, but we disclaim being able to predict the future from past behavior. What we do calculate is a number of probable forward trajectories given the identification of recurring, though not necessarily periodic, patterns and the detailed evolution of their statistical characteristics. Beside the usual measures (moments of price distributions), Spectre extrapolates longer-term measures such as families of “fast – slow” oscillators. These oscillators are true super-cycles which can be conceived as being much more amenable to a meaningful second-order extrapolation than simply time-extending the reconstructed data set composed of FOS periods. Limiting the features identified in the oscillators to the most "energetic", for example above 40 percent of the total mean square error reduction (MSER or energy), prevents short-lived features from dominating the analysis simply because they are riding a strong impulse wave set nearest to the hard right edge. In other words, extreme movements are not viable for very long, and are rapidly overtaken by estimates from the much longer slow trending, consolidation and congestion regions. Such regions provide excellent waveform templates that are often unique to a certain commodity (e.g. seasonals, or strongly supply and demand-driven assets). The analysis is also greatly enhanced if both price AND volume or open interest are available over a "long" time compared to selected "high energy" cycles. A cautionary note: these cycles are not the raw periods, amplitudes and phases precisely identified by Spectre. It's the localized ensemble of these features that form a more complex entity we manipulate as "cycles". A few years ago, at the request of a London silver trader, we applied a Spectre capability originally developed for a 1998 NIH grant (Sleep EEG interpretation) to LFE Silver data, illustrating an under-utilized but potent algorithm, recognition of recurring patterns in data based on their "appearance" or spectral similarity over short sections of data. Spectre scrolls through the data as a "template" clipped from the data by the user is compared in various ways (stretching, energy content, etc.) to the ensemble of cycles found by Spectre in the template. Note that it has an elevated region with characteristic undulations on top, with a ramped upslope and sharp downslope. The actual amplitude or the width of the features identified is not a strong criterion. You may agree that it does a great job, and that those identified regions have an uncanny overall resemblance. These are definite tradingpatterns, with timing info: sell at the trailing edge of the next pattern occurrence (which requires continued re-application of the algorithm on updated prices). There are several of these trading signatures, such as pennants, flags, trend reversals, etc. It is these that are likely to be the most fruitful applications of Spectre in your world. Even the run-up to plateau peaks is a potential long trigger. As you can readily see, even though the only real commonality among these signature waveforms is their overall visual similarity, it is just this similarity that Spectre identifies. The complete spectrum of precisely phased periods is used, not merely the magnitude of a Fourier transform. Although we're not greatly enthused about Elliot Wave, Fibonacci, and friends, Spectre can nontheless identify non-stationary features from impulse waves and the myriad contraction types. Spectre can easily be trained to run on-going analyses as long as the data is compatible with any of several standard and special-party formats. CoDebris just finished refining some long-postponed intricate automated scrolling analysis code written during our 1998-2000 NIH grant. Spectre does multi-variate on-screen analyses, including all the capabilities on the menus. The EEG work used several EEG channels, EKG, CO2 & oxygen levels, etc. But the montage could use any data sources on files, like inflation rates, currency spreads, wheat production, rainfall patterns, etc. A scripting language is available to automate such intricate analyses, try running script files DemoEEG.run and DemoKComplex.run. For the interpreation of sleep EEGs, Spectre captured occurrences of "K-complexes" and "spindles", which are visually distinctive half-second bursts from the cortex which recur every couple of minutes while asleep. When they stop recurring for more than a couple of minutes, you start to waken from deep sleep. Spectre can analyze long data sets by scrolling over short, overlapping "partitions" of the data. The periods, amplitudes and (very important) the phases isolated from each partition are used to "reconstruct" the entire partition. Sort of wavelets. These wavelets are just sine waves optionally tapered to fit the individual partition in which FOS identified their periods, and they are thus localized in both time and frequency, as are synthesized wavelets (Daubechies, Haar, Mathieu). This enables using only the periods that are relevant to a valid extrapolation. The Bartels Cycle Validity Test works perfectly on purely deterministic data, and behaves well even with very noisy data, the noise almost obscuring the known periods (a few pure sines made using Generate Synthetic Data with additive normally distributed noise in Spectre). Armstrong's published Bartels algorithm omitted a key aspect of the test algorithm, see the references in the Spectre menu's Help system. The near-term prediction capability in Spectre is based on an exploitation of non-stationary statistics. Normally, stationary data, where the first few moments (mean, variance, skew) are reasonably constant throughout the data set, allow limited predictions based on that stationarity. When stationarity is often violated, as is the case in almost all financial data, traditional methods for prediction must fail. But by partitioning the data into short segments that are more likely to be stationary and collecting the changes and rates of change in the moments between segments, an interesting picture emerges. Somewhat like the famed VIX indicator, which is based on open interest in puts, the rise and fall of the moments can be used in a predictive tool for non-stationary data. It isn't elegant, just brute force data collection of the varying statistics, but on encountering the "hard right edge" of the data, these rates of change themselves allow formation of extended patterns. First, an extension to the original data is made, using the patterns emerging from the varying moments, and a typical FOS analysis is applied to the data plus its new extension. The extension is formed from random numbers, constrained within the observed ranges of the extended data moments. As an example, if VIX is rising rapidly, traders know "fear" is becoming evident in the marketplace, and once the fear has peaked, it's a good time to buy back in. Crudely, the prediction models this process, building in a positive bias from the predicted peak in the VIX indicator. The process iterates several times to capture ranges of price predictions within established bands. But of course it isn't that simple. The skew, for instance, gives market data a characteristic tilt: prices tend to fall more rapidly than they rise, looking a bit like a shark's fin or the vertical stabilizer on World War II aircraft. Gradual rise, steep fall. Check out "skewness" at Wikipedia. The skew often disappears when a commodity is in a constricted trading range. When a new trend emerges, the details of the skewness distribution will have changed, and this must be incorporated in any valid model. Spectre does all of this, and more, using the periodicities from FOS to form a graphical rendering of the moments. You can see "beneath" the data; the graphs are nicely illustrative, and the patterns are visually obvious. Although not anywhere near stationary, those pattern “events” show a remarkable persistence over reasonable lengths of time. It is the patterns Spectre exploits, not some supposed stationarity that doesn't last for long in financial data. However, at this time, any attempt to project the data forward more than a few percent of the data set length encounters a breakdown in the patterns unless the patterns (the graphical data sets composed of time-varying moments) looks either steady or periodic. In this regard, partitioning at a scale of about 400 points for a data set with 10000 points (on any time scale) works well as a default; experience enables the user to tune the partitioning to roughly match the sizes of characteristic patterns like breakouts from a congested base. The Feature DataBase, which is now loaded with physiology decision-making apparatus, now contains economic and financial patterns. Summarized averages from the partitions enables tracking entities like volume and open interest as candidates for still more patterns. The goal is to achieve event coupling between enough phenomena to represent the contents of a good fundamental or technical analysis book or detailed trader’s experiences over several years in the markets. ---------------- Spectre applications ----------------Spectre was purchased by a Canadian paper mill to early-detect an incipient paper tear on the production line from line-scan camera time series. A NASA Wallops Island rocket motor engineer bought Spectre to early-detect vibration modes growing too near structure resonances, typically coupled at the engine mounts. In both cases, simply extrapolating periods into the future doesn't work at all. But Spectre's Waveform ID means that when you know an event occurred, how far back in the data can you detect the first reliable warnings of an event? And in both cases the prediction is relevant for only a short time, on the order of one or two of the primary (highest MSER) periods employed. The paper mill had been experiencing rips with no time to adjust paper tensioning or slow the feed. Spectre gives them several seconds; a series of characteristic ripples can be seen quite a distance upstream, building toward resonance. The "appearance" of the advancing wavefront is a characteristic precursor, and Spectre can recognize events from their FOS phased periods even when the waves are very small and in weak camera contrast, below what an observer could see. An anthropologist used Spectre to make configurable 3-D charts of rainfall patterns over Mexico from tree ring and hydrologic data spanning twenty centuries. A tidal researcher in Washington examines the effect of salt water intrusions on groundwater tidal motions. Spectre is also used by investors seeking to identify trading patterns assuming cyclical or quasi-cyclical behavior, by vibration control engineers, by animal sound researchers and neurophysiologists, and even by a gentleman in Australia who uses it to help handicap greyhound races. ---------- Spectre User Manual excerpts ----------Overview...Researchers, investment analysts and engineers often want to know if their data contain significant amounts of signal at frequencies of interest to them. These frequencies may correspond to driving forces, cyclical market forces, environmental constraints or system responses, and may result from intrinsically non-linear processes. Often, theoretical or phenomenological models exist to explain some observed behavior, and experimental data is collected to verify whether or not the model is correct. System Identification is the name given to attempts to determine the underlying realities which are sampled in a data set. Similarly, a known but transient waveform may exist in a long data stream: a brief but possibly recurring signal. Your requirement may be to isolate all occurrences of the waveform in time. Examples include characteristic trading patterns or heart-wave EKG signature fluctuations. To some extent a waveform may be partially degraded or obscured by another signal, either deterministic or noise. Spectre is a unique data analysis tool which determines precisely how an ensemble of sine waves make up a data set or time series: periods, amplitudes, phases, and data set "energy" in each sine wave. Simply specify a table of periods (cycle lengths) you believe are present in your data, or let Spectre's AutoGen feature generate a table of relevant periods for you from a quick spectral/temporal estimate. Spectre tests these periods for their optimal contribution to the data set and graphically assembles the selected, precisely phased sine waves into a revealing, evolving portrait: an educational visualization of your data you can't get from a Fast Fourier Transform, even with specialized training and a lot of demanding effort. A plot of the original data and its reconstruction can be printed, and the reconstruction can be saved as a data set on the same scale as the original data. The reconstruction can even be extrapolated into the "future". If you want to get to work right away, Spectre is highly automated. And if you don't have a solid math background, Spectre will be an educational experience. Use the Tools/ Generate Data Set option to learn about sine waves, square waves, sawteeth waves, noise, trends, offsets, etc. in a graphical environment. Spectre does all the work, you just do the thinking to set up new "experiments" and analyze "why". There are many Tools/Pre-Process options to alter and combine the "synthetic" data sets you make with real data of interest to you. You rapidly learn to appreciate what the data is telling you, in a manner no other analysis tool can supply. As a demonstration, on first running Spectre choose from the menu Analyze/Frequency Search. Select the supplied table of candidate periods BREATH.TBL and data file BREATH.DAT, and accept the defaults in the Search Parameters Dialog. This operation is also encapsulated as File/Run Yoga Breath Demo. BREATH.DAT is a physiological data set. The synthesized curve is not a bad approximation to the data, as you can see while the reconstruction builds before your eyes, even though only about half of the "energy" in the data is accounted for by the identified periods, amplitudes and phases, analyzed in well under 1 minute. BREATH.TBL was edited by a researcher using Spectre's built-in editor to contain several periods of interest in his work. Four of those periods are the dominant ones identified. Refining the BREATH.TBL file might find even better periods to use, but the periods supplied are pretty good, or they wouldn't be used in this convincing demonstration. Compare the results using this hand-made table with the AutoGen feature, which generates the period table automatically from a statistical analysis of the data. Frequency Search is a real learning experience to watch in action, which Spectre lets you do. To freeze the action, hit the Alt key to pause the search. Windows halts all action until you hit the escape key (Esc). Also, use Tools/Copy Data Set to capture the analysis at any point into a new Graph Window, including the intermediate Reconstruction. The most important visualization experience while running Spectre is watching the highest-energy, long period sines define major trends in the data and seeing the short-period sines fill in the peaks and valleys. Viewing this unique perpective for the first time yields a very intuitive undertanding of what the data contains. Spectre will line-plot the actual data in bright green, and the evolving "reconstruction" from the selected periods as a yellow dotted line. Plots are auto-rescaled on window resizing, actually recomputed every time the window is painted to optimize the displayed data information content. Each time a new period is selected as containing the next most significant amount of energy, the yellow curve is redrawn to include the new information. You'll get a good sense for periods to include in the .TBL file, and which to exclude, by watching the analysis and then examining the .OUT file that pops up on the screen after each analysis is complete. In fact, watching the earlier, higher-energy sinusoids define the major features of your data is an insightful experience; one researcher termed it a "spectacular visualization" of his data. The side slopes of the major sine curves will lie along major trends in the data, although the peaks won't necessarily correspond to peaks in the data. That is left to shorter period components, which when properly phased with the long period components will ride up and down the peaks in your data with increasing fidelity. The usual, tedious frequency analysis approach is to pre-process your data, apply a Fast Fourier Transform to the series, and plot the spectrum magnitude. Peaks in the FFT spectrum may correspond to interesting frequencies. However, it is difficult for anyone but a signal processing expert to know how much energy (or "activity") in the time series is actually accounted for by a given frequency. It is even harder to resolve nearby, overlapping broad peaks. More often than not, spectral leakage or noise distorts the spectrum and cannot readily be de-coupled from signals of interest. Pre-processing (filtering, tapering and windowing) of data sets is a very demanding discipline, and many of the rules to assure the validity of pre-processing operations are difficult to apply, as such operations actually modify the characteristics of the manipulated data. Further, most researchers with a need for waveform analyses do not have formal training in the subject, and many feel uncomfortable with having to use a host of implicit assumptions. There is an alternative, nearly painless method available to perform such frequency analyses. The researcher first prepares (as a text .TBL file) a table of candidate periods, or Spectre will construct (AutoGen) such a table for you. On initial creation, the table usually contains a fairly large number of entries, as there may be no prior knowledge of what is really important to you in the data. Periods may be longer than the time series, or as short as twice the time interval between points. But the researcher often knows what to look for based on theory or existing work, and the table will contain several periods in the regions of interest. Spectre searches your data using a table of candidate periods in the .TBL file which you chose, selecting those periods which account for the most "energy" or "activity". Spectre is not just a curve-fitter, nor is it simply an FFT. To use Spectre productively, you should know something about your data, but you need nearly no data analysis background. Spectre uses an adaptation, called Fast Orthogonal Search (FOS), of the Orthogonal Search Method developed by Michael J. Korenberg at Queens University in Ontario during the late 1980's (see References). The algorithm is applied to your data set using an associated table of candidate periods. The precise energy, amplitude, and phase of sine waves corresponding to each selected entry in the table are displayed as soon as they are computed. The objective is to determine if frequencies of interest to the researcher are present in significant measure, and report the results both graphically and in a tabular format. The algorithm analyzes a time series stepwise, determining the ability of each period to explain a significant portion of the total Mean Square Error Reduction, or MSER: roughly, the data set's "energy" content in each slected period. It then orthogonally removes the sinusoid explaining the largest percentage of the variance between the data and the sinusoid. This process is repeated on the residuals until there is no further significant error reduction or until a specified number of periods have been identified. The algorithm is capable of much greater time resolution than a Fourier transform, and is not limited to harmonics (multiples) of a fundamental frequency. It is also quite insensitive to noise, as all data elements are used only in series-wide averages over the orthogonal basis functions (sine waves, in our case). Finally, it tolerates missing data points, irregularly-spaced data sets, and short data segments. In many nonlinear or biological systems, the signal frequencies change, or breathe, as the system evolves, so short segments are necessary for system identification. Spectre is a Multiple Document Interface application, so it will display the results from multiple Frequency Searches on-screen at the same time, including the reconstructions (dot plots on top of data line graphs) and text output files. An FFT is included, also used internally for the AutoGen feature to automatically generate a period table (AUTOGEN.TBL), partly from the FFT's crude energy distribution. Time Series ConceptsA time series is a succession of data values at stated intervals. Typically, a time series consists of pairs of numbers: one member of each pair is the value, the other member is the time at which the value was measured. But the values can be from just about any source, and the other member need not be time (for example, in a trading environment you can plot price vs. volume). A time series may be created by an instrument recording sound, heartbeats, light intensity, price fluctuations -- any phenomena at meausred intervals of time, of position, or any event at all which can be recorded. The expectation is that the recording will contain information to assist understanding or controlling the phenomenon. Time series analysis is a collection of techniques to extract that information. The intervals do not have to be regular: a data value may be recorded only on the occurence of a heartbeat, which may be irregular, or even skipped. Many devices log, or record, data in this way. In Spectre, the menu option Tools/Pre-Process/Interpolate may be used to regularize a data set. A data set is made of pairs of X, Y values -- so-called ordered pairs, as the X value is always stated first (X is usually time). The relationship is usually expressed Y = Y (X) and the parantheses here imply Y is to be considered a function of X, to depend on X. X is usually considered the independent variable, Y the dependent variable: each value in a succession of recorded events depends on the time it was recorded. The set of X,Y pairs which comprise a data set is, in graph parlance, referred to the Abscissa and the Ordinate of a graph: the abscissa X values run out along the horizontal axis, and each corresponding ordinate Y value is plotted above its X, at a height proportional to the Y value. In Spectre, the graph is the fundamental element, not the individual X,Y pairs, or even the X or Y values themselves: a data set exists as a collection, and the Graph Window is the simplest, most complete visualization of the data. The main menu option Tools/Generate Synthetic Data allows a researcher to emulate real data using superposed pure sines, square waves, sawtooth waves, squirt waves, a linear trend (constant sloping line), a constant (often called DC, for Direct Current) offset, and additive noise. The researcher first specifies a name for the contrived test data set. The number of data points, temporal resolution (spacing), noise level, trend, the type of wave, and the periods, amplitudes and phases of the waves are then entered. A new Graph Window then appears on screen. Association and VisualizationSpectre plots a data set in green, and the synthesized data (the Reconstruction from a Frequency Search) for the selected periods as a yellow dotted line. Plots are auto-rescaled on window resizing, actually recomputed every time the window is painted to optimize the displayed data information content. Each time a new period is selected as containing the next most significant amount of energy, the yellow curve is redrawn. You can get a good sense for periods to include in the .TBL file, and which to exclude, by watching the analysis and then examining the .OUT file which pops up on the screen after each analysis is complete. In fact, watching the earlier, higher-energy sinusoids define the major features of your data is an insightful experience. The side slopes of the sine curve will lie along major trends in the data, and the peaks won't necessarily correspond to peaks in the data. That is left to shorter period components, which when properly phased with the long period components will ride up and down peaks in the data with increasing fidelity as more sinusoids are identified. Select Data Set as AbscissaSelect Data Set as OrdinateThese features enable the user to associate two data sets, using one as the abscissa, or independent variable, and the other as the ordinate, or dependent variable. Click the System Menu in the upper left corner of each data set graph window, click again on "Select Data Set As Abscissa" or "Select Data Set As Ordinate" (selections are checkmarked on the menus). As soon as a pair has been selected, a new window will be opened showing the relationship. In general, a positive (upward) sloping elliptical blob of green line segments implies a positive correlatiuon between the two data sets. A downward-sloping trend indicates anti-correlated sets. A more or less uniform disk indicates a lack of correlation. Note when running a Fast Fourier Transform on a data set that the spectral magnitude is symmetric (and the spectral phase is anti-symmetric) about zero frequency for real data sets. An asymmetric spectral magnitude would indicate a non-zero imaginary component to the data. You can see this for yorself, and maybe get some ideas how to "package" your data sets, if you try the Graph Window options Select Data Set as Real Part and Select Data Set as Imaginary Part available on the System Menus for two data sets that seem to have a lot in common, though they may not be from identical sources. See also Lissajous Figures. Try Tools/Combine Graphs to subtract or divide two data sets. The resulting graph will emphasize, literally, the differences betwen the data sets. Also, play with the Tools/Pre-Process Data Set options. These can "bend", shift, scale your data. Use the Clip tool to look at a small region of the data (hold and drag the mouse (double-click and hold down, takes practice!) over the region first to define a "rubber-band" rectangle around the region before selecting Clip). The region will appear in a new graph, magnified. At the most elementary level, you don't need to scan columns of numbers to see very subtle differences between two data sets. By juxtaposing two data sets in some way (not necessarily just on the screen or in print), relationships between data sets from very different sources may be seen which would not likely be discovered through conventional analyses. Sometimes, simply tiling the data set's graphs appropriately is sufficient: use Window/Tile Vertically to look at differences in peak and trough values between two graphs, or Window/Tile Horizontally to look at differences between two graphs in the locations of X-axis crossings (zero crossings) and the X-coordinate (abscissa) values at peaks and valleys. Lissajous FiguresChoose a sine wave graph as abscissa, and another, different sine wave graph as ordinate from the two graph windows. Click the upper left corner (System Menu) of one window using Select Data Set as Abscissa and the other using Select Data Set as Ordinate. A new graph is created, where instead of the regular, monotonically increasing X values, the abscissa now is the Y value from the second graph. Striking swirling patterns called Lissajous Figures appear in the new window. They represent the interaction between two quantities which are not usually combined in this way. Even more striking patterns are created by making a Fourier Transform from the new graph. Program EnvironmentDue to the modular, low-level nature of each processing function, Spectre is an in-depth tool-kit for educational purposes as well as serious work . It is not a trivial program to learn to use productively, but the feedback is immediate and visual. Spectre was written using the Windows Multiple Document Interface to allow any number of graph and text windows to appear on-screen at once, either overlapped or tiled, up to memory limits. Functions may be applied to Synthesize (Tools) or Analyze a data set or time series, and the results will be left on-screen as one or more new Graph Windows for assessment. Original data is never modified unless you expressly save a Graph Window by over-writing the original data; see Edit/Rename Data Set. The intermediate data set graphs can themselves be operated on and displayed simultaneously on-screen, allowing a degree of controllability not available in programs which allow only one graph on-screen at a time. Any graph may be saved to disk with a couple of mouse clicks or keystrokes, as an internal naming system keeps track of the operation which generated each intermediate graph. Graphs can be iconized, tiled, or stretched to fill the entire screen. Machine EnvironmentInstalling Spectre does not modify ANY of your system files, and de-installing Spectre is accomplished by simply deleting the Spectre Program Group and the files, which are usually found in C:\Spectre. CAUTION: Disable any Screen Savers (Program Manager/Control Panel/Desktop) before running a Frequency Search on a long data set. Screen Savers suck up large amounts of CPU time. To use Spectre productively, your hardware environment should be at least a 386 running at 33 MHz, with 8 MB of 70 - 80 ns DRAM, a hard drive with better than 20 - 25 ms seek time, and a Windows-compatible mouse. A math co-processor (Intel 80x87 or equivalent) is required: although Spectre will run without one, it would be too slow to be useful on a large data set. Processing speeds are noted (subjectively) where relevant. A Pentium 100 is at least twelve times faster than a 386/33. Windows 3.1 or better is required. Any DOS version 3.5 or later should work. Time series data processing is intrinsically memory-intensive, so having at least 8 megabytes of fast DRAM installed is not a luxury. Spectre video display utilizes the standard Windows video drivers as well as any specialized drivers which are supplied with video/graphics adapters, as long as they were registered with Windows Setup. Even EGA/VGA 16 color video modes are supported. Spectre has been run successfully under Windows 95. Each (X,Y) data point requires 16 bytes from the global heap (2 double-precision floating point numbers) . Another 16 bytes per point, plus change, for a rough total of 30,000 points per MByte, is used for working arrays ONLY during the Analyze/Frequency Search and Waveform Identification. All working arrays are returned to the system (global heap) after each analysis. Stack usage is minimized by block declarations and dynamic allocations on the global heap for most objects. Each graph window stores its own own auxiliary data, such as the data set mean, the extrema, the RMS value, etc. to save recomputing each time a data set statistic is wanted. Spectre began as a DOS program; the main reason for porting to Windows was to get an 84,000 point datafile analyzed for a physiological researcher. Spectre will analyze a 200,000 point data set on an 8 Mbyte system, over half a million points in 16 Mbytes, without swapping to disk for virtual memory. Use Partition Large Data Set to analyze truly huge files with millions of points. Spectre closes all files immediately after each access, but be aware that Windows uses many files internally. Typically, you will want to insert statements into CONFIG.SYS to set the FILES= and BUFFERS= limits to between 30 and 40. Help For Spectre Users Extensive help is available on all features from the menu selection Help/Contents, or hit Shift + F1. For specific menu selections, the ordinary context-sensitive help accesses Help topics directly for any menu option. Hit the Alt key to enter the menu system and hit Shift+F1 (together) to evoke a Help Cursor; use the arrow keys or drag the mouse over the menus to highlight the relevant option. This loads the full WinHelp engine and the Spectre.HLP helpfile, opened to the section selected by the special Help Cursor. Although this is the mechanism preferred by Microsoft to display context-sensitive help, it's a bit intrusive, as the entire screen is taken over. If the Help Cursor is no longer needed, hit the Esc key to restore normal cursor functioning. From within Help, the Search option allows the user to scroll through all Spectre help entries. Finally, some dialog boxes are supported by a Help button which either displays a text window of help information or opens the WinHelp system to the relevant topic. If unusual program behavior occurs, perhaps accompanied by speaker "chirps", Spectre will activate a text diagnostics file, DIAG.TXT. Registered purchasers of Spectre can call CoDebris (619-221-0974 or 858-663-8817, 9am - 7pm Pacific Time) with questions or leave E-mail at codebris@yahoo.com with any questions. We can probably steer you to the right selection or sequence of selections, or make arrangements for new features. As there is often more than one way to do something, discussions about time series processing are usually illuminating to both parties. CoDebris will also recommend one of the other available commercial packages if it becomes clear you need to spend that kind of money (typically over a thousand dollars). We gain insight into what people really need this way, so new features and interfaces can be added to future versions. CoDebris is particularly interested in supporting more data acquisition devices: storage scopes, A/D converters, digital I/O boards, sound boards, etc. Let us know what you're using and if there is sufficient interest we will contact the device manufacturer for interface specifications. Contact CoDebris for information on making user specifications for a custom version upgrade: special purpose filters, proprietary data formats, new features. Source code for portions of Spectre is available, and many parts of Spectre can be exported as an object library (LIB) or Dynamic Link Library (DLL). For a negotiable fee, any desired feature or customization tailored to your requirements can be incorporated, either in the retail or a private version. If the feature is of general utility, the fee may be small or even waived completely. We are very interested in enlarging the variety of data set formats Spectre can recognize: let us know if you want Spectre to read your spreadsheet or data base files or accept data from acquisition hardware: storage scopes, sound boards, DSP boards, A/D or digital I/O boards. If you run Spectre and like it, let us know. If you run it and don't like something about it, let us know that too. Spectre is becoming a fairly comprehensive waveform analysis and synthesis package, and most features are added in response to user suggestions: in the works are more powerful digital filters, multi-variate data sets and non-linear dynamics tools. Users are highly encouraged to help prioritize additions, and to add their own requirements. Future upgrades to Spectre will add these features: Multi-variate data sets in a multi-graph window (e.g. simultaneous heart rate, EEG, and cardiovascular output vs. time, or simultaneous price and volume in different markets vs. time). Stacked "3-D" FFT plots from a partitioned large data file. Full-resolution (not scaled bitmap) printed graphs, color optional, with settable titles, legends, and captioning. Expanded capabilities for Waveform Identification: multiple active templates, adaptive incorporation of user acceptance or veto criteria in succeeding analyses. Frequency Search IntroductionThis is a new and greatly improved version of the Spectre Frequency Search and Data Synthesis program. Install Spectre in a convenient Program Group under Windows and click the icon to run, or select File/Run under Program Manager and fill in C:\Spectre. For a demonstration run, select File/Run Demo..., or choose Analyze/Frequency Search and select a (supplied) ASCII table of candidate periods (TBL) and a supplied data file (DAT). The TBL file can be edited and saved in Spectre. A time-stamped output file (OUT) contains summary information including the selected periods, their computed amplitudes and precise phases. The OUT file can be loaded into Spectre for viewing, editing or printing. Take a look at WHEAT88.DAT, a commodity price dataset. It was Frequency Searched with the AutoGen option, letting Spectre automatically decide which periods are likely to be significant using a crude FFT and a Period/Amplitude estimate. The Reconstruction (dotted yellow) curve is an excellent approximation to the data, with about 90% of the "energy" in the data accounted for by nine identified periods, amplitudes and phases, analyzed in less than a minute. The remaining six periods, all less than 1% of the data, fill in more details. Although the bulk of the data set energy is long-period contributions, the low-energy features are revealing from a technical analysis viewpoint: mixed short- and long-term cyclical behavior (roughly: month, quarter, semi-annual and annual), where several periodic factors converge every so often to yield a significant change even though none of the cycles is individually very strong. Way to subtle for a Fourier Transform. Frequency Search is a real learning experience to watch in action, which Spectre lets you do: to freeze the action, hit the Alt key to pause the search. Windows halts all action until you hit the escape key (Esc). Or, use Tools/Copy Data Set to capture the analysis at any point into a new Graph Window, including the intermediate Reconstruction. True analysis power. Full text editing for files smaller than 32kB is built into Spectre. Note that the actual data files are not meant to be directly editable in Spectre, but they may be altered in the pre-processor (menu selection Tools/Pre-Process Data Set). This allows outlier (artifact) removal, partioning, segmentation, decimation, scaling, filtering, etc. to be applied to named data sets. ReferencesKorenberg, M. J. "Fast orthogonal identification of nonlinear difference equation and functional expansion models" Proc. 30th Midwest Symp. Cir. Sys. 1:270-276 8/87. Korenberg, M. J. "A robust orthogonal algorithm for system identification and time-series analysis" Biol. Cybernetics, 60:267-276 1989. Korenberg, M. J. "Identifying nonlinear difference equation and functional expansion repesentations: The fast orthogonal algorithm" Ann. Biomed. Eng. 16:123-142 1988. Korenberg, M. J.; Bruder, S. B.; McIlroy, P. H. "Exact orthogonal kernel estimation from finite data records: extending Wiener's identification of nonlinear systems" Ann. Biomed. Eng., 1988. Korenberg, M. J. "Orthogonal identification of nonlinear difference equation models" Proc. 28th Midwest Symp. Cir. Sys. 1:90-95 8/85. Ljung, L. "System Identification: Theory for the User", Prentice-Hall, Englewood Cliffs, NJ 1987. Pincus, Steven M. "Approximating Markov Chains", Proc. Natl. Acad. Sci. USA, 89:1-5, 5/92. Uchida, Sunao et al "Computerization of Fujimori's method of Waveform Recognition. A review and methodological considerations for its application to all-night sleep EEG", J. Neuroscience Methods, 64:1-12, 1996. Zawadzki, Eugene M. "Identification and Extraction of Waveform Signatures From Electro-Encephelogram Data", CoDebris 95-2, 8/28/95 2. Program NavigationFinding a comfortable set of methods to use with complex software is usually a trial-and-error process, as documentation is usually specific to a given task and doesn't offer a section which looks at the software from the user's viewpoint. Here is Spectre from the keyboard, mouse button and cursor point of view: Graph WindowA Graph Window contains a plot of the X,Y values from a data set as a bright green line plot on a black background. If a Frequency Search was performed on the Graph Window, the superposed, reconstructed sine waves identified during the search are plotted as a yellow dotted curve on the same graph. If the data file was loaded as "Scrolling", clicking on the left or right side of the graph "moves" the data within the window, loading only enough data into RAM to fill the window. Abscissa and OrdinateAbscissa: the set of X values of a Graph Window. The horizontal "time" axis of a time series. Every X value has a corresponding Y value in a data set. Ordinate: the set of Y values of a Graph Window. The vertical axis of a time series. Every Y value corresponds to a definite X value in a data set. Typically, Y is the "event" and X is the time the event occured. Scrolling Graph WindowA Graph Window which can be scrolled, for data files too large for physical memory (RAM). This is done to avoid using Virtual Memory (swapping to the hard disk) with the associated slowing of routine operations... a factor of up to 100,000 slower! Scrolling can also be used to provide a detailed view of data files too large to be viewed in the few hundred pixels across a screen. Click the left mouse button on the right or left half of the Graph Window to move data forward or back in the window. The keyboard right/left arrow keys also scroll data, and may be held down for nearly-continuous scrolling. All Spectre operations on the data will occur only on the section currently in the window. To operate on an entire large data file, use the Analyze/Partition Large Data Set options. Text WindowA Text Window contains editable text. The text can be altered and saved to disk. If the text is a Period Table (.TBL file), then any edits made to the entries MUST be saved to disk before a Frequency Search is performed using the modified Period Table, as candidate periods are always loaded from file. The same caveat applies to edited script (INI) files. Running SpectreAs with most new programs, use the defaults first time through and play with the options as experience accumulates. Using Spectre can be an insightful and educational experience, giving you great control over some standard time series processing elements, and introducing a few new ones. After loading a data set (File/Open Data File, or creating a synthetic data set with Tools/Generate Synthetic Data), exercise all options which seem even remotely applicable to your line of inquiry, and discard unsatisfactory intermediate graphs. Time series processing is often accompanied by surprises which yield real insight into your data, and is a whole lot more fun than word processing. Save interesting results to file often for comparison with later results. You may want to create an empty subdirectory to ease cleanup and discarding of intermediate results which have been superseded by improved versions. If you don't want to use the built-in file naming convention, choose unique filenames which will allow easy cleanup: AB1.DAT, AB2.DAT, AB2.SYN, etc. are backed up or deleted with a single command COPY AB*.* \SAVE or DEL AB*.*. Use the context-sensitive help (Alt Shift F1) for menu options often while in the menu system. Dialogs requiring user input also have context-sensitive Help buttons. Diagnostic SoundsSounds from your system speaker during a Spectre session are usually intended to tell one of two things: either the system could not satisfy a request of some sort from you or the program, or something needs to be done by you and you're not doing it. The occurrence of a "beep" from the speaker often indicates an action needs to be accomplished before you can move on. Maybe a dialog box or message window needs its "OK" or "Cancel" button pressed. Usually, hitting the keyboard "Enter" key will suffice, as the default option is the one you'd normally take, and it is highlighted with a small, dotted rectangle around the text printed on the key. A sound like a "chirp", with some variation in the tone and duration, is used by Spectre to indicate a degradation in performance. Either too many system resources are in use, or requests for memory are failing. The solution is commonly to shut down any other applications running in background. If things have sunk low enough (windows are not being painted, are broken up or do not appear when expected), save your work, exit Spectre, close down everything else, exit and re-start Windows itself. A diagnostic and log file SpectrreLog.txt is produced whenever a chirp occurs. The file also contains a log of all significant operations run during the session. SpectrreLog.txt is over-written each time Spectre is started. Monitor the status of your system memory, hard drive, and resources occasionally from the Help/About Spectre... menu selection, especially if operations are getting to be slow and uneven. Scripted OperationsTo automate a series of operations, a capability to run scripted commands exists in Spectre. An operation is typically of the form "command, datafile, parameters". A sample INI file is provided with Spectre: ; SPECTRE.RUN ; Sample editable configuration file 1/20/2008 ; Copyright (c) 2008 CoDebris, all rights reserved. ; Backup your configuration file before editing it, and ; save your changes using File/Save or Save As with a ; personalized file name. ; Free-form comments to end of line after a ";" are for your ; use, the ";" may be anywhere. Comments, blank lines, ; tabs, spaces are ignored. Use comments liberally to help ; recall what your entries mean. ; Section header names (tokens) are between brackets [...], ; missing sections use defaults. Data after a section header ; is for that section only. ; Commands are of the form "command, datafile, parameter, ..." ; (note commas) All datafiles opened are saved to [datapath] with ; a Spectre header in binary 32-bit floating point Y-only or X,Y ; data, enabling them to be auto-loaded for scripted commands. [commands] ; Executed in listed sequence longsearch, kW.dat, autogen ; Accumulate detailed partition stats ; pattern-matching identifies ; characteristic waveforms ; embedded in long time serieswaveform, silver89.dat, SilverTemplate.dat, autogen extrapolate, kw.dat, 0.1 ; uptrend confirmed [datapath]; Data files directory, usually whole path c:\freq; from root (first "\"). Can be a ; subdirectory. "c:" may be any drive. ; Trailing "\" optional. ; Default is "." (current dir) [templatepath]; Template files directory (waveform c:\freq; identification samples) Keyboard Mouse EmulationWith no mouse, most processing is difficult, but possible. The keypad arrow keys will move the cursor over a graph. For mouse emulation, make sure the Num Lock key is not active (unlit). The keypad will also permit precise one-pixel-at-a-time position adjustments over a Graph Window when mouse movement is awkward or mouse response is "sticky". The rate of movement increases as an arrow key is held down. The arrow keys can always be used in conjunction with the mouse for greater positional control. By the way, all menu responses can be invoked from the keyboard as an "Alt key sequence": Window/Tile Horizontally is selected with Alt W H. The mnemonic letters are not always initials on pop-up menus with several options, as naming conflicts often result. The relevant key code is underlined on each menu entry. Several Menu selections also have dedicated "virtual keystrokes". Ctrl X, for example, is usually used to Edit/Cut highlighted text from a text window. Graph Cursor A special cursor appears automatically over a Graph Window as the mouse is moved around. It is a cross-hair type, but with an open region in the center to allow precise alignment over any pixel. As long as the mouse left button or a keypad cursor key is depressed, the cursor cannot leave the plot surface, allowing you to get into corners or right up to plot edges without triggering the system move/size cursors. Dragging the mouse draws a "rubber-band" over the graph. The left, right, upper and lower edges are used to select a clipping region for Tools/Pre-Process/Clip Outside Rubber-Band. Clipping Rubber-BandDrag the mouse (double-click and hold down; takes some practice) to draw a rectangular "rubber-band" over part of a graph. The rubber-band edges are used to select a clipping region (Tools/Pre-Process/Clip). Regions left of, right of, above or below the clipping rectangle are excluded. A very useful tool for making a new data set from a piece of an existing Graph Window, either to work on or simply for a "zoom lens" close look. Graph Coordinates DisplaySelect Window/Display Graph Coordinates. Move the mouse over any Graph Window. The x,y coordinates, data set index, and corresponding pixel position can be noted. The keyboard cursor keypad can be used to move in precise increments across the graph. Here is a good place to note that if there are fewer pixels available across the screen than the number of data points, all of then won't get painted (displayed). If you want to see point-to-point variations in your data , try Tools/Pre-Process/Clip to zoom in on a small section of the graph and expand it into a larger Graph Window, as discussed above. Right Mouse ButtonThe right mouse button draws a vertical line over the active graph window at the cursor location. Click at that X value again to remove the bar (use the keypad arrows to align cursor). In all other situations, the right mouse button either has no effect, or produces the same response as the left button. Activating Graph WindowsWhen more than one Graph Window is open on-screen, clicking anywhere over the window surface is sufficient to activate it. Clicking over the Title Bar (also called the Caption Bar) and dragging the mouse is used to move a window. Try Ctrl-Tab to activate graph windows in succession to "thumb through" a screenful of data sets. It can bring up graph windows from the bottom of the "z-order" window stack, and "sinks" the formerly active window to the bottom. See Window Keys. Graph Window SizingYou can, of course, drag the border edges or corners to resize a Graph Window. Spectre will completely recalculate the plot to optimize the information displayed. Keep in mind that a large data set may have many more (or quite a few less) points than there are horizontal pixels on the screen: you can't expect to see every datum unless you slice up some data sets. The Tools/Preprocess/Interpolate, Segment or Clip options can help here. The keyboard can also be used to move or re-size graph windows. See Help/Keyboard for more information on using the sometimes much more effective keyboard in Windows applications. 3. File MenuFile/Open Data FileOpen a data file into a new Graph Window, where it appears as a bright green plot over black background. The data set in any active (highlighted title bar) Graph Window is directly available for any processing option in Spectre. After selecting the data file name, if the data file does not have a Spectre header attached, you are presented with a Data Type Dialog, whre you are asked to specify the data format. Spectre accepts both ASCII text and binary formats. Binary data files are assumed to be Y-only values (ordinates only), but you can select "X,Y Pairs" to over-ride this default if you know your binary data file layout contains data in the abscissa followed by ordinate order. For ASCII data, Spectre auto-detects whether "X,Y Pairs" (2 coulumns) or "Y-Only" (1 column) data is present. If the file already has a Spectre-style header, the Data Type Dialog is skipped, as all the info necessary to load the file is in the header. Very large files (too large for physical memory) can be opened into a Scrolling Graph Window. Enter the number of points to show on-screen in each scroll. Only binary data files may be scrolled. For scrolling, convert long ASCII data files to binary by loading them non-scrolling and saving as binary. Data may be scrolled forward or back by clicking on the Graph Window right or left of center, or by using the right/left arrow keys on the keyboard.. All Spectre operations occur on the data currently in the scrolling window. You select the scroll size, in terms of data points displayed on-screen. Also select the calibration options: Auto-Calibrate (default) means the data is always displayed so it fills the Graph window. If this option is de-selected then you can enter the lower and upper calibration limits. If for example -32768 to 32767 (the range of a 16-bit integer) is chosen, then as the data is scrolled it will maintain a uniform appearance, and only limit-up or limit-down excursions will touch the Graph Window edges. Otherwise, every time you scrolled the data set it would change its vertical scaling to fill the window. Any data outside the calibration range appears clipped, but the actual data is not modified. ASCII data files are not intended to be directly editable as text, but a minor subterfuge will allow you to bring in at least a piece of one for viewing as a Text Window. Just open the data file using the File Filter list box option "All Files (*.*)"in the File/Open Output File... dialog and click on the file name or enter the data file name directly in the file name edit box. If the file is larger than 32kB, only a portion will fit in the text buffer, so don't edit it and don't try to save it. Spectre accepts a few proprietary binary data file formats (EEGs, some instrumentation) in this version. Users with requirements for these or other special formats should contact CoDebris for assistance. Data Type DialogOn opening a data file, you are asked to specify the data type (ASCII text, binary, or proprietary) and the data layout (whether the data is arranged in two columns as pairs of X Y numbers, or as a single column of Y data values). The data may be opened into a normal Graph Window or (for binary data files only) into a Scrolling Graph Window. You need not specify the data layout for ASCII (plain-text) data files, Spectre figures it out based on the number of columns of data. Binary data files consist of 8-, 16-, or 32-bit integers, or 32- bit floating point numbers in the universal IEEE format. Binary data is usually produced as an ordinate-only (Y-only) file. It can be difficult, if you don't know the data type, to choose the correct option; Spectre tries to determine the correct format on loading the file and offers an option to go ahead or try another format if an anomalous data layout is found. The most common binary format is as 16-bit integers, or occasionally as 32-bit floating point. For binary files, the data is assumed to be arranged in rows of X Y pairs, or as ordinate (Y) data only. In the first format, Spectre assumes the X value (the abscissa) precedes the Y value (the ordinate) on each row. Binary data files contain NO delimiters, just a contnuous array of values (XYXYXY... or YYY...). The Y-only format consists of a single number, the Y value, in each row. Many research and engineering analysis instruments produce data files in this format; you are expected to know that the X values are uniformly spaced, and what values to use for the initial X value and the interval between points. If you aren't sure which values to use, use the defaults (initial X = 0.0, interval = 1.0) and proceed. In most cases, the X values determine what numbers to choose as candidate periods in the Period Table (see menu option Analyze/Frequency Search) when performing a Frequency Search. The important consideration is to be internally consistent in your choice. Here again, using the AutoGen feature will greatly help. Knowing the range of your data may help you to decide the data type. 8-bit data has a range of 0-255 or -128 to +127. 16-bit integers have a range of 0 to 65535 or -32768 to 32767. 32 bits extend the range to 0 to over 4 billion, or -2 to +2 billion. As floating point data is in exponential (also called engineering or scientific) notation, e.g: 1.234e-19) form, the range can be huge. Choosing the wrong format can result in a graph which looks like a scatter diagram, or may exhibit a "ladder" or "comb" effect with valid (ordinate) data interspersed with invalid (abscissa) points. Spectre tries to determine the data type and layout when it reads the data file. If Spectre thinks you have chosen the wrong type or layout, you will be informed and may continue or retry loading with another format. If you decide to open the data into a Scrolling Graph Window you can select Auto-Calibration, where Spectre sizes the display of each scrolled section to the minimum and maximum Y-values, or you can set the minimum and maximum limits to keep each scrolled section uniformly scaled. Finally, note that unlike an FFT, Spectre will accept data files with unequally-spaced, or missing, X (abscissa) values. The Frequency Search algorithm doesn't mind, but you will want to convert to equi-spaced data using the Tools/Pre-Process/Interpolate option before using the Fast Fourier Transform. Additional information: Data File (Time Series) FormatPreparing for a Frequency Search Data File (Time Series) FormatData files accepted by Spectre have a time-series formatted either as X,Y pairs of (time, value coordinates) or simply as a list of ordinate (Y) values. Data can be rea