I am familiar with IQ symbols on the IQ plane and how they could represent AM and FM signals, for example.
What I do not understand is how the IQ stream would represent a wide group of signal (say 40 MHz bandwidth) containing many, many signals with a variety of types of modulation simultaneously. Is there some FFT done in IQ to decompose the signal? I presume there are not many simultaneous, separate IQ streams for each signal.
Anyone able to provide an ELI5
IQ is a vector representation of phase and amplitude and nothing else. The IQ data can be in the time domain (time vs amplitude+phase) or, usually post processing with a transform be, in the frequency domain (frequency vs amplitude+phase). So prior to demodulating you would apply a (low pass/high block, bandpass, high pass/low block) filter which can be applied in either the time domain or in the frequency domain. A filter basically removes unwanted information, so that only the required information remains. In the case of IQ data in the frequency domain that could be as simple as setting both the I and Q of the unwanted frequency bins to zero. Applying a similar type of filter to the time domain requires the use of FIR (Finite Impulse Response) or IIR (Infinite Impulse Response) filters. Either way you are removing the unwanted information and focusing in on the wanted information, usually in parallel with the filtering decimation would be applied to reduce the data rate. If you are only looking at a 1MHz wide signal, there is little point in processing an additional 39MHz of data which, post filtering, is all zero that is just wasting processing time.
Read a few books on DSP would be my advise, this is the kind of stuff you would never try and explain to a five year old, unless you were some kind of cruel parent.
Thanks for that: you wrote it exactly as I needed.
I subtext that I presume from your comments is that the higher ADC bit rate in LimeSDR (compared to RTL and HackRF) provides greater depth to a signal, allowing one to extract the information of interest. In the Claude Shannon/Information entropy sense, the channel depth of LimeSDR is as important (if not of greater importance) as the bandwidth.
I am in the process of reading a signal processing book (https://www.amazon.com/Practical-Signal-Processing-Mark-Owen/dp/1107411823?SubscriptionId=AKIAILSHYYTFIVPWUY6Q&tag=duckduckgo-d-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=1107411823) recommended by Michael Ossmann of Great Scott Gadgets/HackRF fame. His educational lectures are excellent but at a much lower level than your apparent competence.
I am currently a retired college prof using SDR to demonstrate evidence for dark matter by using a simple/inexpensive method to “see” the 21 cm monatomic hydrogen H line . The goal is to get the information into a context where a reasonably bright high school student could understand the observation. We have access (remote and on site) to a 60 foot dish that can easily make these observations, but want the students to have hands on activities using a simple horn antenna with “drift scans” to view the galactic plane and measure galactic rotation by doppler.
The observations we make are in the frequency domain where alterations in the apparent frequency of the h line are due to doppler shifts caused by the rotation of our galaxy. The short story is that our galaxy is rotating way too fast to stay together. There is not enough apparent visible matter to hold our galaxy together using conventional gravitational forces alone.
Can you perhaps recommend an beginner/intermediate book on DSP that might help me understand this? Your explanation has provided considerable insight.
Thanks,
-Bob
WA2I
You said “ADC bit rate” which is a confusing term, there is ADC sample rate and ADC resolution (which is measured in bits). A higher sample rate would be directly linked to more bandwidth and higher number of bits to a higher dynamic range. For the hydrogen line even a 1 bit ADC could be good ( http://hawkrao.joataman.net/H1/results/#Comparison_Between_8-bit_and_1-bit_Data_Results ) since the signal is buried so far beneath the noise floor that a massive amount of averaging over a long period of time is usually required to raise the signal above the noise floor with non professional equipment. I’ve heard people interested in the Hydrogen line at 1.42 GHz always looking for more bandwidth, honestly I’ve no idea why, I forgot to ask at the time.
My apparent competence I would say is lower than Michael’s, I may give him a good run for his money in some areas, but to make anything look at simple as he does requires a wide breadth and a good depth of knowledge.
I don’t know of one book, that covers SDR, DSP, filters design, and a million and a half other related chunks of knowledge that fall between the cracks.
Long ago, I read “Understanding Digital Signal Processing” by Richard G. Lyons, but there is no way I could call it a beginner/intermediate book - it is mostly high level maths, or at least the edition of the book I read was.
I currently like “The Scientist and Engineer’s Guide to Digital Signal Processing”, you can read the whole book online or download it chapter by chapter as pdf files and if you like it and feel the author deserves to make a living you can buy a copy http://www.dspguide.com/ch1/1.htm - but it does not cover everything.
If I came across a DSP term that I had not heard before I used to use the search box on https://dspguru.com/ for example if someone mentioned “Parks-McClellan”, I’d use the search engine there to have some idea what it was, or “Decimation”, or “Interpolation”, or …
A lot of the digital theory is only really useful if you are coding your own specific application, most of the time you just need to know how to apply knowledge and not fully understand every single detail. Like what windowing function should you use, where should you use the sinc function. Most people who drive cars do not know that if a 4 stroke engine is running at 1200 RPM that there are 10 explosions a second happening inside each cylinder of the engine block in front of their feet, because they simply don’t need to know that level of detail to drive a car.
Hi @mzs-
I had intended to discuss ADC resolution (12 bit for limeSDR I believe), not sps/bandwidth.
The 8bit/1bit comparison puzzles me (but might be related to the question I am circling toward. How can he get temperature to 4 significant figure(1.008- 1.016)? Even if we drop the first digit, you still have 2 (base 10) digits which by my count requires 7 to 8 bits. Perhaps it is 1 bit per sample but many bits per bin? This then gets to my question which is, how does one do a time integration on FFT transformed data? Is it merely the sum of bits per bin divided by time?
Although your analogy is correct about the details of an engine, as a teacher, I have found that I need to understand how it works “under the hood” even if I will only talk about how fast it goes. Teaching I find, always ends up with me realizing how little I know about things I have been credited to know a lot about (I am a biologist, primarily).
I presume there is an easy way to do an integration in gnuradio / pothos, but I have not yet found it (in a way I understand at least). The excellent report by Joataman that you referred to talks about frequency bins in terms of velocity (not a difficult calculation but, if I will be writing some code it should not be too difficult to sum bits per frequency bin.
Thank you for your kind and detailed responses. I have found a copy of the Richard G Lyons book. A quick glance suggest it is at a level that I can follow along the main points.
Regards,
Bob
Hi Bob,
I’ll add to this thread and hope that, in addition to the other comments, you will see a bunch of different angles on this (no pun intended). A signal referred to as “I/Q” is a specific kind of representation (esp. found in Digital Signal Processing) that makes the math efficient. The I means “In-phase” and the Q is for “Quadrature”. Think of a Cartesian (x-y) plane and rename the y axis Q and the x axis I; but, there is one additional thing - the Q Axis is imaginary and the I axis is real. This plane has a special name and is called and Argand plane because it is a complex plane.
An I/Q signal is also referred to as a complex-valued baseband signal because it is centered at zero frequency, with plus and minus signal content that in general are different. So, in the case of Software Defined Radio, (and there are variations of what I’m about to describe, but I think this will make the point) a analog RF signal (say an FM radio signal) expresses a flux density across an antenna aperture and produces a small signal that goes to the receiver’s band pass filter, then an LNA. This is then split and fed to a dual mixer (multiplier). Each of the split signals are multiplied with an oscillator, but the oscillator provides a quadrature Sin and Cos version of it’s signal separately to each multiplier. A low pass filter is used to retain the desired products of each multiplication and then the result is fed to a A/D converter to digitize the signals at some sample rate, greater than or equal to the total complex bandwidth. The Cos is the I, and the Sin is the Q signal. Mathematically, you can treat the combined A/D signals as Signal = I +j Q
The utility of such a plane is remarkable. Also the math needed to use it requires a good understanding of right angle trig. Additionally, signals are commonly referred to using complex exponential form which seems a bit strange, but if you study Euler’s Identity Ae^jw = A/2 [cos(w) +jsin(w)], you will start to see the beauty of the compact representation. Where w is the instantaneous phase angle of the signal, A is the Amplitude (also called the magnitude when we think of the signal as a time varying vector).
I have to leave some things out here, but since you asked… If an FM signal were coming into the SDR, then the instantaneous signal phase would be changing each time interval according to the modulation signal’s amplitude (in the time domain). So, to make a simple demod, one could compute the difference of the phase element just received with the one before it. This essentially is taking the discrete time derivative, and converting phase differences (freq) in to signal values proportional to the freq change - which un-does the modulation. This is a basic idea and leaves out lots of performance improvements that a real commercial receiver would have.
As for the FFT and I/Q signals, The FFT is a computationally efficient signal processing workhorse that is a mathematical transform that allows you to convert between the frequency-domain and time-domain. It works in a block of N samples (where N is usually a power of 2 - e.g. N=1024 ). FFT’s are most often used to view the signal power and occupancy of some bandwidth of spectrum - e.g. a falling raster plot (waterfall). A a more sophisticated FFT application is digital communications such as WiFi where they are used to make OFDM signaling. BTW, the FFT can accept complex -valued I/Q signals and it can also accept real-valued (no imaginary component), although with real data the spectrum is symmetric so only half of it is unique.
Not sure if that was helpful or not. Good luck in your pursuits. As for references, there are many, but they require a solid understanding of calculus and an undergrad course in signals and systems. Linear algebra and random variables would be also important to understanding how communications signals actually convey information reliably. If your math background is solid I can aim you in a better direction.
Cheers,
John
Hi Bob,
Just saw your question about the FFT integration. So, the output of the FFT is a set of N complex numbers (also called freq bins). Note that it took N complex samples to get N transformed values out. Each FFT complex output value can be converted to the magnitude and phase value for that particular freq bin. If you were to plot these, you would get the frequency response. If you were to average each individual bin and plot than you would see the average frequency response. However, there is a slight catch to the FFT that you need to understand - Windowing.
So, there is a topic called windowing that brings up another thing called spectral leakage. What you need to know is that the FFT is a fast version of the Discrete Fourier Transform, which is a sampled version of the continuous Fourier Transform (FT). I don’t assume you have a background in calculus, but, it turns out that the FT is a summation that expects to have ALL the samples of a signal from -inf to +inf. Not exactly practical. Well, it turns out that we can still use it, but we need to compensate for using it in short term intervals. To do this, we need to apply a time window to our data set to reduce the effects that the sharp boundaries of the rectangular window we create when we just snap up N samples. So, most applications that display power spectrum deal with this by over-lapping and adding, windowed data.
I’ll stop here because I know that was likely a bit more than you expected. If Google this topic, wiki will be your friend.
John
Actually that’s exactly what I am interested in. I am familiar with calculus for the differentiation integration of function however, my understanding breaks down with a numerical integration. Are these the windows that are called Hamming, Hanning, etc? I am also familiar with Euler’s equation that provides the relationship to derive how the time domain can be converted into an infinite series of sine waves.
I believe you are saying that you would use overlapping bins in the freq domain to create an average over say 10 seconds (or 10 minutes, as the case may be). I am surprised to hear that you use overlapping windows. I presume that using non-overlapping windows you would get a step function whereas a moving average (across more than one bin) smoothes the summations over time.
So practically speaking, I use gnuradio to produce a fft over the interval of interest. Since I am looking at the doppler shifted 21cm h-line using a drift scan, I would need a bandwidth of about 2MHz to cover the anticipated doppler shifted frequencies. Each time interval would correlate with a portion of the sky. I believe that gnuradio allows me to define the Hamming or similar windows to which you refer.
Here is my confusion - the sharp boundaries (a kind of quantization artifact) occurs in the frequency domain and will likewise occur in the time domain. To visualize this I would need a 3D plot with the x axis measuring doppler speed (ie f), the y axis would be the summed or averaged magnitudes over the time interval and the z-axis would be position as my drift scan crosses the galactic plane. This would be for each declination I choose and the time interval as the galactic plane rotates into the antenna aperture, this is effectively the RA.
This would also tell me that if I do not obtain a clear galactic rotation curve, if I continue past 24 hours, I would see changes in sidereal time, consistent with signals of galactic origin.
The quantization artifact also occurs in the time domain as one would find jumps in the power from one time bin to the next.
I guess what I really need is to get some real data, warm up my soldering iron, get my LNA working and point it at the sky.
Thanks for helping a noob.
Bob
Hi Bob,
Glad that was of use to you, as for your questions…
Are these the windows that are called Hamming, Hanning, etc?
Yes, those are some of the many different types of windows used to control the data sample set.
I am also familiar with Euler’s equation that provides the relationship to derive how the time domain can be converted into an infinite series of sine waves.
Euler was THE most prolific Mathematician to ever live. Here I am referring more to the work he did to discover and reveal (through series expansions of trig functions) how a complex exponential could represent a complex number on a Argand plane. I think you are referring to how any waveform can be expressed as a summation of basis orthoganal functions which is a Fourier series expansion (e.g. via an analysis function such as the Fourier Transform).
I believe you are saying that you would use overlapping bins in the freq domain to create an average over say 10 seconds (or 10 minutes, as the case may be).
I am saying that to compute the FFT to measure frequency (and avoid spectral leakage), take each group of consecutive N samples, and multiply them with the chosen N-length window then take the FFT of the result. Take the magnitude sqrt(real^2 + imag^2) of each output and plot it. You can also plot the squared mag to display the power spectrum if you want that.
Now, for the average… If your signal is stable in frequency, then you could benefit from a longer time average (say to reduce noise variations). So, you can add up M complex FFT outputs (bin to bin) and divide sum for each bin by M. Then plot the magnitude and you will see the noise variations reduce, revealing the signal. There are other signal processing techniques that can improve your situation beyond a simple average (e.g. coherent integration, correlation, cyclostationary searches, non-parametric Eigen decomposition of the correlation matirx of the input etc.) These can get quite sophisticated as you might imagine.
I am surprised to hear that you use overlapping windows. I presume that using non-overlapping windows you would get a step function whereas a moving average (across more than one bin) smooths the summations over time.
The method I’m describing is called the Welch periodogram - which is a power spectrum estimation. A 50% overlapped window is often used to multiply the input signal. Because the signal data is being reduced on the end of the windows, you back up by 50% (N/2) and window again and FFT. This attempt to keep the average power unaffected by the characteristics of the window.
Here is my confusion - the sharp boundaries (a kind of quantization artifact) occurs in the frequency domain and will likewise occur in the time domain. To visualize this I would need a 3D plot with the x axis measuring doppler speed (ie f), the y axis would be the summed or averaged magnitudes over the time interval and the z-axis would be position as my drift scan crosses the galactic plane. This would be for each declination I choose and the time interval as the galactic plane rotates into the antenna aperture, this is effectively the RA.
This would also tell me that if I do not obtain a clear galactic rotation curve, if I continue past 24 hours, I would see changes in sidereal time, consistent with signals of galactic origin.
The quantization artifact also occurs in the time domain as one would find jumps in the power from one time bin to the next.
Well, I apologize that I’m not following your description of the signal you are processing and I’m not familiar with the kind of science you are performing. I’m gathering that you have a senor of some kind and that you are looking to see things in the frequency domain. Does the data originate in the time or freq domain? Do you use the FFT to get the freq domain info?
By the way, you say there is a quantization artifact that occurs in the freq domain. I must point out that time sampled data has no concept of frequency quantization, only amplitude. I can represent an infinite number of frequencies as I like - practically speaking, up to the floating point precision of the computations used to create my phase change increments - all within the Nyquist range of course (Fs/2 if Real, or Fs if complex - since there are two sample streams per calculation).
Well, I’m again not sure I’ve been much help.
Cheers,
John
There is also a Magnitude Estimator trick that is computationally more efficient, but obviously not as accurate.
Basically Mag ~=Alpha * max(|I|, |Q|) + Beta * min(|I|, |Q|) ( ref: https://dspguru.com/dsp/tricks/magnitude-estimator/ )
For minimum peak error you would choose Alpha = 0.960433870103 and Beta = 0.397824734759 but there are many possible values for Alpha and Beta based on different magnitude criteria. Texas Instruments have even been granted a patent ( US9015452 ) that uses this magnitude approximation.
Very cool. Usually getting the magnitude is not too computationally taxing (compared to other things being calculated), so I’ve always been ok with not approximating… however this is very interesting and nice to know. Thanks!