Issue
I have a device for taking spectrometer measurements. A raspberry Pi is used as a shell for a GUi and communicates over serial (USB) with an Arduino to read spectrometer values. The RPi sends a string to the Arduino to tell it to read the spectrometer. During each measurement all spectrometer pixel values (288) are stored to a uint16 buffer then sent back over serial to the Pi which waits for at least 3 seconds to read data from the Serial. right now it takes about 0.3 seconds for one measurement (including time for RPi to tell the Arduino to take a measurement, the time it takes to actually read the spectrometer, and then send all of the spectrometer values back over Serial to the Pi). I have tried to implement this using I2C but it wasn't much faster (I had to send each value byte by byte). I am trying to implement this with SPI (which will hopefully be faster but don't have any experience with this interface). Is there a simple way to tell the Arduino when to read then send a large buffer of data back to the Pi? (instead of sending it over piece by piece?)
(This is the most important part of my code, there is more to it but hopefully this gets the point across)
Arduino:
#define SPEC_CHANNELS 288 // New Spec Channel
uint16_t data[SPEC_CHANNELS];
void read_value()
{
//Read from SPEC_VIDEO
//analogReadResolution(16);
for(int i = 0; i < SPEC_CHANNELS; i++){
uint16_t readvalue = read_adc();
data[i] = readvalue;
// pulse the spectrometer clock to switch pixels
digitalWriteFast(SPEC_CLK, HIGH);
delayMicroseconds(delayTime);
digitalWriteFast(SPEC_CLK, LOW);
delayMicroseconds(delayTime);
}
// send the buffer info over serial (with some formatting)
for (int i = 0; i < SPEC_CHANNELS-1; i++){
Serial.print(data[i]);
Serial.print(',');
}
Serial.print(data[SPEC_CHANNELS - 1]);
Serial.print("\n");
}
RPi:
self.ser.write(b"read\n") #tell arduino to read spectrometer
data_read = self.ser.readline() #read a full line of serial data
data_temp = np.array([int(p) for p in data_read.split(b",")]) #parse the info by ","
Updated code: Arduino:
void setup() {
Serial.begin(115200);
}
#define SPEC_CHANNELS 288 // New Spec Channel
uint8_t data[SPEC_CHANNELS*2];
void read_value()
{
for(int i = 0; i < SPEC_CHANNELS*2; i+=2){
uint16_t readvalue = read_adc();
data[i] = readvalue & 255;
data[i+1] = (readvalue >> 8) & 255;
digitalWriteFast(SPEC_CLK, HIGH);
delay62ns();
digitalWriteFast(SPEC_CLK, LOW);
delay62ns();
}
Serial.write(data, SPEC_CHANNELS*2);
}
void delay62ns() {
asm("nop");
}
RPi:
self.ser = serial.Serial(port, baudrate = 115200, timeout = 3)
self.ser.write(b"read\n") #tell arduino to read spectrometer
data_read = self.ser.read(288*2) #read a full line of serial data
new_data = []
for x in range(0,288*2,2):
new_data += [256*data_read[x] + data_read[x+1]]
data_plot = np.array(new)
print(data_plot)
output from the arduino Serial monitor:
g ⸮Og⸮⸮⸮ %⸮b⸮ ⸮'p⸮om⸮o ⸮⸮:p`ʏW⸮BOW⸮%�⸮ڊPX⸮z⸮b ⸮r⸮zB
b⸮⸮_⸮⸮ B⸮ ⸮ ⸮ ⸮R⸮rz⸮
7⸮⸮oxJϕ M0⸮% ⸮⸮ ⸮ /⸮ w⸮ ⸮ ⸮?U⸮W⸮%⸮ էW⸮⸮X⸮⸮⸮"⸮H ⸮⸮⸮R⸮⸮J⸮o7`z⸮ ⸮⸮*⸮5bB _⸮⸮Z⸮U U ⸮E⸮⸮⸮⸮ O⸮*⸮ ⸮2x*⸮b⸮o⸮z⸮}⸮_⸮⸮⸮⸮⸮
⸮⸮
=⸮:Ϛ⸮hW/�O-⸮`j⸮⸮
*⸮J⸮woB⸮⸮] ⸮⸮⸮w⸮⸮-⸮ (⸮ ⸮⸮⸮J:⸮eߗ_]⸮
output From Pi (with the working code above):
1586 1344, 1444, 1560 ...
Solution
My suggestion in the comments was to send the data as binary rather than as ASCII strings, because if you imagine the 16-bit sample 10,000 it requires 2 bytes to be transmitted, whereas if you make it an ASCII string, it takes 6 bytes if you include the comma required to separate it from the next sample:
10000,
Glad to see it has improved the performance.
Another idea, if you want to maximise the reads per second, might be to continually read the spectrometer in anticipation of the Raspberry Pi asking for the data and then hand it over when asked. So, instead of:
do forever
sit around idly wasting time till RPi asks for data
read spectrometer
send spectrometer readings to RPi
done
maybe do this:
do forever
read spectrometer
if there are serial bytes available (check in NON-BLOCKING way)
send latest spectrometer readings (which we already have) to RPi
end if
done
Another observation... though you mention delaying 62ns in your code (which corresponds to one cycle on a 16MHz clock), you should be aware that it potentially takes 4+ clock cycles to call your subroutine and 4+ further cycles to return so you are actually incurring 16+ cycles per reading more than you may expect. It doesn't amount to much in the great scheme of things, and your compiler may be inlining your function call anyway... but you get my drift, hopefully.
Imagine your Arduino gathers and sends 288 16-bit values from 0..287. You can synthesize the bytes buffer that you might receive on your serial.read() like this:
import numpy as np
buf = np.arange(288, dtype=np.uint16).tobytes()
b'\x00\x00\x01\x00\x02\x00\x03\x00\x04\x00\x05\x00\x06\x00\x07\x00\x08\x00\t\x00\n\x00\x0b\x00\x0c\x00\r\x00\x0e\x00\x0f\x00\x10\x00\x11\x00\x12\x00\x13\x00\x14\x00\x15\x00\x16\x00\x17\x00\x18\x00\x19\x00\x1a\x00\x1b\x00\x1c\x00\x1d\x00\x1e\x00\x1f\x00 \x00!\x00"\x00#\x00$\x00%\x00&\x00\'\x00(\x00)\x00*\x00+\x00,\x00-\x00.\x00/\x000\x001\x002\x003\x004\x005\x006\x007\x008\x009\x00:\x00;\x00<\x00=\x00>\x00?\x00@\x00A\x00B\x00C\x00D\x00E\x00F\x00G\x00H\x00I\x00J\x00K\x00L\x00M\x00N\x00O\x00P\x00Q\x00R\x00S\x00T\x00U\x00V\x00W\x00X\x00Y\x00Z\x00[\x00\\\x00]\x00^\x00_\x00`\x00a\x00b\x00c\x00d\x00e\x00f\x00g\x00h\x00i\x00j\x00k\x00l\x00m\x00n\x00o\x00p\x00q\x00r\x00s\x00t\x00u\x00v\x00w\x00x\x00y\x00z\x00{\x00|\x00}\x00~\x00\x7f\x00\x80\x00\x81\x00\x82\x00\x83\x00\x84\x00\x85\x00\x86\x00\x87\x00\x88\x00\x89\x00\x8a\x00\x8b\x00\x8c\x00\x8d\x00\x8e\x00\x8f\x00\x90\x00\x91\x00\x92\x00\x93\x00\x94\x00\x95\x00\x96\x00\x97\x00\x98\x00\x99\x00\x9a\x00\x9b\x00\x9c\x00\x9d\x00\x9e\x00\x9f\x00\xa0\x00\xa1\x00\xa2\x00\xa3\x00\xa4\x00\xa5\x00\xa6\x00\xa7\x00\xa8\x00\xa9\x00\xaa\x00\xab\x00\xac\x00\xad\x00\xae\x00\xaf\x00\xb0\x00\xb1\x00\xb2\x00\xb3\x00\xb4\x00\xb5\x00\xb6\x00\xb7\x00\xb8\x00\xb9\x00\xba\x00\xbb\x00\xbc\x00\xbd\x00\xbe\x00\xbf\x00\xc0\x00\xc1\x00\xc2\x00\xc3\x00\xc4\x00\xc5\x00\xc6\x00\xc7\x00\xc8\x00\xc9\x00\xca\x00\xcb\x00\xcc\x00\xcd\x00\xce\x00\xcf\x00\xd0\x00\xd1\x00\xd2\x00\xd3\x00\xd4\x00\xd5\x00\xd6\x00\xd7\x00\xd8\x00\xd9\x00\xda\x00\xdb\x00\xdc\x00\xdd\x00\xde\x00\xdf\x00\xe0\x00\xe1\x00\xe2\x00\xe3\x00\xe4\x00\xe5\x00\xe6\x00\xe7\x00\xe8\x00\xe9\x00\xea\x00\xeb\x00\xec\x00\xed\x00\xee\x00\xef\x00\xf0\x00\xf1\x00\xf2\x00\xf3\x00\xf4\x00\xf5\x00\xf6\x00\xf7\x00\xf8\x00\xf9\x00\xfa\x00\xfb\x00\xfc\x00\xfd\x00\xfe\x00\xff\x00\x00\x01\x01\x01\x02\x01\x03\x01\x04\x01\x05\x01\x06\x01\x07\x01\x08\x01\t\x01\n\x01\x0b\x01\x0c\x01\r\x01\x0e\x01\x0f\x01\x10\x01\x11\x01\x12\x01\x13\x01\x14\x01\x15\x01\x16\x01\x17\x01\x18\x01\x19\x01\x1a\x01\x1b\x01\x1c\x01\x1d\x01\x1e\x01\x1f\x01'
Then to convert it to a Numpy array, all you need is:
data = np.frombuffer(buf, dtype=np.uint16)
Depending on your byte ordering, you may need to set the dtype like this:
dt = np.dtype(np.uint16)
dt = dt.newbyteorder('>') # You may need '<'
data = np.frombuffer(buf, dtype=dt)
data now looks like this:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,
39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103,
104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,
117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129,
130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155,
156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168,
169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,
182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194,
195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207,
208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220,
221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246,
247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259,
260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272,
273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285,
286, 287], dtype=uint16)
On my machine, this is 140x faster than your code that builds a list.
Keywords: Python, serial, transfer, efficient, efficiency, performance, high performance, Raspberry Pi, Arduino, binary, ASCII, Numpy, ADC, spectrometer.
Answered By - Mark Setchell
