📝 Chocolate's Blog

The project had been using FOUR (!) different pieces of hardware to do the job of collecting data (ESP32), parsing data (Raspberry Pi), serving a website (Network-attached storage), and hosting the content (a paid webserver) - which seems a little superfluous. Instead, using Python to run the back-end webserver AND retrieve the data from the ESP32 makes a lo more sense, so welcome to the Python webserver!

Changed the way the log file selector works on the Data Table / Graph page, so now the drop-down shows the dd/mm/yyyy result of the logfile name instead of its raw csv file value. Also allows the inclusion of longtermlog.csv to be referenced, and in a human-friendly way. Behind the scenes the logfiles are still called via their filenames and passed to the datatable.php page. Graph programming was a bit tricky as the original parser script was designed for cumulative data rather than absolute, but this was tweaked using an if/else statement in the code, i.e., if ( $logfile = "longtermlog.csv" ) { parse as non-cumulative } else { parse as cumulative };.

Also now edited is the basicdata.php file, which is now able to summate the daily log values from the longtermlog.csv file enabling long-term totals of data since logging began. Turns out that little Diesel has run quite a long way since the early days: 2.3km and counting!

Well I'm sorry it's been a while - though I somewhat doubt there's anyone actually reading this but if you are there, please drop me a line using the main website Comments page (click here!). I've been letting the logs build up a bit, with keen interest, and notice that Diesel is indeed using both wheels, though preferring his upper one (away from his sleeping quarters), and is active once every three hours of so day or night - which is interesting. I've been editing the logfiles so that they are in keeping with the "up to date" standard of the logfiles now, i.e., changing filename at midnight, and providing only a cumulative log from 0 each day. Via the addition of the following code within the main Raspberry Pi python script, a new file is also being made which saves Diesel's total efforts of the day into a summary logfile, though I haven't yet written a parser for this.

if lasthour > int(datetime.now().strftime("%H")): outstring = (str(time.time()) + ","+ distance1 + ","+ distance2 +","+ motion1count+","+ motion2count+","+ motion3count) outfile = "longtermlog.csv" with open(outfile, 'a') as f: print(outstring, end="\n", file=f) reset = str(requests.get("http://" + esp32IP + "/reset").content)

else: outstring = (str(time.time()) + ","+ distance1 + ","+ distance2 +","+ motion1count+","+ motion2count+","+ motion3count) outfile = datetime.now().strftime("%Y%m%d") + ".csv" with open(outfile, 'a') as f: print(outstring, end="\n", file=f)

Just a quick one. Tried to figure out the best way to do allow the logfiles and subsequent information to display data fresh for each day. I thought about rewriting the python code in the Raspberry Pi to take the difference between two polled readings from the ESP32, and even rewriting the ESP32 code to simply output said difference between each access, but quickly realised the workload doing this would involve re-writing the scripts for every bit of the data analysis side (i.e., scripts for the graphs, tables, and csv file generator) AND refashioning each logfile to incorporate the change to the parsers. Instead, it seems obvious now: at time the value of the current hour is LESS than the previously stored value, i.e., 0 vs 23 then call the RESET programme from the ESP32. In the case of clocks going back, because they only do this by one hour, this should not fail (as 01:59:59 rolls over to 01:00:00).

The limitation to this method is that of power failure, whereupon the ESP32 chip will reset all its data back to 0, so a step-change would be noted within the CSV data, and an according severe negative shift be noted in the table and presented data. I will amend the parser to recognise if a negative result occurs, and simply add on the difference to subsequent lines.

A bit of back-end fiddling today, I have updated the Python script to use outfile = datetime.now().strftime("%Y") + datetime.now().strftime("%m") + datetime.now().strftime("%d") + ".csv" instead of a standard "datalogger.csv" filename, thereby allowing daily logs to be generated on the Raspberry Pi, changing name at midnight. The Pi's webserver now has a simple script simply to find .csv files within the logging directory, which in turn is called by the front-end webpage and presented as a drop-down selection box to choose which logfile you want to view.

Hurrah! After quite a long wait, I had a brainwave while trying to measure how far the reed switch (basically a magnetic on/off sensor) would need to protrude into the cage. I had initially thought I'd 3D print something to lie alongside his wheel, anchored from the outside of the cage, and have the reed switch slide in to. But in measuring the wheel with a biro, it hit me: use a biro! So 10 minutes in the garage cutting 5mm thick slivers of 2x2 pine and drilling four holes for cable ties and one hole for the biro, and sawing the tube of a biro to length, and we now have a working wheel sensor. Plugged it into the breadboard and she worked off the bat!

The motion sensors have two different output modes, depending on the position of Jumper J1 (the external-most setting being signal ON trigger, the inner-most setting being signal WHILE triggered). A re-write of the ESP32 C++ code to summate the time taken while triggered (float), as opposed to just incrementing a trigger count (integer), I feel makes for more granular data. The sensors aren't the world's most reliable methods of determining fine-detected movement, as they have only a limited field of view (especially in something so small as a hamster cage), and also on cessation of movement detection, require three seconds to be reactivated. However, if Diesel keeps moving in front of their fields of view during this time, his movement should be recorded in seconds (to 2 decimal places). The changed code was as follows, to accommodate seconds instead of just a simple count:

if (digitalRead(pinMotion1)) { if (motion1Active == 0) { digitalWrite(ledMotion, 1); motion1Active = 1; motionLevelLast = 1; lastmotion1millis = millis(); lastmotionmillis = lastmotion1millis; Serial.print("Motion detected at "); Serial.println(millis() / 1000); motion1Count++; } } else { if (motion1Active == 1) { motion1Active = 0; duration1Active = (millis() - lastmotion1millis) / 1000.00; totalDuration1 = totalDuration1 + duration1Active; digitalWrite(ledMotion, 0); } } //end if pinMotion1

More front-end updating behind the scenes, now the chart has three lines and is over a date axis rather than UNIX time; also, a legend now exists. The table output has also had a spruce-up with cell background colours reflecting the cell values (via incorporating the formula ((250-$contrast)+(($maxMotion1abs - $data[$row][4])/$maxMotion2abs)*$contrast) within the style="background-color:rgb(r,g,b);" element.

A fair amount of work went into the client-side behind-the-scenes JavaScript today, with the abandonment of the original static SVG-based chart output in preference for an includable publicly-available JavaScript library called chart.js, "one of the simplest visualization libraries for JavaScript, and comes with the many built-in chart types". There's a small amount of server-side work in the PHP file, to generate five XY lists for the scatterplot, which annoyingly cannot accept simply array columns but must be in the format "x: 000, y: 000"; ideally it'd be nice to just have a selector enabling or disabling array column calls (i.e., dataarray[0] vs dataarray[4], but that seems not to be possible.

The challenge accepted - and thankfully renaming a few variables and (perhaps lazily) copying some code to define if() loops for each wheel and each level of the cage has resulted in being able to output several variables where once lay only two, namely, distance1 and distance2, and motion1count, motion2count and motion3count. The info logger on the Raspberry Pi needed updating to record five variables instead of three, and additional variables wheelNumberLast and motionLevelLast were created so that the output can reflect which wheel was used last and which level Diesel was last seen on.

A very excitable wife and children have resulted in Diesel the hamster being gifted a three-layer house, instead of the bungalow cage in which he was inhabiting up until today. His new mansion now has two wheels, slides, and an overhanging sleeping area. This means I'm going to have to adapt my ESP32 wiring and code to account for THREE motion sensors, and TWO reed switches, as opposed to simply one of each! Amazon orders have been made...

This webpage is the end-result of generating the CSV file mentioned above, and parsing it out to a "user-friendly" webpage - which is still a work-in-progress of course. The CSV file is taken by PHP server-side, and broken down into individual rows of the array $data[], using the following code,

$fileData=fopen($csvfile,'r'); while (($line = fgetcsv($fileData)) !== FALSE) { $data[] = $line; } $dataLength = sizeof($data);

Each line of the resulting array is then checked to see if it matches the line above, and if so, ignored. At present the data is simply pumped out into a tabular format using good old fashioned HTML tables, but the plan going forward is to make this data table adjustable in JavaScript to allow a specific time windows, and also eventually to make a vector graphics (SVG) graph of time vs. distance travelled, motion triggers, and possibly even running speed (over the captured data polled windows of 30-45 seconds). As of the time of writing, I've just about managed to make a sine wave out of the SVG with a bit of text overlay, and the data is in very long scrolling table.

It now became highly necessary to not just display the data from the chip in a sensible manner, using PHP calls from a Raspberry Pi running Apache 3 and PHP, but also I thought it would be nice to have a bit more granular data, not just knowing total distances travelled and maximum speeds, but the ability to scroll back through time and see how the data has changed. Thus, Python was invoked, to regularly poll data from the ESP32 and store the variables below into a .csv file: - distance - motionCount

Python (a surprisingly pleasant programming language to use following the data type-pernickety C++), was able to simply loop through its polling code,

import requests import time import os import argparse

def retrieveandsave(i): distance = str(requests.get("http://" + esp32IP + "/d/distance").content) distance = distance.replace("b","") distance = distance.replace("'","") distance = str(float(distance))

motioncount = str(requests.get("http://" + esp32IP + "/d/motioncount").content) motioncount = motioncount.replace("b","") motioncount = motioncount.replace("'","") motioncount = str(int(round(float(motioncount))))

outstring = (str(time.time()) +","+ distance +","+ motioncount) with open(outfile, 'a') as f: print(outstring, end="\n", file=f)

print("Saved line " + str(i)) time.sleep(delay)

either indefinitely or for a certain number of repetitions, as defined by input arguments to the program. It wasn't necessary to compile the script to run stand-alone as it doesn't need to be real-time or processor-intensive, simply pinging the ESP32 every 30 seconds or so is enough to generate a fairly large .CSV file.

Now the rudimentary steps were understood, it was time to take things further and enable a more human interface for the ESP32, as well as the ability to poll it remotely. It wouldn't be sensible to have it accessed straight through a firewall via an open port, considering the potential for easy hacking, so polling data from it via a server call i.e., http://esp32.local/d/motionCount meant that a more secure device, such as a Raspberry Pi or even NAS, could query it and display the data sensibly. So the following variables were created to allow this: - distance - total distance travelled, i.e., motionCount * wheelCircumference - maxspeed - as described above - avespeed - total distance / time spent moving (i.e., ignoring gaps > 10 seconds) - millisnow - current boot epoch time, an unsigned long, for the ESP32, in milliseconds - lastwheelmillis - value of millis() last time a wheel spin was detected - lastmotionmillis - ditto for movement being sensed - motioncount - number of times motion detected

Using an arduino-compatible motion sensor, a second variable, i.e., tracking movement, was possible. The sensor goes high when it detects movement, and drops to low again when movement ceases. The variable motionCount was created to simply log the number of times motion was detected, i.e., motionCount = motionCount++; whenever if (digitalRead(pinMotion)) became true.

Iterative update to the code to calculate wheel circumference (i.e., diameter * pi) and a cumulative log of distance (i.e., distance = distance + circumference every time the wheel spun round by one revolution. Capturing a wheel spin event is not as simple as if (digitalRead(reedSwitch)) {wheelSpins ++;} because this would generate a rapid increase in the value of wheelSpins for however long the switch was active. Instead first a varaible reedSwitchState must be set to high on detecting a trigger, and and if() statement created to only increment wheelSpins if the value was previously == 0. The resetting of the switch to LOW did the opposite, i.e., setting reedSwitchState to LOW as well, and only doing so if the value was previously == 1:

if (digitalRead(pinSwitch)) { if (switchPressed == 0) { digitalWrite(led, 1); digitalWrite(ledReed, 1); delay(10); digitalWrite(led, 0); digitalWrite(ledReed, 0); // at this time an LED was connected as a debugging tool to determine the if statements were working. switchPressed = 1; wheelLast = millis(); Serial.print("Wheel spin at "); Serial.println(millis()/1000); switchCount++; } } else { if (switchPressed == 1) { switchPressed = 0; } } //end if pinSwitch

Instantaneous speed was calculated by taking the millis() value at the time the reed switch was triggered last time and subtracting this from the current millis() value, then dividing the wheel circumference by this difference in time. Top speed was simply determined by if (speednow > maxspeed) {maxspeed = speednow}; The ESP32's webpage at this point simply dumped the variables reed switch state and distance travelled to its only response page.

Iterative update to the code to calculate wheel circumference (i.e., diameter * pi) and a cumulative log of distance (i.e., distance = distance + circumference every time the wheel spun round by one revolution. Instantaneous speed was calculated by taking the millis() value at the time the reed switch was triggered last time and subtracting this from the current value, then dividing the wheel circumference by this difference in time. Top speed was simply determined by if (speednow > maxspeed) {maxspeed = speednow};. The ESP32's webpage at this point simply dumped the variables reed switch state and distance travelled to its only response page.

Identified "need" - how do we know how healthy our hamster is? When does he do his exercise? How fast does he run? How far does he run? Let's get a reed switch and a magnet on his wheel, and do some calculations using C++ on an Espressif ESP32 WROOM32 DEVKIT1 that I've got lying around... (I'd already proved the code concept using a Teensy 2.0 (Arduino-like chip), counting wheel revolutions and basic maths to calculate activity time/etc.) This code step was simply to test the WiFi capabilities of the ESP32 and run a rudimentary webserver displaying the status of a connected switch.

The following will be updated as more progress is made... - Making a suitable case for the veroboard instead of having the breadboards just sitting below Diesel's cage - Data analysis metrics?