Being a fish creep on YouTube: my pond cam

Intro

This is a quick post to give some more information about my pond stream and hopefully answer some of the more common questions.

Backstory

When I moved into my current house, the garden had a pond stocked with a few fish which I suddenly had thrust into my care. After figuring out how filtering, feeding etc. worked, I set to automating the process and monitoring which you can check out here.

All was well for 2 years, then suddenly – once the newly hatched fish had reached their full size – they all started dying off. Once I saw the fish were acting sluggish and refusing food, I purchased everything I could from Amazon – water testing kit, chlorine filter, air stones, new filter media, a pond bomb and new UV bulb. I immediately got the water tested at a local pet store, but they didn’t have a test for ammonia available.

Thinking the water was bad, I scooped out the still wiggling fish into a dechlorinated inflatable pool and hoped that I could triage the survivors. The air stones arrived, and the pond soon looked like a jacuzzi as I tried to raise the oxygen level. The water was crystal clear with the fresh filter/UV bulb and the temperature was good. Clear water doesn’t mean that there isn’t a problem however, and I had no way of knowing without a testing kit.

Unfortunately, most of the fish died, including all the large koi, most of the fish babies from the previous year, and some older goldfish. With all the effort, I was left with Violent Ken the goldfish, a Comet and two dark goldfish hatchlings. It was depressing having to net out and bag up the koi. Do not recommend.

Don’t mess with Violent Ken. He’s seen some things

The water tests were all within spec, but this was only really checked after the other actions were taken as it took a while to arrive.

It was a kick in the ass that I need to care more about what is going on in my pond – I failed as their caretaker. Not having a water testing kit was my biggest mistake, but not identifying when they’re acting off was my next.

I waited a few weeks for the pond to settle, tested the water, and purchased some new fish to keep the remaining 4 company.

This then led to me getting an underwater camera…

Keeping a fish-eye on everything

I have a Synology NAS with their Surveillance system which allows you to connect IP cameras and it’ll save the footage and make it available via their webclient. It’s really simple to set up, although you have to buy additional licences for more than 2 cameras. I already had a few house cameras set up, so adding another camera would be easy enough.

Really need to clean the lens, some of those cameras are looking a bit blurry

I found this camera on Amazon which fit the bill. After getting a CAT6 cable set up and battling the half-translated software, I had everything working.

Me and my girlfriend started to have the Synology webclient running in the background while working as its quite relaxing to watch. I then shared a link with some friends, but it wasn’t intuitive to log into and a bit of faff to set up.

I then found the built-in Synology solution to publish to Youtube – Live Broadcast. You enter the RTMP path from Youtube and your stream key, then choose the camera and quality to send. Done and dusted….?

Well, it seems like my older NAS didn’t have the muscle to re-encode the stream at HD quality and I could just about push the low quality stream at around 480p. The Live Broadcast client also doesn’t handle network drop-outs very well either meaning the stream kept going down.

Embarrassed that I couldn’t show off my pond, I set to work on another solution.

Docker and the over-engineering of

I have a few websites (including this one) that I run from a NUC that I don’t want to pay an external host for. It has an Intel i5-7260U which is a fair bit better than the STM Monaco STiH412 in my NAS. To compartmentalise all the projects I have on there, I use docker.

My goals were to get the stream to Youtube with auto-restart, but that quickly changed to wanting to add overlays and other silliness. Here is the overview:

‘fish’ might be the most popular comment in the live chat

The magic contains the following:

blynk_connector is a python alpine image with a script that repeatedly runs multiple REST API calls against the Blynk server (which my sensor projects publish to) and writes them to the database.

webserver is a php apache image that runs some basic MySQL queries for current and historic data then serves a small HTML page with the stats, something like this:

I have to admit that I used <table> for this

renderer is a browserless/chrome container that is normally used for headless automation like integration testing and QA. Here I’m using it to take a screenshot of the HTML page and return a PNG.

overlay is another python image that orchestrates renderer by asking it for the PNG and moving it into the encoder image. It does this every 15 seconds.

encoder is an image with ffmpeg available. It pulls an internal Synology-provided RTSP feed from the camera, adds the PNG overlay and then puts a subtitle announcement displayed at the top of the stream.

The command it runs is nice and complicated. I’m no expert with ffmpeg, but its working:

ffmpeg \
        -re \
        -f lavfi \
        -i anullsrc \
        -rtsp_transport tcp \
        -stimeout 10000000 \
        -i "$SOURCE" \
        -f image2 -stream_loop -1 -re -i "banner.png" -filter_complex "overlay=x=0:y=main_h-overlay_h+1,drawtext=fontfile=Montserrat-Regular.ttf:textfile=announcement.txt:reload=1:fontcolor=white:fontsize=42:box=1:boxcolor=black@0.5:boxborderw=5:x=(w-text_w)/2:y=(text_h)-20" \
        -r $FPS \
        -g $(($FPS * 2)) \
        -tune zerolatency \
        -preset $QUAL \
        -c:v libx264 \
        -b:v $VBR \
        -c:a aac \
        -strict experimental \
        -f flv "$YOUTUBE_URL/$KEY"

Conclusion

Now I (and you!) can watch the health of my fish from the warmth of the house. By having the camera and some viewers I’ve got an even better incentive to keep the water clear.

I think in the summer I’ll drain the pond and make it a bit more aesthetic.

At the time of writing, we’re celebrating Fishmas, and I hope you can join us in the stream too!

UPDATE:

I’ve updated ffmpeg to output to both Twitch and Youtube and in doing so I noticed the stream failing a few times, especially with YT. Especially irritating is that the YT link changes when the stream returns and I’m assuming anyone subscribed would get an alert that the stream is alive again.

So I created a new container that reads the log output from ffmpeg, compares the encoded frame number and restarts the container if needed. Here is the script panopticon runs:

#!/bin/sh

function log() {
    echo "[$(date +%F_%T)] $1"
    printf "[%s] %s\n" "$(date +%F_%T)" "$1" >> /app/stream.log
}

log "Started! Waiting (15s)"

sleep 15
while true
do
    MOST_RECENT_FFMPEG=`ls -t /app/logs/ffmpeg* | head -n1` 

    frameA=$(tail ${MOST_RECENT_FFMPEG} -n 1 | sed -nr 's/.*frame=(.*)fps.*/\1/p')
    sleep 1
    frameB=$(tail ${MOST_RECENT_FFMPEG} -n 1 | sed -nr 's/.*frame=(.*)fps.*/\1/p')

    log "Frame numbers: $frameA-> $frameB"

    if [ "$frameA" = "$frameB" ]
    then
        log "Stream has hung :("
        docker restart pond-stream_stream_1
        log "Restarted container"
        log "Waiting for container start (30s)"
        sleep 30
    else 
        log "OK"
    fi

    sleep 2
done

This is based of a script I found by chrisstubbs93 here.

I also changed the YT stream to be scheduled instead of using the ‘Go Live’ button. A scheduled stream can be set to not auto-stop and it seems to give a bit more breathing room between loss of data.

Hopefully I’ll be able to get a new uptime highscore with this. I’ll embed it to show my confidence…

EDIT: Spoke too soon, the camera itself crashed 🙁 Did manage 366 hours though.

New ffmpeg script. I specifically want the command to fail if either stream dies as the container will just auto-restart and try again.

LOG_POSTFIX=`date '+%Y%m%d%H%M%S'`

ffmpeg \
        -re \
        -f lavfi \
        -i anullsrc \
        -rtsp_transport tcp \
        -stimeout 10000000 \
        -i "$SOURCE" \
        -flags +global_header \
        -f image2 -stream_loop -1 -re -i "banner.png" -filter_complex "overlay=x=0:y=main_h-overlay_h+1,drawtext=fontfile=Montserrat-Regular.ttf:textfile=announcement.txt:reload=1:fontcolor=white:fontsize=42:box=1:boxcolor=black@0.5:boxborderw=5:x=(w-text_w)/2:y=(text_h)-20" \
        -r $FPS \
        -g $(($FPS * 2)) \
        -tune zerolatency \
        -preset $QUAL \
        -flags +global_header \
        -c:v libx264 \
        -b:v $VBR \
        -c:a aac \
        -maxrate $VBR \
        -bufsize 2000k \
        -strict experimental \
        -flags +global_header \
        -map 0 \
        -f tee "[f=flv]$TWITCH_URL/$TWITCH_KEY|[f=flv]$YOUTUBE_URL/$YOUTUBE_KEY" 2> ./ffmpeg-{$LOG_POSTFIX}.log

EDIT EDIT: In the end the YT stream lasted for months, but my poor home internet connection couldn’t handle streaming to both YT and Twitch.

I found Twitch to provide a better experience – less lag so faster feature usage – so have completely switched to just streaming via Twitch – https://www.twitch.tv/underwaterpondcam

I also stopped using ffmpeg, and instead used an old laptop with OBS running which allows for more customisation, hardware encoding and additional cameras. It also lets my poor NUC cool off.

Increase ATOM Lite range with external antenna

After working on my pond control project, I wanted to be able to increase the range of some M5Stack ATOM Lite modules. These modules are ESP32 based and come in a tiny form-factor.

They open easy enough after removing the sticker on the back and then squeezing the clam-shell case. Only 3 parts make up the module – case halves and the PCB inside.

Flipping over the PCB shows the 3D antenna connected to a capacitor-inductor network and the LNA pin on the ESP32-Pico chip.

The sheet-metal antenna is attached at 4 points – one top left, one top right (obscured) and the two bottom right in the foreground. With a little soldering and flux cleanup, the antenna is removed. I used a quick-clamp as a poor-mans PCB holder.

The left-hand pad is the LNA pin, and the right is a ground pad. The two upper pads are not connected to any trace I could find.

The idea is to get the antenna out the side of the case without interfering with the reset button, and also not coming out the bottom or rear (it would prevent the module from being used in the many modules – e.g. the PoE kit). I drilled a small hole in the case.

I cut back the end of a SMA pigtail connector, and tinned the ends.

I threaded the pigtail through the hole, and soldered the centre pin to the left pad, and the outer shield to the right ground pad.

Closing up the unit, I was ready to test

Before starting the internal antenna removal, I ran a simple sketch to list found AP’s. With the internal antenna, a maximum of 8 networks were discovered. With the external antenna, a maximum of 12 networks were found.

This isn’t the most scientific, but gives an idea of the improvement. I also listed the RSSI along with the AP name, and compared the two. The AP I picked was one that I had configured in the garden with 2 walls between.

The red – external – line clearly shows an improvement over the internal antenna.

The final step will be to add a small amount of epoxy to the antenna penetration in the side of the case for mechanical strength. Quick project done.

Koitrol – Pond monitoring

Nuts out gnome with the old switched junction box

Our pond was slowly losing water over time, and I wanted to be able to track whether it was normal (i.e. evaporation) or if there was a leak in the pond liner. While doing this, I figured adding some temperature sensors would be handy to know when to restart feeding – fish shouldn’t be fed while the water is less than 10C.

FUN FACT 1: Fish go into low activity mode over winter and won’t eat. Anything you feed will just sink and rot; affecting water quality.

Water and electronics don’t mix, so deciding on a water level sensor took a while. Capacitive systems will eventually corrode, ultrasonic ranging would work but drift over time (small error compounds leading to poor consistency) and float switches don’t show continuous change. I picked an etape sensor which is a thin tube with known resistance which changes as the surround liquid presses on it. The sensor has a second resistive element which is not affected by liquid level, but allows you to compensate for temperature changes.

Using an m5stack Atom Lite MCU, I first tried the built-in ADC which has poor linearity and is bad range (0-1.8v from memory?), so I switched to an external ADC121C021 over I2C which was much better. Popped a couple of DS18B20 waterproof temp probes on and we’re good to go.

FUN FACT 2: ADC stands for analog to digital converter. Its actually a very complicated network of resistors and capacitors to turn a wobbly voltage to a digital reading.

All of this was installed into a waterproof junction box with power in, and the two temperature sensors out. Using custom firmware, I had a websocket-backed web GUI and also a Blynk app to use on my phone. The web GUI allows for depth calibration – connect the returned resistance (in ohms) to a depth (mm). The ADC module was conformal coated incase of water ingress.

Sponges for mechanical mounting pressure and a delightful touch of colour.

Pond-ering forward

The next step was to automate the pond filter. The pond has a basic box filter which has a UV lamp in the top which the water must first flow over, then through various compartments containing different foams before being spat out the from two pipes. After a while the bottom of the filter gets full of fish poop and other biological goodness.

To clean the filter, it must first be drained with a tap (quick de-sludge) on the bottom, and then the filters washed. Before draining, the UV lamp should be turned off, then the pump (the water cools the UV lamp in normal operation). Using switches to do this is the boomer way, so we’ll automate it and ~~waste countless hours~~ invest time into speeding this up in the future.

FUN FACT 3: UV light causes algae to die and clump together giving you clear water. A more cheerful name would be the Algae Genocider

Using another Atom Lite, I hooked up a 4 channel relay board, two additional relays and a previous energy monitoring PCB I designed a while ago.

Should have cleaned the flux and that isn’t a bodge wire, I don’t make mistakes

The energy monitor board needs 3.3v but the MCU sends 5v. As the serial lines are not isolated from mains, I used a ADUM1201 and a B0505S isolated DC-DC converter into a linear voltage regulator (5v -> 3.3v) and packed it on a prototype board (gray box with arrows and + – on it).

V1. Energy monitoring board is in the light gray junction box with the serial lines out the bottom.

Using the relays, the filter and the pump (both 240V) can be controlled. Using a 150W waterproof 12V power supply, the remaining pond things can be controlled – lights, ball valve and bilge pump.

Originally I used a 12V to USB power supply (the Atom Lite uses USB C to power it) but I had issues with the 12V PSU browning out with the bilge pump (it doesn’t have soft-start, appears as dead short to the PSU on start up) so I just used a mains USB PSU instead.

Before the ball valve, I intended to use a solenoid value to control the sludge line, but solenoid valves required a minimum pressure to work which this gravity-fed system couldn’t supply. I also intended to use a small 12V pump to move the water, but it didn’t self-prime and kept running dry. In the end, I picked up a motorised ball-valve unit which you control by either sending power to 1 of two wires.