Video Live Streaming with OpenMV and Nicla Vision

Overview

In this tutorial you will use the Arduino® Nicla Vision to capture and stream a live video feed to your local network. For that, you will use the onboard camera sensor and the Wi-Fi® connectivity. For this task you will write a MicroPython script and run it on the Nicla Vision with the help of the OpenMV IDE.

Goals

• Learn how to use the OpenMV IDE to run MicroPython on Nicla Vision
• Learn how to use the Nicla Vision Wi-Fi® connectivity
• Learn how to use MicroPython to capture images and stream them to the network

Required Hardware and Software

• Nicla Vision
• Micro USB cable (either USB-A to Micro USB or USB-C® to Micro USB)
• OpenMV IDE 4.0.14+
• Access to a Wi-Fi® local network (2.4Ghz only)

Nicla Vision and the OpenMV IDE

The OpenMV IDE was built for Machine Vision applications. It is meant to provide an Arduino like experience for simple computer vision tasks using a camera sensor. OpenMV comes with its own firmware that is built on MicroPython. Among other hardware, it supports the Nicla Vision board. OpenMV allows you to easily preview the camera stream and visually inspect color ranges to define thresholds for your machine vision scripts. Here you can read more about the OpenMV IDE.

Instructions

Configuring the Development Environment

Before you can start programming micropython scripts for the Nicla Vision, you need to download and install the OpenMV IDE.

Open the OpenMV download page in your browser, download the latest version available for your operating system, and follow the instructions of the installer.

Open the OpenMV IDE and connect the Nicla Vision to your computer via the USB cable if you have not done so yet.

Click on the "connect" symbol at the bottom of the left toolbar.

If your Nicla Vision does not have the latest firmware, a pop-up will ask you to install it. Your board will enter in DFU mode and its green LED will start fading.

Select

Install the latest release firmware

OK

Nicla Vision's green LED will start flashing while the OpenMV firmware is being uploaded to the board. A loading bar will start showing you the flashing progress.

Wait until the green LED stops flashing and fading. You will see a message saying

DFU firmware update complete!

The board will start flashing its blue LED when it is ready to be connected. After confirming the completion dialog, the Nicla Vision should already be connected to the OpenMV IDE, otherwise, click the "connect" button (plug symbol) once again (the blue blinking should stop).

Live Streaming

In this section you will learn how the MicroPython script works in order to capture live video and stream it to your local network. Live video streaming across local or internet networks is one of the most used applications for connected cameras.

Some application examples are:

• Monitor industrial processes, from analog gauges, food-industry processes, to CNC machinery or 3D printers
• Local video surveillance
• Robotics, from SLAM navigation with AGVs to computer vision-powered robotics arms

The Example Code

This tutorial's main objective is to leverage the Nicla Vision to stream live video reachable from inside the network using your favorite web browser.

The example code we are going to use can also be found in OpenMV by navigating to File > Examples > WiFi > mjpeg_streamer.

In the newer versions of the OpenMV IDE you must connect the board to your computer so see the examples.

You can copy and paste the script below to test this tutorial application.

Copy
1import sensor
2import time
3import network
4import socket
5
6SSID = "*********"  # Network SSID
7KEY = "***********"  # Network key
8HOST = ""  # Use first available interface
9PORT = 8080  # Arbitrary non-privileged port
10
11# Init wlan module and connect to network
12wlan = network.WLAN(network.STA_IF)
13wlan.active(True)
14wlan.connect(SSID, KEY)
15
16while not wlan.isconnected():
17    print('Trying to connect to "{:s}"...'.format(SSID))
18    time.sleep_ms(1000)
19
20# We should have a valid IP now via DHCP
21print("WiFi Connected ", wlan.ifconfig())
22
23# Create server socket
24s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
25s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
26
27# Bind and listen
28s.bind([HOST, PORT])
29s.listen(5)
30
31# Set server socket to blocking
32s.setblocking(True)
33
34# Init sensor
35sensor.reset()
36sensor.set_framesize(sensor.QVGA)
37sensor.set_pixformat(sensor.RGB565)
38
39def start_streaming(s):
40    print("Waiting for connections..")
41    client, addr = s.accept()
42    # set client socket timeout to 5s
43    client.settimeout(5.0)
44    print("Connected to " + addr[0] + ":" + str(addr[1]))
45
46    # Read request from client
47    data = client.recv(1024)
48    # Should parse client request here
49
50    # Send multipart header
51    client.sendall(
52        "HTTP/1.1 200 OK\r\n"
53        "Server: OpenMV\r\n"
54        "Content-Type: multipart/x-mixed-replace;boundary=openmv\r\n"
55        "Cache-Control: no-cache\r\n"
56        "Pragma: no-cache\r\n\r\n"
57    )
58
59    # FPS clock
60    clock = time.clock()
61
62    # Start streaming images
63    # NOTE: Disable IDE preview to increase streaming FPS.
64    while True:
65        clock.tick()  # Track elapsed milliseconds between snapshots().
66        frame = sensor.snapshot()
67        cframe = frame.compressed(quality=35)
68        header = (
69            "\r\n--openmv\r\n"
70            "Content-Type: image/jpeg\r\n"
71            "Content-Length:" + str(cframe.size()) + "\r\n\r\n"
72        )
73        client.sendall(header)
74        client.sendall(cframe)
75        print(clock.fps())
76
77
78while True:
79    try:
80        start_streaming(s)
81    except OSError as e:
82        print("socket error: ", e)
83        # sys.print_exception(e)

Now, let's briefly explain the main code sections of the example script from above.

1. Prepare the Script

Create a new script by clicking the "New File" button in the toolbar on the left side. Delete the example program that initially appears and import the required modules:

Copy
1import sensor   # Import the module for sensor related functions
2import time     # Import module for tracking elapsed time
3import network  # Import module for WiFi connectivity
4import socket   # Import module for sockets communication

2. Network Setup

Define your WiFi® credentials so the Nicla Vision can connect to the internet.

Copy
1SSID = "*********"  # Network SSID
2KEY = "************"  # Network key
3HOST = ""  # Use first available interface
4PORT = 8080  # Arbitrary non-privileged port

Set the

SSID

KEY

The

HOST

You can use

8080

PORT

The code section below searches for a WiFi® network with the supplied credentials and starts a connection process to it. Also, it creates a socket server for video streaming.

Copy
1# Init wlan module and connect to network
2wlan = network.WLAN(network.STA_IF)
3wlan.active(True)
4wlan.connect(SSID, KEY)
5
6while not wlan.isconnected():
7    print('Trying to connect to "{:s}"...'.format(SSID))
8    time.sleep_ms(1000)
9
10# We should have a valid IP now via DHCP
11print("WiFi Connected ", wlan.ifconfig())
12
13# Create server socket
14s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
15s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
16
17# Bind and listen
18s.bind([HOST, PORT])
19s.listen(5)
20
21# Set server socket to blocking (block the operations until completed)
22s.setblocking(True)

3. Preparing the Sensor

Copy
1# Camera setup and initialization
2sensor.reset() # Resets the sensor
3sensor.set_framesize(sensor.QVGA) # Sets the resolution to 320x240 px
4sensor.set_pixformat(sensor.RGB565) # Sets the sensor to RGB

The most relevant functions in this snipped are

set_pixformat

set_framesize

sensor.RGB565

The resolution of the camera needs to be set to a supported format both by the sensor and the algorithm.

QVGA

4. Streaming Function

The

start_streaming()

Eventually, it captures images continuously and sends them to the client compressed and in the right format.

Copy
1def start_streaming(s):
2    print("Waiting for connections..")
3    client, addr = s.accept()
4    # set client socket timeout to 5s
5    client.settimeout(5.0)
6    print("Connected to " + addr[0] + ":" + str(addr[1]))
7
8    # Read request from client
9    data = client.recv(1024)
10    # Should parse client request here
11
12    # Send multipart header
13    client.sendall(
14        "HTTP/1.1 200 OK\r\n"
15        "Server: OpenMV\r\n"
16        "Content-Type: multipart/x-mixed-replace;boundary=openmv\r\n"
17        "Cache-Control: no-cache\r\n"
18        "Pragma: no-cache\r\n\r\n"
19    )
20
21    # FPS clock
22    clock = time.clock()
23
24    # Start streaming images
25    # NOTE: Disable IDE preview to increase streaming FPS.
26    while True:
27        clock.tick()  # Track elapsed milliseconds between snapshots().
28        frame = sensor.snapshot()
29        cframe = frame.compressed(quality=35)
30        header = (
31            "\r\n--openmv\r\n"
32            "Content-Type: image/jpeg\r\n"
33            "Content-Length:" + str(cframe.size()) + "\r\n\r\n"
34        )
35        client.sendall(header)
36        client.sendall(cframe)
37        print(clock.fps())

5. Uploading the Script

Let's program the board with the complete script and test if everything works as expected. Copy the whole example code and paste it into the new script file that you created.

Open the OpenMV Serial Monitor by clicking on Serial Terminal in the lower left corner. Click the Play button at the bottom of the left toolbar. See the board connection progress in the terminal, once connected, the host address will be printed out.

The Nicla Vision IP address is the first one printed,

10.0.0.131

To watch the live stream, open your favorite web browser and enter the Nicla Vision IP address followed by the port, in this case,

10.0.0.131:8080

You can play with the camera resolution by changing it in the

set_framesize

Copy
1sensor.set_framesize(sensor.QVGA) # this is the example default resolution
2sensor.set_framesize(sensor.HD) # this is for HD resolution

You can see all the supported resolutions here to suit your application.

For example, the streaming below is in

HD

Conclusion

In this tutorial, you learned how to use the OpenMV IDE to develop MicroPython scripts that then run on the Nicla Vision. You also learned how to connect it to the network via WiFi®. Last but not least, you learned how to configure the Nicla Vision to be used as a surveillance camera thanks to OpenMV.

Next Steps

• Using a secure domain and a free DNS provider like CloudFlare you can access your live stream from outside your local network wherever you are.
• Familiarize yourself with the OpenMV IDE. There are many other features that were not mentioned in this tutorial (e.g. Machine Learning built-in functions).
• Learn more about the Nicla Vision on its User Manual.