Difference between revisions of "iMX6 OpenRex Environmental chamber testing"

Revision as of 09:49, 18 May 2021

On this page environmental stress testing results of iMX6 OpenRex SBC are displayed. Detailed instructions how to setup boards are also shown.

Hardware configuration

All of the boards used standard specification except of the temperature ranges for the key components. CPU with different frequency and increased operating range was selected for Basic version during the test. Details are described below:

4x iMX6 OpenRex SBC in Max configuration
- i.MX6 Quad 1.0 GHz CPU - automotive temperature range (-40°C to +125°C)
- 2GB DDR3 Memory (4x 4Gb DDR3 Memory chips) - industrial temperature range (-40°C to +95°C)
- 1GBps Ethernet PHY Tranciever KSZ9021RNI - automotive temperature range (-40°C to +85°C)

5x iMX6 OpenRex SBC in modified Basic configuration
- i.MX6 Solo 800 MHz CPU - industrial temperature range (-40°C to +105°C)
- 512MB DDR3 Memory (2x 2Gb DDR3 Memory chips) - industrial temperature range (-40°C to +95°C)
- 1GBps Ethernet PHY Tranciever KSZ9021RNI - automotive temperature range (-40°C to +85°C)

Two versions of heatsink were selected. Both of them covered area occupied by CPU, memories and Ethernet PHY chip. No big performance differences were observed. A standard 14x14x14mm sized heatsink shipped together with SBC after the purchase was tested as well.

The setup in the environmental chamber:

Test description

Three iMX6 OpenRex SBCs (one in Max and two in Basic configuration) were running full peripheral test to check functionality of all the peripherals in different range of temperatures. All the rest of the boards (three in Max and three in Basic version) were running CPU and memory tests to check the reliability of the firmware and hardware design. This included extensive examining of memory settings and used Fly-By Topology layout. Two DDR3 memory chips in Basic configuration were also thoroughly checked.

Testing threads are described below:

3x iMX6 OpenRex peripheral stress testing

1 thread of extensive Memory stress test
1 thread of CPU stress test
SATA stress test (applied for Max version only)
video playing via HDMI output
copying a file from the first USB drive to SD card and vice versa
copying a file from the second USB device to SATA hard drive and vice versa (applied for Max version)
copying a file from the second USB drive to SD card and vice versa (applied for Basic version)
pinging the host PC via Ethernet
all the messages were printed out on the serial console through connected FTDI cable
booting device:
- SATA hard drive in Max configuration
- USB flash drive for Basic version

3x iMX6 OpenRex Max Memory stress testing

4 threads of extensive Memory stress test
4 threads of CPU stress test
reading CPU temperature via SSH Ethernet session
serial FTDI cable used for displaying messages
firmware running from SD card

3x iMX6 OpenRex Basic Memory stress testing

1 threads of extensive Memory stress test
1 threads of CPU stress test
reading CPU temperature via SSH Ethernet session
serial FTDI cable used for displaying messages
firmware running from SD card

USB flash devices and SATA hard drives were place outside the environmental chamber. All the scripts running during the test and the board setup instructions can be found in section How to prepare the test.

Testing Results

The picture below shows the temperature profile during the whole testing process:
800px

Running the boards at -60°C and -40°C – PASS

Test description: CPU, Memory and peripheral stress tests were running at these extremely low temepratures. All the tested kits were working without errors even when temperature reached -60°C.

Note: Initial testing revealed that Ethernet PHY KSZ9031 created the network communication malfunction at very low temperatures. This behaviour was caused by chip manufacturer issue as described in Micrel errata. After changing PHY controller to KSZ9021RN all the boards were working without errors.

Running the boards at +65°C – PASS

Test description: The ambient temperature was gradually increasing (to +55°C, +60°C and +65°C) and we were running the CPU, Memory and peripheral stress tests. Even at +65°C processors were running at full speed (frequency was not lowered by the system to cool down the CPU). Reaching higher temperature is possible by using a bigger and more massive heatsink.

The boards were also tested with a small heatsink which is a part of the webshop iMX6 OpenRex SBC kit. This setup also reached temperatures above +65°C without any errors.

Temperature readout from boards when running at +65°C shows that the temperature inside the CPU was approximately 20°C to 30°C higher. Heatsink design influences this difference the most.

800px

Switch ON/OFF test – PASS

Test description: At temperatures -30°C, -35°C, -40°C and -45°C the boards were switched OFF, left OFF for at least 5 minutes (to cool down completely) and then switched ON to see if they boot up without any problem. Once booted up into Linux, the boards were turned off again. All of the tested boards booted up successfully. Tested at least one time for every temperature.

Temperature stress test – quick change from -60°C to -10°C and back – PASS

Test description: The ambient temperature was quickly increased from -60°C to -10°C and then back to -60°C. The boards were running CPU, Memory and peripheral stress tests. All the boards were working correctly.

Temperature stress test – quick change from +65°C to +40°C and back – PASS

Test description: The ambient temperature was quickly decreased from +65°C to +40°C and back to +65°C while running CPU, Memory and peripheral stress tests. All boards were working without errors.

Boards were tested in a very rugged environment (ice flaked forming on the rear window):

Setup preparation and testing recommendations

It is very important to run the exact same setup for environmental chamber testing in the lab for couple of days. Use exactly the same boards, board setup, cables, PC and all other equipment. The main reason to do this is to minimize possible issues with software, firmware or connections – everything which can cause problem apart from hardware itself. There will not be much time to fix there issues during the testing.

The next very important thing is to make as many notes as possible during the testing. Always add board number and date to the message. This will help to memorize everything and no important detail will be missed in the report.

To get reliable results, running 4 boards is advised at minimum, 6-10 boards is recommended. This amount of boards, cables and equipment require a special precautions to avoid confusion. Thus marking all the boards and important cables is a very useful habit.

These are the most crucial hardware parts, which were marked prior the testing:

Boards: place a sticker on a visible spot (e.g. USB or Ethernet connector). Use simple numbering (e.g. 1, 2, 3…)
Serial consoles: mark both ends of the cable
Adapters or power cables: mark both ends of the cable so resetting a single board becomes much easier
USB memories and hard drives: if there is a problem with e.g. USB flash drive, it can be time saving to know which one to replace (marking used 1-USB1, 1-USB2, 2-USB1…)

Following the same pattern, software and connections was number in similar fashion:

IP addresses: having the same address makes examining possible problems much easier. DHCP server which assigns the same IP address for one particular MAC address was used during the testing (e.g. board with mark 1 will always have IP address 168.192.0.201)
- Be aware: In some situation DHCP server can assign a different IP address. The SSH connection will not be responsible but Ethernet would still be working fine. Check LED status of the router and DHCP server application in this case.
Name of the log files: differentiate between the log files (this is already done in the testing script)
Serial console ports: plug them the way that system will assign the same number as the board number is (e.g. in Linux control PC use directory /dev/ttyUSB1 as a serial port for board number 1)
Visual arrangement of serial and SSH consoles: move them the way the boards are placed inside the chamber

PC setup

During the test DHCP and FTP servers were utilized. The PC was also used during Ethernet ping test while controlling all the boards through SSH and serial console sessions. The control computer was running Windows 7 operating system.

Setting the network

Disable firewall (PC will not be connected to the Internet) and setup a static IP address: Press Windows button -> go to Control Panel -> Network and Internet -> Network and Sharing Center -> Change adapter settings (on the right side in the bar) -> double click on Local Area Connection -> Properties -> In the tab Networking go to Internet Protocol Version 4 (TCP/IPv4) -> type the static IP address and subnet mask as shown below:
500px

Enabling sharing

iMX6 OpenRex SBCs and the control PC are connected via Gigabit Ethernet switch. To ensure correct IP address assignment, turn on the PC first, connect it with the switch and only after that turn the switch on. To be able to download the files from the FTP server, sharing option need to be enabled first. This can be done by setting up the network as a work network. Sharing can be enabled in Control Panel -> Network and Internet -> Network and Sharing Center -> Click to Choose homegroup and sharing options:
800px

Setting up the TFTP and DHCP servers

Tftpd32 software supports both TFTP and DHCP server options. For more details about DHCP configuration, follow the screenshots below:

356px

419px

Starting time server

To assign the correct time for all the boards Time server feature built in Tftpd32 was used. Go to Settings and make sure that SNTP server is checked:
|419px

Assigning custom COM port names

As we mentioned before, it is useful to apply the same numbering for boards and serial ports. To do so plug in the first FTDI cable and go to the control PC, select Start, right click on Computer and select Properties. Now click on the Device Manager in the top right corner. Under the item Ports (COM & LPT), USB ports are visible. Now right click and select Properties:
|500px

Select the Port Settings tab and click on Advanced tab:
|419px

Now select from the COM Port Number (the one which is not in use):
800px

Now this process can be repeated by plugging another serial cable if needed. After all the cables were attached, restart the computer.

SSH and Serial consoles

To control iMX6 OpenRex SBCs, one TeraTerm serial console was opened per board. SSH clients were used to read temperature. For the boards with the peripheral test, an extra SSH client was established to be able to spot possible errors or warnings if occurred.
800px

Preparing the test

Boot device and software

All the boards were using bootloader stored in SPI flash memory with the rest of firmware located on SD card. Only exception was moving the filesystem to either SATA disk (Max version) or USB flash drive (Basic configuration) for peripheral tests.

The only change compared to standard software package was running a multimedia filesystem. U-Boot settings were not adjusted as the default configuration was used. To prepare a fresh SD card follow these instructions. Here is an example of creating a SD card suitable for Max configuration:

mkdir -pv ~/workdir/imx6/rootfs/yocto
cd ~/workdir/imx6/rootfs/yocto
wget http://downloads.voipac.com/files/iMX6_OpenRex_SBC/software/yocto/binaries/fsl-image-gui-imx6-openrexmax-2.4.sdcard.gz
gunzip -c fsl-image-gui-imx6-openrexmax-2.4.sdcard.gz | sudo dd of=/dev/mmcblk0 conv=sync bs=64K

Creating new directories and copying test files

cd /media
wget http://downloads.voipac.com/files/iMX6_OpenRex_SBC/documents/Environmental_chamber_testing/
cp blackbird.wav blackbird2.wav
# prepare directories
cd /home/ubuntu/
mkdir -p openrex/
cd openrex/
mkdir -p env-chamber-testing/
cd env-chamber-testing/
wget http://downloads.voipac.com/files/iMX6_OpenRex_SBC/documents/Environmental_chamber_testing/stressapptest
touch orx-env-chamber-testing.log
touch orx-cpu-temp.log
# setup local time
ln -sf /usr/share/zoneinfo/Europe/Bratislava /etc/localtime

Setup the cron

To be able to check the CPU temperature every minute a new cron job was setup. For Ubuntu file systems postfix package installation is required:

sudo apt-get install postfix

To setup a cron tab, the file containing all cron jobs needs to be altered. Before running the crontab for the first time, selecting a text editor is required:

sudo crontab -e

Paste the following line to the end of the cron list file. The output time format will be set the same as for stressapptest. Cron job performs a current CPU temperature readout and the data are saved in the log file. The log format was selected identical to the one used by stressapptest package:

* * * * * { echo -n $(date +%Y/%m/%d-%T); echo -n "("; echo -n $(date +%Z); echo -n ") "; cat /sys/devices/virtual/thermal/thermal_zone0/temp; } >> /home/ubuntu/openrex/env-chamber-testing/orx-cpu-temp.log

Note: The percent sign '%' has a special purpose in the cron file. As this sign is needed to write the date formats, the backslash is required to be placed before the percent sign: '\%'.

Setup the CPU temperature printout

Open one SSH client per board where CPU temperature will be shown. Tail command is utilized to display the last part of the log file, which is filled up as the CRON job outputs the current readouts:

cd /home/ubuntu/openrex/env-chamber-testing/
tail -f orx-cpu-temp.log

Error checking during peripheral test

To make sure tests are running smoothly, potential errors can be detected by opening another SSH session:

cd /home/ubuntu/openrex/env-chamber-testing/
tail -f orx-env-chamber-testing.log | grep -i "error"

Copying the script file

Before the script is run, the correct permission needs to be set up:

cd /home/ubuntu/openrex/env-chamber-testing/
wget http://downloads.voipac.com/files/iMX6_OpenRex_SBC/documents/Environmental_chamber_testing/openrex-env-chamber-test.sh
chmod 777 openrex-env-chamber-test.sh

Preparing backup script (memory test only)

To store the older files automatically, following script can be used:

#!/bin/sh
 
# obtain board ID from IP address - be sure addresses are allocated based on MAC
boardID=$(/sbin/ip -o -4 addr list eth0 | awk '{print $4}' | cut -d/ -f1 | cut -d'.' -f4 | cut -d'2' -f2);
currentTime=`date +%Y-%m-%d.%H:%M`
   
mv stressapptest.log orx-board-$boardID-stressapptest.log.$currentTime
mv orx-cpu-temp.log orx-board-$boardID-env-cpu-temp.log.$currentTime
 
touch stressapptest.log
touch orx-cpu-temp.log

This script requires permission update as well:

cd /home/ubuntu/openrex/env-chamber-testing/
wget http://www.imx6rex.com/download/software/openrex/others/files/backupFiles.sh
chmod 777 backupFiles.sh

Setting time and date

Before the test, server date and time is automatically obtained from SNTP server:

ntpdate 192.168.0.2

Start CPU and memory stress test

Navigate into the directory, where stressapptest feature and test are stored.

Stress command for Max configuration:

./backupFiles.sh & stressapptest -s 600000 -M 1000 -m 4 -C 4 -W -l /home/ubuntu/openrex/env-chamber-testing/stressapptest.log --printsec 100

Stress command for Basic configuration:

./backupFiles.sh & stressapptest -s 600000 -M 150 -m 1 -C 1 -W -l /home/ubuntu/openrex/env-chamber-testing/stressapptest.log --printsec 100

Start peripheral test

Before the test start it is a good practice to make sure that all the devices were plugged in successfully. To find out where the devices were mounted following command can be used:

fdisk -l

The testing script uses multiple variables to specify its operation. As the first parameter board configuration is used (-max or -basic). As other parameters device names are written in the following order: the first USB stick, the second USB, SD card and SATA drive (if used). All logs are stored into a single file.

Starting peripheral test for Max configuration:

./openrex-env-chamber-test.sh -max sdb1 sdc1 mmcblk1p2 sda1 2>&1 | tee -i orx-env-chamber-testing.log

Starting peripheral test for Basic configuration:

./openrex-env-chamber-test.sh -basic sda1 sdb1 mmcblk1p2 2>&1 | tee -i orx-env-chamber-testing.log

The complete script can be found in the download section or down below:

#!/bin/sh
 
# OpenRex environmental chamber testing script
 
# Example QUAD: ./openrex-env-chamber-test.sh -quad sdb1 sdc1 mmcblk0p1 sda1 2>&1 | tee -i orx-env-chamber-testing.log
# Example SOLO: ./openrex-env-chamber-test.sh -solo sda1 sdb1 mmcblk0p1 2>&1 | tee -i orx-env-chamber-testing.log
 
mountDevice() {
  mount /dev/$1 /media/$2
  cat /etc/mtab | grep -F "/dev/$1 /media/$2"
  if [ "$?" -eq "0" ]; then
    echo "$2 mounted"
  else
    echo "$2 not mounted"; exit 2
  fi
}
 
# prepare files if needed
touch orx-env-chamber-testing.log
touch orx-cpu-temp.log
 
echo -e '\033[9;0]' > /dev/tty1 # disable blanking the display
 
imx6solo=0
imx6quad=0
case $1 in
  -basic)  imx6solo=1 ;;
  -max)  imx6quad=1 ;;
  *)
esac
 
# mount devices
mountDevice $2 usb0
mountDevice $3 usb1
mountDevice $4 mmc0
if [ "${imx6quad}" -eq "1" ]; then
  mountDevice $5 sata
fi
 
updateLogFiles() {
  # obtain board ID from IP address - be sure addresses are allocated based on MAC
  boardID=$(/sbin/ip -o -4 addr list eth0 | awk '{print $4}' | cut -d/ -f1 | cut -d'.' -f4 | cut -d'2' -f2);
  # be sure time server is running on DHCP server
  currentTime=`date +%Y-%m-%d.%H:%M`
 
  mv orx-env-chamber-testing.log orx-board-$boardID-env-chamber.log.$currentTime
  mv orx-cpu-temp.log orx-board-$boardID-env-cpu-temp.log.$currentTime
 
  touch orx-env-chamber-testing.log
  touch orx-cpu-temp.log
}
 
finish_test_now() {
  echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) Ctrl+C Detected: End of the test"
  precced=0;
  #kill -INT $vid_pid $str_pid;
  sleep 3;
  test_status=`cat orx-env-chamber-testing.log | grep -i "error" | grep -v -e "0 errors" -e "no corrected errors"`
  if [ -z "$test_status" ]
  then
    echo "*********\nTEST PASS\n*********"
  else
    echo "*********\nTEST FAIL\n*********\n"
    echo "List of detected errors:"
    cat orx-env-chamber-testing.log | grep -i "error" | grep -v -e "0 errors" -e "no corrected errors" -e "List of detected errors:"
  fi
  updateLogFiles
  exit;
}
 
# kill all processes if Ctrl+C is detected
trap finish_test_now 2
 
# stressapptest - one thread CPU, one thread memory
if [ "${imx6solo}" -eq "1" ]; then
  stressapptest -s 600000 -M 100 -m 1 -C 1 --printsec 10 &
  str_pid=$!
fi
if [ "${imx6quad}" -eq "1" ]; then
  stressapptest -f /media/sata/tmp-file1 -s 600000 -M 500 -m 1 -C 1 --printsec 10 &
  str_pid=$!
fi
echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) Starting stressapptest with PID: " $str_pid
 
# play a video file in an infinite loop
echo 0 > /sys/class/graphics/fb0/blank
mplayer -vo fbdev2:/dev/fb0 -quiet -vf scale -zoom -loop 0 -xy 1024 /home/ubuntu/Clouds.avi < /dev/null &
vid_pid=$!
echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) Starting video file playback with PID: " $vid_pid
 
proceed=1
file1="blackbird.wav"
file2="blackbird2.wav"
 
cp1_from="/media/mmc0/"
cp1_to="/media/usb0/"
 
cp2_from="/home/ubuntu/"
cp2_to="/media/usb1/"
 
#copy files in case they are missing
cp /media/$file1 $cp1_from$file1
cp /media/$file1 $cp1_to$file1
cp /media/$file2 $cp2_from$file2
cp /media/$file2 $cp2_to$file2
 
while [ $proceed -eq 1 ]
do
 
  ping -q -c1 192.168.0.2 >> orx-env-chamber-testing.log
  if [ $? -ne 0 ]
  then
    echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) ERROR: Ping failed"
  fi
 
  cp1_done=`ps cax | grep $cp1_pid | grep cp`
  if [ -z "$cp1_done" ]; then # copy finished
    if cmp -s $cp1_from$file1 $cp1_to$file1; then
      echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) PASS: Copying file from $cp1_from to $cp1_to successful"
    else
      echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) ERROR: Difference between files on $cp1_from and $cp1_to detected"
      cp /media/$file1 $cp1_to$file1
      echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) File copied from /media/ to $cp1_to"
    fi
    cp1_temp=$cp1_from # swap destinations
    cp1_from=$cp1_to
    cp1_to=$cp1_temp
 
    rm $cp1_to$file1 # remove destination file
 
    cp $cp1_from$file1 $cp1_to$file1 &
    cp1_pid=$!
    echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) Started copying file from $cp1_from to $cp1_to"
  fi
 
  cp2_done=`ps cax | grep $cp2_pid | grep cp`
  if [ -z "$cp2_done" ]; then # copy finished
    if cmp -s $cp2_from$file2 $cp2_to$file2; then
      echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) PASS: Copying file from $cp2_from to $cp2_to successful"
    else
      echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) ERROR: Difference between files on $cp2_from and $cp2_to detected"
      cp /media/$file2 $cp2_to$file2
      echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z) File copied from /media/ to $cp2_to"
    fi
    cp2_temp=$cp2_from # swap destinations
    cp2_from=$cp2_to
    cp2_to=$cp2_temp
 
    rm $cp2_to$file2 # remove destination file
 
    cp $cp2_from$file2 $cp2_to$file2 &
    cp2_pid=$!
    echo "$(date +\%Y/\%m/\%d-\%T)($(date +\%Z)) Started copying file from $cp2_from to $cp2_to"
  fi
 
done

The original article can be accessed on the iMX6Rex.com website.

@@ Line 21: / Line 21: @@
 The setup in the environmental chamber:<br />
-[[image:iMX6_OpenRex_SBC-Environmental_chamber-Boards.jpg|800px]]
+[[image:iMX6_OpenRex_SBC-Environmental_chamber-Boards.jpg|700px]]
 == Test description ==