Privacy PolicyLegal Disclosure
New

From Image to JSON: How to Automatically Convert Meal Plans with Python, OpenCV, and Tesseract

May 2, 2023

My vision was to have the meal of the day on my Apple Watch, but the meal plan is only available as a picture and not as structured data.

Picture of a wrist wearing an Apple Watch: The watch shows the meal for tomorrow

Starting point

Each week, an image of the meal plan gets posted on a Confluence* page. I download them with a python script, so that I can process it using OCR.

* Confluence is a wiki software like Wikipedia, but for your company.

Images of the meal plan look like this (including the strange white borders!):

Tabular meal plan

Each week the image is slightly different:

  • sometimes it has white borders, sometimes it doesn’t
  • the resolution is slightly different
  • the font is not always the same
  • rows and columns are not always the same size, rather determined by the text within

Glossary

I’ll use the following terms throughout the article:

  • OCR: Optical Character Recognition, the process of converting images of text to text
  • Tesseract: A popular open-source OCR engine
  • OpenCV: Open Source Computer Vision Library, a library for image processing
  • Python: A programming language

Preprocessing

To account for the image variances, we need to do some preprocessing.

  • upscale the image
  • find contours
  • find rects
  • use biggest rect as new smallest area

We can do that with this script:

# upscale, the factor upscales the image to ~300 dpi
scale = 4.2
scaled = cv2.resize(img, None, fx=scale, fy=scale,
                    interpolation=cv2.INTER_CUBIC)

# convert image to grayscale
grayscale = cv2.cvtColor(scaled, cv2.COLOR_BGR2GRAY)

# convert to binary
(_, tableCellContrast) = cv2.threshold(
    ~grayscale, 5, 255, cv2.THRESH_BINARY)

# find edges
edges = cv2.Canny(tableCellContrast, 5, 10)

The edges image looks like this:

Contours of the table and text

We can use the biggest rectangle as the bounds to crop the image:

contours, _ = cv2.findContours(
    edges, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE
)[-2:]

biggest_contour = None
max_area = 0

for contour in contours:
    area = cv2.contourArea(contour)
    if area > max_area:
        max_area = area
        biggest_contour = contour

x, y, w, h = cv2.boundingRect(biggest_contour)
cropped = scaled[y:y+h, x:x+w]

After these steps, the result is this:

Cropped meal plan

Extracting cells

Using the same tricks as above, we can use contours and lines to find the cells in the table. I created a visualization to show you the bounds of the cells the algorithm found:

Meal plan with each table cell in a different color

The code for this one is a little bit longer, and contains some quirks to get the cells reliably. If you still want to see it, you can expand the next section:

Code block: Extracting cells
grayscale = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
(_, tableCellContrast) = cv2.threshold(
  ~grayscale, 5, 255, cv2.THRESH_BINARY)

# start lines
imgLines = cv2.cvtColor(np.zeros_like(
  tableCellContrast), cv2.COLOR_GRAY2RGB)

imgwidth, imgheight = img.shape[1], img.shape[0]
minLineLength = imgwidth // 10
lines = cv2.HoughLinesP(
  image=~tableCellContrast,
  rho=0.02,
  theta=np.pi / 500,
  threshold=10,
  lines=np.array([]),
  minLineLength=minLineLength,
  maxLineGap=10,
)

a, _, _ = lines.shape
for i in range(a):
  if abs(lines[i][0][0] - lines[i][0][2]) > abs(lines[i][0][1] - lines[i][0][3]):
      # horizontal line
      cv2.line(
          imgLines,
          (0, lines[i][0][1]),
          (imgwidth, lines[i][0][3]),
          (0, 0, 255),
          1,
          cv2.LINE_AA,
      )
  else:
      # vertical line
      cv2.line(
          imgLines,
          (lines[i][0][0], 0),
          (lines[i][0][2], imgheight),
          (0, 0, 255),
          1,
          cv2.LINE_AA,
      )

cv2.rectangle(imgLines, (0, 0), (imgwidth, imgheight), (0, 255, 0), 2)
(thresh, table) = cv2.threshold(
  cv2.cvtColor(imgLines, cv2.COLOR_BGR2GRAY),
  128,
  255,
  cv2.THRESH_BINARY | cv2.THRESH_OTSU,
)
# end lines

# start contours
img_boxes = cv2.cvtColor(grayscale, cv2.COLOR_GRAY2RGB)

# create new cv image with same dimensions as cropped image
contours, hierarchy = cv2.findContours(
  table, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE
)[-2:]
idx = 0

cells = []

for cnt in contours:
  idx += 1
  x, y, w, h = cv2.boundingRect(cnt)
  area = w * h

  if w > imgwidth * 0.9 or h > imgheight * 0.9 or w < 10 or h < 10:
      continue

  roi = grayscale[y: y + h, x: x + w]

  cells.append(Cell(roi, x, y, w, h, None))

  color = list(np.random.random(size=3) * 256)
  cv2.rectangle(img_boxes, (x, y), (x + w, y + h),
                  color, thickness=FILLED)

# end contours
cols = list(sorted(set([cell.x for cell in cells])))
rows = list(sorted(set([cell.y for cell in cells])))

for cell in cells:
  x, y, w, h = cell.x, cell.y, cell.w, cell.h

  cell.y = rows.index(y)
  cell.x = cols.index(x)

Cell is a small dataclass that looks like this:

from dataclasses import dataclass
import numpy as np

@dataclass
class Cell:
    image: np.ndarray
    x: int
    y: int
    w: int
    h: int
    text: str

Preparing for OCR

OCR works best on high-contrast images that might look strange to humans but are easy to work with by computers.

To create this image, we’ll use dilation and erosion to remove artifacts from the letters:

Read more about these operations: docs.opencv.org

grayscale = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

img = grayscale

kernel = np.ones((1, 1), np.uint8)
img = cv2.dilate(img, kernel, iterations=1)
img = cv2.erode(img, kernel, iterations=1)

img = cv2.threshold(cv2.medianBlur(img, 3), 0, 255,
                    cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

This is what the image looks like now:

Meal plan, just pure black and pure white colors, with text looking pixelated

OCR and data transformation

Now we have an image to work with. We already have the cell information, so we can run ocr on each cell instead of the whole image to know exactly which part of the image contains what data. Otherwise, it would be extremely painful to assign the ocr data to the information we need.

custom_config = r"--oem 1 --psm 11 -l deu -c tessedit_write_images=true "

for cell in cells:
  if not cell.text or forceExtract:
      text = pytesseract.image_to_string(
          cell.image, config=custom_config, lang="frak"
      )

      # remove line breaks and form feed chars
      cell.text = text.replace("\n", " ").replace("\f", "").strip()

# sort cells by their coordinates
cells.sort(key=lambda cell: (cell.y, cell.x))

With the cells in the right order, we can transform the data to a dictionary:

table = {}

germanWeekdayToEnglish = {
    "Montag": 0,
    "Dienstag": 1,
    "Mittwoch": 2,
    "Donnerstag": 3,
    "Freitag": 4,
    "Samstag": 5,
    "Sonntag": 6,
}

for cell in [c for c in cells if c.x == 0]:
    table[cell.y] = [c.text for c in cells if c.y == cell.y]

categories = list(table[0])[1:]

days = {}
for y in range(1, len(table), 2):
    weekday = germanWeekdayToEnglish[table[y][0]]

    meals = [
        {
            "category": trim(categories[i]),
            "meal": trim(meal),
            "ingredients": [trim(x) for x in table[y + 1][i + 1].split(",")],
        }
        for i, meal in enumerate(table[y][1:])
    ]

    days[weekday] = meals

The json file for each weekday looks like this:

[
  {
    "category": "Hauptgericht 1",
    "meal": "Gebratene Streifen vom Rind mit Gemüse, abgerundet mit Sojasauce und Honig, dazu Risi-Bisi",
    "ingredients": ["F", "G", "I"]
  },
  {
    "category": "Veganes Hauptgericht",
    "meal": "Spinatlasagne mit getrockneten Tomaten und veganem Käse überbacken und Tomatensauce extra",
    "ingredients": ["A (Weizen)"]
  }
]

REST API

My REST API is extremely simple. I just copy over the jsons to a webserver that hosts these jsons as static files.

Usage works like this: Do a GET-Request for a date like this: GET /2022-05-03.json

Using the data

I’m using the API to show the current meal on my Apple Watch. After 1 p.m., it shows the meal for the next day.

 
Continue reading
 

DIY Ambilight with HDMI 2.1 support

May 2, 2023
Create an ambilight for a comfortable and immersive visual experience using a Raspberry Pi, WLED, and HyperHDR.
 
Continue reading

Save songs from radio to a Spotify playlist

Oct 22, 2020
I like to listen to the radio, and sometimes I want to save the songs I hear there to a Spotify playlist. That process was kind of tedious, so I thought I automate that process.
 
Continue reading

Let Alexa welcome you home

Jun 26, 2020
Using Home Assistant and Node-RED, I created an automation that not only greets me when I come home but also tells me who is home and who is away (and how long).
 
Continue reading

Save and restore light states using Home Assistant

Dec 27, 2019
Use Home Assistant scenes to save and restore the light state in a room
 
Continue reading

Alexa connected to an audio receiver (with Home Assistant)

Jul 16, 2019
Play music on your audio receiver but let Alexa talk through it's internal speaker.
 
Continue reading

Monitor remaining water bottles with ESPHome and Home Assistant

May 7, 2019
I built a scale which sits under the water bottles crate. It measures how many bottles are left and sends a notification when I'm running low.
 
Continue reading

Cover that only opens if there's nothing in it's way in Home Assistant

Mar 8, 2019
I have a screen that goes above my balcony door. If the door has been left open and I accidentally said "Alexa, turn on cinema mode" from downstairs, my screen probably wouldn't be as smooth anymore.
 
Continue reading

Install vaultwarden (formerly bitwarden-rs) on uberspace

Jan 7, 2019

The guide below is outdated. Please use the guide from UberLab instead, which was based on this blog post:

https://lab.uberspace.de/guide_vaultwarden/

Bitwarden is a great open source password manager. Your vault is encrypted with your master key, so even if someone hacks into the Bitwarden Servers (which are hosted on Microsoft Azure), they will only get some unreadable gibberish. If your master password is strong, you should be save.

If you are paranoid about the server security and want to be in full control, or want the premium features for free because you have a webspace anyway, you can self-host Bitwarden.

Bitwarden provides docker containers, but they are big and difficult to install. Uberspace is a web hoster for command line enthusiasts, and while it supports nearly everything, docker isn’t.

In this tutorial, we will use a Rust implementation of the bitwarden api. You can check the project out on GitHub: https://github.com/dani-garcia/bitwarden_rs

Prerequisites

  • Uberspace 7
  • Basic understanding of the command line (the command begins *after *the $ sign)
  • A subdomain configured correctly (see here), e.g. vault.yourdomain.com

Installing Rust

To compile the project, we need to install the rust toolchain.

install via rustup: ~$ curl https://sh.rustup.rs -sSf | sh

press 2 to customize the installation. You can press enter for the host triple to use the default one. When asked for the toolchain, type nightly, as this is required for bitwarden-rs. Add rust to the PATH by pressing y.

Then, proceed with the installation.

To finish the setup, logout and login again or run ~$ source $HOME/.cargo/env.

Install Bitwarden-rs

clone the project: ~$ git clone https://github.com/dani-garcia/bitwarden_rs.git

to build bitwarden-rs, you’ll need to set an environment variable pointing to the sqlite3 header files: ~$ export SQLITE3_LIB_DIR=/var/lib64

cd into the project: ~$ cd bitwarden_rs

build the server executable: ~/bitwarden_rs $ cargo build --release --features sqlite

if that doesn’t work the first time, just try again.

now, we will have to download the newest build (check this page for the newest build number and replace it in the following snippet: https://github.com/dani-garcia/bw_web_builds/releases):

~/bitwarden_rs $ mkdir web-vault && cd web-vault
~/bitwarden_rs/web-vault $ wget https://github.com/dani-garcia/bw_web_builds/releases/download/v2.11.0/bw_web_v2.11.0.tar.gz
~/bitwarden_rs/web-vault $ tar -xvzf bw_web_v2.11.0.tar.gz

After that, go back to the project folder: ~/bitwarden_rs/web-vault $ cd ..

We need to add a .env-file.

~/bitwarden_rs $ nano .env

add this:

ADMIN_TOKEN=CHuPAsoYgykByUpqVrjRYG/MeYO+jdnmZskgTsBa9kj2MnP7QrQ0GelJ7Lqixph8 # generate one with ~$ openssl rand -base64 48 ROCKET_PORT=62714 # your port here

SMTP_HOST=yourhost.uberspace.de SMTP_FROM=[email protected] SMTP_PORT=587 SMTP_SSL=true SMTP_USERNAME=[email protected] SMTP_PASSWORD=yourpassword

SMTP_USERNAME and SMTP_PASSWORD must be the login data from a valid uberspace mail account (SMTP_FROM must be correct too). You can also use a mail account from another service, like GMail. Alter the values like the port accordingly.

Press CTRL+O to save, and CTRL+X to exit.

You can edit other options, look into .env.template to see a list of available options.

Now, we just have to add a redirection to the port:

~/bitwarden_rs $ uberspace web backend set / —http —port 62714

If you want to use a subdomain, read more about web backends in the uberspace wiki: https://manual.uberspace.de/web-backends.html#specific-domain

Now it’s time to test if everything works: ~/bitwarden_rs $ target/release/bitwarden_rs

If there is no error, you are good to go. You should be able to access your vault on yourdomain.com.

Auto start and run in background

We will use supervisord to run the server and automatically restart it on crash.

Create a new file for your service: ~$ touch ~/etc/services.d/bitwarden_rs.ini with the following content:

[program:bitwarden_rs] directory=/home/YOURUSERNAME/bitwarden_rs command=/home/YOURUSERNAME/bitwarden_rs/target/release/bitwarden_rs autostart=yes autorestart=yes

Add the service to supervisor:

~ supervisorctl update ~$ supervisorctl start bitwarden_rs

Now the server should be running again.

Updating

Updating bitwarden is really easy. Just stop the server, pull everything and download the new web vault, build the executable and start the server again:

~/bitwarden_rs $ supervisorctl stop bitwarden_rs
~/bitwarden_rs $ git pull
~/bitwarden_rs $ mv web-vault web-vault.old && mkdir web-vault && cd web-vault
~/bitwarden_rs/web-vault $ wget new-release.tar.gz
~/bitwarden_rs/web-vault $ tar -xvzf new-release.tar.gz
~/bitwarden_rs/web-vault $ cd ..
~/bitwarden_rs $ cargo build --release
~/bitwarden_rs $ supervisorctl start bitwarden_rs
 
Continue reading
 

Making the windows smart

Dec 13, 2018
We will add magnet contacts to every window and wire them all to an ESP8266. Then, we flash a sketch with esphomeyaml to connect everything to home assistant.
 
Continue reading

Building a smart home theater with Home Assistant

Nov 15, 2018
I wanted to control the screen I bought for my projector with Home Assistant, so I replaced the controller with a Sonoff Dual and used esphomeyaml to create the program.
 
Continue reading
Mastodon@[email protected]
GitHub@vigonotion
  • © 2023 · vigonotion
  •  
  • Privacy Policy
  • Legal Disclosure
  • Colophon