Extract receipt data with Mindee’s API using reactjs

 

In this tutorial, you'll learn how to create an optimized and interactive user interface for automatic receipt data extraction using ReactJS and Mindee. We are going to use both Mindee API and open source ReactJS SDK. The SDK has a lot of features built-in such as shapes visualisation and interactions, zoom, multi pages ... It will help you create any dynamic user experience based on the receipt image to help your users validate very quickly the results predicted by the API.

 

In 10 minutes, you'll get something like this:

 

Mindee receipt parsing react js

 

Let's get started!

 

 

API Prerequisites

 

  1. We are using create-react-app framework for bootstrapping the app. If you don't have it installed, you can do it here. You can also use another React framework for creating the app but some code may differ from what is written in this tutorial.
  2. You’ll need a free Mindee account. Sign up and confirm your email to login.
  3. A receipt.  Look in your bag/wallet for a recent one, or do a Google Image search for a receipt and download a few to test with.

 

 

Setup your project

 

First, we need to create a new react application using create-react-app. Open up your terminal and go to the directory you want to create your project in, and create your app with the name you want (we call ours receipt_parsing):

 

npx create-react-app receipt_parsing

 

To make sure everything went well, go to your directory and run the app

 

cd receipt_parsing
yarn start

 

Or using npm

 

cd receipt_parsing
npm run start

 

Your browser is going to open a new window and run the app.

 

A last package install before we can start coding! As we are going to use Mindee ReactJS SDK, we need to install it. It's open-source and you can find all the documentation on the Github repo. To install in your React app, just run in your terminal:

 

yarn add react-mindee-js

 

Or using npm:

 

npm install react-mindee-js

 

Now, using your favourite IDE, open the App.js file that was created by create-react-app, inside the src folder of your project directory. In the whole tutorial, we are going to work only on this file. Once you access the App.js file in your terminal, delete everything and just paste this code:

 

import React, {useState} from 'react';

import {
  AnnotationViewer,
  createPolygonFromCoordinates
} from "react-mindee-js";

const HelloWorld = () => {
  const [image, setImage] = useState(null)

  const shapes = [
    {
      polygon: createPolygonFromCoordinates([[0.66, 0.3], [0.12, 0.5], [0.52, 0.8], [0.82, 0.24]]),
      color: "#fd3246",
      data:"This is red shape",
      index: 0,
      active: {
        color: '#00f0f0',
        lineWidth: 3
      }
    }
  ]

  return (
    <div>
      <input type="file" onChange={e => setImage(URL.createObjectURL(e.target.files[0]))}/>
      {
        image && <div style={{ height: '500px', width: '400px'}}>
          <AnnotationViewer
            shapes={shapes}
            image={image}
            onShapeClick={shape => console.log('shapeClicked', shape)}
            onShapeHover={shape => console.log('shapeHovered', shape)}
          />
        </div>
      }
    </div>
  );
}
export default HelloWorld;

 

We define a polygon by its [x,y] coordinates and colors.  When an image is added from the form, we call the onChange to apply the polygon, and write to the console on Click and Hover actions.

You're all setup to start. When you run the app, you should see a file input button on your web page, and once you upload an image using it, you get something like:

 

The ploygon is defined but the four [x,y] vertices, and the color is red, but on hover, it turns blue (00f0f0).

 

 

Parse the receipt

 

Log onto the platform and get into your Expense receipt API environment by clicking the following card.

Expense receipt API card

 

If you don’t have an API token yet for this API, go to the “credentials” section and create a new API Key. 

 

new api token

 

Inside your App.js file, replace everything with this code, and replace the 'my-token-here' placeholder with the one you just created on the platform:

 

import React, {useState} from 'react';
import {
  AnnotationViewer,
  formatPrediction
} from "react-mindee-js";

const HelloWorld = () => {
  const [image, setImage] = useState(null)
  const [shapes, setShapes] = useState([])

  const parseReceipt = (e) => {
    let myFile = e.target.files[0]

    if (!myFile) {return}
    
    setImage(URL.createObjectURL(myFile))

    let data = new FormData();
    data.append("file", myFile, myFile.name);

    let xhr = new XMLHttpRequest();

    xhr.addEventListener("readystatechange", function () {
      if (this.readyState === 4) {
        let json_response = JSON.parse(this.responseText)
        setShapes(formatPrediction(json_response.predictions[0]))
      }
    });

    xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
    xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
    xhr.send(data);
  }

  return (
    <div>
      <input type="file" onChange={e => parseReceipt(e)}/>
      {
        image && <div style={{ height: '500px', width: '400px'}}>
          <AnnotationViewer
            shapes={shapes}
            image={image}
            onShapeClick={shape => console.log('shapeClicked', shape)}
            onShapeHover={shape => console.log('shapeHovered', shape)}
          />
        </div>
      }
    </div>
  );
}
export default HelloWorld;

 

In this iteration, we have replaced the onChange callback function to a new function parseReceipt. Basically, this function will call the Receipt parsing API (and send the image to Mindee to extract the receipt details), and transform the results to very easily generate shapes using formatPrediction() function from the SDK.

 

Run the app, you should see something like this when you upload a receipt image:

 

react mindee parse receipt

 

All boxes displayed on the image are interractive and you can bind custom onClick and onMouseOver mouse events functions to them. You can also try to zoom in the image, try to move the image inside the viewer etc.. All features are explained on the Github repo.

 

Important note: Calling the receipt parsing API client-side is not secure, as your API token is present in the browser data, potentially exposing this to the world.  This approach should only be used for prototyping, you'll need a back-end middleware to make a secure production app.

 

Now we are just going to change a few lines of codes to display the data extracted by the API. To do so, we are going to add a new state object 'receiptData', and display a few features on the web page. In your code, change the HelloWorld component to this (don't forget to fix your api token):

 

const HelloWorld = () => {
  const [image, setImage] = useState(null)
  const [shapes, setShapes] = useState([])
  const [receiptData, setReceiptData] = useState(null)

  const parseReceipt = (e) => {
    let myFile = e.target.files[0]

    if (!myFile) {
      return
    }
    setImage(URL.createObjectURL(myFile))

    let data = new FormData();
    data.append("file", myFile, myFile.name);

    let xhr = new XMLHttpRequest();

    xhr.addEventListener("readystatechange", function () {
      if (this.readyState === 4) {
        let json_response = JSON.parse(this.responseText)
        setShapes(formatPrediction(json_response.predictions[0]))
        setReceiptData(json_response.predictions[0])
      }
    });

    xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
    xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
    xhr.send(data);
  }

  return (
    <div>
      <input type="file" onChange={e => parseReceipt(e)}/>
      {
        image && <>
          <div style={{height: '500px', width: '400px'}}>
            <AnnotationViewer
              shapes={shapes}
              image={image}
            />
          </div>
          {
            receiptData &&
            <div style={{marginLeft: "50px"}}>
              <p>Total amount : {receiptData.total.amount}</p>
              <p>Date: {receiptData.date.iso}</p>
              <p>Time: {receiptData.time.iso}</p>
              <p>Category: {receiptData.category.value}</p>
            </div>
          }
        </>
      }
    </div>
  );
}

 

Run the app, upload a receipt, you should get:

 

Reactjs mindee parsing receipt

 

We're almost done!

 

Bind interractions between images boxes and corresponding receipt text features

 

To make sure your user can validate very easily data extracted by the API, the ReactJS SDK allows you to add interactions with visual features on the images and the rest of your app.

 

First, we'll create a function that change the isActive attribute inside a shape object, from it's name. The SDK will understand this attribute and will modify the shape layout (you can modify the active layout by changing colors and line width in the active attribute of a shape).

 

const highlightFeature = (featureName) => {
  let newShapes = []
  shapes.forEach(shape => {
      shape.isActive=(shape.featureName === featureName)
      newShapes.push(shape)
    }
  )
  setShapes(newShapes)
}

 

We'll then modify a bit the rendered code, and bind on the onMouseOver event of the text features the highlight function.

 

return (
    <div>
      <input type="file" onChange={e => parseReceipt(e)}/>
      {
        image && <>
          <div style={{height: '500px', width: '400px'}}>
            <AnnotationViewer
              shapes={shapes}
              image={image}
            />
          </div>
          {
            receiptData &&
            <div style={{marginLeft: "50px"}}>
              <p onMouseOver={() => highlightFeature('total')}>Total amount : {receiptData.total.amount}</p>
              <p onMouseOver={() => highlightFeature('date')}>Date: {receiptData.date.iso}</p>
              <p onMouseOver={() => highlightFeature('time')}>Time: {receiptData.time.iso}</p>
              <p>Category: {receiptData.category.value}</p>
            </div>
          }
        </>
      }
    </div>
  );

 

Final App.js

import React, {useState} from 'react';
import './App.css';

import {
  AnnotationViewer,
  formatPrediction
} from "react-mindee-js";

const HelloWorld = () => {
  const [image, setImage] = useState(null)
  const [shapes, setShapes] = useState([])
  const [receiptData, setReceiptData] = useState(null)

  const parseReceipt = (e) => {
    let myFile = e.target.files[0]

    if (!myFile) {
      return
    }
    setImage(URL.createObjectURL(myFile))

    let data = new FormData();
    data.append("file", myFile, myFile.name);

    let xhr = new XMLHttpRequest();

    xhr.addEventListener("readystatechange", function () {
      if (this.readyState === 4) {
        let json_response = JSON.parse(this.responseText)
        setShapes(formatPrediction(json_response.predictions[0]))
        setReceiptData(json_response.predictions[0])
      }
    });

    xhr.open("POST", "https://api.mindee.net/products/expense_receipts/v2/predict");
    xhr.setRequestHeader("X-Inferuser-Token", "my-token-here");
    xhr.send(data);
  }

  const highlightFeature = (featureName) => {
    let newShapes = []
    shapes.forEach(shape => {
        shape.isActive=(shape.featureName === featureName)
        newShapes.push(shape)
      }
    )
    setShapes(newShapes)
  }

  return (
    <div>
      <input type="file" onChange={e => parseReceipt(e)}/>
      {
        image && <>
          <div style={{height: '500px', width: '400px'}}>
            <AnnotationViewer
              shapes={shapes}
              image={image}
            />
          </div>
          {
            receiptData &&
            <div style={{marginLeft: "50px"}}>
              <p onMouseOver={() => highlightFeature('total')}>Total amount : {receiptData.total.amount}</p>
              <p onMouseOver={() => highlightFeature('date')}>Date: {receiptData.date.iso}</p>
              <p onMouseOver={() => highlightFeature('time')}>Time: {receiptData.time.iso}</p>
              <p>Category: {receiptData.category.value}</p>
            </div>
          }
        </>
      }
    </div>
  );
}
export default HelloWorld;

 

Running your app, you should see now that when you hover over a text line at the bottom, the color of the box highlighting the value on the receipt will change. You can modify the colours as you want to make the change more or less dramatic, as you please.  All of the details for this is explained in the Github readme.

 

Conclusion

 

Using our ReactJS library makes identifying regions in the image easy.  The Mindee Receipt Parsing API extracts the relevant details (and the location in the image). Then the react library highlights the locations with the AnnotationViewer. Combined, you have a powerful tool to extract data from images, and display them for verification to your users.