dataLoader - how to skip insertion of data what already exist in database?


I’m try ti load a data using @unitybase/base/dataloader module. How to skip insertion of data what already exist in the database?


You should manually load data into the array, filter it using your business rules and when pass the array to dataLoader.loadArrayData

In the sample below, we load a CSV file:

1;code10;caption 10;10;0;1.11;1.1111
2;code20;caption 20;10;0;2.12;2.1112

And filter data using 2nd column as a key (so if database contains rows with codes from CSV file we skip it during loading)

const {dataLoader, csv, argv} = require('@unitybase/base')
const path = require('path')
const fs = require('fs')

let session = argv.establishConnectionFromCmdLineAttributes()
conn = session.connection

let fn = path.join(__dirname, 'tst_dictionary-TST.csv')
let fContent = fs.readFileSync(fn)
if (!fContent) { throw new Error(`File ${fn} is empty or not exist`) }
fContent = fContent.trim()
let csvData = csv.parse(fContent)

csvData.splice(0, 1) // remove first row

// for each row try to lookup value with specified code in database. If found - skip row
let notExisted = csvData.filter(
  (row) => !conn.lookup('tst_dictionary', 'ID',
      conn.Repository('tst_dictionary').where('code', '=', row[1]).ubql().whereList

// load only non-existed rows
dataLoader.loadArrayData(conn, notExisted, 'tst_dictionary',