dataLoader - how to skip insertion of data what already exist in database?


#1

I’m try ti load a data using @unitybase/base/dataloader module. How to skip insertion of data what already exist in the database?


#2

You should manually load data into the array, filter it using your business rules and when pass the array to dataLoader.loadArrayData

In the sample below, we load a CSV file:

ID;code;caption;filterValue;booleanColumn;currencyValue;floatValue
1;code10;caption 10;10;0;1.11;1.1111
2;code20;caption 20;10;0;2.12;2.1112

And filter data using 2nd column as a key (so if database contains rows with codes from CSV file we skip it during loading)

const {dataLoader, csv, argv} = require('@unitybase/base')
const path = require('path')
const fs = require('fs')

let session = argv.establishConnectionFromCmdLineAttributes()
conn = session.connection

let fn = path.join(__dirname, 'tst_dictionary-TST.csv')
let fContent = fs.readFileSync(fn)
if (!fContent) { throw new Error(`File ${fn} is empty or not exist`) }
fContent = fContent.trim()
let csvData = csv.parse(fContent)

csvData.splice(0, 1) // remove first row

// for each row try to lookup value with specified code in database. If found - skip row
let notExisted = csvData.filter(
  (row) => !conn.lookup('tst_dictionary', 'ID',
      conn.Repository('tst_dictionary').where('code', '=', row[1]).ubql().whereList
    )
)

// load only non-existed rows
dataLoader.loadArrayData(conn, notExisted, 'tst_dictionary', 
  'ID;code;caption;filterValue;booleanColumn;currencyValue;floatValue'.split(';')
)