Invalid Input Error: Mismatch between the schema of different files

How to fix mismatch schema error when using read_csv_auto.

    concat(replace("<colname>",'.UK',''),'.L') as sym, cast(strptime("<DATE>"::string, '%Y%m%d') AS DATE) AS date, *
1
2
3
4
 
INSERT INTO table SELECT  
    concat(replace("<colname>",'.UK',''),'.L') as sym, cast(strptime("<DATE>"::string, '%Y%m%d') AS DATE) AS date, *
    FROM read_csv_auto('c:\temp\data\**\*.txt', header=true);
 

If you get an error saying "Invalid Input Error: Mismatch..." when running code similar to the above, you have a few options:

Delete any empty files

Empty files cause a mismatch error. Search all directories for empty files and delete them.

Debug - By filtering which files get loaded. Either:

  • Move files in/out of the loading directory until it works to narrow down which file is causing the problem.
  • Use the rejects_table feature to record which records are failing to load.
    rejects_table = 'rejects_table', ignore_errors = true