Skip to content

SQL Commiter with SCRAM-based authentication fails #64

Description

@schudoku

I am using norconex filesystem crawler 2.9.1 and want to commit to a postgre database 13.4. The database requires user authentication which is SCRAM-based. In the collector config I specified the following commiter:

  <committer class="com.norconex.committer.sql.SQLCommitter">
    <driverPath>C:\dev\databases\drivers\postgresql-42.3.5.jar</driverPath>
    <driverClass>org.postgresql.Driver</driverClass>
    <connectionUrl>jdbc:postgresql://localhost:64000/databasename</connectionUrl>
    <tableName>crawler.crawl_test</tableName>
    
    <createTableSQL>
  	CREATE TABLE {tableName} (
  		{primaryKey} VARCHAR(32672) NOT NULL,
  		content CLOB,
  		PRIMARY KEY ( {primaryKey} ),
  		title   VARCHAR(256)
  		author  VARCHAR(256)
  	)
    </createTableSQL>
    
    <createFieldSQL>
  	ALTER TABLE {tableName} ADD {fieldName} VARCHAR(5000)
    </createFieldSQL>
    
    <fixFieldValues>true</fixFieldValues>
    
    <credentials>
  	<username> ---PLAIN-USERNAME-HERE--- </username>
  	<password> ---PLAIN-PW-HERE--- </password-->
    </credentials>
    
  </committer>

And this is the stacktrace I get:

[..]
INFO [AbstractCrawler] Crawler For ProSi Reports: Crawler finishing: committing documents.
INFO [AbstractFileQueueCommitter] Committing 266 files
INFO [SQLCommitter] Sending 100 commit operations to SQL database.
INFO [SQLCommitter] Checking if table "crawler.crawl_1" exists...
INFO [SQLCommitter] Table "crawler.crawl_1" does not exist. Attempting to create it...
INFO [AbstractCrawler] Crawler For ProSi Reports: Crawler executed in 4 seconds.
ERROR [JobSuite] Execution failed for job: Crawler For ProSi Reports
com.norconex.committer.core.CommitterException: Could not commit batch to database.
at com.norconex.committer.sql.SQLCommitter.commitBatch(SQLCommitter.java:480)
at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179)
at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208)
at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143)
at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222)
at com.norconex.committer.sql.SQLCommitter.commit(SQLCommitter.java:425)
at com.norconex.collector.core.crawler.AbstractCrawler.execute(AbstractCrawler.java:274)
at com.norconex.collector.core.crawler.AbstractCrawler.doExecute(AbstractCrawler.java:228)
at com.norconex.collector.core.crawler.AbstractCrawler.startExecution(AbstractCrawler.java:184)
at com.norconex.jef4.job.AbstractResumableJob.execute(AbstractResumableJob.java:49)
at com.norconex.jef4.suite.JobSuite.runJob(JobSuite.java:357)
at com.norconex.jef4.suite.JobSuite.doExecute(JobSuite.java:294)
at com.norconex.jef4.suite.JobSuite.execute(JobSuite.java:167)
at com.norconex.collector.core.AbstractCollector.start(AbstractCollector.java:150)
at com.norconex.collector.core.AbstractCollectorLauncher.launch(AbstractCollectorLauncher.java:95)
at com.norconex.collector.fs.FilesystemCollector.main(FilesystemCollector.java:76)
Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (The server requested SCRAM-based authentication, but no password was provided.)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2294)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2039)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
at org.apache.commons.dbutils.AbstractQueryRunner.prepareConnection(AbstractQueryRunner.java:204)
at org.apache.commons.dbutils.QueryRunner.update(QueryRunner.java:419)
at com.norconex.committer.sql.SQLCommitter.ensureTable(SQLCommitter.java:607)
at com.norconex.committer.sql.SQLCommitter.commitBatch(SQLCommitter.java:463)
... 15 more
Caused by: org.postgresql.util.PSQLException: The server requested SCRAM-based authentication, but no password was provided.
at org.postgresql.core.v3.ConnectionFactoryImpl.lambda$doAuthentication$4(ConnectionFactoryImpl.java:791)
at org.postgresql.core.v3.AuthenticationPluginManager.withPassword(AuthenticationPluginManager.java:81)
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:788)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:180)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:223)
at org.postgresql.Driver.makeConnection(Driver.java:402)
at org.postgresql.Driver.connect(Driver.java:261)
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2304)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2290)
... 21 more
INFO [JobSuite] Running Crawler For ProSi Reports: END (Mon May 23 09:41:40 CEST 2022)

It says no password provided, but I did. Could you help me?

Regards!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions