Extract data from the Twitter Application Programming Interface (API) to a PostgreSQL table.
(Object
= {})
options for this function.
| Name | Description |
|---|---|
options.twitter Object
(default {})
|
options for twitter (see twitter2return ). |
options.twitter.stream function
(default function(err,data){})
|
callback function on a stream 'data' event for the returned
Twitter stream
.
|
options.pg Object
(default {})
|
contains options for queries in pg . |
options.pg.table string
(default process.env.PGTABLE||'twitter2pg_table')
|
PostgreSQL table name. |
options.pg.column string
(default process.env.PGCOLUMN||'tweets')
|
PostgreSQL column name for
options.pg.table
.
|
options.pg.query string
(default process.env.PGQUERY||'INSERT INTO $options.pg.table ($options.pg.column) VALUES ($1);')
|
PostgreSQL parameterized query to insert Twitter data in JSON format.
|
options.pg.connection Object
(default {})
|
PostgreSQL connection details. |
options.pg.connection.host string
(default process.env.PGHOST||'localhost')
|
Host address of PostgreSQL instance. |
options.pg.connection.port number
(default process.env.PGPORT||5432)
|
Port number of PostgreSQL instance. |
options.pg.connection.database number
(default process.env.PGDATABASE||process.env.PGUSER||process.env.USER||'postgres')
|
Database name for PostgreSQL instance. |
options.pg.connection.user string
(default process.env.PGUSER||process.env.USER||'postgres')
|
User name for PostgreSQL instance. |
options.pg.connection.password string
(default process.env.PGPASSWORD)
|
Password of user for PostgreSQL instance. |
options.jsonata string
(default process.env.JSONATA)
|
jsonata
query for the recieved tweet object in JSON format before inserting into the PostgreSQL table (
options.pg.table
).
|
(Promise | stream):
Returns a stream if
options.twitter.method
is 'stream', otherwise returns a Promise:
If options.twitter.method == 'stream'
stream.on('data', function): calls function when a tweet is available stream.on('error', function): calls function when there is an error Else
Return a Promise object that resolves a data object in the form {twitter: {client: ..., tweets: ...}, pg: {client: ..., results: ...}}
data.twitter.client: contains a Twitter client object created from options.twitter.connection
data.twitter.tweets: contains the tweets in JSON format
data.pg.client: contains the PostgreSQL client from options.pg.connection
data.pg.results: contains the PostgreSQL query results of options.pg.query
var twitter2pg = require('twitter2pg');
// (options) Initialize options object
var options = {
twitter: {},
pg: {}
};
// *** CONNECTION SETUP ***
// (options_twitter_connection) Twitter API connection keys
options.twitter.connection = {
consumer_key: '***',
consumer_secret: '***',
access_token_key: '***',
access_token_secret: '***'
};
// (options_pg_connection) PostgreSQL connection details
options.pg.connection = {
host: 'localhost',
port: 5432,
database: 'postgres',
user: 'postgres',
password: '***'
};
// *** SEARCH TWEETS ***
// (options_twitter_rest) Search for keyword 'twitter' in path 'GET search/tweets'
options.twitter.method = 'get'; // get, post, or stream
options.twitter.path = 'search/tweets'; // api path
options.twitter.params = {q: 'twitter'}; // query tweets
// (options_pg) PostgreSQL options
options.pg.table = 'twitter_data';
options.pg.column = 'tweets';
options.pg.query = 'INSERT INTO $options.pg.table($options.pg.column) SELECT * FROM json_array_elements($1);';
// (options_jsonata) Filter for statuses array using jsonata
options.jsonata = 'statuses';
// (twitter2pg_rest) Query tweets using REST API into PostgreSQL table
twitter2pg(options)
.then(data => {
console.log(data);
}).catch(err => {
console.error(err.message);
});
// *** STREAM TWEETS ***
// (options_twitter_connection) Track keyword 'twitter' in path 'POST statuses/filter'
options.twitter.method = 'stream'; // get, post, or stream
options.twitter.path = 'statuses/filter'; // api path
options.twitter.params = {track: 'twitter'}; // query tweets
// (options_pg) PostgreSQL options
options.pg.table = 'twitter_data';
options.pg.column = 'tweets';
options.pg.query = 'INSERT INTO $options.pg.table($options.pg.column) VALUES($1);';
// (options_jsonata) Remove jsonata filter
delete options.jsonata;
// (twitter2pg_stream) Stream tweets into PostgreSQL table
var stream = twitter2pg(options);
stream.on('error', function(error) {
console.error(error.message);
});