bash - Save sqoop incremental import id -

-

i have lot of sqoop jobs running in aws emr, need turn off instance.

there's way save last id incremental import, maybe localy , upload s3 via cronjob.

my first idea is, when create job send request redshift, data stored , last id or last_modified, via bash script.

another idea output of sqoop job --show $jobid, filter parameter of last_id , using create job again.

but don't know if sqoop offer way more easily.

as per sqoop docs,

if incremental import run command line, value should specified --last-value in subsequent incremental import printed screen reference. if incremental import run saved job, value retained in saved job. subsequent runs of sqoop job --exec someincrementaljob continue import newer rows imported.

so, need store nothing. sqoop's metastore take care of saving last value , avail next incremental import job.

example,

sqoop job \ --create new_job \ -- \ import \ --connect jdbc:mysql://localhost/testdb \ --username xxxx \ --password xxxx \ --table employee \ --incremental append \ --check-column id \ --last-value 0

and start job --exec parameter:

sqoop job --exec new_job

Comments

Post a Comment

Popular posts from this blog

mysql - Dreamhost PyCharm Django Python 3 Launching a Site -

-
i'm new python/django , created simple django website using pycharm. site works fine running on computer, i'm lost when comes taking site onto host (dreamhost). i'm able upload files correct directories fine, , believe made correct changes database on mysql. i'm not sure go there though. right now, page shows files rather site. direction appreciated. much! you should not run django application using python manage.py runserver on production. how run application on django depends on server (apache, nginx) want use for apache - https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/modwsgi/ for nginx - https://www.digitalocean.com/community/tutorials/how-to-serve-django-applications-with-uwsgi-and-nginx-on-ubuntu-14-04 the mysql database should, of course, exist , have correct credentials. or, if choose use sqlite, file should accessible, readable , writable www-user (may different depending on linux distributive)
Read more

java - Sending SMS with SMSLib and Web Services -

-
i have made rest web service send smss using hsdpa usb modem. using smslib in java send smss. every time web service invoked, create gateway start service, send message, stop service , remove gateway. takes 20 seconds each message. found starting service takes lot of time. part of code use send smss service.getinstance().addgateway(gateway); service.getinstance().startservice(); outboundmessage msg = new outboundmessage(phonenumber, message); if (service.getinstance().sendmessage(msg)) { result = "message sent successfully!!"; } else { result = "could not send message."; } service.getinstance().stopservice(); service.getinstance().removegateway(gateway);//remove gateway is there way can start service once if not started , use send message when ever web service invoked? why dont group messages , send in 1 go? service.getinstance().sendmessages(messa...
Read more

python 3.5 - Pyqtgraph string in x tick -

-
Image
so have been playing pyqtgraph , came problem needed add strings x axis tick label (instead of default float). looked around , found show string values on x-axis in pyqtgraph the solution kind of need (the first solution on page, second 1 cause problems because on python 3.5). tried modified first solution, did not work. from pyqt4 import qtcore import pyqtgraph pg x = ['a', 'b', 'c', 'd', 'e', 'f'] y = [1, 2, 3, 4, 5, 6] xdict = dict(enumerate(x)) win = pg.graphicswindow() stringaxis = pg.axisitem(orientation='bottom') stringaxis.setticks([xdict.items()]) plot = win.addplot(axisitems={'bottom': stringaxis}) curve = plot.plot(xdict.keys(),y) if __name__ == '__main__': import sys if sys.flags.interactive != 1 or not hasattr(qtcore, 'pyqt_version'): pg.qtgui.qapplication.exec_() the error getting typeerror: 'dict_keys' object not support indexing on line curve = plot.plo...
Read more