Druid.io: update/override existing data via streams from Kafka (Druid Kafka indexing service)
I'm loading streams from Kafka using the Druid Kafka indexing service.
But the data I uploaded is always changed, so I need to reload it again and avoid duplicates and collisions if data was already loaded.
I research docs about Updating Existing Data in Druid.
But all info about Hadoop Batch Ingestion, Lookups .
Is it possible to update existing Druid data during Kafka streams?
In other words, I need to rewrite the old values with new ones using Kafka indexing service (streams from Kafka).
May be any kind of setting to rewrite duplicates?
See also questions close to this topic
-
How to pass OnClickListener from Activity to Fragment
I need to pass an OnClickListener object from activity to a Fragment. I can't pass argument to Fragment constructor, so i want to do something like:
Bundle bundle = new Bundle(); bundle.putCustomObject(new OnClickListener(){...}); -
generate nginx secure link from android activity
i know how to generate nginx secure link from php but how i do that from android i have searched everywhere but couldn't find a code or a clue about this. i will appreciate your help guys i am new to android
here is my nginx configuration:
location /securetest { alias html/mhmsecure/securetest; secure_link $arg_md5,$arg_expires; secure_link_md5 "$secure_link_expires$uri$remote_addr secret"; if ($secure_link = "") { return 403; } if ($secure_link = "0") { return 410; } }Thank you
-
Unable to Update UI using LiveData and Room
I am writing and reading to a local SQLite database using
RoomandLiveData. I have successfully read and wrote to the database, as I am usingStethoto confirm.I understand
LiveDatahandles functionality on a separate thread. I am struggling with updating myRecyclerViewbased on the database information I retrieve usingLiveData:private void retrieveTasks() { final LiveData<List<Movie>> movies = mDb.movieDao().loadAllMovies(); movies.observe(this, new Observer<List<Movie>>() { @Override public void onChanged(@Nullable List<Movie> movies) { for (Movie movie : movies) { String photoURL = "http://image.tmdb.org/t/p/w185" + movie.getPoster_path(); Log.v("FAVORITE_URL", photoURL); mMovieURLS.add(photoURL); } runOnUiThread(new Runnable() { @Override public void run() { mMovieResultsAdapter.swapDataSet(mMovieURLS); mMovieResultsAdapter.notifyDataSetChanged(); } }); } }); } -
kafka.com:9092/0: Connect to ipv4# failed: Connection refused
I use Kafka with one node (version 1.0.0).
why when I set
listeners=PLAINTEXT://138.201.YYYY.YYY:9092, everything is OK and I can produce and consume well in local and client enviroment , but when I setlisteners=PLAINTEXT://kafka1.dev:9092, in the client side, I cannot produce and consume any message and this error will appear :%3|1531686738.672|FAIL|rdkafka#producer-1| [thrd:kafka1.dev:9092/0]: kafka1.dev:9092/0: Failed to resolve 'kafka1.dev:9092': Name or service not known %3|1531686738.672|ERROR|rdkafka#producer-1| [thrd:kafka1.dev:9092/0]: kafka1.dev:9092/0: Failed to resolve 'kafka1.dev:9092': Name or service not knownand I have to add
138.201.YYY.YYY kafka1.devin theetc/hostsfile (in local) to fix the problem and produce fine. -
Apache-Apex : How to write data from kafka topic to hdfs filesystem?
I am trying to read dat from kafka topic and writing it to HDFS filesystem, I buid my project using the apex malhar from [ https://github.com/apache/apex-malhar/tree/master/examples/kafka]. Unfortunally, after setting up the kafka properties and hadoop config the data are not created in my hdfs 2.6.0 system. PS: the console dosen't show any error and everything seems to work fine
here the code I am using for my app
public class TestConsumer { public static void main(String[] args) { Consumer consumerThread = new Consumer(KafkaProperties.TOPIC); consumerThread.start(); ApplicationTest a = new ApplicationTest(); try { a.testApplication(); } catch (Exception e) { e.printStackTrace(); } } }Here the ApplicationTest class example from apex malhar
package org.apache.apex.examples.kafka.kafka2hdfs; import org.apache.log4j.Logger; import javax.validation.ConstraintViolationException; import org.junit.Rule; import org.apache.apex.malhar.kafka.AbstractKafkaInputOperator; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.net.NetUtils; import com.datatorrent.api.LocalMode; import info.batey.kafka.unit.KafkaUnitRule; /** * Test the DAG declaration in local mode. */ public class ApplicationTest { private static final Logger LOG = Logger.getLogger(ApplicationTest.class); private static final String TOPIC = "kafka2hdfs"; private static final int zkPort = NetUtils.getFreeSocketPort(); private static final int brokerPort = NetUtils.getFreeSocketPort(); private static final String BROKER = "localhost:" + brokerPort; private static final String FILE_NAME = "test"; private static final String FILE_DIR = "./target/tmp/FromKafka"; // broker port must match properties.xml @Rule private static KafkaUnitRule kafkaUnitRule = new KafkaUnitRule(zkPort, brokerPort); public void testApplication() throws Exception { try { // run app asynchronously; terminate after results are checked LocalMode.Controller lc = asyncRun(); lc.shutdown(); } catch (ConstraintViolationException e) { LOG.error("constraint violations: " + e.getConstraintViolations()); } } private Configuration getConfig() { Configuration conf = new Configuration(false); String pre = "dt.operator.kafkaIn.prop."; conf.setEnum(pre + "initialOffset", AbstractKafkaInputOperator.InitialOffset.EARLIEST); conf.setInt(pre + "initialPartitionCount", 1); conf.set(pre + "topics", TOPIC); conf.set(pre + "clusters", BROKER); pre = "dt.operator.fileOut.prop."; conf.set(pre + "filePath", FILE_DIR); conf.set(pre + "baseName", FILE_NAME); conf.setInt(pre + "maxLength", 40); conf.setInt(pre + "rotationWindows", 3); return conf; } private LocalMode.Controller asyncRun() throws Exception { Configuration conf = getConfig(); LocalMode lma = LocalMode.newInstance(); lma.prepareDAG(new KafkaApp(), conf); LocalMode.Controller lc = lma.getController(); lc.runAsync(); return lc; } } -
connect failed between containers with zookeeper and kafka
i use kafka,zookeeper,elasticsearch,3 nodes of kafka,3 nodes of zkthis is i run docker-compose.yml on my mac,
123.345.567is the output of below commandifconfig | grep 'inet 192'| awk '{ print $2}'docker-compose.yml is
version: '2' services: zookeeper: ports: - "2181:2181" - "2888:2888" - "3888:3888" build: context: /docker/zookeeper dockerfile: Dockerfile args: clientPort: 2181 follow_to_leader_port: 2888 leader_select_port: 3888 environment: service1: 0.0.0.0:2888:3888 service2: 123.345.567:2889:3889 service3: 123.345.567:2890:3890 clientPort: 2181 ZOO_MY_ID: 1 initLimit: 2 zookeeper2: ports: - "2182:2182" - "2889:2889" - "3889:3889" build: context: /docker/zookeeper dockerfile: Dockerfile args: clientPort: 2182 follow_to_leader_port: 2889 leader_select_port: 3889 environment: service1: 123.345.567:2888:3888 service2: 0.0.0.0:2889:3889 service3: 123.345.567:2890:3890 clientPort: 2182 ZOO_MY_ID: 2 initLimit: 2 zookeeper3: ports: - "2183:2183" - "2890:2890" - "3890:3890" build: context: /docker/zookeeper dockerfile: Dockerfile args: clientPort: 2183 follow_to_leader_port: 2890 leader_select_port: 3890 environment: service1: 123.345.567:2888:3888 service2: 123.345.567:2889:3889 service3: 0.0.0.0:2890:3890 clientPort: 2183 initLimit: 2 ZOO_MY_ID: 3 kafka: build: context: /docker/kafka dockerfile: Dockerfile ports: - "9096:9096" environment: broker_id: 1 listeners: PLAINTEXT://123.345.567:9096 advertised_listeners: PLAINTEXT://123.345.567:9096 zookeeper_connect: 123.345.567:2181,123.345.567:2182,123.345.567:2183/kafka_test kafka2: build: context: /docker/kafka dockerfile: Dockerfile ports: - "9097:9097" environment: broker_id: 2 listeners: PLAINTEXT://123.345.567:9097 advertised_listeners: PLAINTEXT://123.345.567:9097 zookeeper_connect: 123.345.567:2181,123.345.567:2182,123.345.567:2183/kafka_test kafka3: build: context: /docker/kafka dockerfile: Dockerfile ports: - "9098:9098" environment: broker_id: 3 listeners: PLAINTEXT://123.345.567:9098 advertised_listeners: PLAINTEXT://123.345.567:9098 zookeeper_connect: 123.345.567:2181,123.345.567:2182,123.345.567:2183/kafka_testdocker ps output
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 82c0f30cac77 compose_zookeeper3 "/opt/startzookeeper…" About an hour ago Up 40 minutes 0.0.0.0:2183->2183/tcp, 0.0.0.0:2890->2890/tcp, 0.0.0.0:3890->3890/tcp compose_zookeeper3_1 ebd93d99d0d2 compose_kafka "/opt/startkafka.sh" About an hour ago Up 40 minutes 0.0.0.0:9096->9096/tcp compose_kafka_1 be56ebaee602 compose_kafka2 "/opt/startkafka.sh" About an hour ago Up 40 minutes 0.0.0.0:9097->9097/tcp compose_kafka2_1 8a68874e52f9 compose_kafka3 "/opt/startkafka.sh" About an hour ago Up 40 minutes 0.0.0.0:9098->9098/tcp compose_kafka3_1 000a686a2772 compose_zookeeper2 "/opt/startzookeeper…" About an hour ago Up 40 minutes 0.0.0.0:2182->2182/tcp, 0.0.0.0:2889->2889/tcp, 0.0.0.0:3889->3889/tcp compose_zookeeper2_1 d3fc5ad9b8f8 compose_zookeeper "/opt/startzookeeper…" About an hour ago Up 40 minutes 0.0.0.0:2181->2181/tcp, 0.0.0.0:2888->2888/tcp, 0.0.0.0:3888->3888/tcp compose_zookeeper_1some configuration in kafka
broker.id=2 listeners=PLAINTEXT://123.345.567:9097 advertised.listeners=PLAINTEXT://123.345.567:9097 zookeeper.connect=123.345.567:2181,123.345.567:2182,123.345.567:2183/kafka_testsome configuration in zookeeper2
clientPort=2182 server.1=123.345.567:2888:3888 server.2=0.0.0.0:2889:3889 server.3=123.345.567:2890:3890some error log
[32mzookeeper3_1 |[0m 2018-07-15 09:54:19,083 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2183:FastLeaderElection@813] - New election. My id = 3, proposed zxid=0x100000047 [32mzookeeper3_1 |[0m 2018-07-15 09:54:19,113 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1025] - Connection broken for id 1, my id = 3, error = [32mzookeeper3_1 |[0m java.io.EOFException [32mzookeeper3_1 |[0m at java.io.DataInputStream.readInt(DataInputStream.java:392) [32mzookeeper3_1 |[0m at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1010) [32mzookeeper3_1 |[0m 2018-07-15 09:54:19,123 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1028] - Interrupting SendWorker [32mzookeeper3_1 |[0m 2018-07-15 09:54:19,125 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@595] - Notification: 1 (message format version), 3 (n.leader), 0x100000047 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state) [32mzookeeper3_1 |[0m 2018-07-15 09:54:19,126 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@941] - Interrupted while waiting for message on queue [32mzookeeper3_1 |[0m java.lang.InterruptedException [32mzookeeper3_1 |[0m at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) [32mzookeeper3_1 |[0m at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) [32mzookeeper3_1 |[0m at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) [32mzookeeper3_1 |[0m at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1094) [32mzookeeper3_1 |[0m at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:74) [32mzookeeper3_1 |[0m at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:929)kafka
[35mkafka_1 |[0m [2018-07-15 09:54:38,137] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) [35mkafka_1 |[0m kafka.common.KafkaException: Socket server failed to bind to 123.345.567:9096: Cannot assign requested address. [35mkafka_1 |[0m at kafka.network.Acceptor.openServerSocket(SocketServer.scala:404) [35mkafka_1 |[0m at kafka.network.Acceptor.<init>(SocketServer.scala:308) [35mkafka_1 |[0m at kafka.network.SocketServer$$anonfun$createAcceptorAndProcessors$1.apply(SocketServer.scala:126) [35mkafka_1 |[0m at kafka.network.SocketServer$$anonfun$createAcceptorAndProcessors$1.apply(SocketServer.scala:122) [35mkafka_1 |[0m at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) [35mkafka_1 |[0m at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) [35mkafka_1 |[0m at kafka.network.SocketServer.createAcceptorAndProcessors(SocketServer.scala:122) [35mkafka_1 |[0m at kafka.network.SocketServer.startup(SocketServer.scala:84) [35mkafka_1 |[0m at kafka.server.KafkaServer.startup(KafkaServer.scala:247) [35mkafka_1 |[0m at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) [35mkafka_1 |[0m at kafka.Kafka$.main(Kafka.scala:92) [35mkafka_1 |[0m at kafka.Kafka.main(Kafka.scala) [35mkafka_1 |[0m Caused by: java.net.BindException: Cannot assign requested address [35mkafka_1 |[0m at sun.nio.ch.Net.bind0(Native Method) [35mkafka_1 |[0m at sun.nio.ch.Net.bind(Net.java:433) [35mkafka_1 |[0m at sun.nio.ch.Net.bind(Net.java:425) [35mkafka_1 |[0m at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) [35mkafka_1 |[0m at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) [35mkafka_1 |[0m at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67) [35mkafka_1 |[0m at kafka.network.Acceptor.openServerSocket(SocketServer.scala:400) [35mkafka_1 |[0m ... 11 more [35mkafka_1 |[0m [2018-07-15 09:54:38,144] INFO [KafkaServer id=1] shutting down (kafka.server.KafkaServer) [35mkafka_1 |[0m [2018-07-15 09:54:38,147] INFO [SocketServer brokerId=1] Stopping socket server request processors (kafka.network.SocketServer) [35mkafka_1 |[0m [2018-07-15 09:54:38,149] INFO [SocketServer brokerId=1] Stopped socket server request processors (kafka.network.SocketServer) [35mkafka_1 |[0m [2018-07-15 09:54:38,160] INFO Shutting down. (kafka.log.LogManager) [35mkafka_1 |[0m [2018-07-15 09:54:38,208] INFO Shutdown complete. (kafka.log.LogManager)i wonder whether these containers does not need the ip:123.345.567.maybe they need some
inner ipto communicate between containers?or i need some command to check or make docker network setting? thanks your any suggestion! -
How to validate a Target System_Bigdata for below e.g
Table 1 (Source data) Column name (name, dept, coll, grade)
above column data from single table will fit in to multiple tables in target system after some transformation as below
table A (Target) - Name_1 (column name), Table B (target) - dept_1 (column name), table c (target) - coll_1 (column name), table d (target) - grade_1 (column name),
How to test above scenario as data from single table goes to multiple tables and ensure all data is perfectly loaded to target.
-
Semantic Segmentation using dl4j with FCN Networks
I have checked out dl4j examples and executed AnimalsClassification example for testing successfully.
I have to train, evaluate and predict(use) a semantic segmentation algorithm like UNet, because input images are not the same size FCN is required.
and changed the network of AnimalsClassification example to UNet from this link
but getting error. Could you please help me to solve this error?
The Error:
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 19:22:39,951 INFO ~ Load data.... 19:22:40,696 INFO ~ Build model.... 19:22:41,363 INFO ~ Loaded [CpuBackend] backend 19:22:44,955 INFO ~ Number of threads used for NativeOps: 2 19:22:45,555 INFO ~ Number of threads used for BLAS: 2 19:22:45,562 INFO ~ Backend used: [CPU]; OS: [Linux] 19:22:45,562 INFO ~ Cores: [2]; Memory: [1.3GB]; 19:22:45,562 INFO ~ Blas vendor: [OPENBLAS] 19:22:56,425 WARN ~ Layer "Layer not named" distribution is set but will not be applied unless weight init is set to WeighInit.DISTRIBUTION. Exception in thread "main" java.lang.IllegalStateException: Invalid configuration: network has no inputs. Use .addInputs(String...) to label (and give an ordering to) the network inputs at org.deeplearning4j.nn.conf.ComputationGraphConfiguration.validate(ComputationGraphConfiguration.java:279) at org.deeplearning4j.nn.conf.ComputationGraphConfiguration$GraphBuilder.build(ComputationGraphConfiguration.java:918) at org.deeplearning4j.examples.convolution.AnimalsClassification.graphBuilder(AnimalsClassification.java:443) at org.deeplearning4j.examples.convolution.AnimalsClassification.run(AnimalsClassification.java:145) at org.deeplearning4j.examples.convolution.AnimalsClassification.main(AnimalsClassification.java:447) Process finished with exit code 1The code I have changed is:
package org.deeplearning4j.examples.convolution; import lombok.Builder; import org.apache.commons.io.FilenameUtils; import org.datavec.api.io.filters.BalancedPathFilter; import org.datavec.api.io.labels.ParentPathLabelGenerator; import org.datavec.api.split.FileSplit; import org.datavec.api.split.InputSplit; import org.datavec.image.loader.NativeImageLoader; import org.datavec.image.recordreader.ImageRecordReader; import org.deeplearning4j.api.storage.StatsStorage; import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator; import org.deeplearning4j.datasets.iterator.MultipleEpochsIterator; import org.deeplearning4j.eval.Evaluation; import org.deeplearning4j.nn.api.OptimizationAlgorithm; import org.deeplearning4j.nn.conf.*; import org.deeplearning4j.nn.conf.distribution.Distribution; import org.deeplearning4j.nn.conf.distribution.GaussianDistribution; import org.deeplearning4j.nn.conf.distribution.NormalDistribution; import org.deeplearning4j.nn.conf.distribution.TruncatedNormalDistribution; import org.deeplearning4j.nn.conf.graph.MergeVertex; import org.deeplearning4j.nn.conf.inputs.InputType; import org.deeplearning4j.nn.conf.layers.*; import org.deeplearning4j.nn.graph.ComputationGraph; import org.deeplearning4j.nn.multilayer.MultiLayerNetwork; import org.deeplearning4j.nn.weights.WeightInit; import org.deeplearning4j.optimize.listeners.ScoreIterationListener; import org.deeplearning4j.ui.api.UIServer; import org.deeplearning4j.ui.stats.StatsListener; import org.deeplearning4j.ui.storage.InMemoryStatsStorage; import org.deeplearning4j.util.ModelSerializer; import org.nd4j.linalg.activations.Activation; import org.nd4j.linalg.dataset.DataSet; import org.nd4j.linalg.dataset.api.iterator.DataSetIterator; import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization; import org.nd4j.linalg.dataset.api.preprocessor.ImagePreProcessingScaler; import org.nd4j.linalg.learning.config.AdaDelta; import org.nd4j.linalg.learning.config.IUpdater; import org.nd4j.linalg.learning.config.Nesterovs; import org.nd4j.linalg.lossfunctions.LossFunctions; import org.nd4j.linalg.schedule.ScheduleType; import org.nd4j.linalg.schedule.StepSchedule; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.File; import java.util.List; import java.util.Random; import static java.lang.Math.toIntExact; /** * Animal Classification * * Example classification of photos from 4 different animals (bear, duck, deer, turtle). * * References: * - U.S. Fish and Wildlife Service (animal sample dataset): http://digitalmedia.fws.gov/cdm/ * - Tiny ImageNet Classification with CNN: http://cs231n.stanford.edu/reports/2015/pdfs/leonyao_final.pdf * * CHALLENGE: Current setup gets low score results. Can you improve the scores? Some approaches: * - Add additional images to the dataset * - Apply more transforms to dataset * - Increase epochs * - Try different model configurations * - Tune by adjusting learning rate, updaters, activation & loss functions, regularization, ... */ public class AnimalsClassification { protected static final Logger log = LoggerFactory.getLogger(AnimalsClassification.class); protected static int height = 100; protected static int width = 100; protected static int channels = 3; protected static int batchSize = 20; // protected static long seed = 42; private static long seed = 1234; protected static Random rng = new Random(seed); protected static int epochs = 50; protected static double splitTrainTest = 0.8; protected static boolean save = false; protected static String modelType = "AlexNet"; // LeNet, AlexNet or Custom but you need to fill it out private int numLabels; public void run(String[] args) throws Exception { log.info("Load data...."); /**cd * Data Setup -> organize and limit data file paths: * - mainPath = path to image files * - fileSplit = define basic dataset split with limits on format * - pathFilter = define additional file load filter to limit size and balance batch content **/ ParentPathLabelGenerator labelMaker = new ParentPathLabelGenerator(); File mainPath = new File(System.getProperty("user.dir"), "dl4j-examples/src/main/resources/animals/"); FileSplit fileSplit = new FileSplit(mainPath, NativeImageLoader.ALLOWED_FORMATS, rng); int numExamples = toIntExact(fileSplit.length()); numLabels = fileSplit.getRootDir().listFiles(File::isDirectory).length; //This only works if your root is clean: only label subdirs. BalancedPathFilter pathFilter = new BalancedPathFilter(rng, labelMaker, numExamples, numLabels, batchSize); /** * Data Setup -> train test split * - inputSplit = define train and test split **/ InputSplit[] inputSplit = fileSplit.sample(pathFilter, splitTrainTest, 1 - splitTrainTest); InputSplit trainData = inputSplit[0]; InputSplit testData = inputSplit[1]; /** * Data Setup -> transformation * - Transform = how to tranform images and generate large dataset to train on **/ // ImageTransform flipTransform1 = new FlipImageTransform(rng); // ImageTransform flipTransform2 = new FlipImageTransform(new Random(123)); // ImageTransform warpTransform = new WarpImageTransform(rng, 42); // ImageTransform colorTransform = new ColorConversionTransform(new Random(seed), COLOR_BGR2YCrCb); // List<ImageTransform> transforms = Arrays.asList(new ImageTransform[]{flipTransform1, warpTransform, flipTransform2}); /** * Data Setup -> normalization * - how to normalize images and generate large dataset to train on **/ DataNormalization scaler = new ImagePreProcessingScaler(0, 1); log.info("Build model...."); // Uncomment below to try AlexNet. Note change height and width to at least 100 // MultiLayerNetwork network = new AlexNet(height, width, channels, numLabels, seed, iterations).init(); // MultiLayerNetwork network; // switch (modelType) { // case "LeNet": // network = lenetModel(); // break; // case "AlexNet": // network = alexnetModel(); // break; // case "custom": // network = customModel(); // break; // default: // throw new InvalidInputTypeException("Incorrect model provided."); // } ComputationGraph network = new ComputationGraph(graphBuilder()); network.init(); // network.setListeners(new ScoreIterationListener(listenerFreq)); UIServer uiServer = UIServer.getInstance(); StatsStorage statsStorage = new InMemoryStatsStorage(); uiServer.attach(statsStorage); network.setListeners(new StatsListener( statsStorage),new ScoreIterationListener(1)); /** * Data Setup -> define how to load data into net: * - recordReader = the reader that loads and converts image data pass in inputSplit to initialize * - dataIter = a generator that only loads one batch at a time into memory to save memory * - trainIter = uses MultipleEpochsIterator to ensure model runs through the data for all epochs **/ ImageRecordReader recordReader = new ImageRecordReader(height, width, channels, labelMaker); DataSetIterator dataIter; MultipleEpochsIterator trainIter; log.info("Train model...."); // Train without transformations recordReader.initialize(trainData, null); dataIter = new RecordReaderDataSetIterator(recordReader, batchSize, 1, numLabels); scaler.fit(dataIter); dataIter.setPreProcessor(scaler); trainIter = new MultipleEpochsIterator(epochs, dataIter); network.fit(trainIter); // Train with transformations /* for (ImageTransform transform : transforms) { System.out.print("\nTraining on transformation: " + transform.getClass().toString() + "\n\n"); recordReader.initialize(trainData, transform); dataIter = new RecordReaderDataSetIterator(recordReader, batchSize, 1, numLabels); scaler.fit(dataIter); dataIter.setPreProcessor(scaler); trainIter = new MultipleEpochsIterator(epochs, dataIter); network.fit(trainIter); }*/ log.info("Evaluate model...."); recordReader.initialize(testData); dataIter = new RecordReaderDataSetIterator(recordReader, batchSize, 1, numLabels); scaler.fit(dataIter); dataIter.setPreProcessor(scaler); Evaluation eval = network.evaluate(dataIter); log.info(eval.stats(true)); // Example on how to get predict results with trained model. Result for first example in minibatch is printed dataIter.reset(); DataSet testDataSet = dataIter.next(); List<String> allClassLabels = recordReader.getLabels(); int labelIndex = testDataSet.getLabels().argMax(1).getInt(0); // int[] predictedClasses = network.predict(testDataSet.getFeatures()); String expectedResult = allClassLabels.get(labelIndex); // String modelPrediction = allClassLabels.get(predictedClasses[0]); // System.out.print("\nFor a single example that is labeled " + expectedResult + " the model predicted " + modelPrediction + "\n\n"); if (save) { log.info("Save model...."); String basePath = FilenameUtils.concat(System.getProperty("user.dir"), "src/main/resources/"); ModelSerializer.writeModel(network, basePath + "model.bin", true); } log.info("****************Example finished********************"); } private ConvolutionLayer convInit(String name, int in, int out, int[] kernel, int[] stride, int[] pad, double bias) { return new ConvolutionLayer.Builder(kernel, stride, pad).name(name).nIn(in).nOut(out).biasInit(bias).build(); } private ConvolutionLayer conv3x3(String name, int out, double bias) { return new ConvolutionLayer.Builder(new int[]{3,3}, new int[] {1,1}, new int[] {1,1}).name(name).nOut(out).biasInit(bias).build(); } private ConvolutionLayer conv5x5(String name, int out, int[] stride, int[] pad, double bias) { return new ConvolutionLayer.Builder(new int[]{5,5}, stride, pad).name(name).nOut(out).biasInit(bias).build(); } private SubsamplingLayer maxPool(String name, int[] kernel) { return new SubsamplingLayer.Builder(kernel, new int[]{2,2}).name(name).build(); } private DenseLayer fullyConnected(String name, int out, double bias, double dropOut, Distribution dist) { return new DenseLayer.Builder().name(name).nOut(out).biasInit(bias).dropOut(dropOut).dist(dist).build(); } public MultiLayerNetwork lenetModel() { /** * Revisde Lenet Model approach developed by ramgo2 achieves slightly above random * Reference: https://gist.github.com/ramgo2/833f12e92359a2da9e5c2fb6333351c5 **/ MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder() .seed(seed) .l2(0.005) .activation(Activation.RELU) .weightInit(WeightInit.XAVIER) .updater(new Nesterovs(0.0001,0.9)) .list() .layer(0, convInit("cnn1", channels, 50 , new int[]{5, 5}, new int[]{1, 1}, new int[]{0, 0}, 0)) .layer(1, maxPool("maxpool1", new int[]{2,2})) .layer(2, conv5x5("cnn2", 100, new int[]{5, 5}, new int[]{1, 1}, 0)) .layer(3, maxPool("maxool2", new int[]{2,2})) .layer(4, new DenseLayer.Builder().nOut(500).build()) .layer(5, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD) .nOut(numLabels) .activation(Activation.SOFTMAX) .build()) .backprop(true).pretrain(false) .setInputType(InputType.convolutional(height, width, channels)) .build(); return new MultiLayerNetwork(conf); } public MultiLayerNetwork alexnetModel() { /** * AlexNet model interpretation based on the original paper ImageNet Classification with Deep Convolutional Neural Networks * and the imagenetExample code referenced. * http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf **/ double nonZeroBias = 1; double dropOut = 0.5; MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder() .seed(seed) .weightInit(WeightInit.DISTRIBUTION) .dist(new NormalDistribution(0.0, 0.01)) .activation(Activation.RELU) .updater(new Nesterovs(new StepSchedule(ScheduleType.ITERATION, 1e-2, 0.1, 100000), 0.9)) .biasUpdater(new Nesterovs(new StepSchedule(ScheduleType.ITERATION, 2e-2, 0.1, 100000), 0.9)) .gradientNormalization(GradientNormalization.RenormalizeL2PerLayer) // normalize to prevent vanishing or exploding gradients .l2(5 * 1e-4) .list() .layer(0, convInit("cnn1", channels, 96, new int[]{11, 11}, new int[]{4, 4}, new int[]{3, 3}, 0)) .layer(1, new LocalResponseNormalization.Builder().name("lrn1").build()) .layer(2, maxPool("maxpool1", new int[]{3,3})) .layer(3, conv5x5("cnn2", 256, new int[] {1,1}, new int[] {2,2}, nonZeroBias)) .layer(4, new LocalResponseNormalization.Builder().name("lrn2").build()) .layer(5, maxPool("maxpool2", new int[]{3,3})) .layer(6,conv3x3("cnn3", 384, 0)) .layer(7,conv3x3("cnn4", 384, nonZeroBias)) .layer(8,conv3x3("cnn5", 256, nonZeroBias)) .layer(9, maxPool("maxpool3", new int[]{3,3})) .layer(10, fullyConnected("ffn1", 4096, nonZeroBias, dropOut, new GaussianDistribution(0, 0.005))) .layer(11, fullyConnected("ffn2", 4096, nonZeroBias, dropOut, new GaussianDistribution(0, 0.005))) .layer(12, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD) .name("output") .nOut(numLabels) .activation(Activation.SOFTMAX) .build()) .backprop(true) .pretrain(false) .setInputType(InputType.convolutional(height, width, channels)) .build(); return new MultiLayerNetwork(conf); } public static MultiLayerNetwork customModel() { /** * Use this method to build your own custom model. **/ return null; } @Builder.Default private int[] inputShape = new int[] {3, 512, 512}; @Builder.Default private int numClasses = 0; @Builder.Default private WeightInit weightInit = WeightInit.RELU; @Builder.Default private IUpdater updater = new AdaDelta(); @Builder.Default private CacheMode cacheMode = CacheMode.NONE; // @Builder.Default private WorkspaceMode workspaceMode = WorkspaceMode.ENABLED; @Builder.Default private WorkspaceMode workspaceMode = WorkspaceMode.SINGLE; @Builder.Default private ConvolutionLayer.AlgoMode cudnnAlgoMode = ConvolutionLayer.AlgoMode.PREFER_FASTEST; public ComputationGraphConfiguration graphBuilder() { ComputationGraphConfiguration.GraphBuilder graph = new NeuralNetConfiguration.Builder().seed(seed) .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) .updater(updater) .weightInit(weightInit) .dist(new TruncatedNormalDistribution(0.0, 0.5)) .l2(5e-5) .miniBatch(true) .cacheMode(cacheMode) .trainingWorkspaceMode(workspaceMode) .inferenceWorkspaceMode(workspaceMode) .graphBuilder(); graph .addLayer("conv1-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(64) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "input") .addLayer("conv1-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(64) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv1-1") .addLayer("pool1", new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX).kernelSize(2,2) .build(), "conv1-2") .addLayer("conv2-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(128) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "pool1") .addLayer("conv2-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(128) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv2-1") .addLayer("pool2", new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX).kernelSize(2,2) .build(), "conv2-2") .addLayer("conv3-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(256) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "pool2") .addLayer("conv3-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(256) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv3-1") .addLayer("pool3", new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX).kernelSize(2,2) .build(), "conv3-2") .addLayer("conv4-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(512) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "pool3") .addLayer("conv4-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(512) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv4-1") .addLayer("drop4", new DropoutLayer.Builder(0.5).build(), "conv4-2") .addLayer("pool4", new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX).kernelSize(2,2) .build(), "drop4") .addLayer("conv5-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(1024) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "pool4") .addLayer("conv5-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(1024) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv5-1") .addLayer("drop5", new DropoutLayer.Builder(0.5).build(), "conv5-2") // up6 .addLayer("up6-1", new Upsampling2D.Builder(2).build(), "drop5") .addLayer("up6-2", new ConvolutionLayer.Builder(2,2).stride(1,1).nOut(512) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "up6-1") .addVertex("merge6", new MergeVertex(), "drop4", "up6-2") .addLayer("conv6-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(512) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "merge6") .addLayer("conv6-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(512) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv6-1") // up7 .addLayer("up7-1", new Upsampling2D.Builder(2).build(), "conv6-2") .addLayer("up7-2", new ConvolutionLayer.Builder(2,2).stride(1,1).nOut(256) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "up7-1") .addVertex("merge7", new MergeVertex(), "conv3-2", "up7-2") .addLayer("conv7-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(256) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "merge7") .addLayer("conv7-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(256) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv7-1") // up8 .addLayer("up8-1", new Upsampling2D.Builder(2).build(), "conv7-2") .addLayer("up8-2", new ConvolutionLayer.Builder(2,2).stride(1,1).nOut(128) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "up8-1") .addVertex("merge8", new MergeVertex(), "conv2-2", "up8-2") .addLayer("conv8-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(128) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "merge8") .addLayer("conv8-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(128) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv8-1") // up9 .addLayer("up9-1", new Upsampling2D.Builder(2).build(), "conv8-2") .addLayer("up9-2", new ConvolutionLayer.Builder(2,2).stride(1,1).nOut(64) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "up8-1") .addVertex("merge9", new MergeVertex(), "conv1-2", "up9-2") .addLayer("conv9-1", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(64) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "merge9") .addLayer("conv9-2", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(64) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv9-1") .addLayer("conv9-3", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(2) .convolutionMode(ConvolutionMode.Same).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.RELU).build(), "conv9-2") .addLayer("conv10", new ConvolutionLayer.Builder(3,3).stride(1,1).nOut(1) .convolutionMode(ConvolutionMode.Truncate).cudnnAlgoMode(cudnnAlgoMode) .activation(Activation.SIGMOID).build(), "conv9-3") .addLayer("output", new CnnLossLayer.Builder(LossFunctions.LossFunction.MCXENT).build(), "conv10") .setOutputs("output").backprop(true).pretrain(false); return graph.build(); } public static void main(String[] args) throws Exception { new AnimalsClassification().run(args); } }Thanks a lot.
-
"How to calculate the Hadoop cluster size? "
How Can Hadoop cluster size be calculated? I'm working on a small college project and facing problem in a Hadoop Cluster size.
-
Copying data incrementally (delta data) from an API endpoint by HTTP GET to an Azure SQL DB on Azure Data Factory
I am trying to perform an ETL activity in which data hosted in JSON at an API is incrementally copied into an Azure SQL Database table. The problem I'm having is that I am not sure how to account for the new/changed entries. I don't want to delete everything and do a massive copy each time the pipeline is run... Are there any suggestions? The only help I've been able to find thus far cover scenarios in which an Azure SQL Database table is the source instead of the sink...
-
How to call stored procedure on Teradata with SSIS sql execution task?
I have a problem with calling a stored procedure with parameter on Teradata. I use SSDT 2015 and Teradata 15.10.1.10. The stored procedure has one parameter:
in VALID_FROM varchar(10)To escape date-conversion problems I choose to make input variable with type varchar and then inside the stored procedure I make conversion
cast(VALID_FROM as date)and use it in queries.So, I have two blocks is SSIS: 1) "GetInputParameter" (SQL execution task) that returns string with date. 2) "ExecuteProcedureOnDest" (SQL execution task) that calling stored procedure on Teradata through ODBC driver.
I checked that input parameter is a string with format 'YYYY-MM-DD', so it seems to me, that everything is ok. But when I run the package, I see an error on task "ExecuteProcedureOnDest":
[Teradata][ODBC Teradata Driver][Teradata Database] SPINTORGLISTPREPARE:Invalid date.
I will be very grateful for help in solving this problem!
-
Azure Data Factory HTTP connector to parse a webpage
A bit new to using Azure for ETL and Machine Learning.
I want to parse a webpage such as here and convert it into a labeled catalog of structured data, to which I can apply ML transforms.
I was reading up the Azure documentation on HTTP Connector but I am unclear on a stepwise process to do so using either the Azure Data factory UI or scripts.
Can Azure Data Factory be used to such a parsing task and if yes, is there clear documentation on how the Azure Data Factory UI can be used to do so?
-
Druid count differ when we run same query on daliy and row data
When run query to ABS Data Source in Druid.I got some count but that differ when same query run with ABS_DAILY data source. And we make ABS_DAILY from ABS.
{ "queryType" : "groupBy", "dataSource" : "ABS", "granularity" : "all", "intervals" : [ "2018-07-12T00:00:00.000Z/2018-07-13T00:00:00.000Z" ], "descending" : "false", "aggregations" : [ { "type" : "count", "name" : "COUNT", "fieldName" : "COUNT" } ], "postAggregations" : [ ], "dimensions" : [ "event_id" ] }Below json used for submit Daily job for druid which will create segments for ABS_DALIY for specific time
{ "spec": { "ioConfig": { "firehose": { "dataSource": "ABS", "interval": "2018-07-12T00:00:00.000Z/2018-07-13T00:00:00.000Z", "metrics": null, "dimensions": null, "type": "ingestSegment" }, "type": "index" }, "dataSchema": { "granularitySpec": { "queryGranularity": "day", "intervals": [ "2018-07-12T00:00:00.000Z/2018-07-13T00:00:00.000Z" ], "segmentGranularity": "day", "type": "uniform" }, "dataSource": "ABS_DAILY", "metricsSpec": [], "parser": { "parseSpec": { "timestampSpec": { "column": "server_timestamp", "format": "dd MMMM, yyyy (HH:mm:ss)" }, "dimensionsSpec": { "dimensionExclusions": [ "server_timestamp" ], "dimensions": [] }, "format": "json" }, "type": "string" } } }, "type": "index" }I quired to ABS_DAILY with below it return different result than ABS Count. Which it should not.
{ "queryType" : "groupBy", "dataSource" : "ERS_DAILY", "granularity" : "all", "intervals" : [ "2018-07-12T00:00:00.000Z/2018-07-13T00:00:00.000Z" ], "descending" : "false", "aggregations" : [ { "type" : "count", "name" : "COUNT", "fieldName" : "COUNT" } ], "postAggregations" : [ ], "dimensions" : [ "event_id" ] } -
how to do pre-calculate in Druid
I have more than 10TB data ingested to Druid every day, The data format is: srcIP,DestIP,Traffic.
There would be some frequent queries, such as:
to get the max/min/average of traffic every minute.Actually we have about 100K records ingested to Druid every minute. So I need to pre-calculate the metrics on every minute other wise the query would be slow.Therefore is there any efficient way I can very quickly pre-calculate min/max/ave of traffic for every minute. Also, the user may like to get the
min/max/averageof traffic by hour or half day. -
Resource limit exceeded in druid groupBy Query
I am trying to run groupBy Query above the limit of 500k data. I am getting this error.
{ "error": "Resource limit exceeded", "errorMessage": "Not enough dictionary space to execute this query. Try increasing druid.query.groupBy.maxMergingDictionarySize or enable disk spilling by setting druid.query.groupBy.maxOnDiskStorage to a positive number.", "errorClass": "io.druid.query.ResourceLimitExceededException", "host": "ubuntu:8083" }How can i use groupBy Query above the limit of 500k data? I am using druid 0.11.0 version and groupBy v2 engine. I have 1 million row data that i am running query on it.
I have tried increasing the
druid.query.groupBy.maxMergingDictionarySizeanddruid.query.groupBy.maxOnDiskStoragebut this does not solve my problem.