How to limit number of threads all concurrent methods at once
I am working on a legacy project which uses Java 8, Spring, HikariCP, and MySQL. Microservices' methods are triggered with a Kafka topic and start a reporting operation. Almost all triggered methods have this and some of them have the same usage inside their blocks.
new ForkJoinPool().submit(() -> { users.parallelStream().forEach(user ->
The application creates 8-9k threads and all of them try to get or create a record. However, the database couldn't handle these requests and started to throw exceptions and Zabbix sends mails about heap memory usage above %90:
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30000ms.
When I check the database and see the variable for max_connections = 600, but this is not enough.
I want to set a limit for thread count for the application level. I tried setting these parameters but the thread size doesn't decrease.
SPRING_TASK_EXECUTION_POOL_QUEUE-CAPACITY , SPRING_TASK_EXECUTION_POOL_MAX-SIZE, -Djava.util.concurrent.ForkJoinPool.common.parallelism
Is there any property to solve this problem?
1 answer
-
answered 2022-04-28 07:27
oguz
I have changed all
new ForkJoinPool() to ForkJoinPool.commonPool()
and use this parameter to control thread creation-Djava.util.concurrent.ForkJoinPool.common.parallelism
after that I have fixed my problem.
do you know?
how many words do you know
See also questions close to this topic
-
Read each name in Array list to create seperate object for
I have a file that has student names, age, and an id number. I have a student class that holds the everything above for each student object. I stored all the names, id numbers. and age separately in an array list. Now im trying to assign the info to create a student object.
public class Student { private String lName; private int idNumber; private int age; public Student() { lName = ""; idNumber = 0; age = 0; } public Student(String l, int i, int a) { lName = l; idNumber = i; age = a; } public void setName(String last) { lName = last; } public String getName() { return lName; } public void setIdNum(int num) { idNumber = num; } public int getIdNum() { return idNumber; } public void setAge(int a) { age = a; } public int getAge() { return age; } }
My Text File looks something like this: This info is stored in parallel array lists. I don't quite get how to implement this into an object to pass it into my second contractor method.
Josh 2134 19 Smith 5256 21 Rogers 9248 19 Andrew 7742 20
Here's what I've tried;
public static void main(String[] args) { String file = "studentData.txt"; Scanner reader = new Scanner(file); ArrayList<String> lastNames = lNames(file); ArrayList<Integer> idNumbers = idNum(file); ArrayList<Integer> ageList = ages(file); Scanner input = new Scanner(System.in); Student s1 = new Student(); // confused about how to implement this constructor with the textile info for (int i = 0; i<idNumbers.size(); i++) { Student user = new Student(lastNames.get(i), idNumbers.get(i), ageList.get(i)); } //user enters idNumber to display age System.out.println("Enter ID Number"); //exception handling to be added int idNum = input.nextInt(); for (int i = 0; i<idNumbers.size(); i++) { if (idNum == idNumbers.get(i)) { s1.setAge(ageList.get(i)); System.out.println(s1.getAge()); } } }
-
Using EdittextPreference for Goto search
sorry for my poor English. I want to use EditTextPreference in my bottom nav like the pic below, ![screenshot][1]
I have recycleview xml in my project with in many sub cardview layouts(which is scrollable) and I want to create item in the bottom nav which is called "Goto". When the "Goto" item is clicked i want it to pop-up like the screenshot. And when user enters a number(according to the cardviews i.e if the number of cardview is 40 user must enter 1-40) I want to search the cardview by its ID. Thank you and I hope u got it, If u have any questions let me know [1]: https://i.stack.imgur.com/grK8P.jpg
My xml format look like this. As you see in the blow since the cardviews are huge in number it is not cool to scroll all the way down that is why i need Goto item in the bottom nav to search it by its ID when the user click number in the EditTextPreference as u see in the screenshot. i.e The screenshot is not from my app
<LinearLayout> <LinearLayout> <androidx.cardview.widget.CardView> <RealtiveLayout> <Textview/> <RealtiveLayout> </androidx.cardview.widget.CardView> </LinearLayout> <LinearLayout> <androidx.cardview.widget.CardView> <RealtiveLayout> <Textview/> <RealtiveLayout> </androidx.cardview.widget.CardView> </LinearLayout> <LinearLayout> <androidx.cardview.widget.CardView> <RealtiveLayout> <Textview/> <RealtiveLayout> </androidx.cardview.widget.CardView> </LinearLayout> <LinearLayout> <androidx.cardview.widget.CardView> <RealtiveLayout> <Textview/> <RealtiveLayout> </androidx.cardview.widget.CardView> </LinearLayout> .. .. .. .. many more..
-
How to get remaining time of the day in java?
I would like to calculate the time remaining for next day 00:00:00 from the current date time.
For e.g. time difference between 2
022-05-07T05:49:41.883807900Z
and2022-05-08T00:00:00Z
Expected answer:
18:10:19
or 65419 (in seconds).How can I achieve this with efficiently using java 8?
-
Hibernate handles id assigning instead of database
I have an auto-increment PK in a table and I want hibernate to handle id assigning instead of database. As my understanding the
@GeneratedValue(strategy = GenerationType.IDENTITY)
lets the database generate a new value with each insertion operation. So do we have any different solution to handle it? -
How can I delete a row by its SKU instead of its ID?
I try to delete the row using the sku of the product. I'm using spring boot and angular. I got an error when I added the sku on my button like this one
(click)="onDeleteProductBySku(deleteClick?.sku)"
it said that theProperty 'sku' does not exist on type '(product: Product) => void'.
. On my command prompt, I got this error. How can I solve this problem?Error: product/product.component.html:50:109 - error TS2339: Property 'sku' does not exist on type '(product: Product) => void'. 50 <button class="btn btn-outline-danger btn-sm me-2" (click)="onDeleteProductBySku(deleteClick?.sku)">Delete</button> product/product.component.ts:12:16 12 templateUrl: './product.component.html', ~~~~~~~~~~~~~~~~~~~~~~~~~~ Error occurs in the template of component ProductComponent.
ProductsController.java --> This is working on the postman.
//Delete a product record using sku //http://localhost:8080/products/deletebysku?sku=12345678 @DeleteMapping("/products/deletebysku") @ResponseBody private void deleteProductBySku(@RequestParam String sku){ productsService.deleteProductBySku(sku); }
product.component.ts
public deleteProduct!: Product; public onDeleteProductBySku(sku: string): void { this.productServive.deleteProductBySku(sku).subscribe( (response: void) => { this.messageShow(); console.log(response); this.getAllProduct(); }, (error: HttpErrorResponse) => { this.errorMessage(error.message); } ); } public deleteClick(product: Product) { this.deleteProduct = product; console.log("delete by sku"); }
product.service.ts
public deleteProductBySku(sku: string): Observable<void> { return this.http.delete<void>(`${this.apiServerUrl}/products/deletebysku?sku=${sku}`); }
product.component.html
<button class="btn btn-outline-danger btn-sm me-2" (click)="onDeleteProductBySku(deleteClick?.sku)">Delete</button>
-
Multithreading using springboot
I’m new to multithreading and I want to know how to approach it for my code, currently files are getting generated and processed into fielder using single thread hence unable to handle bulk file generation abs this needs to be handled using multithreading . How do I approach this ? Using java 11 and xml format springboot approach .
-
Write a program using Multithreading in C++ while using pthread_join()?
Write a program in the shell using threads. Each thread must be doing one of the tasks mentioned below. One task must be completed and then another task should be started. Use pthread_join() system call to make the thread wait for the other thread's completion. Get the input at the start and use these numbers in all threads. The numbers must consist of four digits at least. i.e 1234. a) Finding the largest of three numbers b) Reversing the largest number c) Sum of individual digits of a 4-digit number (1234 -> 1+2+3+4=10)
-
Will volatile work for Collections and objects?
Trying to understand the theory about
volatile
. They say,volatile
provides happens-before, so changes done in thread 1 should be visible in thread 2.What if we will put volatile for Collection? Will java provide visibility in Thread 2 for changes done in Thread 1 ? And why?
public class Vol { volatile List<String> list = new ArrayList<>(); // thread 1 public void put(String s) { list.add(s); } // thread 2 public List<String> get(){ // all changes are visible? "Why" for No\Yes answer return list; } }
-
Spring Cloud Dataflow - Set max-connection-pool for Composed Task Runner
I've encountered an issue on Spring Cloud Dataflow when running multiple composed tasks at once.
Hikari DataSource takes 10 connections from the connection pool by default. When running for example 10 composed tasks at once, this means 100 connections + connections required for every task on each composed task.
I tried running the Composed Task Runner locally with
spring.datasource.hikari.maximum-pool-size=1
and it worked.Is there any way how to set this property to every Composed Task Runner by default ? I did not find any documentation related to modifying things like this for composed tasks.
-
Hikari connection pook with oracle on azure
We are using Hikari Connection pool to connect to Oracle on Azure. We tried increasing the timeout value to 12000000 after consecutive connection time out error with a value 600000.
Error is intermittent 2022-04-06 08:32:18 java.lang.Exception: Error while getting connection from connection pool for:#### Connection is not available, request timed out after 300001ms.
What ever be the time we get this error intermittently.
-
JDBC Connection not being released
I'm calling a stored procedure below, everything works fine but I have observed that (through visualizing connections on db) for some reason application does not releases the connection after executing below code and every time this code is executed, a new connection is created until the limit reaches and its not able to acquire the JDBC connection anymore.
NOTE: the em (entity manager) is autowired here:
Session session = em.unwrap(Session.class); try{ ProcedureCall call = session.createStoredProcedureCall("sp_name"); Output outputs = call.getOutputs().getCurrent(); List<Object[]> resultList = ((ResultSetOutput) outputs).getResultList(); session.close(); return resultList; }catch(Exception e){ session.close(); return null; }
Below is the HikariCP configuration
spring.datasource.hikari.maximumPoolSize=10 spring.datasource.hikari.minimumIdle=2 spring.datasource.hikari.idleTimeout=15000 spring.datasource.hikari.maxLifetime=600000 spring.datasource.hikari.connectionTimeout=30000
-
Make a parallel process by modifying my original code as little as possible
I have this class:
package metodo_java8_moderno; import java.util.ArrayList; import java.util.List; public class SingleThreadClass { public List<Object> editAndAdd(List<List<Long>> matrix){ Long limit = (long) matrix.get(0).size()*matrix.size()/2; List<Long> numbers = new ArrayList<>(); for(int i=0; i<matrix.get(0).size(); i++){ for(int j=0; j<matrix.size(); j++){ if(matrix.get(j).get(i).longValue() <= limit){ numbers.add(matrix.get(j).get(i)); matrix.get(j).set(i,null); } } } List<Object> objectList = new ArrayList<>(); objectList.add(matrix); objectList.add(numbers); return objectList; } }
I want to parallel only the following piece by exploiting all the cores of my CPU:
for(int j=0; j<matrix.size(); j++){ if(matrix.get(j).get(i).longValue() <= limit){ numbers.add(matrix.get(j).get(i)); matrix.get(j).set(i,null); } }
I get an error and I believe it is due to the sharing of objects between different threads.
I post all the code of my work:
ElementMutator.java
package metodo_java8_moderno; @FunctionalInterface public interface ElementMutator<T> { void apply(int i); }
ForkJoinListMutator.java
package metodo_java8_moderno; import java.util.List; import java.util.concurrent.ForkJoinPool; import java.util.concurrent.RecursiveAction; public class ForkJoinListMutator { public static final int DEFAULT_SEQ_THRESHOLD = 10000; private static final ForkJoinListMutator defaultInstance = new ForkJoinListMutator(ForkJoinPool.commonPool()); private final ForkJoinPool forkJoinPool; public ForkJoinListMutator(ForkJoinPool forkJoinPool) { this.forkJoinPool = forkJoinPool; } public static ForkJoinListMutator getDefault() { return defaultInstance; } public <T> void mutate(List<T> list, ElementMutator<T> mutator) { mutate(list, DEFAULT_SEQ_THRESHOLD, mutator); } public <T> void mutate(List<T> list, int seqThreshold, ElementMutator<T> mutator) { MutateTask<T> mainTask = new MutateTask<>(list, seqThreshold, mutator); forkJoinPool.invoke(mainTask); } private static class MutateTask<T> extends RecursiveAction { private final List<T> list; private final int start; private final int end; private final int seqThreshold; private final ElementMutator<T> mutator; public MutateTask(List<T> list, int seqThreshold, ElementMutator<T> mutator) { this(list, 0, list.size(), seqThreshold, mutator); } public MutateTask(List<T> list, int start, int end, int seqThreshold, ElementMutator<T> mutator) { this.list = list; this.start = start; this.end = end; this.seqThreshold = seqThreshold; this.mutator = mutator; } @Override protected void compute() { final int length = end - start; if (length <= seqThreshold) { computeSequentially(); } else { MutateTask<T> leftTask = new MutateTask<>(list, start, start+length/2, seqThreshold, mutator); leftTask.fork(); leftTask.join(); MutateTask<T> rightTask = new MutateTask<>(list, start+length/2, end, seqThreshold, mutator); rightTask.compute(); } } private void computeSequentially() { for (int i = start; i < end; i++) { mutator.apply(i); } } } }
ForkJoinListMutatorExample.java
package metodo_java8_moderno; import java.util.List; public class ForkJoinListMutatorExample { public static void main(String args[]) { GenerateMatrix generateMatrix = new GenerateMatrix(); List<List<Long>> matrix = generateMatrix.generate(3,4); System.out.println(matrix); long t1 = System.nanoTime(); SingleThreadClass singleThreadClass = new SingleThreadClass(); List<Object> output = singleThreadClass.editAndAdd(matrix); long t2 = System.nanoTime(); System.out.println("Time taken single thread process: " + (t2-t1)/100000000); List<List<Long>> newMatrix = (List<List<Long>>) output.get(0); List<Long> numbers = (List<Long>) output.get(1); System.out.println(newMatrix); System.out.println(numbers); t1 = System.nanoTime(); MultiThreadClass multiThreadClass = new MultiThreadClass(); output = multiThreadClass.editAndAdd(matrix); t2 = System.nanoTime(); System.out.println("Time taken multi thread process: " + (t2-t1)/100000000); List<List<Long>> newMatrix2 = (List<List<Long>>) output.get(0); List<Long> numbers2 = (List<Long>) output.get(1); System.out.println(newMatrix2); System.out.println(numbers2); } }
GenerateMatrix.java
package metodo_java8_moderno; import java.util.ArrayList; import java.util.List; import java.util.Random; public class GenerateMatrix { public List<List<Long>> generate(int columns, int rows){ List<List<Long>> matrix = new ArrayList<>(); List<Long> row; Random randomGenerator = new Random(); for(int i=0; i<rows; i++){ row = new ArrayList<>(); for(int j=0; j<columns; j++){ row.add((long) randomGenerator.nextInt(columns*rows+1)); } matrix.add(row); } return matrix; } }
MultiThreadClass.java
package metodo_java8_moderno; import java.util.ArrayList; import java.util.List; import java.util.concurrent.locks.ReentrantReadWriteLock; public class MultiThreadClass { private ReentrantReadWriteLock reentrantReadWriteLock = new ReentrantReadWriteLock(); public List<List<Long>> matrix; public void setMatrix(List<List<Long>> matrix) { reentrantReadWriteLock.writeLock().lock(); try { this.matrix = matrix; } catch (Exception e){ this.matrix = null; } finally { reentrantReadWriteLock.writeLock().unlock(); } } public List<List<Long>> getMatrix() { reentrantReadWriteLock.readLock().lock(); List<List<Long>> matrix; try { matrix = this.matrix; } catch (Exception e){ matrix = null; } finally { reentrantReadWriteLock.readLock().unlock(); } return matrix; } public int i; public void setI(int i) { reentrantReadWriteLock.writeLock().lock(); try { this.i = i; } catch (Exception e){ this.i = -1; } finally { reentrantReadWriteLock.writeLock().unlock(); } } public int getI() { reentrantReadWriteLock.readLock().lock(); int i; try { i = this.i; } catch (Exception e){ i = -1; } finally { reentrantReadWriteLock.readLock().unlock(); } return i; } public List<Long> numbers; public void setNumbers(List<Long> numbers) { reentrantReadWriteLock.writeLock().lock(); try { this.numbers = numbers; } catch (Exception e){ this.numbers = null; } finally { reentrantReadWriteLock.writeLock().unlock(); } } public List<Long> getNumbers() { reentrantReadWriteLock.readLock().lock(); List<Long> numbers; try { numbers = this.numbers; } catch (Exception e){ numbers = null; } finally { reentrantReadWriteLock.readLock().unlock(); } return numbers; } public Long limit; public void setLimit(Long limit) { reentrantReadWriteLock.writeLock().lock(); try { this.limit = limit; } catch (Exception e){ this.limit = null; } finally { reentrantReadWriteLock.writeLock().unlock(); } } public Long getLimit() { reentrantReadWriteLock.readLock().lock(); Long limit; try { limit = this.limit; } catch (Exception e){ limit = null; } finally { reentrantReadWriteLock.readLock().unlock(); } return limit; } public List<Object> editAndAdd(List<List<Long>> matrix){ this.matrix = matrix; this.limit = (long) this.matrix.get(0).size()*this.matrix.size()/2; this.numbers = new ArrayList<>(); int core = Runtime.getRuntime().availableProcessors(); for(int i=0; i<this.matrix.get(0).size(); i++){ this.i = i; ForkJoinListMutator listMutator = ForkJoinListMutator.getDefault(); listMutator.mutate(this.matrix,Math.max(1,this.matrix.size()/(core*1+1)),(j) -> parallelFor(j)); } List<Object> objectList = new ArrayList<>(); objectList.add(this.matrix); objectList.add(this.numbers); return objectList; } public void parallelFor(int j){ try{ List<List<Long>> matrix = getMatrix(); int i = getI(); List<Long> numbers = getNumbers(); Long limit = getLimit(); if(matrix.get(j).get(i).longValue() <= limit){ numbers.add(matrix.get(j).get(i)); matrix.get(j).set(i,null); } setMatrix(matrix); setI(i); setNumbers(numbers); setLimit(limit); //System.out.println(">> "+this.matrix); //System.out.println(">> "+this.numbers); }catch (Exception e){ System.out.println("Errore!"); System.out.println(e.getMessage()); System.out.println(e.getCause()); } } }
SingleThreadClass.java
package metodo_java8_moderno; import java.util.ArrayList; import java.util.List; public class SingleThreadClass { public List<Object> editAndAdd(List<List<Long>> matrix){ Long limit = (long) matrix.get(0).size()*matrix.size()/2; List<Long> numbers = new ArrayList<>(); for(int i=0; i<matrix.get(0).size(); i++){ for(int j=0; j<matrix.size(); j++){ if(matrix.get(j).get(i).longValue() <= limit){ numbers.add(matrix.get(j).get(i)); matrix.get(j).set(i,null); } } } List<Object> objectList = new ArrayList<>(); objectList.add(matrix); objectList.add(numbers); return objectList; } }
-
Forkjoinpool VS sequential perofrmance
I am comparing sequential and parallel performance(using ForkJoinPool) of an algorithm(sum of first n numbers):
public class ForkJoinSumCalculator extends RecursiveTask<Long> { private static final ForkJoinPool FORKJOINPOOL = new ForkJoinPool(); private final long[] numbers; private final int start; private final int end; public static final long THRESHOLD = 10_000; public static void main(String[] args) { long startTime = System.currentTimeMillis(); int numLoops = 40; for(int i = 1; i <= numLoops; i++) { ForkJoinSumCalculator forkJoinSumCalculator = new ForkJoinSumCalculator(LongStream.rangeClosed(1, 100000000).toArray()); FORKJOINPOOL.invoke(forkJoinSumCalculator); } System.out.println("Total time parallel:"+ (System.currentTimeMillis() - startTime)); startTime = System.currentTimeMillis(); for(int i = 1; i <= numLoops ; i++) { long seqSum = 0L; for(int j = 1; j <= 100000000 ; j++) { seqSum = seqSum + j; } } System.out.println("Total time sequential:"+ (System.currentTimeMillis() - startTime)); } public ForkJoinSumCalculator(long[] numbers) { this(numbers, 0, numbers.length); } private ForkJoinSumCalculator(long[] numbers, int start, int end) { this.numbers = numbers; this.start = start; this.end = end; } @Override protected Long compute() { ....splitting the task ....or calculating the sum if size is less than THRESHOLD } }
I tried to vary numLoops for a wide range of values, but always sequential approach perform better and that too by order of 3-4.
Shouldn't parallel version perform better here given that array size is not that small.