Artem Konovalov – Software Engineer

3 Concurrency Models

2024-08-11T00:00:00+00:00

First, the goal of the post isn’t covering all existing concurrency models. The post is inspired by the book “Seven Concurrency Models in Seven Weeks When Threads Unravel” by Paul Butcher.

Before diving into the topic, it seems important to remind the difference between concurrency and parallelism. These two terms are pretty similar but have different meanings. The good explanation I have found in the book “Solution Architect’s Handbook” by Shrivastava:

People often get confused between parallelism and concurrency by thinking they are both the same thing; however, concurrency is different from parallelism. In parallelism, your application divides an enormous task into smaller subtasks, which it can process in parallel with a dedicated resource for each subtask. In concurrency, however, an application processes multiple tasks simultaneously by utilising shared resources among the threads.

In other words, concurrency is a part of implementation, but parallelism is a way of combining different tasks and isn’t part of internal implementation.
Now, we are ready to move forward.

Mutex and blockers

The model is straightforward and intuitive. When we work with this model, we have to deal with threads face-to-face. We have to think about how to split a task into subtasks, how to initiate threads, how to synchronize threads by mutex and locks. Convenience and in which level programming is error-prone highly depends on tools that programming language gives and guarantees that its memory model provides.

To make it clear, let’s write a simple example that calculates the count of words. We can do it concurrently. We use here the classic Producer-Consumer Pattern. There are two types of threads: Readers and Counters.

The Readers fetches file content, splits text into words and put them in the common queue. The last readers operation is decreasing number of running readers.

@AllArgsConstructor  
private static class Reader implements Callable<Boolean> {  
    private final Queue<String> queue;  
    private final String file;  
    private final AtomicInteger countOfRunningReaders;  
  
    @Override  
    @SneakyThrows   
    public Boolean call() {  
        Files.lines(Path.of(file))  
                .flatMap(line -> Arrays.stream(line.split(" ")))  
                .forEach(queue::add);  
  
        countOfRunningReaders.decrementAndGet();  
        return true;  
    }  
}

The Counters read words from the queue and calculate numbers of words. If the count of running readers equals 0, Counters stop working.

@AllArgsConstructor  
private static class Counter implements Callable<Map<String, Integer>> {  
    private final BlockingQueue<String> queue;  
    private final AtomicInteger countOfRunningReaders;  
  
    @Override  
    @SneakyThrows    
    public Map<String, Integer> call() {  
        var counter = new HashMap<String, Integer>();  
  
        while (countOfRunningReaders.get() > 0) {  
            var word = queue.poll(200, TimeUnit.MILLISECONDS);  
            if (word == null)  
                continue;  
            counter.merge(word, 1, (k, v) -> v + 1);  
        }  
  
        return counter;  
    }  
}

The last step is to initiate the queue and thread pool. It will be used to submit our readers and counters. When Counters finish working, we get word numbers calculated by each Counter and combine their results into a sharable map.

var files = List.of("file1.txt", "file2.txt", "file3.txt");  
var countOfRunningReaders = new AtomicInteger(files.size()); 
var queue = new LinkedBlockingQueue<String>();  

var readers = files  
        .stream()  
        .map(file -> new Reader(queue, file, countOfRunningReaders))  
        .toList();  
var counters = IntStream.range(0, 3)  
        .mapToObj(e -> new Counter(queue, countOfRunningReaders))  
        .toList();  
  
var threadPool = Executors.newFixedThreadPool(readers.size() + counters.size());  
threadPool.invokeAll(readers);  
var totalMap = new HashMap<String, Integer>();  
for (Future<Map<String, Integer>> future : threadPool.invokeAll(counters)) {  
    var result = future.get();  
    result.forEach((k, v) -> totalMap.merge(k, v, (k1, v1) -> v1 + v));  
}  
  
System.out.println(totalMap);

Functional programming

Last years, functional programming is getting popular. Its elements are appearing even in object-oriented programming languages. It has several key concepts:

Functions are first-citizen objects. They might be created, passed as arguments, returned as a result.
Functions tend to be pure functions, i.e., they don’t have side effects and deal only with data taken as parameters.
It encourages immutable values. It means if we need to change state, we have to create a new one with different values.

The mentioned above information is enough for us to rewrite the program from the previous chapter in functional style.

Alike the previous example, we need a function that reads file content.

@SneakyThrows  
private static Stream<String> readWords(String file) {  
    return Files
			.lines(Path.of(file))  
            .flatMap(e -> Arrays.stream(e.split(" ")));  
}

Then we create a parallel stream that performs operations in parallel. As we can see the code is more laconic and readable in comparison to the previous example. Thanks to pure functions and their composition.

var result = List
		.of("file1.txt", "file2.txt", "file3.txt")  
        .parallelStream()  
        .flatMap(file -> readWords(file))  
        .map(word -> Pair.create(word, 1))  
        .collect(
			Collectors.toConcurrentMap(
						        Pair::getLeft, 
						        Pair::getRight, 
						        Integer::sum)
		);

Actors

Actor model is another way of working concurrency. An actor stands for an operation. Each operation is performed by single thread associated to the operation. This fact lets us not think about synchronization between threads, because code looks like a one-threaded code. Actors communicate each other by sending events. An event contains input data for another actor. All sent events are added to an actor queue and handled one by one. The diagram below depicts how our Word Counter program works:

File Reader actor receives events that contain file names and then sends events to Line Splitter. Events contain only single line. According to the name, the actor splits lines and sends events. Each event involves a word. When Words Counter actor receives the event, it changes internal state to keep current number of words.

Also, worth to mention, with the actor model we can create distributed programs easily. Events might be sent over the network as well as within one host.

Is 3-tier Architecture anti-pattern?

2023-10-24T00:00:00+00:00

In this topic, I would like to discuss about 3-tier architecture, what kind of drawbacks it has and what might be used instead of it.

Three-tier Architecture

Let’s talk about three-tier architecture. It’s a well-known pattern that defines in which way we should organise modules of our programs. The 3-tier architecture is very popular across web applications, but it might be applied to not only implementing this type of application. Based on the name, you might guess that there are 3 tiers:

Presentation Layer
Logic Layer
Data Layer

Presentation Layer contains logic that is in charge of providing data via REST or other protocols. The code of this layer handles requests, takes needed data, transforms the data to required formats and sends responses. Usually, a framework is responsible for most of these actions but a programmer still needs to declare API and to define from which place data will be taken and where data should be passed to.
Logic Layer contains business logic e.g. creating new objects, changing values of existing domain objects and so on. It’s the most interesting part of the application.
The last one is the Data Layer. This layer provides access to a database in a convenient way. Due to the Data Layer, we don’t have to think about how to persist and retrieve objects from a database. Moreover, in theory, this abstraction tier lets switch database type without changing business logic e.g. from MySQL to PostgreSQL, or even to a NoSQL solution.
As a rule, each tier is represented by a package that contains all classes associated with a corresponding tier. Let’s look at an example. It is a simple content management system. The system provides a Web Interface and REST API to operate posts, comments, and users. The picture below depicts the structure of the 3-tier architecture of the application. For convenience, I mark the related classes with the same emojis.

.
└──ru
   └── izebit
           ├── ApplicationLauncher.java 
           ├── controller
           │   ├── UserController.java 👽
           │   ├── PostController.java 🗒️
           │   └── CommentController.java ✏️
           ├── repository
           │   ├── UserRepository.java 👽
           │   ├── PostRepository.java 🗒️
           │   ├── CommentRepository.java ✏️
           │   └── model
           │       ├── User.java 👽
           │       ├── Post.java 🗒️
           │       └── Comment.java ✏️
           └── service
               ├── UserService.java 👽
               ├── AuthenticationService.java 👽
               ├── StopSpamService.java ✏️
               ├── CommentService.java ✏️
               ├── PostService.java 🗒️
               ├── SpellCheckerService.java 🗒️
               └── dto
                   ├── User.java 👽
                   ├── Post.java 🗒️
                   └── Comment.java ✏️

As we can see, there are classes grouped in packaged by type, not by domain. All classes must be public because they should be accessible from other packages, otherwise, controllers can’t work with services and services can’t work with repositories.
Moreover, not related classes are also accessible to each other inside a package, for example, AuthenticationService and SpellCheckerService. In this way, based only on the structure, it’s tough to understand the relationship between classes. In addition, the current package structure contradicts the Encapsulation principle.

A language mechanism for restricting direct access to some of the object’s components.

Wikipedia

To restrict access to components from other packages, we might place all related classes into the same “business logic” package and expose only classes that are used by other components. Other classes might have restricted modifiers e.g. package-private or private modifiers. This approach can be implemented with Hexagonal Architecture.

Hexagonal Architecture

The backbone of Hexagonal Architecture is Ports and Adapters. Business logic has boundaries and is isolated from other components. There is no direct access to it from other packages.
Ports provide a shareable API that might be used to work with our business logic and vice versa i.e. to define API for interacting with the external world as well. Thus, it stands for “WHAT our system can do and other systems can do with”.
Adapters define in which way ports might be invoked and which way our system might work with other components. One port might have many adapters. In this way, adapter stands for “HOW the system and other systems interact each other”.
This approach lets us write maintainable code with isolated domain-specific logic. All dependencies are represented as interfaces and might be mocked if necessary. It makes writing tests easier. Also, it affects flexibility because we can change adapters without touching core logic. In other words, this design encourages Orthogonality.

Let’s look at the structure of the app with Hexagonal Architecture:

.
└── ru
    └── izebit
        ├── ApplicationLauncher.java
        ├── comment
        │   ├── CommentServiceImpl.java
        │   ├── StopSpamService.java
        │   └── ports
        │       ├── in
        │       │   ├── CommentService.java
        │       │   └── adapters
        │       │       └── CommentController.java
        │       ├── models
        │       │   └── Comment.java
        │       └── out
        │           ├── CommentRepository.java
        │           ├── UserService.java
        │           └── adapters
        │               ├── CommentRepositoryImpl.java
        │               └── UserServiceImpl.java
        ├── post
        │   ├── PostServiceImpl.java
        │   ├── SpellCheckerService.java
        │   └── ports
        │       ├── in
        │       │   ├── PostService.java
        │       │   └── adapters
        │       │       └── PostController.java
        │       ├── models
        │       │   └── Post.java
        │       └── out
        │           ├── PostRepository.java
        │           ├── UserService.java
        │           └── adapters
        │               ├── PostRepositoryImpl.java
        │               └── UserServiceImpl.java
        └── user
            ├── AuthenticationService.java
            ├── UserServiceImpl.java
            └── ports
                ├── in
                │   ├── UserService.java
                │   └── adapters
                │       └── UserController.java
                ├── models
                │   └── User.java
                └── out
                    ├── UserRepository.java
                    └── adapters
                        └── UserRepositoryImpl.java

In comparison to the previous one, this structure brings a few additional interfaces and packages, but at the same time, it gives more information on the relationships between components.
The protected package visibility modifier hides all domain-specific classes. Only components of packages */ports/in/ and */ports/model/ must be public i.e. accessible from other packages. By now, there’s still no clear relationship between packages. We need to provide public access to only particular packages.
For instance, during working with comments and posts, we need to check users if they are authenticated, they present in blacklist etc. It means that classes from the post and comment packages should have access to public classes from the user package, but the reverse is false.
Limiting access might be implemented with Java Modules, but we discover another approach.

Spring Modulith

This Spring component lets developers implement logical modules in Spring Boot applications. It supports structural validation, generation of Module Component diagrams etc. In this post, we are going to use a few modules of this Spring component.

First of all, we need to mark our public packages with the NamedInterface annotation.
By default, all top-level packages are considered as modules and there is no need to define them explicitly. We do it only for sub-packages.

@org.springframework.modulith.NamedInterface("in")  
package ru.izebit.user.ports.in;

Then we should define dependencies for each package. For this purpose, we use the following annotation:

@org.springframework.modulith.ApplicationModule(  
        allowedDependencies = "user::in"  
)  
package ru.izebit.comment;

This information on packages are stored in package-info.java files placed in the relevant directories.

After these two steps, we can write a simple test that checks correct packages access.

@Test  
void shouldBeCompliant() {  
    ApplicationModules.of(ApplicationLauncher.class)
				      .verify();  
}

If access is broken, we can see the following output:

org.springframework.modulith.core.Violations: - Module 'comment' depends on module 'user' via ru.izebit.comment.ports.in.adapters.CommentController -> ru.izebit.user.ports.out.UserRepository. Allowed targets: user::in.

The error message is descriptive and gives enough information to find out the root of the issue.
In addition to checking package access violations, one of the features of Spring Modulith is generation of UML diagrams. If you make the code below a part of build pipeline, you will always have up-date diagrams of your system.

void writeDocumentationSnippets() {  
    new Documenter(modules)  
            .writeModuleCanvases()  
            .writeModulesAsPlantUml()  
            .writeIndividualModulesAsPlantUml();  
}

The picture depicts C4 PlantUML diagram of the application.

You can find the source code of the application here.

Garbage Collectors in JVM-based languages

2023-10-16T00:00:00+00:00

In this topic I would like to talk about garbage collectors (GCs). The aim that I target is to give a brief overview of main concepts of Garbage Collectors and list available GC implementations.

What is Garbage Collector?
Serial Garbage Collector
Parallel Garbage Collector
Concurrent Mark Sweep (CMS) Collector
G1 - Garbage First Collector
ZGC - The Z Garbage Collector
ShenandoahGC
Azul C4 Garbage Collector
Epsilon Garbage Collector

What is Garbage Collector?

Garbage Collector is a form of automatic memory management
— Wikipedia

The component takes care about reclaiming memory that was allocated for objects needed earlier. There are programming languages that don’t provide this functionality and a programmer has to manage memory manually. Java Virtual Machine (JVM) includes garbage collector. It means that all JVM-based languages e.g. Java, Kotlin, Scala, Clojure supports automatic reclaiming unused objects.
There are many algorithms that might be applied to check if an object is not needed and memory for the object might be reused for other objects. The main idea of them is if there are no references to an object, the object is unreachable and can’t be used. By this way, this type of objects might be deleted safely.

Garbage Collector traverses the reference graph from the several start points. They are called GC Roots. It might be static variables, classes loaded by system class loaders, active threads etc.

There are some common phases of the GC work that encounter in listed below garbage collectors:

Identify live objects that are reachable from the GC Roots
Reclaim memory held by dead objects
Relocate live objects or compact them

Let’s talk about what type of garbage collectors you can use if you write code in JVM-based language.

Serial Garbage Collector

It’s one of the oldest garbage collector and exists from the first version of JVM. Serial Garbage Collector works simply. All running threads are suspended and all references are checked by one thread. It occurs stop-the-world pause.

To enable the GC, use the following command:

java \
-XX:+UseSerialGC # enable Serial Garbage Collector

Parallel Garbage Collector

Based on the name, you might guess how it works. The Parallel Garbage Collector works as Serial Garbage Collector works, but we can set number of threads. By this way, the total stop-the-world pause must be less if garbage collector works in multi-thread mode.

There are several JVM flags that works with Parallel Garbage Collector.

java \
-XX:+UseParallelGC        # enable Parallel Garbage Collector
-XX:MaxGCPauseMillis=<N>  # maximum time interval between GC
-XX:GCTimeRatio=<N>       #the ratio between GC time and time of execution application
-XX:ParallelGCThreads=<N> # number of threads 

Concurrent Mark Sweep (CMS) Collector

The idea behind a work of the garbage collector is The Weak Generational Hypothesis. It says

Objects will die young

i.e. the longer an object lives, the longer it will be reachable.

By this way, there is no need of scanning the whole set of objects. If memory is split into different categories and check them separately, it might be possible to reduce the stop-the-world pause by scanning regions of memory at different rates.

CMS Collector works in pretty the same way. All memory is split into 3 regions:

Permanent Generation
Young Generation
Old Generation

Permanent Generation is a memory region where JVM locates static data e.g. loaded classes metadata, static methods, static variables.
Young Generation is a segment where JVM allocates memory for a new objects. After several GC collections, if an object is still alive, it will be moved to Old Generation.
Old Generation is memory region where “old” objects are located. At some point, object might be placed there directly if capacity of Young Generation doesn’t let do it.

This garbage collector is a combination of two previous ones. Serial Garbage Collector and Parallel Garbage Collector get on the stage at different time.
Initially, Parallel Garbage Collector pauses application threads twice. At initial pause it marks objects that are reachable from the GC Roots as live. Then Parallel Garbage Collector traces the available references concurrently and identifies reachable objects without suspending application threads. Since application is running during this step, checked references might be changed. That’s why Parallel Garbage Collector needs the second remark pause. After it, all objects that aren’t reachable might be removed and live objects are compacted. This operation happens during Concurrent Sweep phase.
When Old Generation becomes full Serial Collector starts working. In this case, all application threads are stopped while collection is done.

There are several JVM flags that might come in handy to work with CMS Collector:

java \
-XX:+UseConcMarkSweepGC      # enable Concurrent Mark & Sweep Garbage Collector
-XX:ConcGCThreads=<N>        # total count of threads for GC
-XX:+CMSScavengeBeforeRemark # run remark stage after GC execution on new generation area
-XX:CMSWaitDuration          # run GC on tenured are after GC execution on new generation
-XX:ParallelGCThreads=<N>    # count of threads for parallel GC execution
-XX:+UseParNewGC             # use parallelism during GC on new generation area

G1 - Garbage First Collector

Making latency predictable for application running on JVM is a challenging task. G1 Collector tries to keep stop-the-world pause in fixed bounders.
In order to do it, all generations are split into equal-sized regions of memory. Based on calculated statistics, G1 can estimate how many regions it can check in set interval.

At some cases, If an object requires larger and equal the size of a half region, this type of object gets allocated as sequence of several regions. This objects are called Humongous Objects .

G1 concentrates efforts on regions of Young Generation and checks regions of Old Generation occasionally. After initial mark the Garbage Collector pick the regions that will yield a large amount of free space. This approach lets the garbage collector be most effective. That’s why it is called Garbage First - blocks with garbage will be handled first.
After each Young Generation Collection, all remaining live objects are evacuated and compacted to an available region of Young Generation or if the threshold is met, objects are moved to a region of Old Generation accordingly.

The following flags might be used to tune G1 settings:

java \
-XX:+UseG1GC                       # enable Garbage First Collector
-XX:MaxGCPauseMillis=<N>           # Maximum interval between GC pauses
-XX:GCPauseIntervalMillis=<N>      # Maximum interval between G1 executions
-XX:G1HeapRegionSize=<size>[k|m|G] # heap memory size
-XX:ParallelGCThreads=<N>          # count of threads for parallel GC execution
-XX:ConcGCThreads=<N>              # total count of threads for GC

ZGC - The Z Garbage Collector

This garbage collector targets to provide low latency, less 10ms and to work with multi-terabytes heap. It guarantees O(1) pause duration. In case of big capacity of heap, a phase with moving objects and compaction takes too much time. Moreover it requires suspending application threads, in order to change references to new locations of relocated objects. ZGC solves this problem concurrently without stop-the-world pause. The trick that lets Z Garbage Collector do it is load GC barriers and coloured references.

There are 3 stages of GCs work:

Concurrent Mark - identify objects as reachable
Concurrent Relocate - to avoid memory fragmentation or promote object to Old Generation, move objects to another region
Concurrent Remap - change references to new locations of evacuated objects.

All these stages are performed without stop-the-world pause, thus application threads are not suspended. Let’s look at how it might be done. 64-bit architecture provides 64 bits for memory addressing. However, not all 64 bits are used. Only 42 bits are used for addressing. By this way, other bits might store meta information. Coloured References are references where 4 bits are used by ZGC.

Finalizable Bit means that object is available through a finalizer Remapped Bit denotes that reference is up to date and points to the current location Marked 0 and Marked 1 bits are used to mark reachable objects.

Load GC barrier is a piece of code injected by JIT Compiler for all references. Each time when ZGC traverse the reference graph or application thread loads a variable through a reference, a load GC barrier checks a coloured reference and there can be 2 cases:

if Remapped bit is 0 then variable might be loaded without any additional actions safely.
otherwise there are following actions must be done
1. reference colour must be fixed
2. some actions must be done with the object atomically depends on the current stage of Garbage Collector, e.g. moving object to another location, change reference to a new location etc.

Like in case of G1 Collector, there are generational regions but have different size. 2 phases:

Minor Collection happens when ZGC works with Young Generation.
Major Collection - the Garbage Collector works with both generations.

As we can see we pays additional cost for loading references but this cost is not huge and it might be a trade-off between slight decreasing of application performance and extremely low GC pauses.

The following command is used to enable the ZGC:

java \
-XX:+UseZGC         # enable ZGC
-XX:+ZGenerational  # Generational ZGC is enabled

ShenandoahGC

This garbage collector is available in RedHat OpenJDK & AdoptOpenJDK builds. It uses the same approach like Z Garbage Collector. The notable difference between two garbage collectors is ShenandoahGC works also on 32-bit architecture.

Instead of using Coloured references, ShenandoahGC stores metadata in HostSpot Object Header. The header involves Forward Pointer that points to a current location or a new location of objects after performing relocate stage. The garbage collector alike ZGC uses Load GC Barriers. The logic of work is the same - to check the current phase of the GC work and to perform some actions depends on the phase or just to follow to the point.

There are 3 stages of the GC work:

Concurrent Mark - mark all reachable objects as alive
Concurrent Evacuation - move live objects to new regions or compact them
Concurrent Update References - update references to new location All these stages might be merged into one. It is called Traversal Mode.

To use this ShenandoahGC, enable the JVM flag below:

java 
-XX:+UseShenandoahGC # enable Shenandoah Garbage Collector

Azul C4 Garbage Collector

The full name of GC is Continuously Concurrent Compacting Collector, or shortly C4. This Garbage Collector might be used with Azul Zulu Builds of OpenJDK. It is generational pauseless GC. It uses read barrier called The Loaded Value Barrier (LVB). Like ZGC stores metadata in unused bits of references. It’s field named Not Marked Through state. Each time when reference is loaded, the barrier checks the field and if additional actions are needed, they are applied. There are 3 phases of C4 algorithm.

Marking - mark all reachable objects
Relocation - move objects in order to free up larger consequent memory regions.
Remapping - change the references to a new location of relocated objects.

Epsilon Garbage Collector

This garbage collector is pretty simple in comparison to the mentioned earlier ones. It doesn’t collector garbage. With enabled Epsilon GC sooner or later java.lang.OutOfMemoryError occurs. For sure, if you don’t have infinite capacity of RAM or your program doesn’t allocate new objects at all. At some case, it might be a good option, if latency is important and overhead from a garbage collector is not acceptable e.g. microbenchmarking, garbage free or short-running programs.

With following flags you can enable this Epsilon Garbage Collector:

java \
XX:+UnlockExperimentalVMOptions   # unlock experimental features
-XX:+UseEpsilonGC                 # enable Epsilon Garbage Collector

Life expectancy visualization

2021-11-26T00:00:00+00:00

country where you live:

your age:

Life behind proxy

2021-10-02T00:00:00+00:00

This topic is a cheat sheet for people who have to work behind proxy.

There are some placeholders that will be used later. It is obvious to guess what they mean, but I give some details to avoid misunderstanding.

PROXY_URL - proxy url, that all your traffic goes through.
PROXY_PORT - proxy port.

PROXY_USER - your login for a proxy.
PROXY_PASS - your password.

apt (package manager of debian-based distribution)

cat /etc/apt/apt.conf.d/proxy.conf

Acquire::https::Verify-Peer "false";
Acquire::https::Verify-Host "false";

Acquire {
  AllowUnauthenticated "true";
  HTTP::proxy "http://<proxy_user>:<proxy_pass>@<PROXY_URL>:<PROXY_PORT>/";
  HTTPS::proxy "http://<proxy_user>:<proxy_pass>@<PROXY_URL>:<PROXY_PORT>/";
}

gradle (dependency manager and build automation tool)

cat gradle.properties

systemProp.http.proxyHost=<PROXY_URL>
systemProp.http.proxyPort=<PROXY_PORT>
systemProp.http.proxyUser=<proxy_user>
systemProp.http.proxyPassword=<PROXY_USER>

systemProp.https.proxyHost=<PROXY_URL>
systemProp.https.proxyPort=<PROXY_PORT>
systemProp.https.proxyUser=<PROXY_USER>
systemProp.https.proxyPassword=<PROXY_PASS>

maven (dependency manager for java projects)

cat ~/.m2/settings.xml

Take your notice, file below involves environment variables. If you are not going to use them, just put ordinal values instead of ${env.*}.

<settings>
  <proxies>
   <proxy>
      <id>http-proxy</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>${env.PROXY_URL}</host>
      <port>${env.PROXY_PORT}</port>
      <username>${env.PROXY_USER}</username>
      <password>${env.PROXY_PASS}</password>
      <nonProxyHosts>localhost|my-domain</nonProxyHosts>
    </proxy>
    <proxy>
      <id>https-proxy</id>
      <active>true</active>
      <protocol>https</protocol>
     <host>${env.PROXY_URL}</host>
      <port>${env.PROXY_PORT}</port>
      <username>${env.PROXY_USER}</username>
      <password>${env.PROXY_PASS}</password>
      <nonProxyHosts>localhost|my-domain</nonProxyHosts>
    </proxy>
  </proxies>

  ...
</settings>

sbt (build tool for scala projects, preemptively)

export SBT_OPTS=" \
-Dhttp.proxySet=true \
-Dhttp.proxyHost=${PROXY_URL} \
-Dhttp.proxyPort=${PROXY_PORT} \
-Dhttp.proxyUser=${PROXY_USER} \
-Dhttp.proxyPassword=${PROXY_PASS} \
-Dhttp.nonProxyHosts=my-domain \
-Dhttps.proxySet=true \
-Dhttps.proxyHost=${PROXY_URL} \
-Dhttps.proxyPort=${PROXY_PORT} \
-Dhttps.proxyUser=${PROXY_USER} \
-Dhttps.proxyPassword=${PROXY_PASS} \
-Dhttps.nonProxyHosts=my-domain \

git (you, definitely, know what it is) 🙂

cat ~/.gitconfig

[http]
    proxy = http://<PROXY_USER>:<PROXY_PASS>@<PROXY_URL>:<PROXY_PORT>
    sslVerify = false
[https]
    proxy = http://<PROXY_USER>:<PROXY_PASS>@<PROXY_URL>:<PROXY_PORT>
    sslVerify = false
[http "local-git-repo"]
    proxy = ""
    sslVerify = false
[https "local-git-repo"]
    proxy = ""
    sslVerify = false

curl (tool for downloading some stuff from the internet)

export http_proxy=http://<PROXY_USER>:<PROXY_PASS>@<PROXY_URL>:<PROXY_PORT>
export https_proxy=http://<PROXY_USER>:<PROXY_PASS>@<PROXY_URL>:<PROXY_PORT>

pip (dependency manager for python projects)

Pip uses environment variables that curl uses, but there is another way to work with it.
The program has a flag called --proxy:

pip install --proxy http://<PROXY_USER>:<PROXY_PASS>@<PROXY_URL>:<PROXY_PORT> ...

You may have some troubles because of wrong SSL certificates. To say PIP to ignore this type of errors, you can also pass the param with all used repositories: --trusted-host=pipy.org --trusted-host=files.pythonhosted.org ....

npm package manager for javascript projects.

To work with npm behind proxy, you should type only 3 commands:

npm config set strict-ssl false
npm config set https-proxy http://<PROXY_USER>:<PROXY_PASS>@<PROXY_URL>:<PROXY_PORT>
npm config set http-proxy http://<PROXY_USER>:<PROXY_PASS>@<PROXY_URL>:<PROXY_PORT>

Reactive spring [ru]

2021-01-12T00:00:00+00:00

In this video, I talk about the reactive features of the spring framework. My goal is to figure out what the reason for reactive programming is, why it is so actual nowadays and how to code in reactive programming paradigm’s way with spring framework.

How to stop java application gracefully

2020-10-26T00:00:00+00:00

_{^{Photo by Jack Anstey on Unsplash}}

Today we are talking about how to stop java process gracefully.

How can we stop your application? The first thing comes to my mind is to kill the application by next shell command: kill -9 <application-pid>. It is possible to save PID app to somewhere, for instance into a file, and then read PID and execute the command. However, in reality sometimes it’s an appropriate way. Imagine the situation - there is an application and this app has some work to do. We can’t interrupt its work because, in this case, the state would be inconsistent. For example, the application is copying files or performing some transactions or something like that. We have to wait the moment when all work has been done, or the app will be ready to stop working on its own.

The code below illustrates the logic of app work:

private boolean isRunning = true;
public void perform(){
    while(isRunning){
      val task = getNextTask();
      if (task == null)
        TimeUnit.SECONDS.sleep(1)
      else
        task.execute();
    }
}

What steps do we have to do in order to stop app gracefully?

First of all, we need to set isRunning to true, then we wait until a task is done. It’s pretty clear, but what initiates this activity? We can send a system termination signal to an application. As I mentioned earlier, kill is an excellent candidate to do it. Please be attentive, the command without flag `-9` . It's crucial because `kill -9` doesn't let us handle this signal at all.

In java, we can handle a termination signal and add some callback if an application received the signal from the operating system or somebody else. java.misc.Signal does exactly want we need. The code below creates callback.

Signal.handle(new Signal("TERM"), new SignalHandler() {
             @Override
             public void handle(final Signal signal) {
                 stopServiceAndWaitUtilTaskIsDone();
             }
});

stopServiceAndWaitUtilTaskIsDone should set isRunning to false and wait until a task is done. The second one can be implemented with flags as well. I leave it to do on your own.

By now, we have everything to stop an application properly. Thank you for reading, see you soon 🖖

TF-IDF with Hadoop Streaming

2020-07-29T00:00:00+00:00

_{^{Photo by Mike Benna on Unsplash}}

Today I want to talk about how we can calculate tf-idf with hadoop streaming.

First of all, for those who don’t know what TF-IDF is, I can explain. It is statistical metrics of words, which reflects the importance of each word to a document. The bigger TF-IDF value of a particular word, and a particular document the more frequently this word appears in a document and the rarely in other documents. You can gather more information from the Wikipedia article. It is widely used in data science tasks, especially related to text processing problems. In this article, I will use a term and a word interchangeably.

Before starting, I give some information about Hadoop streaming. It’s a member of big Hadoop family tools. The component lets process data in a map-reduce way. As you know, the native language for Hadoop is java, but it isn’t necessary to know this language to work with it. In this article, we will use python. Unfortunately, debugging python scripts can’t be a pain in your neck. I highly recommend testing your scripts locally before running them on a Hadoop cluster. More information about Hadoop streaming you can find on the official website

Let’s get to the point, in order to calculate TF-IDF metric, we need to know how we can do it. Look at the formula below

\[TFIDF(term,document) = TF(term, document)*IDF(term)\]

TF is called Term Frequency. So you can look at it as if it is a function TF(term, document) which has several parameters: term and document.

\[TF = \frac{\text{term count in a document}}{\text{total count of words in the same document}}\]

IDF is called Inverse document frequency. It depends on а term and shows how many documents contain a particular term. In other words, It is a function:

\[IDF(term) = \frac{1}{\log_e(1+ \text{count of documents which contain a particular term})}\]

That’s all that I wanted to say about TF-IDF. The next step is the implementation of this formula.

We will compute the metric for each word and an article in 3 steps.

The first stage.

Originally, input data will be a dump of Wikipedia articles. It has a pretty straightforward format:

12	Anarchism is often defined as a political philosophy which holds the state to be undesirable, unnecessary, or harmful.  ...

i.e.

<article_id>\tab<text>

The input data should be split into separated words. Also, we have a list of stop words we should take it into account during parsing text. After the mapping phase, output data will have the next format:

<article_id>\tab<word>\tab<number>

However, map output data contains 3 different types of information.

The first of them contains a count of words in the article, i.e. we will write for each word a row like that:

<article_id>\tab<word>\tab1

The second type contains information about the total count of words. For each word, we will write a row like that:

<article_id>\tab\ \tab1

As you can see, a word is missed.

Finally, the third type contains information about the count of articles contains a particular word. During parsing text, we will write a row for each word which appear for the first time in a text. Its format bellow:

 \tab<word>\tab1

Аn article id would be missed here.

There is a part of mapper-1.py which do all this stuff:

import sys
import re

reload(sys)
sys.setdefaultencoding('utf-8')  # required to convert to unicode

path = 'stop_words_en.txt'
stop_words = set()

# Your code for reading stop words here
for l in open(path):
    line = l.strip()
    stop_words.add(line.strip())

for line in sys.stdin:
    try:
        article_id, text = unicode(line.strip()).split('\t', 1)
    except ValueError as e:
        continue

    text = re.sub("^\W+|\W+$", "", text, flags=re.UNICODE)
    words = re.split("\W*\s+\W*", text, flags=re.UNICODE)

    total_count = 0
    unique_words = set()
    for w in words:
        word = w.lower().strip()
        if word not in stop_words:
            print("%s\t%s\t%s" % (article_id, word, 1))
            print("%s\t%s\t%s" % (article_id, ' ', 1))

            if word not in unique_words:
                print("%s\t%s\t%s" % (" ", word, 1))
                unique_words.add(word)

In a reduce phase, we will group all values by keys: article id and word and sum all of them.

import sys
import re

reload(sys)
sys.setdefaultencoding('utf-8')  # required to convert to unicode

current_article_id = None
current_word = None
current_count = 0

for line in sys.stdin:
    try:
        article_id, word, count = unicode(line).split('\t', 2)
    except ValueError as e:
        raise e

    if article_id == current_article_id and word == current_word:
        current_count = int(count) + current_count
    else:
        if current_article_id:
            print("%s\t%s\t%s" % (current_article_id, current_word, current_count))

        current_article_id = article_id
        current_word = word
        current_count = int(count)

if current_count > 0:
    print("%s\t%s\t%s" % (current_article_id, current_word, current_count))

In order to run hadoop job, we should execute a shell command:

OUT_DIR="intermediate-result-1"
hdfs dfs -rm -r ${OUT_DIR} > /dev/null

yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar \
    -D mapred.output.key.comparator.class=org.apache.hadoop.mapred.lib.KeyFieldBasedComparator \
    -D mapred.jab.name="tf-idf-job" \
    -D mapreduce.job.reduces=5 \
    -D stream.num.map.output.key.fields=2 \
    -D mapred.text.key.partitioner.options="-k1,1" \
    -D mapred.text.key.comparator.options="-k1,1 -k2,2" \
    -files mapper-1.py,reducer-1.py,/datasets/stop_words_en.txt \
    -mapper "python2 mapper-1.py" \
    -reducer "python2 reducer-1.py" \
    -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \
    -input /data/wiki/en_articles_part \
    -output ${OUT_DIR} > /dev/null

The most interesting part here is sorting parameters. They are required to sort rows by 2 keys. In our case, 2 keys are article id and words. A reducer consumes sorted data, and it lets make reducer code simpler. Another interesting thing is to pass a stop_words_en.txt file that contains stop words. This file would be present in the current working directory of the task and would be available to read.

As a result, output data of the first stage would look like that:

varying	3
version	1
video	4
voltage	5
vs	2
waveform	1

The second stage.

In this stage, we will calculate the TF value for each word. The output of the previous stage contains all required information. All work will be done in a reducer. A mapper does nothing. Its output data will be the same as input data.

Below there is a code of mapper-2.py:

import sys

reload(sys)
sys.setdefaultencoding('utf-8')  # required to convert to unicode

for line in sys.stdin:
    print(unicode(line))

In this case, you can try to use in-build mapper org.apache.hadoop.mapred.lib.IdentityMapper and org.apache.hadoop.mapred.lib.IdentityReducer reducer. Unfortunately, I have caught some exceptions related to wrong typecasting, so I had to write own mapper with the same logic. 😕

As I said earlier, a reducer will calculate TF values and skip rows with blank article id values. We will use skipped rows to calculate IDF values the next time.

import sys
import re

reload(sys)
sys.setdefaultencoding('utf-8')  # required to convert to unicode

current_article_id = None
count_terms_in_article = 0
current_word = None

for line in sys.stdin:
    try:
        article_id, word, count = unicode(line).split('\t', 2)
    except ValueError as e:
        print(line)
        continue

    if article_id == " ":
        print("%s\t%s\t%s" % (word, article_id, int(count)))
    elif current_article_id == article_id:
        tf = float(count) / count_terms_in_article
        print("%s\t%s\t%s" % (word, article_id, tf))
    else:
        current_article_id = article_id
        count_terms_in_article = float(count)
        current_word = word

The shell command to run a Hadoop job is the same as the shell command of the previous stage. We sort mapper output data by 2 keys as well.

IN_DIR="intermediate-result-1"
OUT_DIR="intermediate-result-2"
hdfs dfs -rm -r ${OUT_DIR} > /dev/null

yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar \
    -D mapred.output.key.comparator.class=org.apache.hadoop.mapred.lib.KeyFieldBasedComparator \
    -D mapred.jab.name="tf-idf-job-2" \
    -D mapreduce.job.reduces=5 \
    -D stream.num.map.output.key.fields=2 \
    -D mapred.text.key.partitioner.options="-k1,1" \
    -D mapred.text.key.comparator.options="-k1,1 -k2,2" \
    -files reducer-2.py,identity-mapper.py \
    -mapper "python2 identity-mapper.py" \
    -reducer "python2 reducer-2.py" \
    -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \
    -input ${IN_DIR} \
    -output ${OUT_DIR} > /dev/null

As the result, we would have the output data like that:

valued	988	0.00263157894737
various	988	0.00526315789474
view	988	0.00263157894737

The third stage.

There is nothing to do for a mapper, so we would use identity-mapper like in the previous stage. We need to sort mapper output data by 2 keys - word and article id. Input data contains rows with blank article id. We need them to compute ** IDF** then we will multiply IDF values with all values of other rows grouped by word. By this way, we will have TF-IDF metric values calculated for each word and document.

Reducer code is below.

import sys
import re
from math import log

reload(sys)
sys.setdefaultencoding('utf-8')  # required to convert to unicode

count_of_articles_with_term = 0
current_word = None

for line in sys.stdin:
    try:
        word, article_id, num = unicode(line).split('\t', 2)
    except ValueError as e:
        continue

    if current_word == word:
        idf = 1.0 / log(1 + count_of_articles_with_term)
        tf = float(num)
        result = str(tf * idf)
        print("%s\t%s\t%s" % (word, article_id, result))
    else:
        count_of_articles_with_term = int(num)
        current_word = word

A shell command to run a hadoop job is the same.

IN_DIR="intermediate-result-2"
OUT_DIR="result"
hdfs dfs -rm -r ${OUT_DIR} > /dev/null

yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar \
    -D mapred.output.key.comparator.class=org.apache.hadoop.mapred.lib.KeyFieldBasedComparator \
    -D mapred.jab.name="tf-idf-job-3" \
    -D mapreduce.job.reduces=5 \
    -D stream.num.map.output.key.fields=2 \
    -D mapred.text.key.partitioner.options="-k1,1" \
    -D mapred.text.key.comparator.options="-k1,1 -k2,2" \
    -files reducer-3.py,identity-mapper.py \
    -mapper "python2 identity-mapper.py" \
    -reducer "python2 reducer-3.py" \
    -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \
    -input ${IN_DIR} \
    -output ${OUT_DIR} > /dev/null

Hooray 🎉 Everything is done, and we can have a look at the result:

hdfs dfs -cat result/* | grep -P 'labor\t12' | head -1 | awk '{print $3}'

For labor and article id 12 TF-IDF equals 0.00035046896211

There is a docker playground to reproduce all step on your own. Moreover, you can get a Jupyterhub notebook that encompasses all stages on the Github repo.

Not evident JIT optimizations

2020-06-11T00:00:00+00:00

I would like to write about an interesting thing that I have found at my work.

The code is given:

public String pack(final byte[] data) {
        int bufferIdx = 0;
        char[] buffer;
        if (data.length % 2 != 0) {
            buffer = new char[(data.length >> 1) + 2];
            buffer[0] = flag;
            bufferIdx = 1;
        } else {
            buffer = new char[data.length >> 1];
        }
        int idx = 0;
        for (int i = bufferIdx; i < buffer.length; i++) {
            int bpos = idx << 1;
            char c;
            if (bpos + 1 < data.length) {
                c = (char) (((data[bpos] &amp; 0x00FF) << 8) + (data[bpos + 1] &amp; 0x00FF));
            } else {
                c = (char) (((data[bpos] &amp; 0x00FF) << 8));
            }
            buffer[i] = c;
            idx++;
        }
        return new String(buffer);
}

As you can see, the method deserializes byte array into string. The main point here is to convert 2 bytes to 1 char since the size of char in java is 2 bytes.

I have faced that execution of the method is significantly slower on byte arrays with odd length than on byte arrays with not odd length.

It is clear that some job in the loop takes the most time of execution. However, discovering the bytecode is pointless because the compiler does not have a clue what length of a byte array passed as a parameter.

According to this fact, there are some optimizations can be applied only in runtime by JIT compiler. To confirm my guess I used the tool JitWatcher The program shows generated assembler code after each stage of optimization passed in runtime.

Before going ahead, I have to give more details about JitWatcher works. It sets some specific jvm flags for your program:

-XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation -XX:+TraceClassLoading -XX:+PrintAssembly

and collects all information that jvm prints while your application is running. Hence, there is no magic and you can manage without it but without JitWatcher you have to analyze a bunch of assembler code. I bet you would not want to do it 😏. JitWatcher does it instead of you.

To set the flag -XX:+PrintAssembly jvm requires hsdis library. It has to be located in a directory $JAVA_HOME/jre/lib/server/ There are to ways to get the library. First of them is to build it on your own and the second one is to download already compiled artifact.

After spending some time on analysing, I found out that JIT makes some strange speculations. It is about vector instructions. In case with odd length of byte array they are used. However, in case with not odd length they are not used by JIT.

It is well known, usage of the vector instructions can boost performance of your application significantly. I decided to rewrite the loop in order to make JIT use them in all cases. This technique is called loop unrolling

int countOfIteration = data.length / 2;
for (int index = 0; index < countOfIteration; index++) {
      final int position = index * 2;
      char firstPart = (char) ((data[position] &amp; 0x00FF) << 8);
      char secondPart = (char) (data[position + 1] &amp; 0x00FF);
      buffer[index] = (char) (firstPart + secondPart);
}

After making the changes, JIT does not have any chances. It started generate assembler code with vector instructions all the time regardless of a length of a byte array.

The table below shows the difference.

Benchmark	Size of array	Mode	Score	Units
with optimization	100000	avgt	46777.0331	ms/op
with optimization	100001	avgt	47718.458	ms/op
without optimization	100000	avgt	47218.527	ms/op
without optimization	100001	avgt	71312.515	ms/op

CPU optimizations

2020-05-15T00:00:00+00:00

The goal of this article is to cover some cpu optimizations from java developer’s view. We will talk about:

Branch prediction
Cache levels
False sharing
Prefetching
Loop unrolling

Branch prediction

Let’s look at the next code:

private static int countOfElementsLessThen(final int value, final int[] array){
    int result=0;
    for(int anArray : array)
        if (anArray < value)
            result++;
    return result;
}

The method looks pretty good and at the first sight, there is nothing to worry about. Yet this point is delusive.

We can estimate the average time taken for invocation the method (for this purpose JMH benchmark is used). Let’s take an array that contains numbers. Yet in the first case, the array is unordered and in the second one, the array is sorted. Surprisingly, the results of our estimations are different significantly.

Benchmark	Mode	Count	Score	Error	Units
sorted array	avgt	10	0.631	± 0.035	ms/op
unsorted array	avgt	10	3.957	± 0.142	ms/op

As we can see, type of input array matters. The thing is modern cpu executes instructions not sequentially. They do it with so-called Instruction pipelining. Thus, an instruction goes through several stages and, as the rule, different instructions take place on different stages at the same time.

The picture below helps understand how it works more clearly.

However, the workflow of execution the method countOfElementsLessThen does not let use the instruction pipelining effectively. Each command needs the result of the execution the previous instruction in order to figure out what the instruction will be executed next. It is slow and there is an optimization for this case . It is called branch prediction. The point of it is trying to predict in which way the workflow will go next time and executing it. At the moment, when CPU has all necessary information to check which branch of the workflow is right. There are two possible scenarios. The first one is CPU prediction is incorrect, so CPU rollbacks the execution result and execute the instructions of the correct branch. The second scenario is the prediction is right, so CPU just uses the current execution result.

I suppose, CPUs predictions are not too sophisticated. In case with the method countOfElementsLessThen, CPU decides to execute instructions in if block based on statistical measures about previous executions. By this way, all instructions are executed sequentially. At some loop iteration when a current number of array is bigger than a parameter value, prediction can be changed. This explanation is quite naive and simplified.

Cache levels

CPU is very performance, and the difference between time to access to RAM memory and CPU registers is enormous. The picture below shows it:

In order to smooth the difference, CPU has several cache levels. As the rule, there are 3 levels: the first level cache, the second level cache and the third level cache. Each level has different capacity and access latency. For sure, it can range from a manufacturer and platform. For example, my laptop has the next configuration:

Cache level	Capacity
First level	256KB
Second level	1MB
Third level	6MB

Also, it should be noted that each CPU core has the first and the second level caches, but the third level cache is only one for all cores.

I don’t want to show a code due to it is quite cumbersome. If you wish to check it, it is here The results I have got is below:

Benchmark	Mode	Count	Score	Error	Units
CpuCacheLevelBenchmark.firstCacheLevel	ss	10	12.528	± 1.804	ms/op
CpuCacheLevelBenchmark.secondCacheLevel	ss	10	15.938	± 1.361	ms/op
CpuCacheLevelBenchmark.thirdCacheLevel	ss	10	21.886	± 6.798	ms/op
CpuCacheLevelBenchmark.ramCacheLevel	ss	10	49.141	± 11.880	ms/op

The tendency is clear - the higher cache level, the more latency we get.

First of all, I would like to say that it is not an optimization. False sharing is an effect that can affect your program performance significantly. To some extends, it follows from the previous optimization.
All data take place in caches in form of cache lines. Sometimes, we can face the situation where we are extremely unlucky and one cache line contains non-related data. Different threads update the data and CPU needs to sync caches of different cores.

To make the explanation clear, suppose there is an object:

Tuple unoptimizedTuple = new Tuple(){
        private volatile int firstValue;
        private volatile int secondValue;
        //other methods
    };

In case, if one tread updates field firstValue and another one updates field secondValue of the same object, we can observe the false sharing effect. To avoid it, we can align an object size of the object with respect to cache lines. After it, the fields take place in different cache lines. There is one of possible solutions:

Tuple optimizedTuple = new Tuple(){
        private volatile int firstValue;
        //non-used fields for aligning
        private long p1,p2,p3,p4,p5,p6,p7,p8;
        private volatile int secondValue;
        //other
    };

Let’s compare the difference:

Benchmark	Mode	Count	Score	Error	Units
FalseSharingBenchmark.invokeWithOptimization	ss	10	62.584	±6.625	ms/op
FalseSharingBenchmark.invokeWitoutOptimization	ss	10	289.808	±26.294	ms/op

Prefetching

As I said early, CPU uses cache to minimize the latency of access to RAM. Initially, CPU tries to get data from the first level cache and in case if the data is absent, CPU tries to do the same with higher level caches. Eventually, if data is not found in the caches (cache miss), CPU loads it from RAM. Obviously, the process of fetching data from caches and especially RAM is not fast. In order to avoid it, CPU uses prefetching.

The logic of the work is pretty simple. The data that is closed to used data is highly likely to be required coming soon. Therefore, CPU should care about existence of this data in a cache.

We can observe the effect of the optimization when we use a linked and array list. An array is allocated sequentially in a memory and there is no problem for CPU to detect the next memory segment for loading. However, prefetching does not work with linked lists 😕. Thus, each node of linked list refers to an arbitrary segment of memory.

The graph and table depict the difference:

Benchmark	Mode	Count	Score	Error	Units
PrefetchingBenchmark.invokeWithOptimization	avgt	10	5.570	± 0.229	ms/op
PrefetchingBenchmark.invokeWithoutOptimization	avgt	10	8.341	± 0.206	ms/op

Loop unrolling

The last CPU optimization that we are talking about in this article is loop unrolling. The essence of it is quite simple. The modern CPUs can apply vectorization techniques to improve performance. If CPU supports it, it has extended registers. For example, CPU that does not, has registers with 8 or 32 bit capacity, but CPU that does, has registers with 128 or 256 bit. Thus, the last one can store a set of data and work with them like it is one value at once.

Let’s look at the code:

int result = 0;
for (long v : array)
    result += v

The code is quite trivial. It sums up all values of an array with odd size (I set to 1 000 001). In my case, CPU doesn’t use vectorization instructions, so we can help it by slightly rewrite the code.

int result = 0;
for (int i = 0; i < array.length - 1; i += 4)
    result += (
        array[i]
        + array[i + 1]
        + array[i + 2]
        + array[i + 3]
        );

result += array[array.length - 1];

After changes, CPU applies vectorization instructions. The results prove our suggestions:

Benchmark	Mode	Cnt	Score	Error	Units
LoopUnrollingBenchmark.invokeWithOptimization	avgt	10	0.390	± 0.021	ms/op
LoopUnrollingBenchmark.invokeWithoutOptimization	avgt	10	0.440	± 0.038	ms/op

That’s all. Thank you for reading. I hope the article will come in handy.

Getting started with Selenide

2020-04-05T00:00:00+00:00

_{^{Photo by Denys Nevozhai on Unsplash}}

Today we are talking about how to create a simple application based on selenide framework.

Show me the code!©

Above all, the selenide framework is a wrapper of selenium framework with a more convenient API. Very often it is used to test UI. However, there is another way of using selenide. Recently, I have faced that our partner did not provide any API except UI panel. It was very sad news for me because I have to scope with that strict.

Within bounds of this task, I had to make a process of getting some reports automatic. The first thing which came to my mind is using selenium or something like that. Unfortunately, selenium framework has quite rigid and cumbersome API and I had a look at selenide. Otherwise, work with it seems to be more enjoyable. It has fluent API and simple and clear work mechanics.

The first step is adding selenide to our dependency manager. In this article we use apache maven.

<dependency>
    <groupId>com.codeborne</groupId>
    <artifactId>selenide</artifactId>
    <version>5.2.4</version>
</dependency>

Next step is to implement some business logic. In this example, we create a program, which would enter LinkedIn.com and accept all friend requests. As you can see, it was pretty trivial and not rocket science.

The code below does what we want to do.

First of all, we need to login to the site. Method login does it.

private static void login(String username, String password) {
        open("http://www.linkedin.com");
        $$(By.tagName("a"))
                .filter(attribute("data-tracking-control-name", "guest_homepage-basic_nav-header-signin"))
                .first()
                .waitUntil(appear, TimeUnit.SECONDS.toMillis(30))
                .click();

        $(By.id("username")).setValue(username);
        $(By.id("password")).setValue(password);

        $$(By.tagName("button"))
                .filter(text("Sign in"))
                .first()
                .click();
}

Then we need to go to a page «My network» and find all requests there and accept them. According to name, the method acceptRequests does what we expect:

private static int acceptRequests() {
        $(By.id("mynetwork-tab-icon")).click();

        $(By.className("mn-community-summary")).waitUntil(appear, TimeUnit.SECONDS.toMillis(10));

        int acceptedRequests = 0;
        while (!Thread.currentThread().isInterrupted()) {
            ElementsCollection elements = $$(By.tagName("button"))
                    .filter(attribute("data-control-name", "accept"));
            if (elements.isEmpty())
                break;

            elements.first().click();
            screenshot("page-" + acceptedRequests);
            refresh();
            acceptedRequests++;
        }

        return acceptedRequests;
}

In addition, we have to create a web driver. A web driver provides some API for working with a web browser. Almost every popular browser has own web driver implementation. However, we use a remote web driver. I will explain why we do it later.

@SneakyThrows
private static WebDriver initWebDriver(String seleniumUrl){
        Configuration.timeout = TimeUnit.SECONDS.toMillis(10);
        Configuration.reportsFolder = Files.createTempDirectory("selenide-build").toAbsolutePath().toString();

        ChromeOptions chromeOptions = new ChromeOptions();
        chromeOptions.setExperimentalOption("useAutomationExtension", false);
        chromeOptions.addArguments(
                "disable-infobars", 
                "--disable-extensions",
                "--disable-dev-shm-usage",
                "--no-sandbox",
                "--incognito",
                "--window-size=1920,1080",
                "--headless",
                "--disable-gpu",
                "--ignore-certificate-errors");

        DesiredCapabilities capabilities = DesiredCapabilities.chrome();
        capabilities.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
        capabilities.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.ACCEPT);
        WebDriver driver = new RemoteWebDriver(new URL(seleniumUrl), capabilities);
        WebDriverRunner.setWebDriver(driver);
        return driver;
}

How it works?

For sure, we do not run our app outside a docker container. In order to do it, we need a docker image. I have chosen the official selenium docker image , you can choose what you wish. This docker image contains a chrome browser and a web driver for working with this browser.

I feel I have to give more details about how it works. Have a look at the picture below.

This diagram shows how we work with a chrome browser. There are several elements of an interaction chain. On the client-side (our program) we use remote driver API. Apparently, a remote driver connects to a chrome driver. A chrome driver is provided by browser maintainers so it knows exactly how to deal with a chrome browser.

You can ask me: «Why it’s so difficult?». I agree with you. On the face of it, this architecture is quite overwhelmed, but there are several reasons to do it. We use a remote browser because it lets us connect to a browser running on a docker container. Moreover, this approach has some pros. For instance, we want to use a browser which does not have a version for our target platform. (A target platform is Linux or Mac OS, but we want to use Microsoft Edge Browser located on a remote host)

Run app and enjoy

The last step is to run our application and see a result of its work. In order to do it, we should run the docker image:

$ docker run -d -p 4444:4444 selenium/standalone-chrome
$ mvn exec:java -Dexec.args="--username <your_username> --password <your_password>"

I hope this topic was useful to you 😉. The source code is below.

How to throw an exception with type `java.lang.Object`

2020-03-03T00:00:00+00:00

_{^{Photo by Hello I’m Nik on Unsplash}}

As you might know, we can throw exceptions that inherit from java.lang.Throwable. This class is a parent for all exception in java. ATHROW is a byte instruction for throwing exceptions. I am curious, are there some type checks for this instruction after compilation. In the article we will generate a small method with bcel lib.

First of all, create a new class ru.izebit.Test. With bcel, it looks pretty easy:

String className = "ru.izebit.Test";
ClassGen classGenerator = new ClassGen(className, "java.lang.Object", "Test", Const.ACC_PUBLIC, new String[0]);
InstructionFactory instructionFactory = new InstructionFactory(classGenerator);

The class involves public static void main(Object...args) method. Inside the method main there is only one command. It is an invocation test method.

InstructionList instructionListOfMainMethod = new InstructionList();
InstructionHandle invokeHandler = instructionListOfMainMethod.append(
instructionFactory.createInvoke(className, "test", Type.VOID, Type.NO_ARGS, Const.INVOKESTATIC));
instructionListOfMainMethod.append(InstructionFactory.createReturn(Type.VOID));

The next code block defines the method main with input parameters.

MethodGen mainMethodGenerator = new MethodGen(
                                              Const.ACC_PUBLIC | Const.ACC_STATIC,
                                              Type.VOID, 
                                              new Type[]{new ArrayType(Type.STRING, 1)}, new String[]{"args"},
                                              "main", 
                                              className,
                                              instructionListOfMainMethod, 
                                              classGenerator.getConstantPool());

Also, it might come in handy to catch all exceptions occur in the method test and print information about them.

InstructionHandle catchBlock = instructionListOfMainMethod.append(
                                               instructionFactory.createFieldAccess(
                                                                            "java.lang.System", 
                                                                            "out",
                                                                            new ObjectType("java.io.PrintStream"), 
                                                                            Const.GETSTATIC));
 instructionListOfMainMethod.append(InstructionConst.DUP);
 instructionListOfMainMethod.append(
                                              new PUSH(classGenerator.getConstantPool(), 
                                              "An exception has been caught"));
 instructionListOfMainMethod.append(
                                               instructionFactory.createInvoke(
                                                                            "java.io.PrintStream", 
                                                                            "println", 
                                                                            Type.VOID, 
                                                                            new Type[]{Type.STRING}, 
                                                                            Const.INVOKEVIRTUAL));
 instructionListOfMainMethod.append(InstructionFactory.createReturn(Type.VOID));

For the catch block, we should declare a try block. This block catches all exceptions with type java.lang.Object. It sounds ridicules, but nothing stops us from doing it, so why not?

mainMethodGenerator.addExceptionHandler(
                                          invokeHandler,
                                          invokeHandler.getNext(), 
                                          catchBlock, 
                                          ObjectType.getInstance("java/lang/Object"));
classGenerator.addMethod(mainMethodGenerator.getMethod());

The next step is to implement method test. All this method does is create a new java.lang.Object and throws it. As you can see, it is not big deal. The code below generates this logic.

InstructionList instructionListOfTestMethod = new InstructionList();
instructionListOfTestMethod.append(instructionFactory.createNew("java/lang/Object"));
instructionListOfTestMethod.append(InstructionConst.DUP);
instructionListOfTestMethod.append(
                instructionFactory.createInvoke(
                                       "java/lang/Object",
                                       "<init>",
                                       Type.VOID, 
                                       Type.NO_ARGS, 
                                       Const.INVOKESPECIAL));
instructionListOfTestMethod.append(InstructionConst.ATHROW);
 instructionListOfTestMethod.append(InstructionFactory.createReturn(Type.VOID));

Also, we should define a declaration of the method public static test() throw Object:

MethodGen testMethodGenerator = new MethodGen(
                                    Const.ACC_PUBLIC | Const.ACC_STATIC,
                                    Type.VOID, 
                                    Type.NO_ARGS, 
                                    new String[0],
                                    "test", 
                                    className,
                                    instructionListOfTestMethod, 
                                    classGenerator.getConstantPool());
 testMethodGenerator.addException("java/lang/Object");
 testMethodGenerator.stripAttributes(true);
 testMethodGenerator.setMaxStack();
 testMethodGenerator.setMaxLocals();
 classGenerator.addMethod(testMethodGenerator.getMethod());

That is all about generation code. We can save our byte code to file.

String outputPath = this.getClass().getProtectionDomain().getCodeSource().getLocation().getPath() + "ru/izebit/Test.class";
try (FileOutputStream fos = new FileOutputStream(outputPath)) {
      classGenerator.getJavaClass().dump(fos);
}

Just in case, if you got lost, I show the generated class completely:

package ru.izebit;

public class Test {
    public static void main(String[] args) {
        try {
            test();
        } catch (Object var1) {
            System.out.println("An exception has been caught");
        }
    }

    public static void test() throws Object {
        throw new Object();
    }
}

Trying to execute our program:

java ru.izebit.Test

Error: Unable to initialize main class ru.izebit.Test
Caused by: java.lang.VerifyError: (class: ru/izebit/Test, method: test signature: ()V) Can only throw Throwable objects

The java byte code validation does not let us execute the program. Fortunately, there is an option to disable this check:

java -Xverify:none ru.izebit.Test

Hooray, after it, you can see the result:

An exception has been caught

Webinar on spring boot [ru]

2020-02-13T00:00:00+00:00

During this webinar you will learn how to create a simple web blog with spring framework.

The video covers next topics:

Spring Boot
Spring Data JPA
Spring MVC and Thymeleaf
Spring Security

Why should you try android development?

2020-01-18T00:00:00+00:00

_{^{Photo by Rami Al-zayat on Unsplash}}

I know, I had to write this article several years ago, but nothing can be too late 😀

It is for you if you write java for backend and have never tried to do something. If you are already a mobile developer, I am not sure you will find something useful in the article, so you can surely skip it.

I try to describe why you should try to develop apps for Android. Currently, I am a backend developer, so I very often compare mobile development with backend development. The reason for it is an absence of any experience except working in the backend.

Ok, let’s go.

The first of the reasons are easy to do. It’s true, because, both have the same stack (JVM), the same languages (Java, Kotlin, Scala or any JVM-based language), even some libs, of course not at all. You will have to learn something specific to mobile development and Android API, but it’s not tough. Nowadays, there is a lot of documentation, articles, posts on the internet about questions, which you will face. You don’t have to spend much time on it, only basic concepts and common knowledge about the android platform. It will be enough to start and write a simple app, then, sure, you will be learning something new again and again 😏

The second thing, you will be isolated from the OS. You will not think about how the operating system launches your apps, how to close etc. Actually, it doesn’t matter. I agree that in every activity you can dig as deep as possible, and Android development isn’t an exception. Nevertheless, it’s not absolutely obligatory to know these details to create beautiful and amazing apps. In backend development, you have to know, for instance, how to install packages in Linux, how to set environment variables, how to run your applications, how to launch аn app as a service etc. Also, if you don’t have special people for it (they are called system admins or DevOps), it will be nice to know how you plan to update your application without downtime, to transfer data, how services interact with each other. You can trust me, you will have to know all these things. Maybe I am mistaken, but mobile development strikes me easier than backend development.

The third thing, you can share it with your friends, parents, and others. All these people don’t have to be programmers to understand and appreciate your work result. You just create an app, upload one to Google Play or just sent it to someone, install it, and it’s all. Everybody can see it, touch it, and use it. If you were a backend developer, you would create functions, modules, services. You couldn’t show your work, maybe, only to other programmers like you. These abstractions are not comprehensive for the average person. It’s happiness to create something and share one with somebody. Mobile development gives you this opportunity in extremely the easiest way.

The fourth thing, creating mobile apps is actually funny. Android OS provides an API for doing a lot of things. You can get access to a camera, GPS, gyroscopes, microphone etc. It gives you many opportunities, which are limited only by your imagination. Moreover, you can create an application, which you can install on your smartphone and use every day. To my mind, it’s a perfect experience and good basics to continue improving.

I hope I have persuaded you to try it. Thanks for reading, take care and good luck.

Code generation at runtime

2019-12-05T00:00:00+00:00

_{^{Photo by Sigmund on Unsplash}}

At some points, we need to generate classes with a particular implementation. Unfortunately, Java is not as so dynamic as Groovy, JavaScript or Python. Anyway, there are several ways for doing it. One of them is generating bytecode outright but this way is sophisticated.

In this article we consider another way. It involves four steps:

to write java code
to compile
to load
to execute.

The standard java library contains javax.tools.JavaCompiler class. We need it to compile code. The method below does it and stores a class in a temp directory. It returns the path to the compiled class.

private static String compile(String className, String code) throws Exception {
    JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
    StandardJavaFileManager fileManager = compiler.getStandardFileManager(null, null, null);
    Path output = Files.createTempDirectory("_" + System.currentTimeMillis());

    JavaCompiler.CompilationTask task = compiler
                .getTask(null,
                        fileManager,
                        null,
                        Arrays.asList("-d", output.toAbsolutePath().toString()),
                        null,
                        singletonList(new JavaSourceFromString(className, code))
                );

    boolean result = task.call();
    if(!result) throw new IllegalStateException("something wrong happened");

    return output.toAbsolutePath().toString();
}

As you can see, method getTask takes a collection of objects as the last parameter. These objects are resources contain information where source code can be gathered. For instance, it can be a file or something else.

Unfortunately, there is no implementation for simple string resources, so we can do it on our own.

private static class JavaSourceFromString extends SimpleJavaFileObject {
     private final String code;

     private JavaSourceFromString(String name, String code) {
          super(URI.create("string:///" + name.replace('.', '/') + Kind.SOURCE.extension), Kind.SOURCE);
          this.code = code;
     }

     public CharSequence getCharContent(boolean ignoreEncodingErrors) {
           return code;
     }
} 

When we have a compiled class, we should load it with a class loader:

private static ClassLoader getClassLoader(String classPath) throws Exception {
  return new URLClassLoader(
                new URL[]{Paths
                        .get(classPath)
                        .toUri()
                        .toURL()
                },
                null);
} 

After it, having ClassLoader, we can create a new object with type Class<?>. The class is not defined in compile time, so we can’t invoke any methods except listed in java.lang.Object directly. Thus, we should use Reflection.

private static void method(Class<?> type, String methodName) throws Exception {
    Method method = type.getDeclaredMethod(methodName);
    method.setAccessible(true);
    System.out.println(method.invoke(type.newInstance()));
}

The code below creates two new classes with the same name ru.izebit.Person. These classes have a method say with different implementation. After loading the classes, it invokes these methods.

ClassLoader classLoader = CustomCompiler.getClassLoader("ru.izebit.Person",
                        "package ru.izebit;\n" +
                        "public class Person {\n" +
                        "    public String say() {\n" +
                        "        return \"i am happy 🤠\";\n" +
                        "    }\n" +
                        "}");

method(classLoader.loadClass("ru.izebit.Person"), "say");

classLoader = CustomCompiler.getClassLoader("ru.izebit.Person",
                        "package ru.izebit;\n" +
                        "public class Person {\n" +
                        "    public String say() {\n" +
                        "        return \"i am tired 😫 \";\n" +
                        "    }\n" +
                        "}");
method(classLoader.loadClass("ru.izebit.Person"), "say");

I would like to notice, that despite having the same names, the program works without any errors. There is no collision because of using different class loaders.
The result of work is below:

i am happy 🤠
i am tired 😫 

The valid parentheses problem

2019-10-25T00:00:00+00:00

_{^{Photo by Clay Banks on Unsplash}}

This problem is about checking all the parentheses are matched, i.e., each open parentheses has a closing parentheses in right order. It’s also so-called “balanced parenthesis problem”. Given a string that contains some symbols and some parentheses. We should say “correct” if all the parentheses are matched, otherwise “invalid”. To make it clear, let’s look at some examples.
() - correct
()( - invalid
(()()) - correct
)()( - invalid

In order not to make the solutions cumbersome, in this article you don’t observe cases with more than one type of parentheses. However, you can extend these solutions for these cases easily.

First of all, there are several solutions for this problem. The less sophisticated of them is an approach with using a stack. We should check each symbol of the string, and we can have 3 cases:

A symbol is an open parentheses (. In this case, we put it into a stack
If a symbol is a closing parentheses ), we take an element from a stack if it is not empty, otherwise the answer is “invalid”
A symbol is a something else, so we just ignore it.

After all, we check the stack and if it is not empty, so the answer is “invalid” otherwise “correct”.

The implementation is quite simple:

public static boolean isCorrect(String str) {
    if(str == null ||  str.strim().length() == 0)
        return false;
    
    Stack<Character> stack = new Stack<>();
    for (int i = 0; i < str.length(); i++)
        if (str.charAt(i) == '(')
            stack.push('(');
        else if (str.charAt(i) == ')') {
            if (stack.isEmpty())
                return false;
            stack.pop();
        }
    
    return stack.isEmpty();
}

We can use the same approach but use a counter instead of a stack. By this way, we should check each symbol from the very beginning. If a symbol is an open parentheses, we increase a counter. If a symbol is a closing parentheses, we decrease the counter. In case if a counter is less than zero, we can give an answer “invalid”, otherwise keep checking symbols. The solution is great to implement with using recursion. The code in scala is below:

def isCorrect(str: String): Boolean = {
  @scala.annotation.tailrec
  def isCorrect(index: Int, result: Int): Boolean = {
    if (str.length == index)
      return result == 0

    str(index) match {
      case '(' => isCorrect(index + 1, result + 1)
      case ')' => result - 1 >= 0 && isCorrect(index + 1, result - 1)
      case _ => isCorrect(index + 1, result)
    }
  }

  isCorrect(0, 0)
}

The last solution is with using multithreading. The main point of it is 2 statements:

(1) count of open parentheses of each prefix of the string must not be less than count of closing parentheses of the rest string.

and the second one is pretty obvious:

(2) count of open parentheses of the whole string must equal count of closing parentheses of the whole string.

Some details are necessary to understand it.
Given: $S_1, S_2, S_3 ... S_n$ - parts of the original string, i.e., $\sum_{i=1}^{n}S_i = S$ is a concatenation of these n parts, and it equals the whole string $S$.

Each $S_i$ has 2 variables:

finalValue - final value. It is like a counter, at least it is estimated in the same way. Each time when we meet ( we increase it, and decrease when we meet )
minValue - minimum value is the minimum of final value when we estimate during the processing of the whole string. $minValue = \min_{1 \leq i \leq n}S_i$

By this way, the two equations should be right:

(1) $\sum_{j=1}^{i-1} finalValue_j + minValue_i \geq 0$ and

(2) $\sum_{j=1}^{i-1} finalValue_j = 0$

the implementation of this solution in scala is below:

def parIsCorrect(str: String) = {
  val threshold = 100

  private def traverse(start: Int, end: Int): (Int, Int) = {
    var finalValue = 0
    var minValue = 0

    var i = start
    while (i < end) {
      if (str(i) == '(')
        finalValue += 1
      else if (str(i) == ')')
        finalValue -= 1

      minValue = Math.min(finalValue, minValue)
      i += 1
    }

    (minValue, finalValue)
  }

  private def reduce(start: Int, end: Int): (Int, Int) = {
    if (end - start <= threshold)
      traverse(start, end)
    else {
      val middle = (end + start) / 2

      val firstTask = new Callable[(Int, Int)] {
        override def call(): (Int, Int) = reduce(start, middle)
      }
      val secondTask = new Callable[(Int, Int)] {
        override def call(): (Int, Int) = reduce(middle, end)
      }

      val results = ForkJoinPool.commonPool().invokeAll(List(firstTask, secondTask))

      val r1 = results(0).get()
      val r2 = results(1).get()

      var finalValue = 0
      var minValue = 0
      minValue = Math.min(minValue, finalValue + r1._1)
      finalValue += r1._2
      minValue = Math.min(minValue, finalValue + r2._1)
      finalValue += r2._2

      (minValue, finalValue)
    }
  }

  reduce(0, str.length) == (0, 0)
}

Migration from mysql to postgresql with liquibase

2019-07-03T00:00:00+00:00

Recently, I had problem about migration tables and data from one relation database to another one. I solved it with one remarkable tool called liquibase. It is cross-platform and written in java.

MySQL database is given, and we want to transfer data from it to Postgresql. Initially, we have to create a change log file for existing tables:

sh liquibase.sh --driver=com.mysql.jdbc.Driver \
      --classpath=<path to jdbc driver> \
      --changeLogFile=<path to some directory>/changelogs.xml \
      --url="<database url, example jdbc:mysql://localhost:3306/test_bd>" \
      --username=<username> \
      --password=<password> \
      --diffTypes="tables, views, columns, indexes, foreign keys, primary keys, unique constraints, data" 
      generateChangeLog

When we have a generated file changelogs.xml, we can restore our database structure from that file. The command bellow creates the tables, indexes etc. in postgresql.

sh liquibase.sh \
      --driver=org.postgresql.Driver \
      --classpath=<path to jdbc driver>\
      --changeLogFile=<path to some directory>/changelogs.xml \
      --url="jdbc:postgresql://localhost:5432/test_bd" \
      --username=<username> \
      --password=<password> \
      update

You should keep in your mind that new database sequences does not keep previous values, and we have to set them manually.

CREATE OR REPLACE FUNCTION setval_schema(schema_name name, raise_notice boolean = false)
    RETURNS VOID AS
-- Sets all the sequences in the schema "schema_name" to the max(id) of every table
$BODY$
 
DECLARE
    row_data RECORD;
    sql_code TEXT;
 
BEGIN
    IF ((SELECT COUNT(*) FROM pg_namespace WHERE nspname = schema_name) = 0) THEN
        RAISE EXCEPTION 'The schema "%" does not exist', schema_name;
    END IF;
 
    FOR sql_code IN
        SELECT 'SELECT SETVAL(' ||quote_literal(N.nspname || '.' || S.relname)|| ', MAX(' ||quote_ident(C.attname)|| ') ) FROM ' || quote_ident(N.nspname) || '.' || quote_ident(T.relname)|| ';' AS sql_code
            FROM pg_class AS S
            INNER JOIN pg_depend AS D ON S.oid = D.objid
            INNER JOIN pg_class AS T ON D.refobjid = T.oid
            INNER JOIN pg_attribute AS C ON D.refobjid = C.attrelid AND D.refobjsubid = C.attnum
            INNER JOIN pg_namespace N ON N.oid = S.relnamespace
            WHERE S.relkind = 'S' AND N.nspname = schema_name
            ORDER BY S.relname
    LOOP
        IF (raise_notice) THEN
            RAISE NOTICE 'sql_code: %', sql_code;
        END IF;
        EXECUTE sql_code;
    END LOOP;
END;
$BODY$
LANGUAGE 'plpgsql' VOLATILE;

SELECT setval_schema('public');

Types of references

2019-04-29T00:00:00+00:00

_{^{Photo by Shane Rounce on Unsplash}}

Today, I am going to talk on type of references. There are four types of them:

strong
weak
phantom
soft

To use them, except the first type, you should use a particular class from package java.lang.refs.

So, what is the difference between them and what special traits do they have? The difference is how garbage collector deals with them.

Strong reference

This type of reference is an ordinary reference that is used each time when you declare an object.

Object obj = new Object();

The garbage collector does not collect the object until at least one strong reference exists to the object. I hope everything is clear about it.

Weak reference

This type is completely ignored by the garbage collector. Even if there are some weak references point to an object, the object will be removed by it. Usually, this kind of references can be useful in cases when we want to avoid memory leaks. One of the cases is caches. When there are no references except weak ones, we do not want to store objects in our caches and want the garbage collector to remove them.

Soft reference

It is similar to the previous type of reference, except one thing. The garbage collector can remove an object that some soft references point to in case OutOfMemoryError occurs. It means if JVM does not have enough memory, the garbage collector deletes these objects.

The usage is them is easy:

SoftReference<String> softReference = new SoftReference<>("it is my answer: ");
WeakReference<BigInteger> weakReference = new WeakReference<>(new BigInteger("42"));
System.out.println(softReference.get() + weakReference.get());

Phantom references

Before staring talking about phantom references, I should give some details about in which way garbage collector releases memory. It goes over a graph of references started from the special vertices GC roots and checks their availability. These vertices denote local variables, method parameters active threads, JNI references, loaded classes loaded by the system loader.

By this way, if the graph does not involve an object, it is considered to be unused. From the garbage collector’s view, it might be removed safely. If a method finalize() is implemented, an object will be removed in two stages:

At the first stage, a method finalize() is called,
Then the garbage collector makes sure that the object is still unavailable and then deletes the object.

Thus, in a method finalize() we might by mistake or on purpose, make an object alive again by passing another object the reference to it.

Getting back to phantom reference, it seems a bit of strange type of reference for one thing, a method get() returns always null. This kind of references is used to implement some logic after calling method finalize(). Thus, we can implement clean up an object in two steps.

Look at creating a phantom reference closely:

PhantomReference<String> phantomReference 
        = new PhantomReference<>("I am a mystical reference type", new ReferenceQueue<>());

As you might have noticed, a constructor takes 2 arguments. First of them is an object that a reference points to and the second one is a queue.
What is the point of it? It will be clear when we figure out how garbage collector deals with references.

when the garbage collector detects an object in one of the cases below:

weak reference exists or
soft reference exists for the object and OutOfMemoryError occurs or
at least, one phantom reference exists.

It calls a method finalize() and then adds the object to the queue. The object is not removed until the queue contains it. By this way, we can create a daemon thread, that retrieves elements from the queue and do some stuff. Moreover, it is already implemented in a class java.misc.Cleaner.

Look at the code below:

Object obj = new Object();
Cleaner.create(obj, () -> System.out.println("the object is removed!"));

The code looks very robust and work with phantom references is hided.

Pay your attention that usage of finalize() is not recommended in many cases and if you do not know exactly what you do, you should avoid using finalization mechanism.

Hand-made collector

2019-04-02T00:00:00+00:00

_{^{Photo by Snejina Nikolova on Unsplash}}

After release java 1.8, we can work with collection with stream API. By this way, we are able to create a sequence of operations that are applied to elements from a some source such as collection, generating function to create either finite or infinite sequence, lines from a file etc. This topic is very broad and discussing it is out of the scope of the current article, and we are not going to talk about it.

In this article we are going to look at the Collectors closely. This kind of objects that let us create a result of performing the whole sequence of operations.

Different types of collectors produce result in different way and different formats. There are already several collectors in java.util.stream.Collectors such as toList that creates a list of elements. Also, some collectors has more complex implementation, for instance, groupBy that groups elements by some criteria.

Our goal is to write a custom collector by our own. The new collector will group strings by the first letter.

First, we should get to know the interface java.util.stream.Collector that we will implement.

interface Collector<T,A,R> {
        Supplier<A>          supplier();
        BiConsumer<A,T>      accumulator();
        BinaryOperator<A>    combiner();
        Function<A,R>        finisher();
        Set<Characteristics> characteristics();
}

There are only five signatures. Some of them are not supposed to be implemented. It is optional and needed only in some specific cases. These cases will be covered later.

supplier() creates additional helper object that performs basic operations on elements.
acumulator() . It combines the result that we calculated earlier and the current element of the stream. It takes an object helper and the current element as parameters.
combiner() . In case if stream is parallel, this function combines the intermediate results performed by different threads.
finisher() returns the final result of execution.
characteristics() According to the name, it returns collector’s charactristics. It worths to list them:
- CONCURRENT the collector can work with parallel stream.
- UNORDERED the collector does not garantee the order of elements. It means if a stream has some order, the final result does not nessasary have the same order.
- IDENTITY_FINISH If a collector has this characteristic, execution finisher() function is skipped.

By now, we know everything what to need to implement our own collector. Let us start with a class helper.

public final class CustomCollectorBuilder {
    private final Map<Character, Set<String>> set;
 
    public CustomCollectorBuilder() {
        this.set = new HashMap<>();
    }
  
    public void add(String str) {
        Set<String> subSet = getSubset(str);
        
        if (subSet == null) 
            return;
        
        subSet.add(str);
    }
  
    public CustomCollectorBuilder addAll(Map<Character, Set<String>> collection) {
        for (Map.Entry<Character, Set<String>> entry : collection.entrySet()) {
            Set<String> subSet = getSubset(entry.getKey());
            subSet.addAll(entry.getValue());
        }
        return this;
    }
            
    public Map<Character, Set<String>> build() {
        return this.set;
    }
  
    private Set<String> getSubset(String str) {
        if (str == null || str.length() == 0) 
            return null;
        return getSubset(str.charAt(0));
    }
    private Set<String> getSubset(Character key) {
        Set<String> subSet = set.get(key);
        if (subSet == null) {
            subSet = new HashSet<>();
            set.put(key, subSet);
        }
        return subSet;
    }
}

I hope this code is clear for you and you do not have problems with understanding it.

The next step is to write the collector.

public final class CustomCollector implements Collector<String, 
                                                        CustomCollectorBuilder, 
                                                        Map<Character, Set<String>>> {

        @Override
        public Supplier<CustomCollectorBuilder> supplier() {
            return CustomCollectorBuilder::new;
        }

        @Override
        public BiConsumer<CustomCollectorBuilder, String> accumulator() {
            return CustomCollectorBuilder::add;
        }

        @Override
        public BinaryOperator<CustomCollectorBuilder> combiner() {
            return (first, second) -> first.addAll(second.build());
        }

        @Override
        public Function<CustomCollectorBuilder, Map<Character, Set<String>>> finisher() {
            return CustomCollectorBuilder::build;
        }

        @Override
        public Set<Characteristics> characteristics() {
            return EnumSet.of(Characteristics.CONCURRENT, Characteristics.UNORDERED);
        }
}

As you can notice, according to collector’s characteristics, the collector works with parallel streams and does not guarantee any order of elements.

That is all what I wanted to share with you. The example of usage of the new collector is demonstrated below:

Collection<String> strings = Arrays.asList("apple", "orange", "banana", "pear", "peach");
Map<Character, Set<String>> result = strings
                                            .stream()
                                            .collect(new CustomCollector());

for (Character character : result.keySet())
    System.out.println(result.get(character));

The result:

[pear, peach]
[apple]
[banana]
[orange]

Caching of hibernate

2019-03-05T00:00:00+00:00

Recently, I faced a task related to caching queries in Hibernate framework. I would like to tell about various types of caches in hibernate and how to use and tune them.

First, there are tree cache levels:

session level cache;
second level cache;
query level cache.

Session level cache

According to its name, it is used during one session to database. Each session has own cache and when a session is closed, a cache is cleaned up. To be more clear, I list an example below:

Session session = HibernateUtil.getSessionFactory().openSession();
try {
    Criteria criteria = session.createCriteria(Account.class);
    criteria.add(Restrictions.eq("id", id));
    Account firstAccount = (Account) criteria.uniqueResult();
    Account secondAccount = (Account) criteria.uniqueResult();
} finally {
    session.close();
}

In this case, we see only query to a database. The second call criteria.uniqueResult() returns a value from a cache. The session cache level is enabled by default, and you have to do nothing to use it in contrast to other ones.
Hibernate framework does not provide implementations for second cache level and query cache level. In order to use them, you have to add some third-party libraries to your project.

The second level cache

This type of cache, or it is also so called sessionFactory affects all sessions. By default, it is disabled. As I said before, you should add some implementation of the cache. I use ehcache but you can use whatever you want. It is worth to mention, the configuration in the article is tested with hibernate 4.0.

I add some dependencies to my pom.xml:

<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-core</artifactId>
    <version>${hibernate.version}</version>
</dependency>
<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-ehcache</artifactId>
    <version>${hibernate.version}</version>
</dependency>
<dependency>
    <groupId>net.sf.ehcache</groupId>
    <artifactId>ehcache-core</artifactId>
    <version>2.6.11</version>
</dependency>

and also add some parameters to hibernate configuration:

<property name="hibernate.cache.use_second_level_cache">
  true
</property>
<property name="hibernate.cache.region.factory_class">
  org.hibernate.cache.ehcache.SingletonEhCacheRegionFactory
</property>
<property name="net.sf.ehcache.configurationResourceName">
  ehcache.xml
</property>

To tune Ehcache, you should create a file ehcache.xml with parameters:

<ehcache>
  <diskStore path="java.io.tmpdir"/>
  <defaultCache
            maxElementsInMemory="100000"
            eternal="false"
            statistics="true"
            timeToIdleSeconds="120"
            timeToLiveSeconds="120"
            overflowToDisk="true"
            diskPersistent="false"
            diskSpoolBufferSizeMB="500"
            diskExpiryThreadIntervalSeconds="120"
            memoryStoreEvictionPolicy="LRU"/>
</ehcache>

To avoid misunderstanding, I should to give some remarks.

if eternal option is true, objects will never be removed from a cache.
timeToLiveSeconds - how many seconds, an object lives in a cache before removing from it.
diskStore option contains path parameter that should be set to a path, where you want objects will be saved. In my case, it is a temporary directory.

It is almost all what we have to do. The last thing is to say hibernate that which entities we want to cache.
If we set mapping via xml, add an option <cache usage="read-write"/>, but if you use annotations, you should add @Cache(usage = CacheConcurrencyStrategy.READ_WRITE) to an entity class.

A usage parameter can have parameters:

when read-write is set, objects are updates concurrently according to a transaction level.
nonstrict-read-write is supposed to update object in cache without blocking in multithreading mode. However, using this mode, objects retrieved from cache might be outdated. This mode is a trade-off between performance and consistency.
read-only can be useful for object that never changed.

Query level cache

To use the query level cache, you should add an implementation to your project as well. Ehcache is an appropriate one and the configuration from previous section is legal for a query level cache. Also, you should add <property name="hibernate.cache.use_query_cache">true</property> to your hibernate configuration and that is all. Each time when you want queries to be cached, you should just call setCacheable(true) for them.

How to create init.d system service

2019-03-03T00:00:00+00:00

Sometimes it is needed not to only run program, but be sure that this process will work after reboot a host or in some unexpected fails. There are several ways. One of them is to look after the process manually and another one is to let an operating system takes care about it on its own. All you need for it is to make your program a system service. This article is about exactly this case. To be more precisely, it says how to create a system service with update-rc.d.

Look at it closely and break the process down into several steps:

create a script to rule life cycle of the program.
grant all needed permissions to the script
register the created script as a linux system service
enable the system service

As you can see, it is not big deal.

1. create a script

On the first stage, we should create a file my-service located in a directory /etc/init.d/.

touch /etc/init.d/my-service

The template of the script is as follows:

#!/bin/bash
#
### BEGIN INIT INFO
# Provides:             my-service
# Required-Start:       $syslog $remote_fs
# Required-Stop:        $syslog $remote_fs
# Should-Start:         $local_fs
# Should-Stop:          $local_fs
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    My Service
# Description:          My Service
### END INIT INFO
#
### BEGIN CHKCONFIG INFO
# chkconfig: 2345 55 25
# description: My Service
### END CHKCONFIG INFO
 
NAME="my-service"
APPLICATION_PATH="/opt/my-super-program"
PID_FILE="$APPLICATION_PATH/file.pid"
COMMAND="/usr/bin/python $APPLICATION_PATH/script.py 2> &1 > /dev/null & "


# run the program and write its pid to a file.
start() {
    echo "Starting $NAME"
    eval $COMMAND
    PID=$(pgrep -f $NAME) 
    echo $PID >> $PID_FILE
    RETVAL=0
}

# stop the process 
# get pid value from the file and kill the process with it.
stop() {
    if [ -f $PID_FILE ]; then
        echo "Shutting down $NAME"
        # kill the process
        kill -9 $(cat $PID_FILE)
        rm -f $PID_FILE
        RETVAL=0
    else
        echo "$NAME is not running."
        RETVAL=0
    fi
}
 
# restart
# nothing complex
restart() {
    stop
    start
}

# in case if list of running processes contains a process run from the a certain directory,
# it mean that our program has been already started otherwise it is stopped.
status() {
    echo `ps -ef` | grep -q "$APPLICATION_PATH"
    if [ "$?" -eq "0" ]; then
        echo "$NAME is running."
        RETVAL=0
    else
        echo "$NAME is not running."
        RETVAL=3
    fi
}

# a function will be called according to a given command 
case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    status)
        status
        ;;
    restart)
        restart
        ;;
    *)
        echo "Usage: {start|stop|status|restart}"
        exit 1
        ;;
esac
exit $RETVAL

Given template is denoted to create a service that is started from the directory APPLICATION_PATH by the command COMMAND. To check the status of the process, PID_FILE is needed. The file contains a pid value of running process. Important, the program has to be run as a daemon and does not print information to console. The prefix _2> &1 > /dev/null & does exactly what we need.

From the previous script, we can notice that the script involves 3 parts: 1) header with comments and system information. This part contains, for example, dependencies of the service and in which order they have to be started and stopped. Also, it contains the system title and system description. 2) the main part that handles all received commands. 3) functions that implement stop, start, status check and so on operations.

2. grant permissions

To run the script, we have to grant some permissions to the file. The command below does it:

chmod +x /etc/init.d/my-service

3. register a script as a system service

In order to make operating system know about existence of our service, we have to register it with next command:

update-rc.d my-service defaults

4. enable a system service

However, it is not all. After registering, our service will be known to the OS and we can execute some command on it. For instance, to run or to stop:

service my-service start

But, the service will not be started automatically after reboot a host. It is default behavior for Linux. To change it, we have to execute a command with next parameters:

update-rc.d my-service enable

After it, your service is completely read to work. 😉

SQL join visualization

2019-02-05T00:00:00+00:00

The tables structure:

select *
from workers;
---------------
name | type
---------------

select *
from phones;
------------------
name | phone
------------------

contacts of Equals and Hashcode methods

2019-01-01T00:00:00+00:00

It is well known that equals() method is declared in java.lang.Object class. It has a naive implementation that compares references of two objects and in case if they are the same it returns true otherwise false.

There are many situations when we need to override this method. If you want your program works in way you expected, you should follow some contracts.

One of them is related to the companion of equals() method. It is hashCode() method. Both methods are extremely important for correct work with data structures based on hash algorithm such as java.util.HashMap or java.util.HashSet. The rule for overriding is simple. If equals method returns true for 2 objects, it means that hashCode should return the same value, but equality of hash codes does not mean that objects are the same. The last part of the statement is explained by hash collision.

It is not only thing that you should do. There is one more contact. According to it, your implementation of equals method should be:

reflexive
Given:
a is an object,
so if we call a.equals(a), the method should always return true
symmetric
Given:
a and b are different objects.
If a.equals(b) returns true, b.equals(a) must return true as well and vice versa. If b.equals(a) returns false, a.equals(b) must return false.
transitive
Given:
a, b, c are different objects.
If a.equals(b) returns true and b.equals(c) returns true, a.equals(c) must be true.
consistent
The result of invocation does not depend on how many times we call a method until we do not change the state of objects. In other words, if a.equals(b) returns true, it will be true until a and b has not been changed.
null
Given:
a is an object.
The result of a.equals(null) is always false.

JVM flags

2018-12-29T00:00:00+00:00

Recently, I needed to decrease the time of stop-the-world pause that happened while Garbage Collector worked. During working on the problem, I found out many interesting jvm flags. In order to keep the information in one place, I created the following table.

type	category	name	description
memory	common	`-Xms<size>[k\|m\|G]`	the initial heap size
		`-Xmx<size>[k\|m\|G]`	maximum heap size
		`-Xss<size>[k\|m\|G]`	execution stack size
		`-XX:NewRatio=<N>`	the ratio between old generation (tenured) and new generation
		`-XX:MinHeapFreeRatio=<N>`	minimum ratio between free space to decrease size of generation
		`-XX:MaxHeapFreeRatio=<N>`	minimum ratio between free space to increase size of generation
	new generation	`-Xmn<size>[k\|m\|G]`	size of new generation memory
		`-XX:NewSize=<size>[k\|m\|G]`	minimum size of young generation memory
		`-XX:MaxNewSize=<size>[k\|m\|G]`	maximum size of young generation memory
		`-XX:SurvivorRatio=<N>`	$$ ratio = \frac{\text{eden}}{\text{survivor} * 2} $$
		`-XX:YoungGenerationSizeIncrement=<N>`	percent of increasing young generation
	old generation	`-XX:TenuredGenerationSizeIncrement=<N>`	percent of increasing old generation
	metaspace	`-XX:MaxMetaspaceSize=<size>[k\|m\|G]`	size of metaspace
	metaspace	`-XX:+CMSClassUnloadingEnabled`	unload unused classes from the memory
garbage collection	common	`-XX:+ScavengeBeforeFullGC`	perform GC for young generation area before Full GC
		`-XX:+CMSScavengeBeforeRemark`	perform GC for young generation area before remark> stage
		`-XX:-DisableExplicitGC`	disable `System.gc()`
		`-XX:CMSInitiatingOccupancyFraction=<N>`	load factor of Tenured> area to run GC
		`-XX:InitiatingHeapOccupancyPercent=<N>`	load factor of heap area to run mark stage
		`-XX:GCHeapFreeLimit=<N>`	free memory limit after full GC. In case if size of free memory is less, OutOfMemoryError occurs
		`-XX:-UseGCOverheadFreeLimit`	activity limit of GC. In case if GC takes more time, OutOfMemoryError occurs
		`-XX:GCTimeLimit=<N>`	time limit for GC. If GC time exceeds this limit, OutOfMemoryError occurs. This ratio is between summary time of execution app and GC time
	Serial GC	`-XX:+UseSerialGC`	enable Serial GC
	Parallel GC	`-XX:+UseParallelGC`	enable Parallel GC
		`-XX:MaxGCPauseMillis=<N>`	maximum time interval between GC
		`-XX:GCTimeRatio=<N>`	the ratio between GC time and time of execution application. $$ \text{GC} = \frac{1}{1+\text{N}} $$
	Concurrent Mark and Swap	`-XX:+UseConcMarkSweepGC`	enable Concurrent Mark and Swap GC
		`-XX:+UseParNewGC`	use parallelism during GC on new generation area
		`-XX:ParallelGCThreads=<N>`	count of threads for parallel GC execution
		`-XX:ConcGCThreads=<N>`	total count of threads for GC
		`-XX:+CMSScavengeBeforeRemark`	run remark stage after GC execution on new generation area
		`-XX:CMSWaitDuration`	run GC on tenured are after GC execution on new generation
	Garbage First	`-XX:+UseG1GC`	enable G1
		`-XX:MaxGCPauseMillis=<N>`	Maximum interval between GC pauses
		`-XX:GCPauseIntervalMillis=<N>`	Maximum interval between G1 executions
		`-XX:G1HeapRegionSize=<size>[k\|m\|G]`	heap memory size
		`-XX:ParallelGCThreads=<N>`	count of threads for parallel GC execution
		`-XX:ConcGCThreads=<N>`	total count of threads for GC
	GC logging	`-verbose:gc`	enable printing GC information
		`-Xlog:gc`	enable garbage collector logs
		`-Xlog:gc*`	enable garbage collector detailed logs
		`-Xloggc:<filename>`	save logs into a file
		`-XX:+UseGCLogFileRotation>`	enable log rotation
		`-XX:+HeapDumpBeforeFullGC`	create memory before full GC stage
		`-XX:NumberOfGCLogFiles=<N>`	count of log files
		`-XX:+PrintGCDetails`	print more details about GC
		`-XX:+PrintGCDateStamps`	add time to logs
		`-XX:+PrintGCApplicationConcurrentTime`	time that taken by GC in case when all treads of application were working
		`-XX:+PrintGCApplicationStoppedTime`	stop the world time
other		`-XX:+PrintFlagsFinal`	print passed jvm flags
		`-XX:+HeapDumpOnOutOfMemoryError`	make dump before unexpected termination due to OutOfMemoryError
		`-XX:HeapDumpPath=dump.hprof`	make dump before unexpected termination due to OutOfMemoryError
		`-XX:+CompileThreshold=<N>`	count of executions that have been done until JIT optimizations are applied
		`-XX:+UseCompressedOops`	compact references in order to decrease memory usage
		`-XX:+OptimizeStringConcat`	use StringBuilder instead of `+` operator
		`-XX:+PrintCompilation`	show optimization levels of JIT
		`-XX:+AggressiveOpts`	aggressive optimization level
		`-XX:+UseStringCache`	cache strings
jmap tool	run	`jmap <pid>`	get information about memory usage
	flags	`-heap`	information about a whole heap
		`-histo`	information about memory usage in terms of each classes
		`-histo:live`	information about live objects
		`-dump:live,format=b,file=heap.hprof`	save dump into a file

$\text{heap} = \text{young generation} + \text{old generation}$
$\text{young generation} = \text{eden} + \text{survivor} * 2$
$\text{survivor ratio} = \frac{ \text{eden}}{\text{survivor} * 2}$

how to get rid of hibernate.properties

2018-12-01T00:00:00+00:00

In this article we find out how to move properties from hibernate.properties to spring configuration.

First, we need to define properties for sessionFactory like that:

<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
    <property name="dataSource" ref="dataSource"/>
    <property name="annotatedClasses">
        <list>
            <value>ru.izebit.Model</value>
        </list>
    </property>
    <property name="hibernateProperties">
        <props>
            <prop key="enable_lazy_load_no_trans">false</prop>
            <prop key="hibernate.hbm2ddl.auto">validate</prop>
        </props>
    </property>
</bean> 

We can go further and move common properties to tomcat context.xml.

<?xml version='1.0' encoding='utf-8'?>

<context>
    <!--database source-->
    <resource name="jdbc/myDataSource" auth="Container"
              type="javax.sql.DataSource"
              driverClassName="oracle.jdbc.OracleDriver"
              url="jdbc:oracle:thin:@izebit.ru:1521:orcl"
              username="user" password="password"
              maxActive="100" maxIdle="10" maxWait="-1" initialSize="10"/>

    <!--hibernate dialect-->
    <environment name="hibernate.dialect" type="java.lang.String" value="org.hibernate.dialect.Oracle10gDialect"/>
    <!--hibernate schema-->
    <environment name="hibernate.default_schema" type="java.lang.String" value="user_schema"/>
</context>

then add props to spring context configuration:

<jee:jndi-lookup id="dataSource" jndi-name="jdbc/myDataSource" expected-type="javax.sql.DataSource"/>
<jee:jndi-lookup id="dialect" jndi-name="java:comp/env/hibernate.dialect" expected-type="java.lang.String"/>
<jee:jndi-lookup id="schema" jndi-name="java:comp/env/hibernate.default_schema" expected-type="java.lang.String"/>
   
<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
        <property name="hibernateProperties">
            <props>
                <prop key="enable_lazy_load_no_trans">false</prop>
                <prop key="hibernate.hbm2ddl.auto">validate</prop>
                <prop key="hibernate.dialect">#{dialect}</prop>
                <prop key="hibernate.default_schema">#{schema}</prop>
            </props>
        </property>    
</bean>

Sometimes, we can not get rid of hibernate.properties completely, but some parameters will be set dynamically from tomcat context. In order to do it, we will create a bean, that will set needed parameters during application bootstrap.

<!--load props from hibernate.properties-->
<util:properties id="hibernateProperties" location="classpath:spring/hibernate.properties"/>

<!-- getting parameters from tomcat context.xml -->
<jee:jndi-lookup id="dialect" jndi-name="java:comp/env/hibernate.dialect" expected-type="java.lang.String"/>

<bean lazy-init="false" id="dialectSetter" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
        <property name="targetObject" ref="hibernateProperties"/>
        <property name="targetMethod" value="setProperty"/>
        <property name="arguments">
            <array>
                <value>hibernate.dialect</value>
                <ref bean="dialect"/>
            </array>
        </property>
</bean>

<bean id="sessionFactory" 
      class="org.springframework.orm.hibernate4.LocalSessionFactoryBean" 
      depends-on="dialectSetter">
        <property name="hibernateProperties" ref="hibernateProperties"/>
</bean>

There is one important point is order of bean initialization. The bean dialectSetter must to earlier than sessionFactory. For that reason, we set its dependencies depends-on = "dialectSetter" explicitly.

Producer Extends Consumer Super (PECS) principle

2018-10-14T00:00:00+00:00

This approach is used in order to make your API more flexible if you use generics. Before explaining it, I should remind what generic wildcards are in java.

The wildcards - are some bounders for types when we use generics. They can be two types: upper or lower bounded generics. A lower bounded generics can be used in case if we want to use all classes that are parents of a particular class T.

<? super T>

A upper bounded generics can be applied when we want to work with only classes that inherit from a particular class T.

<? extends T>

Ok, that’s all what we need to know to keep going. Let’s return to PECS. The PECS approach says:

Producer - extends, consumer - super

I need to give more details here. If we have a parameter that produces some object with a generic type, we should use a lower bounded generics. But, if a parameter consumes a some value with a generic type, we should use a upper bounded generic generics for this.

As you can see, it is quite easy to remember. However, it is still not clear how to apply it. Have a look at an example.

/**
 * add numbers from 0 to count to the collection
 * @param collection - collection that numbers will be add to
 * @param count - number of elements
 */
void addElements(Collection<? super Integer> collection, int count) {
    for (int i = 0; i < count; i++) {
        collection.add(i);
    }
}

The method above is a consumer, because it fills a collection with a generic type with some elements. According to the PECS approach, in this case we should use super. Indeed, we don’t care about what a certain type of elements in a collection. We just want to know exactly that elements with type Integer can be put into it safely, in terms of type-safety. A upper bounded generic let us do it.

Another method below is a consumer.

/**
 * find an index of the element
 *
 * @param array   - list of elements
 * @param element - element that we need to find index of 
 * @param <T>     - type of elements
 * @return index of the element if the collection contains it otherwise -1 
 */
<T> int getIndex(List<? extends T> array, T element) {
    int index = -1;
    int currentIndex = 0;
    for (T e : array) {
        if (e.equals(element)) {
            index = currentIndex;
            break;
        }
        currentIndex++;
    }

    return index;
}

In opposite to the previous example, we don’t put elements into a collection. The method just takes elements from a list and compares it to a given parameter. The parameter produces elements.

To sum up, in short words,

if we take elements from a parameter, it is a producer, and if we put elements into a parameter, the parameter is a consumer.

Proof of concept of incremental compilation with maven

2018-09-22T00:00:00+00:00

In this article, I am going to talk about proof of concept of incremental compilation with maven. Its aim to boost compilation maven multimodule project.

Let’s look at the typical process of build project. In our favorite terminal we type mvn clean install than maven builds our whole project and copies new created artifact to local repository. It is not big deal and looks trivial. However, there are some problems. One of them is the process might take much time. Even if we skip running tests, maven still spends time on creating jar archive or other type of artifact and finally coping it to $HOME/.m2/repository

We can do it faster if we do not perform some step of build process for our multimodule project. As the rule, not all modules are affected. According to this fact, we can build modules that actually required to build. But doing it manually is too tedious because we have to detect what modules are changed and run build command with special maven flags -pl.

There is another solution with usage of git. I hope everyone uses scv for his or her projects and this approach can be adapted for any type of it, not only git.
The key idea is simple. After each time when maven installs artifact, we will add some information about last git commit. The information lets us distinguish different artifact builds with the same project version. In case if revision is the same, to build a submodule again does not make sense, because there is already actual version of artifact in a local repository.

How can we implement it? To do it, we need several maven plugins and to code in xml 😃.

To detect git revision, we add a plugin to deal with git:

<plugin>
    <groupId>pl.project13.maven</groupId>
    <artifactId>git-commit-id-plugin</artifactId>
    <version>2.2.4</version>

    <executions>
        <execution>
            <id>get-the-git-information</id>
            <goals>
                <goal>revision</goal>
            </goals>
            <phase>validate</phase>
        </execution>
    </executions>
    <configuration>
        <validationProperties>
            <validationProperty>
                <name>validating git dirty</name>
                <value>${git.dirty}</value>
                <shouldMatchTo>false</shouldMatchTo>
            </validationProperty>
        </validationProperties>
    </configuration>
</plugin>

git.dirty parameter will be set to true if there are some not committed changes. In that case, we will not compare revision versions and build will be required anyway.

Comparing revision versions will be implemented with ant tasks. It is the most “sophisticated” and cumbersome part of the article.

<!-- wp:code -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-antrun-plugin</artifactId>
    <version>1.8</version>
    <executions>
        <execution>
          <configuration>
            <exportAntProperties>true</exportAntProperties>
            <target>
              <!--add task defintions from ant-contrib-->
              <taskdef resource="net/sf/antcontrib/antcontrib.properties"/>
              <echo>Git revision: ${git.commit.id}</echo>
              <echo>Does repo contain any changes? ${git.dirty}</echo>
    
              <!--получаем путь до артифакта в локальном репозитории-->
              <propertyregex property="directory.name"
                                       input="${project.groupId}"
                                       regexp="[.]"
                                       replace="/"
                                       global="true"/>
              <property name="artifact.path"
                        value="${settings.localRepository}/${directory.name}/${project.artifactId}/${project.version}"/>
              <echo>Artifact path: ${artifact.path}</echo>
    
              <!-- detect if local changes do not exist  and two files: jar archive and file with revision version exist-->
              <condition property="skip.build" value="true" else="false">
                            <and>
                                <isfalse value="${git.dirty}"/>
                                <resourceexists>
                                    <file file="${artifact.path}/${project.build.finalName}.${project.packaging}"/>
                                </resourceexists>
                                <resourceexists>
                                    <file file="${artifact.path}/${git.commit.id}"/>
                                </resourceexists>
                            </and>
              </condition>
    
              <!--if build is required, remove all old files-->
              <if>
                <equals arg1="${skip.build}" arg2="true"/>
                <then>
                  <echo message="Skip build for ${project.artifactId}"/>
                </then>
                <else>
                  <echo message="Rebuild ${project.artifactId}"/>
                  <delete dir="${artifact.path}/" failonerror="false"/>
                  <mkdir dir="${artifact.path}"/>
                  <touch file="${artifact.path}/${git.commit.id}"/>
                </else>
              </if>
            </target>
          </configuration>
          <phase>validate</phase>
          <goals>
            <goal>run</goal>
          </goals>
        </execution>
      </executions>
    <dependencies>
            <dependency>
                <groupId>ant-contrib</groupId>
                <artifactId>ant-contrib</artifactId>
                <version>1.0b3</version>
                <exclusions>
                    <exclusion>
                        <groupId>ant</groupId>
                        <artifactId>ant</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
    </dependencies>
</plugin>

As you might notice, the implementation is simple. The local repository beside artifacts also contains one more file with revision versions. The file name involves the version of the last commit. If all changes are committed and all required files (artifact file and file with revision version) exist, the build process of submodule will be skipped otherwise clean up and rerun build.

Unfortunately, there are no ways to skip build for a particular submodule in maven. Thus, we have to pass skip parameter to all plugins that are used in a build process. It is not taught, but it makes our result pom ponderous.

<plugins>
    <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <skipMain>${skip.build}</skipMain>
                <skip>${skip.build}</skip>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
     </plugin>
     <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-resources-plugin</artifactId>
            <version>3.0.0</version>
            <configuration>
                <skip>${skip.build}</skip>
            </configuration>
     </plugin>
     <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>2.4</version>
            <configuration>
                <skipIfEmpty>true</skipIfEmpty>
            </configuration>
     </plugin>
     <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-install-plugin</artifactId>
            <configuration>
                <skip>${skip.build}</skip>
            </configuration>
     </plugin>
</plugins>

I hope you find the article interesting and useful 🖖.

How to deal with jar hell

2018-06-01T00:00:00+00:00

One day during deploy new a version of an app on test stage, I faced an exception NoSuchMethodException.
A log file contained the next stacktrace:

java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
    at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
    at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:221)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)</code></pre>

I was confused because it had worked perfectly on my local machine. The most likely cause of that error was a conflict of different versions of the same dependency. In java world this situation is called java hell. So how to deal with it?

The first step is run your app with an option: -verbose:class. A jvm print a list of all loaded classes. You may see something like the output below.

java -jar -verbose:class my_super_app.jar

[Opened /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.Object from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.io.Serializable from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.Comparable from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.CharSequence from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.String from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.reflect.GenericDeclaration from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar]

So, after it, we have information about a need class and a jar name that it comes from. The next step is find out how the dependency turns out in your project. In my case I use maven, but I am sure, other build tools have a command with pretty the same functionality.
The command mvn dependency:tree shows a three of dependencies of your project. It is useful because we can see really all dependencies (include transitive ones!).

[INFO] +- org.glassfish.jersey.media:jersey-media-json-jackson:jar:2.12:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.12:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.2:compile
[INFO] |  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.12:compile
[INFO] |  |  +- org.glassfish.hk2:hk2-api:jar:2.3.0-b10:compile
[INFO] |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.3.0-b10:compile
[INFO] |  |  +- org.glassfish.hk2:hk2-locator:jar:2.3.0-b10:compile
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile

The final step is find a dependency with wrong version and name that we know from the previous step. In most cases, the problem goes away after excluding the dependency in your pom.xml.

<dependencies>
    ... 
       <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>commons-logging</groupId>
                    <artifactId>commons-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    ...
</dependencies>      

If it does not help, repeat the process again 🙂.

The best singleton

2018-05-13T00:00:00+00:00

This article is not about what the singleton is and even not about design patterns. Today, I talk on what is the best implementation of this admirable pattern.

To be honest, I used to think that double-check is the best choose. However, after reading the article on java memory model I realized that the world would be never the same. It turns up, volatile is not cheap, and it is barely better than synchronized block. According to this fact, if you need lazy initialization and thread-safe implementation, you should use the approach below:

public final class Singleton {
      private Singleton(){};
      
      public static Singleton getInstance(){
          return SingletonCreator.INSTANCE;
      }
      
      private static class SingletonCreator {
          private static final Singleton INSTANCE = new Singleton();
      }      
}

This code is lazy, because class SingletoneCreator is not initialized until it is not needed and not used in other places of program. But there is only one place where this class might be used. This place is a method getInstance.

This code is thread safe because the field INSTANSE is final and java memory model guarantees us that all final fields have been initialized only once and will be consistent for all threads.

Types of joins

2018-04-16T00:00:00+00:00

To my shame I still do not know all types of joins. This article is my try to figure out in this topic.

First, we should understand what join is.

Join is an operation that union tables by some columns.

There are several types of them:

left inner
right inner
left outer
right outer
some other types.

The main difference between all of them is what rows the result includes. Later, we will look at some examples in order to comprehend it better.

We will use 2 tables workers and phones in our examples.

select * from workers;

output:

---------------
name  | type
---------------
john  | developer
steve | manager
bruce | seller
neal  | architect

select * from phones;

output:

------------------
name   | phone
------------------
john   | 100
jeramy | 103
steve  | 102
zach   | 101

To better understand, we will execute sql queries with different type of joins.

left inner join

A query is given:

select * from workers left inner join phones on workers.name = phones.name;

The result of execution of the query includes workers who has phone numbers.

-------------------------------
name  |type        | phone
-------------------------------
john  | developer  | 100
steve | manager    | 102

Venn Diagram illustrates result very well.

left outer join

select * from workers left outer join phones on workers.name = phones.name

output:

-------------------------------
name  |type         | phone
-------------------------------
john  | developer   | 100
steve | manager     | 102
bruce | seller      | null
neal  | architect   | null

The result involves, as you can see, list of all workers.

right outer join

A query with this type of join works similarly to the previous query, but there is only difference is that the result contains all rows of the second table.

select * from workers right outer join phones on workers.name = phones.name

-------------------------------
name   |type         | phone
-------------------------------
john   | developer   | 100
steve  | manager     | 102
jeramy | null        | 103
zach   | null        | 101

inner join and right inner join

The result of executions of these queries with inner join, right inner join and left inner join are identical.

select * from workers inner join phones on workers.name = phones.name

-------------------------------
name  |type        | phone
-------------------------------
john  | developer  | 100
steve | manager    | 102

full outer join

Beside basic types, there are several additional types. They are used rarely, but sometimes you can find them useful. One of them is full outer join.

select * from workers full outer join phones on workers.name = phones.name

After execution the query abow, we will have rows from the results of left outer join and right outer join.

-------------------------------
name   |type         | phone
-------------------------------
john   | developer   | 100
steve  | manager     | 102
jeramy | null        | 103
zach   | null        | 101
bruce  | seller      | null
neal   | architect   | null

Moreover, we can use where for the query and get rows that do not match.

select * from workers full outer join phones on workers.name = phones.name where phones.phone is null or workers.type is null;

The result:

-------------------------------
name   |type         | phone
-------------------------------
jeramy | null        | 103
zach   | null        | 101
bruce  | seller      | null
neal   | architect   | null

cross join

It is one of specific type of joins. The result of execution involves all possible combinations of rows from two tables. It is called Cartesian product. The count of the result equals a product of count of rows of both tables.

select * from workers cross join phones

others

This part is about some useful queries with different types of joins.

If we can get a list of workers who do not have phone numbers, we can use the query below:

select * from workers left outer join phones where phones.phone is null

The result:

-------------------------------
name   |type         | phone
-------------------------------
bruce  | seller      | null
neal   | architect   | null

You can get a set of workers who do not work and who have a phone number with the next query:

select * from workers right outer join phones where workers.type is null

The result:

-------------------------------
name   |type         | phone
-------------------------------
jeramy | null        | 103
zach   | null        | 101

That is all what I want to share with you. I hope the article will come in handy.

Visualization of join types

The order of object initialization in java

2018-03-23T00:00:00+00:00

Have you ever thought about in which order objects in java initialize? If you haven’t, this article can you help to find it out.

Given: two classes: Child and Parent. According to their names, one inherits from another one. Each class involves static and ordinal init blocks, static and non-static fields. Let’s look at this case more closely.

Parent.java

public class Parent {
    private static final String staticField = staticMethod();
    static {
        System.out.println("parent static block");
    }

    {
        System.out.println("parent non-static block");
    }
    private final String field = initMethod();
    private String initMethod() {
        System.out.println("parent non static field");
        return "";
    }

    public static String staticMethod() {
        System.out.println("parent static field");
        return "";
    }

    public Parent() {
        System.out.println("parent constructor");
    }
}

Child.java

public final class Child extends Parent {
     private final static String staticField = staticMethod();
     private final String field = initMethod();

     static {
            System.out.println("child static block");
     }

     {
            System.out.println("child non-static block");
     }


     public static String staticMethod() {
            System.out.println("child static field");
            return null;
     }

     private String initMethod() {
            System.out.println("child non static field");
            return null;
     }

     public Child() {
            System.out.println("child constructor");
     }
}

If we run compile and run this code by this way:

public static void main(String[] args) throws Exception {
        new Child();
}

we will see the next output:

parent static field
parent static block
child static field
child static block
parent non-static block
parent non-static field
parent constructor
child non-static field
child non-static block
child constructor

To sum up, we can see that objects inits in the particular order:

all static blocks and fields of a parent class
the same happens in a child class
all non-static blocks and fields of a parent class
invocation of a parent constructor
initialization of non-static fields and blocks in a child class
invocation of a child constructor

As we can see, the order is quite logical and easy to remember.

Friendship between CDATA and JAXB

2018-01-21T00:00:00+00:00

Hello, today we are talking about JAXB. This technology lets us serialize java objects into xml. It is pretty old one and exists for long time since java ee . Its API is simple and easy to use, however there are some nuances as somewhere else. They occur each time when you want to do something nontrivial. In my case, I have to put some fields into CDATA blocks and leave their content without any changes.

There is an example below. It is a class Person. Just in case, all examples are written in Kotlin.

@XmlRootElement(name = "person")
@XmlAccessorType(XmlAccessType.FIELD)
class Parent {
    @XmlJavaTypeAdapter(CDATAAdapter::class)
    @XmlElement(name = "name")
    var name: String = ""
    @XmlElementWrapper(name = "children")
    @XmlElements(
            XmlElement(name = "boy", type = Boy::class),
            XmlElement(name = "girl", type = Girl::class))
    var children: Collection<Child> = emptyList()
}

open class Child {
    var name: String = ""
}

class Girl : Child()
class Boy : Child()

The field name is need to be put into a CDATA block. In order to do it, we add a @XmlJavaTypeAdapter annotation and implement own CDATAAdapter interface on ourselves. Have a look at this implementation:

class CDATAAdapter : XmlAdapter<String, String>() {
    override fun unmarshal(value: String?): String {
        return if (value == null || value.isBlank())
            return ""
        else
            value.trim().removePrefix("&lt![CDATA[").removeSuffix("]]&gt")
    }

    override fun marshal(value: String?): String {
        return if (value == null || value.isBlank())
            ""
        else
            "&lt![CDATA[$value]]&gt"
    }
}

You might notice that it is not sophisticated at all - just simple manipulations with strings, and will be right. The next step is say our serializator or our marshaller don’t change strings that are already changed by CDATAAdapter.

private val MARSHALLER: Marshaller by lazy {
    val context = JAXBContext.newInstance(Parent::class.java)
    val marshaller = context.createMarshaller()
    marshaller.setProperty(Marshaller.JAXB_ENCODING, StandardCharsets.UTF_8.toString())
    marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true)
    marshaller.setProperty(com.sun.xml.internal.bind.marshaller.CharacterEscapeHandler::class.java.canonicalName,
            CharacterEscapeHandler { chars, start, end, isAttribute, writer ->
                val value = String(chars, start, end).trim()
                if (value.startsWith("&lt![CDATA[") && value.endsWith("]]&gt"))
                    writer.write(chars, start, end)
                else
                    MinimumEscapeHandler.theInstance.escape(chars, start, end, isAttribute, writer)
            })

    marshaller
}

If we don’t pay attention to some configurations about encoding and “pretty” formatting, we will see a parameter for setting a string handler. We use our custom string handler.

What does it do? The logic is simple. It handles strings in two different ways depend upon an input string is changed by CDATAAdapter or not. If it is not changed, it delegates the invocation to an internal handler otherwise it adds CDATA tags to the input string.

That’s all and now we are ready to check the work:

val parent = Parent().apply {
        this.name = "John"
        this.children = arrayListOf(
                Girl().apply {
                    this.name = "Margaret"
                },
                Boy().apply {
                    this.name = "Steve"
                }
        )
    }
 
MARSHALLER.marshal(parent, System.out)

If we execute the code above, we will see the next output:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<person>
    <name><![CDATA[John]]></name>
    <children>
        <girl>
            <name>Margarette</name>
        </girl>
        <boy>
            <name>Steve</name>
        </boy>
    </children>
</person>

About design patter builder

2018-01-17T00:00:00+00:00

Today we are going to talk on a design pattern Builder. We start from the very begging, so try to find out for what the design patter is required.
The pattern belongs to a creation patterns category. It means you can create objects with a builder. It lets you split creation an object into several phases. Have a look at an example below. It demonstrates a typical use-case of Builder.

public class Person {
    public final String name;
    public final int age;

    public Person(String name, int age) {
        this.age = age;
        this.name = name;
    }
}

In order to create the instance of class Person, we should know values of name and age parameters. A builder of this class is an intermediate class that contains all this values.

public class PersonBuilder {
    private int age;
    private String name;

    public PersonBuilder setAge(int age) {
        this.age = age;
        return this;
    }

    public PersonBuilder setName(String name) {
        this.name = name;
        return this;
    }

    public static PersonBuilder builder() {
        return new PersonBuilder();
    }

    public Person build() {
        return new Person(name, age);
    }
}

We can see, that this class looks pretty surplus. However, when we have to deal with many parameters and/or check them with some rules, we see all advantages of use of Builder more clearly. We move all constructions related to creation new instances from the class Person to its builder class PersonBuilder. By this way, we do not overwhelm the first one with additional functionality.

By now, it is possible to create an object by this way:

Person person = PersonBuilder
        .builder()
        .setAge(26)
        .setName("Artem")
        .build();

Have a notice, that parameters can be set arbitrarily. Sometimes, it is not what we actually want. For instance, in case when some parameter check depends on several parameters. Thus, we use another approach:

public class PersonBuilder {
    private int age;
    private String name;

    public static PersonNameBuilder builder() {
        return new PersonNameBuilder(new PersonBuilder());
    }

    public Person build() {
        return new Person(name, age);
    }


    public static class PersonNameBuilder {
        private final PersonBuilder builder;

        private PersonNameBuilder(PersonBuilder builder) {
            this.builder = builder;
        }

        public PersonAgeBuilder setName(String name) {
            builder.name = name;
            return new PersonAgeBuilder(builder);
        }
    }

    public static class PersonAgeBuilder {
        private final PersonBuilder builder;

        private PersonAgeBuilder(PersonBuilder builder) {
            this.builder = builder;
        }

        public PersonBuilder setAge(int age) {
            builder.age = age;
            return builder;
        }
    }
}

What’s going here? Each time, when some parameter is set, a builder returns an instance of a new builder with only one method. A method lets set a parameter and return a new builder and so on. By this way, we can guarantee the order of operations:

set a name
set an age
create a new person.

Moreover, the order is checking at compile time.

Person person = PersonBuilder
        .builder()
        .setAge(26) // <-- compile error
        .setName("Artem")
        .build();</code></pre>

Without any doubts, the code above is not good to read. By renaming methods, we can create DSL looks like a natural language.

Person man = Person
                   .withName("Artem")
                   .andSurname("Konovalov")
                   .Lives()
                         .in("Saratov")
                         .at("Moscovskay st.")
                   .Works()
                         .as("Programmer")
                         .foR("Some company")
                         .earning(1_000_000)
                         .inCurrency(DOLLARS)
                   .Hobbies()
                         .is("programming")
                         .and()
                         .is("sport");

But it requires more code and time to think about your API properly 🙂. I want not to overburden the article, so I skip the implementation of the example above.

From time to time, a necessity of extending already existing builder classes occurs. A basic class can be extended and more and more child classes emerge. In this case, we expect to reuse an existing builder and add some additional methods. A code of a new child class Citizen below:

public class Citizen extends Person {
    public final String address;

    public Citizen(String name, String surname, String address) {
        super(name, surname);
        this.address = address;
    }
}

The basic builder:

public abstract class BasePersonBuilder<B extends BasePersonBuilder<B, P>, P> {
    protected String name;
    protected String surname;

    @SuppressWarnings("unchecked")
    public B setSurname(String surname) {
        this.surname = surname;
        return (B) this;
    }

    @SuppressWarnings("unchecked")
    public B setName(String name) {
        this.name = name;
        return (B) this;
    }

    public abstract P build();
}

Thus, the builder of the class Person looks like this:

class PersonBuilder extends BasePersonBuilder<PersonBuilder, Person> {
    @Override
    public Person build() {
        return new Person(name, surname);
    }
}

and a builder of class Citizen:

class CitizenBuilder extends BasePersonBuilder<CitizenBuilder, Citizen> {
    private String address;

    public CitizenBuilder setAddress(String address) {
        this.address = address;
        return this;
    }

    @Override
    public Citizen build() {
        return new Citizen(name, surname, address);
    }
}

Creating new instances of these classes remains as easy and simple as it was.

Person person = new PersonBuilder()
        .setName("Max")
        .setSurname("Pain")
        .build();

Citizen citizen = new CitizenBuilder()
        .setAddress("USA, NY")
        .setName("Join")
        .setSurname("Smith")
        .build();</code></pre>

That is all what I wanted to share with you today. Thank you for reading 😉

Converting dates from XMLGregorianCalendar to GregorianCalendar

2017-12-02T00:00:00+00:00

During parsing xml file, I have faced an issue. It was about date converting. An object with XMLGregorianCalendar type and with value that was earlier than 1582 year had been converted into XMLGregorianCalendar object wrongly.

XMLGregorianCalendar beforeDate = DatatypeFactory
                                                .newInstance()
                                                .newXMLGregorianCalendar();
beforeDate.setDay(11);
beforeDate.setMonth(11);
beforeDate.setYear(1581);
XMLGregorianCalendar alterDate = DatatypeFactory
                                                .newInstance()
                                                .newXMLGregorianCalendar();
alterDate.setYear(1582);
alterDate.setMonth(11);
alterDate.setDay(11);

SimpleDateFormat dateFormat = new SimpleDateFormat("dd:MMM:yyyy");
System.out.println(dateFormat.format(beforeDate.toGregorianCalendar().getTime()));

System.out.println(dateFormat.format(alterDate.toGregorianCalendar().getTime()));

output:

11:Jan:1582

This problem happened because of changing Julian calendar to Gregorian calendar. To disable it we need to set date value explicitly as follows:

GregorianCalendar calendar = new GregorianCalendar();
calendar.clear();
calendar.set(xmlCalendar.getYear(), xmlCalendar.getMonth() - 1, xmlCalendar.getDay());

SimpleDateFormat dateFormat = new SimpleDateFormat("dd:MMM:yyyy");
System.out.println(dateFormat.format(calendar.getTime()));

output:

11:Jan:1580

String interning

2017-10-10T00:00:00+00:00

In java, as well known, strings are stored in a cache. This technique of optimization is called string interning. To use memory effectively, all the same strings that encounters in a source code will refer to the same object during execution.

Give a look at the code below:

String s1 = "hi";
String s2 = new String("h") + "i";
System.out.println(s1.equals(s2));

false

The first string s1 is known to the compiler, the second string s2 is created in runtime hence the compiler does not apply the optimization and there are two different objects with type string.

By this way, all strings “hi” that are known to the compiler (created with new operator or set explicitly) will refer to the same object. We can change the first object s1 and will see that other strings with the same value will be affected.

Field field = s1
                .getClass()
                .getDeclaredField("value");
field.setAccessible(true);

Field modifiers = Field
                    .class
                    .getDeclaredField("modifiers");
modifiers.setAccessible(true);
modifiers.setInt(field, field.getModifiers() & ~Modifier.FINAL);

field.set(s1, "bye".toCharArray());

After it, all string objects with value “hi” refers to the string object with value “bye” 😃.

System.out.println(s1.equals("bye"));

true

Artem Konovalov – Software Engineer

3 Concurrency Models

Mutex and blockers

Functional programming

Actors

Is 3-tier Architecture anti-pattern?

Three-tier Architecture

Hexagonal Architecture

Spring Modulith

Links:

Garbage Collectors in JVM-based languages

Contents

What is Garbage Collector?

Serial Garbage Collector

Parallel Garbage Collector

Concurrent Mark Sweep (CMS) Collector

Links:

G1 - Garbage First Collector

Links:

ZGC - The Z Garbage Collector

Links:

ShenandoahGC

Links:

Azul C4 Garbage Collector

Links:

Epsilon Garbage Collector

Links:

Life expectancy visualization

Life behind proxy

apt (package manager of debian-based distribution)

gradle (dependency manager and build automation tool)

maven (dependency manager for java projects)

sbt (build tool for scala projects, preemptively)

git (you, definitely, know what it is) 🙂

curl (tool for downloading some stuff from the internet)

pip (dependency manager for python projects)

npm package manager for javascript projects.

Reactive spring [ru]

How to stop java application gracefully

What steps do we have to do in order to stop app gracefully?

TF-IDF with Hadoop Streaming

The first stage.

The second stage.

The third stage.

Not evident JIT optimizations

CPU optimizations

Branch prediction

Cache levels

False sharing

Prefetching

Loop unrolling

Getting started with Selenide

Show me the code!©

How it works?

Run app and enjoy

How to throw an exception with type `java.lang.Object`

Webinar on spring boot [ru]

Why should you try android development?

Code generation at runtime

The valid parentheses problem

Migration from mysql to postgresql with liquibase

Types of references

Strong reference

Weak reference

Soft reference

Phantom references

Hand-made collector

Caching of hibernate

Session level cache

The second level cache

Query level cache

How to create init.d system service

1. create a script

2. grant permissions

3. register a script as a system service

4. enable a system service

SQL join visualization

contacts of Equals and Hashcode methods

JVM flags

how to get rid of hibernate.properties