Category - Technicals

How to Make ETL Simple and Intuitive with NiFi

Is it possible to implement ETL with an opportunity to make changes during execution and a completely visual processing process without writing code at all?

Yes, it is. With NiFi.

Implementation of ETL is one of the most common tasks now. Creating an aggregator site or simply integrating several enterprise applications leads to the need to solve the ETL task. You can solve this problem with the help of well-known frameworks, such as Apache Camel for example. But let’s try doing it with NiFi.
Read More

async/await syntax encourages frontend developers to develop incorrect code

await operator is an incredible new feature in ECMAScript 2017. There are many articles demonstrating how much you can benefit from it. I agree: asynchronous code was never so brief and clear as when it is written via async/await syntax.

Nevertheless, there is a significant pitfall you should be aware of while developing asynchronous front end code. Backend developers are safe: they can easily copy code from any of the articles above to their NodeJS application, and it will work just fine. Really! However, if you are going to run this code in a browser environment as is, it would be a serious mistake! None of the examples is correctly adjusted for the front end. This is a disaster: in the top 10 Google references on “async await”, I failed to find any examples correctly adjusted for the front end. Junior/middle JS developers and newcomers use these examples as the basis for their work and develop incorrect code that results in unexpected bugs and performance loss all over the place.

Read More

Flexible Data Model Design

Introduction

Data often changes. When it comes to integration, the data can change unpredictably. Of course, large vendors notify the community about the breaking changes in advance or give us some period for migration. However, if you implement your own component of a (distributed) system then you likely face a situation when, for example, Tom changed something, while you were working on your cool feature of a Java-application and now you have to fix some unexpected failures on your side. Or a customer asked you to include some small (or big) changes in the initial specifications and next day Tom found out that his part did not seem to be working.

Read More

Database: How To Make a Bad Thing Work Well

Our team uses software development and database best practice and it turned out that developers got used to some of it and stopped looking at it with a critical eye.

There is a widely known opinion that implementing a business logic on the DB side is a bad idea. In most cases it is true, and can be explained in many ways.

Read More

Multipartial Upload or How to Bring Amazon S3 and Legacy Code Together

A large part of software development projects today are either being developed for quite a long time, or received as a legacy from another team. Such code has been in most cases written by people no longer working in the team. Another pitfall can be the requirement to create an unchangeable and backward compatible API product. Under such conditions, the team have to provide the product support and even introduce new features. I encountered a similar problem connected with legacy code on one of ISSArt’s projects.

Read More

Search functionality design practices. Part 2

Implementation of the search functionality is a challenge for a software developer. Search-related requirements influence significantly on the final architecture of the system. In the first part we studied different filters and some practices of their implementation. In this part I will tell about the full text search.

Read More

Search functionality design practices. Part 1

The humanity had started searching long before the beginning of modern computer century. Thanks to the common sense people invented indices. Libraries, dictionaries, maps made the life easier for the next generations of knowledge hunters. And now you as a software developer face the necessity to implement your own search functionality. Usually you can meet search-related requirements at the end of the customer’s list. But, in my opinion, they influence significantly on the final architecture of the system. Furthermore, the search index is the foundation of searching, its design errors can lead to the wrong choice of storage or database that can break your system in general.

Read More

High Load Java App Server: Things to consider when building

Focusing on the front end demand is where many developers start scoping a project. This coupled with the backend requirements often point to the most obvious choices for the application architecture. As we’ll see it’s often useful to do a bit more digging and mining before settling on a particular design plan. There are many pitfalls that can be avoided when the proper time and consideration is put in to technology and architecture selection. Some questions worth asking include: “What do we mean by high load?”, “Will we ever need to scale?”, “Are there predetermined hardware and/or operating system requirements?”, “What demands are being placed on the backend?”, “Is code base maintenance a priority?”, “Will the code base ever need to be ported to new platforms?”, “Is this a one off deployment? While your gut may be screaming high load javaapplication server a little preliminary consideration will pay off big time in the long run to validating your decision and/or help you defend a different path.

Read More

Challenges of Amazon MWS API: ISS Art experience

This article is focused on Amazon MWS API and some challenges related to its integration.
If you plan to integrate Amazon MWS into your PHP project, you should think twice whether you need this functionality or not.
In our Symfony project such integration is needed. However, there are a lot of hidden dangers. In the article I will try to give you a heads up how to avoid the pitfalls.

Read More