Introduction
This article is aimed at persons who are interested in administrating a webMethods Integration Server and need to know how to deploy and maintain integrations to the server via webMethods packages.
Contents
webMethods packages are the building blocks for integrations and the primary delivery mechanism for provisioning Software AG’s integration runtime webMethods. This is why webMethods Integration Server is ideally situated to responding to the rapidly shifting architectural changes in our production environment, from monolithic high availability clustered servers to modern micro-services running in Kubernetes. You can define very precise runtime environments for specific needs by simply choosing what packages you want you deploy to your runtime.
The subjects to be covered are;
- What is a webMethods package precisely
- Installing, Archiving and removing packages
- Package life cycle
- Updating & reloading packages in a production environment
- Deployment in a production environment
- The package home page
- Access & Ports
What is a webMethods package
A package is simply a grouping of services and the necessary configuration, code and other stuff that can be installed, archived and removed from your webMethods Integration Runtime. Packages are the smallest deployable unit, and your solutions will normally comprise a number of different packages.
Services cannot be provided without having first a package and even Software AG provides about 90% of all the features of webMethods Integration via packages. Our packages are always prefixed with ‘Wm’, albeit you may see some packages provided by our experts in the field with the prefix ‘Wx’ too.
Packages are not binary objects and can be easily browsed via the file system, which also makes it easier to share via a shared VCS system (more on this in our next article on packages). You can find them on the file system of your server or see them via the admin page of your server
Packages are persisted to disk either in;
- MicroServices Runtime - /IntegrationServer/packages
- Integration Server - /IntegrationServer/instances//packages
Where is the name of instance that you have created, which will be ‘default’ 99% of the time.
You can also see if the packages have been successfully loaded or not from the webMethods Integration Server admin portal as show here
Developers should ideally provide a README in the package directory to outline what the package does and any post-installation setup steps that are required.
NOTE: as of 10.15 a README.md file will be created automatically whenever a package is created and developers will be encouraged to document it!
Installing, Archiving and removing packages
Out of the box a webMethods Integration Server is almost an empty box, with most features provided as we said via packages. Software AG packages are primarily installed via the Software AG installer. However, your package and third party packages will either be managed manually as described here or through some kind of automation, which we describe in the section ‘hot deployment’
Packages can be deployed either via the file system (server will require a restart) or via the webMethods Integration Server admin page (as shown above). Installing the package through the admin portal should allow you to avoid having to restart the server, but not always!
Package management shows a complete list of all the packages that have been found in the aforementioned packages folder. You can choose to enable disable packages without having to remove them from the server. In which case the package will be removed from memory.
Packages can be installed either;
- Copy the package folder into the packages directory and restart the server
- Copy a package archive (effectively a zip) into the replicate/inbound folder and via the package management page, click “Install inbound releases” and then select the package from the pull down list.
The advantage of the latter option is that you should not need to restart the server. The replicate folder is found at the same level as the packages directory
- MicroServices Runtime - /IntegrationServer/replicate
- Integration Server - /IntegrationServer/instances//replicate
In the directory you will the find the following folder
- inbound - place package zip here for installation.
- outbound - zip archives are written here when exporting packages.
- archive - copy of installed package zip file including date/time stamp.
- salvage - Safe deleted packages, can restore them “recover packages option”.
- _ autodeploy _ - Have to create this folder manually, will discuss in the section Deployment in a production environment
The package should appear immediately after installing a package via the replicate/inbound folder in the list. However check that the package was loaded correctly and that that there are no errors or warnings associated with the package, you can do this by clicking on the package name e.g.
Here you can see an example of a package that has only “partially” loaded and perhaps requires that the server is restarted. Ideally developers should provide packages that do not require restarts and if they do, that it is clearly documented in the associated README.md
Likewise a package may show “No” as the enabled status, which means that the server failed to load the package into memory. This could be for a number of reasons and as stated you should refer to the server log more for more information.
In many cases it will be because the package is dependent on another package that is not yet loaded. You can view the package dependencies via the same package details page as before.
Therefore make sure that you install packages in the correct order. In most cases you will probably want to install packages in an automated manner and we will examine that below.
Archiving packages
Installing a package via the admin portal requires that you first archive the package and this is done via the same page. Each package has a archive icon
Which can be used to create a zip archive, which in turn in written to the replicate/outbound directory of your Integration Server instance.
Package life cycle
Packages are loaded 100% into memory for performance reasons and the files on the disk other than config, are not referenced after that. It is important to understand how packages are loaded and initialised when starting a server, because a badly behaving package can seriously impact startup time or even stop a server from starting.
Firstly a package will only be loaded into memory if it is in the packages folder and has a valid manifest.v3 file.
The ‘ enabled ’ attribute must be set to ‘true’, otherwise the package will be ignored. Packages are first loaded into memory (alphabetically) and then initialised in order of dependencies.
Huh, what initialisation and what dependencies you may ask ??
Well, a dependent package is simply a package that reference another package. Virtually all package are dependent on the ‘WmPublic’ package as you will certainly have made service calls to it. However, you may have your own packages that have services or references that are used in other packages and they will be included in the “requires” list. Additionally developers can choose to have startup or shutdown services that are automatically executed when a package is loaded or removed. A badly behaving service can directly impact the startup or shutdown of your server either delaying it or blocking it completely. The server will not continue until all services have completed or failed.
You can see if a package is taking too long to startup by examining the server log at startup e.g.
The log shows the package being loaded, followed by the startup services which are executed in the order as dictated by the package dependencies. Triggers and inbound ports are only activated once all packages are loaded and initialised properly.
Disabling packages via the file system
If a package is blocking or stopping your server i.e. the last message in the log say something like ‘JcNastyPackage: Startup service (jc.nasty.tools:startup)’ then you can always disable it manually via the ‘enabled’ attribute in the manifest.v3 file as shown above. However, this in turn could cause other packages to fail if they are in turn dependent on it. Still, this can be your ‘get out of jail’ card if you need to urgently get a server up and running.
Don’t keep all your eggs in one basket
This leads onto an interesting question; “How many packages can be hosted in a single instance ?”
Well, how long is a piece of string ? Seriously there is no fixed answer to this, other than it depends? It also leads into a rather big subject about your chosen architecture and even a debate about the pro’s and con’s of micro-services versus the monolith. However, before we go down that path let’s say it does make sense to breakup your integrations into discrete packages to ensure that your server can startup in a timely manner and that if a server does need maintenance or indeed fails that not all of your integrations will be blocked.
Remember the more packages you host in a single server, the longer the server will take to start and the more memory it will consume and hence have less working memory for your processing.
I don’t want this article to turn into a discussion on tuning, but remember you will improve performance if you fix your min and max memory JVM attributes to be the same. You probably knew that, but did you know that packages are actually loaded initialised in multiple threads in order to improve performance too ? You can change the available number of threads via an extended setting called.
- watt.server.package.parallel.threads
By default it is set to 6, evidently dependent packages are only initialised in the correct order.
Updating & reloading packages
Now let’s look at what happens when we decide to reload a package as it can have an important impact on service availability, with the potential to incur lots of exceptions or even data loss.
First why would we need to reload a package ?
Primarily because we want to be able to update a server without requiring a restart.
It’s important to understand what actually happens when a package is reloaded in order to understand the impact. The steps are as follows if you DON’T have any package dependencies
- Shutdown service of package is executed.
- Package is removed from memory once shutdown has completed.
- Package is read from disk and loaded into memory.
- Startup service is executed.
- Package is now available.
It’s important to appreciate that any invokes to the package during this process will result in a service not found exception (404). In addition if the package has any dependent package(s) then by default these get reloaded too, which you can see if you refer to the server log under admin → logs → server.
As you can see reloading package B, causes package A to also reload. This can cause a cascade of packages to reload. During which time, services in all the related packages will trigger service exceptions.
Therefore we do not recommend that you reload packages manually in a production environment.
Deployment in a production environment
As stated you should not reload a package without understanding the impact in a production environment. However you can update a package without having the service not found exception being triggered.
How ?
By enabling hot deploy in the settings section from the admin page of your server.
The same process as above will trigger when installing a package. However, the server will keep a graph of all the dependent services that are effected and allow incoming calls to be initiated for them. it will then immediately block the threads until all of the relevant packages have finished reloading. After which the threads will then be allowed to proceed and execute the new version of the services.
Once done you can install a new version of a package via the admin portal under packages → manage → “Install inbound release”
You will need to put the zipped version of the package into the replicate/inbound of your webMethods instance in order to see it in the list of packages to install.
The upside is no more “Service not found” exceptions in production. The downside; well potentially there are two issues;
-
Client timeout - the client will trigger a timeout if the reload takes too long and so error out. In addition we will still uselessly invoke the service
- Exhaust thread limit - Threads still have to be allocated and hence we could cause the server to max out. This issue can be mitigated by ensuring that you use private thread pools and dedicated ports (see later).
So don’t use this feature to deploy utility packages with lots of dependent packages and certainly not during peak activity. We will be introducing a new package property in 10.15 to allow you to choose whether dependent packages should be reloaded or not. In most cases the reload isn’t required and so we have decided to make it optional. However, you will have to choose to activate it via the package properties when it becomes available in 10.15
NOTE: The hot deploy feature does not work for reload, so you should still avoid reloading packages ini a production environment.
Auto deploying a package
We talked about manually updating a package, but deployment can also be automated via the following techniques;
- Subscribing to packages
Little used feature to allow one server to publish packages to one or more subscribing servers. Personally I have never used this feature, but if you are interested you can refer to our official online document in empower
Copying Packages from One Server to Another
- Deployer
webMethods deployer is Software AG’s solution for packaging complete solutions and deploying them, where a solution comprises multiple package and other assets. You can find more information in our empower documentation centre
- Auto deploy
You need a micro services runtime license to use this feature, but it allows you to drop a package into a directory, where it will be automatically picked up by the server. Refer to our empower documentation centre
- Packages API
WebMethods Integration Server has an admin API that allow you to install/archive packages e.g.
- POST - http://localhost:5555/admin/package?action=install&filename=MyPackage.zip&enable=true This would still use the replicate/inbound directory as the source directory, likewise for archiving e.g.
- POST - http://localhost:5555/admin/package/MyPackage?action=archive
You can fetch the swagger definition for the complete Integration Server administrative API from you local webMethods Integration Server with the following link
http://localhost:5555/admin/swagger/integrationServer
All the above techniques leverage hot deployment if it is enabled.
Server clusters
Remember that servers in a cluster should not present different versions of the same package and hence you will need to upgrade all nodes together. It is best practice that you adopt a new namespace if the
A word on micro-services
Please do not try to deploy packages to directly to runtime containers if you are using a micro-service architecture. The golden rule is to ensure that containers are immutable, which means you should not try modifying them when they are running. Instead employ a build and replace strategy to replace containers of an older version with a new version.
The analogy is that you don’t try to replace the wheels on you car whilst driving. Better to replace your car with a loaner whilst it is updated. In our case the car being an image. I will talk about this in more detail in another tech article at a future date.
The package home page
Did you know that each package has its own home page? You can access it very easily from either package management admin web page or directly by adding the name of the package to the end of the url for your webMethods runtime server.
By default the home page is not very interesting
However, in many cases the home page will be have been modified by the developer to either provide some kind of status page or perhaps to provide online configuration or settings. So make sure to check a package’s home page!
In fact it is perfectly possible to host a full fledged modern React/Angular/javaScript application via a package through the home page!
NOTE: In 10.15 we will be providing a new public service to allow to make your home page much more interesting than above with only a single click. The new service will autogenerate a html page to document all of the API and services (only commented services). This will be of great help to developers who want to use your package and ensure that you can easily keep in sync your development with your documentation. The only requirement on you ? Ensure that you comment your services and APIs in Designer!
Access & Ports
Another recommendation is to ensure that any ingresses into the server for external access are managed through dedicated ports. These ports should be then associated with the proper package. This means that you may see additional ports that added to your server after installing a package. You can view all of the servers ports via the admin portal under Server → Ports.
Here you can see some examples of several ports that are configured via a specific package. The configuration file can be found in the packages config directory and is named ‘*listeners.cnf’
It is important to make sure that these ports are secured properly, but also accessible via any firewalls. Package ports should be ideally set up to “Deny by default” and also use a separate thread pool.
- Deny by default - Access mode that only allows access to the listed services.
- Private thread pool - Click on port details link to see if private thread pool is allocated. This is recommended to ensure that the server threads cannot be exhausted by one bad client.
Likewise a package may present inbound triggers and adapter notifications, which should also be reviewed and then checked periodically.
NOTE: The admin port (default: 5555) should not be used other than for accessing the admin portal. Ideally firewalls should block all external access to this port!
Summary
There you have it, webMethods packages are the building blocks for your integrations. Providing you with a simple means for deploying a well defined set of features, that are self contained and reliable across a number of different runtime environments.
In my next article we will further examine packages through eyes of a developer to ensure that developers provide packages that are robust, reliable and are easy to install and setup
Top comments (0)